2024-03-21T12:41:05.493 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-21T12:41:05.493 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_gabrioux@teuthology --description orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} --archive /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614953 --name gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi -- /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614953/orig.config.yaml 2024-03-21T12:41:05.527 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T12:41:05.632 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614953 branch: wip-guits-testing-2024-03-21-0818 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} email: gabrioux@ibm.com first_in_suite: false job_id: '7614953' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-testing-2024-03-21-0818 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: 960fbbe4645c2becc8084669be194c95183c7753 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 960fbbe4645c2becc8084669be194c95183c7753 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-testing-2024-03-21-0818 sha1: 960fbbe4645c2becc8084669be194c95183c7753 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 187 sha1: 960fbbe4645c2becc8084669be194c95183c7753 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-testing-2024-03-21-0818 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_960fbbe4645c2becc8084669be194c95183c7753/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 960fbbe4645c2becc8084669be194c95183c7753 targets: smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN589b3RczkarYEMJ7Gi2FAuMmWJFf7GwYb25ciTKTGKJ+++Bt6IO38ss1jDZ4R7lrqEun5K7RMeuMfdZ85ma8U= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJ2UNdgj78yltC2wNujQGNpQzoyP4jJ6xFd72mvl9HnPOFewfqDGbEfEQqNGhpwO++brsqfqGzM12X2DLDrI+Q= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-21_12:04:17 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-21T12:41:05.633 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_960fbbe4645c2becc8084669be194c95183c7753/qa; will attempt to use it 2024-03-21T12:41:05.633 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_960fbbe4645c2becc8084669be194c95183c7753/qa/tasks 2024-03-21T12:41:05.634 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-21T12:41:05.635 INFO:teuthology.task.internal:Checking packages... 2024-03-21T12:41:05.660 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '960fbbe4645c2becc8084669be194c95183c7753' 2024-03-21T12:41:05.661 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-21T12:41:05.661 INFO:teuthology.packaging:ref: None 2024-03-21T12:41:05.661 INFO:teuthology.packaging:tag: None 2024-03-21T12:41:05.661 INFO:teuthology.packaging:branch: wip-guits-testing-2024-03-21-0818 2024-03-21T12:41:05.661 INFO:teuthology.packaging:sha1: 960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:41:05.661 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-testing-2024-03-21-0818 2024-03-21T12:41:05.807 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2267.g960fbbe4 2024-03-21T12:41:05.808 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-21T12:41:05.815 INFO:teuthology.task.internal:no buildpackages task found 2024-03-21T12:41:05.816 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-21T12:41:05.855 INFO:teuthology.task.internal:Saving configuration 2024-03-21T12:41:05.875 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-21T12:41:05.888 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T12:41:05.914 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614953', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 12:35:15.305809', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN589b3RczkarYEMJ7Gi2FAuMmWJFf7GwYb25ciTKTGKJ+++Bt6IO38ss1jDZ4R7lrqEun5K7RMeuMfdZ85ma8U='} 2024-03-21T12:41:05.939 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614953', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 12:35:15.304290', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJ2UNdgj78yltC2wNujQGNpQzoyP4jJ6xFd72mvl9HnPOFewfqDGbEfEQqNGhpwO++brsqfqGzM12X2DLDrI+Q='} 2024-03-21T12:41:05.939 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-21T12:41:05.945 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-21T12:41:05.946 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-21T12:41:05.946 INFO:teuthology.run_tasks:Running task console_log... 2024-03-21T12:41:06.054 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbfdd0aef70>, signals=[15]) 2024-03-21T12:41:06.055 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-21T12:41:06.061 INFO:teuthology.task.internal:Opening connections... 2024-03-21T12:41:06.062 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-03-21T12:41:06.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:41:06.159 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-03-21T12:41:06.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:41:06.255 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-21T12:41:06.262 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-03-21T12:41:06.280 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-03-21T12:41:06.280 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-03-21T12:41:06.337 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-03-21T12:41:06.338 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="9" 2024-03-21T12:41:06.338 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-03-21T12:41:06.338 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-03-21T12:41:06.338 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="9" 2024-03-21T12:41:06.338 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el9" 2024-03-21T12:41:06.338 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-21T12:41:06.338 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-03-21T12:41:06.338 INFO:teuthology.orchestra.run.smithi163.stdout:LOGO="fedora-logo-icon" 2024-03-21T12:41:06.338 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-21T12:41:06.338 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-03-21T12:41:06.338 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-21T12:41:06.338 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-21T12:41:06.338 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-21T12:41:06.339 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-03-21T12:41:06.372 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-03-21T12:41:06.390 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-03-21T12:41:06.391 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-03-21T12:41:06.450 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-03-21T12:41:06.450 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-03-21T12:41:06.450 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-03-21T12:41:06.451 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-03-21T12:41:06.451 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-03-21T12:41:06.451 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-03-21T12:41:06.451 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-21T12:41:06.451 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-03-21T12:41:06.451 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-03-21T12:41:06.451 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-21T12:41:06.451 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-03-21T12:41:06.451 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-21T12:41:06.451 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-21T12:41:06.451 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-21T12:41:06.451 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-03-21T12:41:06.472 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-21T12:41:06.481 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-21T12:41:06.497 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-21T12:41:06.497 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-03-21T12:41:06.500 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-03-21T12:41:06.517 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-21T12:41:06.523 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-21T12:41:06.523 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-03-21T12:41:06.558 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-03-21T12:41:06.612 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-21T12:41:06.672 INFO:teuthology.run_tasks:Running task kernel... 2024-03-21T12:41:06.686 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-21T12:41:06.686 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-21T12:41:06.686 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T12:41:06.687 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T12:41:06.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T12:41:06.704 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-03-21T12:41:06.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T12:41:06.705 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-03-21T12:41:06.761 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-361.el9.x86_64 2024-03-21T12:41:06.761 INFO:teuthology.task.kernel:Running kernel on smithi163: 5.14.0-361.el9.x86_64 2024-03-21T12:41:06.761 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum install -y kernel 2024-03-21T12:41:06.762 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-361.el9.x86_64 2024-03-21T12:41:06.762 INFO:teuthology.task.kernel:Running kernel on smithi204: 5.14.0-361.el9.x86_64 2024-03-21T12:41:06.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum install -y kernel 2024-03-21T12:41:09.339 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - AppStream 7.7 MB/s | 6.7 MB 00:00 2024-03-21T12:41:09.695 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - AppStream 5.4 MB/s | 6.7 MB 00:01 2024-03-21T12:41:11.006 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-21T12:41:11.078 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-21T12:41:12.165 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.0 MB 00:00 2024-03-21T12:41:14.242 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-21T12:41:14.843 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-03-21T12:41:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-03-21T12:41:18.468 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 6.1 MB 00:00 2024-03-21T12:41:20.074 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-21T12:41:20.929 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-21T12:41:21.134 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-21T12:41:22.396 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-03-21T12:41:26.176 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-21T12:41:26.297 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 5.5 MB/s | 21 MB 00:03 2024-03-21T12:41:28.258 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-21T12:41:28.258 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-21T12:41:28.345 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:41:28.354 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:41:28.355 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-03-21T12:41:28.355 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:41:28.355 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:41:28.355 INFO:teuthology.orchestra.run.smithi204.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-21T12:41:28.355 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-21T12:41:28.355 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-21T12:41:28.355 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-21T12:41:28.355 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-21T12:41:28.355 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:41:28.355 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:41:28.355 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:41:28.355 INFO:teuthology.orchestra.run.smithi204.stdout:Install 4 Packages 2024-03-21T12:41:28.356 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:41:28.356 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 76 M 2024-03-21T12:41:28.356 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 123 M 2024-03-21T12:41:28.356 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:41:29.027 INFO:teuthology.orchestra.run.smithi204.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 112 kB/s | 44 kB 00:00 2024-03-21T12:41:31.338 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-21T12:41:33.436 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-21T12:41:33.437 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-21T12:41:33.523 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:41:33.533 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:41:33.534 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-03-21T12:41:33.534 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:41:33.534 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:41:33.534 INFO:teuthology.orchestra.run.smithi163.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-21T12:41:33.534 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-21T12:41:33.534 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-21T12:41:33.534 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-21T12:41:33.534 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-21T12:41:33.534 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:41:33.534 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:41:33.534 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:41:33.534 INFO:teuthology.orchestra.run.smithi163.stdout:Install 4 Packages 2024-03-21T12:41:33.535 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:41:33.535 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 76 M 2024-03-21T12:41:33.535 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 123 M 2024-03-21T12:41:33.535 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:41:33.989 INFO:teuthology.orchestra.run.smithi163.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 131 kB/s | 44 kB 00:00 2024-03-21T12:41:44.001 INFO:teuthology.orchestra.run.smithi204.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 993 kB/s | 15 MB 00:15 2024-03-21T12:41:47.046 INFO:teuthology.orchestra.run.smithi163.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 1.1 MB/s | 15 MB 00:13 2024-03-21T12:41:48.984 INFO:teuthology.orchestra.run.smithi204.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 1.4 MB/s | 27 MB 00:19 2024-03-21T12:41:54.346 INFO:teuthology.orchestra.run.smithi163.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 1.3 MB/s | 27 MB 00:20 2024-03-21T12:41:56.629 INFO:teuthology.orchestra.run.smithi163.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 1.5 MB/s | 33 MB 00:22 2024-03-21T12:41:56.629 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:41:56.629 INFO:teuthology.orchestra.run.smithi163.stdout:Total 3.3 MB/s | 76 MB 00:23 2024-03-21T12:41:57.292 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:41:57.670 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:41:57.671 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:41:58.654 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:41:58.659 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:42:00.200 INFO:teuthology.orchestra.run.smithi204.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 1.1 MB/s | 33 MB 00:31 2024-03-21T12:42:00.200 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:42:00.200 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.4 MB/s | 76 MB 00:31 2024-03-21T12:42:00.862 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:42:01.125 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:42:01.231 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:42:01.231 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:42:01.749 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-21T12:42:01.829 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:42:02.245 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:42:02.246 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:42:04.589 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:42:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:42:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:42:05.413 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-21T12:42:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:42:08.259 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:42:08.406 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:42:09.426 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:42:09.465 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:42:13.118 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:42:13.356 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:42:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:42:18.056 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:42:43.825 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:42:43.825 INFO:teuthology.orchestra.run.smithi163.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-21T12:42:43.825 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:42:43.868 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:42:44.303 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:42:44.303 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-21T12:42:44.303 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:42:44.303 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:42:44.740 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:42:44.740 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:42:44.740 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:42:44.740 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-21T12:42:44.740 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-21T12:42:44.740 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-21T12:42:44.740 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-21T12:42:44.740 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:42:44.740 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:42:45.084 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T12:42:45.187 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T12:42:45.187 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T12:42:45.188 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-21T12:42:45.188 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-21T12:42:45.188 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-21T12:42:45.188 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-21T12:42:45.188 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi163.front.sepia.ceph.com, path=None, version=distro) 2024-03-21T12:42:45.188 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T12:42:45.267 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T12:42:45.268 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T12:42:45.268 DEBUG:teuthology.orchestra.run.smithi163:> sudo rpm -qi grub2-tools 2024-03-21T12:42:45.302 INFO:teuthology.orchestra.run.smithi163.stdout:Name : grub2-tools 2024-03-21T12:42:45.302 INFO:teuthology.orchestra.run.smithi163.stdout:Epoch : 1 2024-03-21T12:42:45.302 INFO:teuthology.orchestra.run.smithi163.stdout:Version : 2.06 2024-03-21T12:42:45.302 INFO:teuthology.orchestra.run.smithi163.stdout:Release : 68.el9 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:Architecture: x86_64 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:Group : Unspecified 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:Size : 8264254 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:License : GPLv3+ 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:Packager : builder@centos.org 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:Vendor : CentOS 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:Summary : Support tools for GRUB. 2024-03-21T12:42:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:Description : 2024-03-21T12:42:45.304 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:42:45.304 INFO:teuthology.orchestra.run.smithi163.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-21T12:42:45.304 INFO:teuthology.orchestra.run.smithi163.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-21T12:42:45.304 INFO:teuthology.orchestra.run.smithi163.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-21T12:42:45.304 INFO:teuthology.orchestra.run.smithi163.stdout:hardware devices. 2024-03-21T12:42:45.304 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:42:45.304 INFO:teuthology.orchestra.run.smithi163.stdout:This subpackage provides tools for support of all platforms. 2024-03-21T12:42:45.305 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-21T12:42:45.305 INFO:teuthology.task.kernel:Updating grub on smithi163 to boot 5.14.0-430.el9.x86_64 2024-03-21T12:42:45.305 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-21T12:42:45.978 INFO:teuthology.orchestra.run.smithi163.stderr:Generating grub configuration file ... 2024-03-21T12:42:47.392 INFO:teuthology.orchestra.run.smithi163.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-21T12:42:47.411 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:42:47.411 INFO:teuthology.orchestra.run.smithi204.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-21T12:42:47.411 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:42:47.418 INFO:teuthology.orchestra.run.smithi163.stderr:done 2024-03-21T12:42:47.420 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2024-03-21T12:42:47.455 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:42:47.483 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.wLJ83Y35h3 2024-03-21T12:42:47.483 DEBUG:teuthology.orchestra.run.smithi163:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wLJ83Y35h3 2024-03-21T12:42:47.553 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.wLJ83Y35h3 2024-03-21T12:42:47.699 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.wLJ83Y35h3 is 6KB 2024-03-21T12:42:47.747 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.wLJ83Y35h3 2024-03-21T12:42:47.761 DEBUG:teuthology.orchestra.run.smithi163:> sudo /bin/ls /boot/loader/entries || true 2024-03-21T12:42:47.834 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-21T12:42:47.834 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-21T12:42:47.834 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-21T12:42:47.834 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-21T12:42:47.837 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-21T12:42:47.864 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:42:47.864 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-21T12:42:47.864 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:42:47.864 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:42:47.979 DEBUG:teuthology.orchestra.run.smithi163:> sudo shutdown -r now 2024-03-21T12:42:48.266 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:42:48.266 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:42:48.266 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:42:48.266 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-21T12:42:48.266 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-21T12:42:48.266 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-21T12:42:48.266 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-21T12:42:48.266 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:42:48.266 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:42:48.589 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T12:42:48.682 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T12:42:48.683 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T12:42:48.683 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-21T12:42:48.683 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-21T12:42:48.683 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-21T12:42:48.683 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-21T12:42:48.683 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2024-03-21T12:42:48.683 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T12:42:48.764 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T12:42:48.764 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T12:42:48.764 DEBUG:teuthology.orchestra.run.smithi204:> sudo rpm -qi grub2-tools 2024-03-21T12:42:48.798 INFO:teuthology.orchestra.run.smithi204.stdout:Name : grub2-tools 2024-03-21T12:42:48.798 INFO:teuthology.orchestra.run.smithi204.stdout:Epoch : 1 2024-03-21T12:42:48.798 INFO:teuthology.orchestra.run.smithi204.stdout:Version : 2.06 2024-03-21T12:42:48.799 INFO:teuthology.orchestra.run.smithi204.stdout:Release : 68.el9 2024-03-21T12:42:48.799 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: x86_64 2024-03-21T12:42:48.799 INFO:teuthology.orchestra.run.smithi204.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-21T12:42:48.799 INFO:teuthology.orchestra.run.smithi204.stdout:Group : Unspecified 2024-03-21T12:42:48.799 INFO:teuthology.orchestra.run.smithi204.stdout:Size : 8264254 2024-03-21T12:42:48.799 INFO:teuthology.orchestra.run.smithi204.stdout:License : GPLv3+ 2024-03-21T12:42:48.799 INFO:teuthology.orchestra.run.smithi204.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-21T12:42:48.799 INFO:teuthology.orchestra.run.smithi204.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-21T12:42:48.799 INFO:teuthology.orchestra.run.smithi204.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-21T12:42:48.799 INFO:teuthology.orchestra.run.smithi204.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-21T12:42:48.799 INFO:teuthology.orchestra.run.smithi204.stdout:Packager : builder@centos.org 2024-03-21T12:42:48.799 INFO:teuthology.orchestra.run.smithi204.stdout:Vendor : CentOS 2024-03-21T12:42:48.799 INFO:teuthology.orchestra.run.smithi204.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-21T12:42:48.800 INFO:teuthology.orchestra.run.smithi204.stdout:Summary : Support tools for GRUB. 2024-03-21T12:42:48.800 INFO:teuthology.orchestra.run.smithi204.stdout:Description : 2024-03-21T12:42:48.800 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:42:48.800 INFO:teuthology.orchestra.run.smithi204.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-21T12:42:48.800 INFO:teuthology.orchestra.run.smithi204.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-21T12:42:48.800 INFO:teuthology.orchestra.run.smithi204.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-21T12:42:48.800 INFO:teuthology.orchestra.run.smithi204.stdout:hardware devices. 2024-03-21T12:42:48.800 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:42:48.800 INFO:teuthology.orchestra.run.smithi204.stdout:This subpackage provides tools for support of all platforms. 2024-03-21T12:42:48.801 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-21T12:42:48.801 INFO:teuthology.task.kernel:Updating grub on smithi204 to boot 5.14.0-430.el9.x86_64 2024-03-21T12:42:48.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-21T12:42:49.409 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2024-03-21T12:42:50.828 INFO:teuthology.orchestra.run.smithi204.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-21T12:42:50.868 INFO:teuthology.orchestra.run.smithi204.stderr:done 2024-03-21T12:42:50.870 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-03-21T12:42:50.885 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.6BkXlSw8PW 2024-03-21T12:42:50.885 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6BkXlSw8PW 2024-03-21T12:42:50.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.6BkXlSw8PW 2024-03-21T12:42:51.097 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.6BkXlSw8PW is 6KB 2024-03-21T12:42:51.146 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.6BkXlSw8PW 2024-03-21T12:42:51.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo /bin/ls /boot/loader/entries || true 2024-03-21T12:42:51.225 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-21T12:42:51.225 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-21T12:42:51.225 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-21T12:42:51.225 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-21T12:42:51.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-21T12:42:51.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2024-03-21T12:43:18.008 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-21T12:43:18.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-21T12:43:18.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:43:21.387 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-21T12:43:21.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-21T12:43:21.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:43:36.351 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-21T12:43:39.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-21T12:43:45.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-21T12:43:45.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:43:48.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-21T12:43:48.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-21T12:43:48.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:43:51.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-21T12:44:00.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-21T12:44:00.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:44:03.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-21T12:44:03.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-21T12:44:04.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:44:18.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-21T12:44:18.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:44:22.430 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-21T12:44:37.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-21T12:44:37.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:44:40.915 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-03-21T12:44:41.387 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-21T12:44:41.387 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-21T12:44:41.387 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-03-21T12:44:41.401 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-430.el9.x86_64 2024-03-21T12:44:41.402 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-21T12:44:41.402 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-21T12:44:41.402 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-21T12:44:42.403 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-21T12:44:42.403 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-21T12:44:42.496 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2024-03-21T12:44:42.549 DEBUG:teuthology.parallel:result is None 2024-03-21T12:45:18.503 DEBUG:teuthology.orchestra.remote:timed out 2024-03-21T12:45:36.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-21T12:45:36.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:45:36.777 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-03-21T12:45:37.156 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-21T12:45:37.156 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-21T12:45:37.157 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-03-21T12:45:37.172 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-430.el9.x86_64 2024-03-21T12:45:37.173 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-21T12:45:37.173 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-21T12:45:37.173 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-21T12:45:38.174 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-21T12:45:38.174 DEBUG:teuthology.orchestra.run.smithi163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-21T12:45:38.547 INFO:teuthology.orchestra.run.smithi163.stdout:ttyS1 2024-03-21T12:45:38.571 DEBUG:teuthology.parallel:result is None 2024-03-21T12:45:38.572 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-21T12:45:38.579 INFO:teuthology.task.internal:Creating test directory... 2024-03-21T12:45:38.579 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T12:45:38.582 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T12:45:38.601 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-21T12:45:38.615 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-21T12:45:38.648 INFO:teuthology.task.internal:Creating archive directory... 2024-03-21T12:45:38.649 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T12:45:38.651 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T12:45:38.723 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-21T12:45:38.730 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-21T12:45:38.730 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T12:45:38.732 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T12:45:38.770 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T12:45:38.773 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T12:45:38.784 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T12:45:38.787 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T12:45:38.789 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-21T12:45:38.796 INFO:teuthology.task.internal:Configuring sudo... 2024-03-21T12:45:38.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T12:45:38.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T12:45:38.862 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-21T12:45:38.871 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-21T12:45:38.871 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T12:45:38.901 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T12:45:38.918 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T12:45:39.000 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T12:45:39.044 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T12:45:39.125 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T12:45:39.168 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:45:39.168 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T12:45:39.235 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T12:45:39.280 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T12:45:39.330 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T12:45:39.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T12:45:39.458 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-21T12:45:39.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T12:45:39.527 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-03-21T12:45:39.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-03-21T12:45:39.568 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T12:45:39.630 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T12:45:40.046 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-21T12:45:40.054 INFO:teuthology.task.internal:Starting timer... 2024-03-21T12:45:40.054 INFO:teuthology.run_tasks:Running task pcp... 2024-03-21T12:45:40.086 INFO:teuthology.run_tasks:Running task selinux... 2024-03-21T12:45:40.118 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-21T12:45:40.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-03-21T12:45:40.181 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-03-21T12:45:40.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-03-21T12:45:40.244 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-03-21T12:45:40.245 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-21T12:45:40.246 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-03-21T12:45:40.265 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-03-21T12:45:40.265 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-03-21T12:45:40.290 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-03-21T12:45:40.290 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi163.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-03-21T12:45:40.291 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-21T12:45:40.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T12:45:40.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-21T12:45:40.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T12:45:40.362 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-21T12:45:40.362 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-03-21T12:45:40.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-03-21T12:45:40.451 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-21T12:45:40.465 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-21T12:45:40.471 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-21T12:45:40.624 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-21T12:45:40.640 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-21T12:45:40.642 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi163.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-21T12:49:51.680 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi163.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-03-21T12:49:51.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-21T12:49:51.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:49:51.772 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-03-21T12:49:51.849 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-21T12:49:51.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-21T12:49:51.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:49:51.940 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-03-21T12:49:52.020 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-21T12:49:52.021 INFO:teuthology.run_tasks:Running task clock... 2024-03-21T12:49:52.031 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-21T12:49:52.031 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T12:49:52.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T12:49:52.034 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T12:49:52.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T12:49:52.064 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-21T12:49:52.080 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-21T12:49:52.091 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-21T12:49:52.108 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-21T12:49:52.110 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2024-03-21T12:49:52.124 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2024-03-21T12:49:52.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-03-21T12:49:52.140 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-21T12:49:52.150 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-03-21T12:49:52.156 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-21T12:49:52.167 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-21T12:49:52.183 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-21T12:49:52.205 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-03-21T12:49:52.214 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T12:49:52.214 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-03-21T12:49:52.214 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:49:52.214 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:49:52.214 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:49:52.215 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:49:52.238 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-03-21T12:49:52.246 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T12:49:52.246 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-03-21T12:49:52.246 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:49:52.246 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:49:52.246 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:49:52.246 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:49:52.246 INFO:teuthology.run_tasks:Running task install... 2024-03-21T12:49:52.255 DEBUG:teuthology.task.install:project ceph 2024-03-21T12:49:52.255 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '960fbbe4645c2becc8084669be194c95183c7753'}} 2024-03-21T12:49:52.255 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '960fbbe4645c2becc8084669be194c95183c7753'} 2024-03-21T12:49:52.255 INFO:teuthology.task.install:Using flavor: default 2024-03-21T12:49:52.261 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-21T12:49:52.261 INFO:teuthology.task.install:extra packages: [] 2024-03-21T12:49:52.261 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '960fbbe4645c2becc8084669be194c95183c7753', 'tag': None, 'wait_for_package': False} 2024-03-21T12:49:52.262 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:49:52.264 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '960fbbe4645c2becc8084669be194c95183c7753', 'tag': None, 'wait_for_package': False} 2024-03-21T12:49:52.264 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:49:52.488 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/ 2024-03-21T12:49:52.488 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2267.g960fbbe4 2024-03-21T12:49:52.509 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/ 2024-03-21T12:49:52.510 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2267.g960fbbe4 2024-03-21T12:49:52.660 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-21T12:49:52.660 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-21T12:49:52.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-21T12:49:52.677 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-21T12:49:52.677 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:49:52.677 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-21T12:49:52.687 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-21T12:49:52.687 DEBUG:teuthology.orchestra.run.smithi204:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/960fbbe4645c2becc8084669be194c95183c7753/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-21T12:49:52.706 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-21T12:49:52.706 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/960fbbe4645c2becc8084669be194c95183c7753/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-21T12:49:52.755 DEBUG:teuthology.orchestra.run.smithi204:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-21T12:49:52.774 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-21T12:49:52.835 DEBUG:teuthology.orchestra.run.smithi204:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-21T12:49:52.855 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-21T12:49:52.901 INFO:teuthology.orchestra.run.smithi204.stdout:check_obsoletes = 1 2024-03-21T12:49:52.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum clean all 2024-03-21T12:49:52.922 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2024-03-21T12:49:52.924 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2024-03-21T12:49:53.155 INFO:teuthology.orchestra.run.smithi204.stdout:74 files removed 2024-03-21T12:49:53.173 INFO:teuthology.orchestra.run.smithi163.stdout:74 files removed 2024-03-21T12:49:53.197 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw 2024-03-21T12:49:53.204 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw 2024-03-21T12:49:53.965 INFO:teuthology.orchestra.run.smithi204.stdout:ceph packages for x86_64 178 kB/s | 83 kB 00:00 2024-03-21T12:49:53.977 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 176 kB/s | 83 kB 00:00 2024-03-21T12:49:54.307 INFO:teuthology.orchestra.run.smithi204.stdout:ceph noarch packages 40 kB/s | 12 kB 00:00 2024-03-21T12:49:54.403 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-03-21T12:49:54.666 INFO:teuthology.orchestra.run.smithi204.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-21T12:49:54.720 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 6.8 kB/s | 1.9 kB 00:00 2024-03-21T12:49:55.157 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-21T12:49:55.410 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - AppStream 9.5 MB/s | 6.7 MB 00:00 2024-03-21T12:49:56.657 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-21T12:49:56.944 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-03-21T12:49:57.631 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-21T12:49:57.937 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-21T12:49:59.542 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-21T12:50:00.636 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-21T12:50:06.054 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.1 MB 00:01 2024-03-21T12:50:07.562 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-03-21T12:50:07.949 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 1.2 MB/s | 6.1 MB 00:04 2024-03-21T12:50:08.697 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-21T12:50:09.291 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 100 kB/s | 16 kB 00:00 2024-03-21T12:50:11.086 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-21T12:50:13.606 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-21T12:50:16.017 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:50:16.025 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:50:16.025 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-21T12:50:16.025 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:50:16.025 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:50:16.025 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 9.9 M 2024-03-21T12:50:16.025 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2024-03-21T12:50:16.025 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 3.4 M 2024-03-21T12:50:16.026 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 3.1 M 2024-03-21T12:50:16.026 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-21T12:50:16.026 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options 2024-03-21T12:50:16.026 INFO:teuthology.orchestra.run.smithi204.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-21T12:50:16.026 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 5.4 M 2024-03-21T12:50:16.026 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 20 M 2024-03-21T12:50:16.026 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 25 k 2024-03-21T12:50:16.026 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-21T12:50:16.026 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-21T12:50:16.026 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-21T12:50:16.026 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 746 k 2024-03-21T12:50:16.026 INFO:teuthology.orchestra.run.smithi204.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-21T12:50:16.026 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 501 k 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 5.1 M 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 45 k 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 131 k 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 162 k 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 320 k 2024-03-21T12:50:16.027 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 298 k 2024-03-21T12:50:16.028 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 99 k 2024-03-21T12:50:16.028 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-21T12:50:16.028 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-21T12:50:16.028 INFO:teuthology.orchestra.run.smithi204.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-21T12:50:16.028 INFO:teuthology.orchestra.run.smithi204.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-21T12:50:16.028 INFO:teuthology.orchestra.run.smithi204.stdout:Installing weak dependencies: 2024-03-21T12:50:16.028 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-21T12:50:16.028 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:50:16.028 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:50:16.028 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:50:16.028 INFO:teuthology.orchestra.run.smithi204.stdout:Install 30 Packages 2024-03-21T12:50:16.028 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 2 Packages 2024-03-21T12:50:16.028 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:50:16.029 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 59 M 2024-03-21T12:50:16.029 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:50:16.042 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-21T12:50:17.074 INFO:teuthology.orchestra.run.smithi204.stdout:(1/32): ceph-base-19.0.0-2267.g960fbbe4.el9.x86 6.6 MB/s | 5.4 MB 00:00 2024-03-21T12:50:17.132 INFO:teuthology.orchestra.run.smithi204.stdout:(2/32): ceph-selinux-19.0.0-2267.g960fbbe4.el9. 433 kB/s | 25 kB 00:00 2024-03-21T12:50:17.249 INFO:teuthology.orchestra.run.smithi204.stdout:(3/32): libcephfs2-19.0.0-2267.g960fbbe4.el9.x8 6.2 MB/s | 746 kB 00:00 2024-03-21T12:50:17.374 INFO:teuthology.orchestra.run.smithi204.stdout:(4/32): libradosstriper1-19.0.0-2267.g960fbbe4. 3.9 MB/s | 501 kB 00:00 2024-03-21T12:50:17.592 INFO:teuthology.orchestra.run.smithi204.stdout:(5/32): ceph-radosgw-19.0.0-2267.g960fbbe4.el9. 7.4 MB/s | 9.9 MB 00:01 2024-03-21T12:50:17.667 INFO:teuthology.orchestra.run.smithi204.stdout:(6/32): python3-ceph-argparse-19.0.0-2267.g960f 604 kB/s | 45 kB 00:00 2024-03-21T12:50:17.742 INFO:teuthology.orchestra.run.smithi204.stdout:(7/32): python3-ceph-common-19.0.0-2267.g960fbb 1.7 MB/s | 131 kB 00:00 2024-03-21T12:50:17.826 INFO:teuthology.orchestra.run.smithi204.stdout:(8/32): python3-cephfs-19.0.0-2267.g960fbbe4.el 1.9 MB/s | 162 kB 00:00 2024-03-21T12:50:17.918 INFO:teuthology.orchestra.run.smithi204.stdout:(9/32): python3-rados-19.0.0-2267.g960fbbe4.el9 3.4 MB/s | 320 kB 00:00 2024-03-21T12:50:17.984 INFO:teuthology.orchestra.run.smithi204.stdout:(10/32): librgw2-19.0.0-2267.g960fbbe4.el9.x86_ 8.4 MB/s | 5.1 MB 00:00 2024-03-21T12:50:18.010 INFO:teuthology.orchestra.run.smithi204.stdout:(11/32): python3-rbd-19.0.0-2267.g960fbbe4.el9. 3.2 MB/s | 298 kB 00:00 2024-03-21T12:50:18.068 INFO:teuthology.orchestra.run.smithi204.stdout:(12/32): python3-rgw-19.0.0-2267.g960fbbe4.el9. 1.2 MB/s | 99 kB 00:00 2024-03-21T12:50:18.294 INFO:teuthology.orchestra.run.smithi204.stdout:(13/32): ceph-common-19.0.0-2267.g960fbbe4.el9. 10 MB/s | 20 MB 00:02 2024-03-21T12:50:18.352 INFO:teuthology.orchestra.run.smithi204.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 563 kB/s | 159 kB 00:00 2024-03-21T12:50:18.386 INFO:teuthology.orchestra.run.smithi204.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 281 kB/s | 106 kB 00:00 2024-03-21T12:50:18.411 INFO:teuthology.orchestra.run.smithi204.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 379 kB/s | 44 kB 00:00 2024-03-21T12:50:18.487 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:50:18.494 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:50:18.494 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-21T12:50:18.494 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:50:18.495 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:50:18.495 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 9.9 M 2024-03-21T12:50:18.495 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2024-03-21T12:50:18.495 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 3.4 M 2024-03-21T12:50:18.495 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 3.1 M 2024-03-21T12:50:18.495 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-21T12:50:18.495 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options 2024-03-21T12:50:18.495 INFO:teuthology.orchestra.run.smithi163.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-21T12:50:18.495 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 5.4 M 2024-03-21T12:50:18.495 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 20 M 2024-03-21T12:50:18.495 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 25 k 2024-03-21T12:50:18.495 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-21T12:50:18.495 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 746 k 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 501 k 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 5.1 M 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 45 k 2024-03-21T12:50:18.496 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 131 k 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 162 k 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 320 k 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 298 k 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 99 k 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout:Installing weak dependencies: 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:50:18.497 INFO:teuthology.orchestra.run.smithi163.stdout:Install 30 Packages 2024-03-21T12:50:18.498 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 2 Packages 2024-03-21T12:50:18.498 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:50:18.498 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 59 M 2024-03-21T12:50:18.498 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:50:18.545 INFO:teuthology.orchestra.run.smithi204.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 661 kB 00:00 2024-03-21T12:50:18.570 INFO:teuthology.orchestra.run.smithi204.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 282 kB 00:00 2024-03-21T12:50:18.587 INFO:teuthology.orchestra.run.smithi204.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 234 kB/s | 41 kB 00:00 2024-03-21T12:50:18.804 INFO:teuthology.orchestra.run.smithi204.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 149 kB/s | 35 kB 00:00 2024-03-21T12:50:18.821 INFO:teuthology.orchestra.run.smithi204.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 191 kB/s | 44 kB 00:00 2024-03-21T12:50:18.846 INFO:teuthology.orchestra.run.smithi204.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 720 kB/s | 216 kB 00:00 2024-03-21T12:50:18.963 INFO:teuthology.orchestra.run.smithi204.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-03-21T12:50:19.064 INFO:teuthology.orchestra.run.smithi204.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-21T12:50:19.080 INFO:teuthology.orchestra.run.smithi204.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-03-21T12:50:19.122 INFO:teuthology.orchestra.run.smithi204.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-21T12:50:19.239 INFO:teuthology.orchestra.run.smithi204.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-21T12:50:19.264 INFO:teuthology.orchestra.run.smithi204.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-21T12:50:19.298 INFO:teuthology.orchestra.run.smithi204.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-03-21T12:50:19.399 INFO:teuthology.orchestra.run.smithi204.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.9 MB/s | 1.6 MB 00:00 2024-03-21T12:50:19.460 INFO:teuthology.orchestra.run.smithi163.stdout:(1/32): ceph-base-19.0.0-2267.g960fbbe4.el9.x86 6.8 MB/s | 5.4 MB 00:00 2024-03-21T12:50:19.518 INFO:teuthology.orchestra.run.smithi163.stdout:(2/32): ceph-selinux-19.0.0-2267.g960fbbe4.el9. 433 kB/s | 25 kB 00:00 2024-03-21T12:50:19.677 INFO:teuthology.orchestra.run.smithi163.stdout:(3/32): libcephfs2-19.0.0-2267.g960fbbe4.el9.x8 4.6 MB/s | 746 kB 00:00 2024-03-21T12:50:19.807 INFO:teuthology.orchestra.run.smithi204.stdout:(31/32): librbd1-19.0.0-2267.g960fbbe4.el9.x86_ 6.2 MB/s | 3.1 MB 00:00 2024-03-21T12:50:19.827 INFO:teuthology.orchestra.run.smithi163.stdout:(4/32): libradosstriper1-19.0.0-2267.g960fbbe4. 3.3 MB/s | 501 kB 00:00 2024-03-21T12:50:20.049 INFO:teuthology.orchestra.run.smithi204.stdout:(32/32): librados2-19.0.0-2267.g960fbbe4.el9.x8 4.3 MB/s | 3.4 MB 00:00 2024-03-21T12:50:20.055 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:50:20.055 INFO:teuthology.orchestra.run.smithi204.stdout:Total 15 MB/s | 59 MB 00:04 2024-03-21T12:50:20.092 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:50:20.130 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:50:20.131 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:50:20.461 INFO:teuthology.orchestra.run.smithi163.stdout:(5/32): librgw2-19.0.0-2267.g960fbbe4.el9.x86_6 8.1 MB/s | 5.1 MB 00:00 2024-03-21T12:50:20.527 INFO:teuthology.orchestra.run.smithi163.stdout:(6/32): python3-ceph-argparse-19.0.0-2267.g960f 679 kB/s | 45 kB 00:00 2024-03-21T12:50:20.660 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:50:20.661 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:50:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:(7/32): ceph-common-19.0.0-2267.g960fbbe4.el9.x 9.8 MB/s | 20 MB 00:02 2024-03-21T12:50:20.795 INFO:teuthology.orchestra.run.smithi163.stdout:(8/32): python3-ceph-common-19.0.0-2267.g960fbb 493 kB/s | 131 kB 00:00 2024-03-21T12:50:20.820 INFO:teuthology.orchestra.run.smithi163.stdout:(9/32): python3-cephfs-19.0.0-2267.g960fbbe4.el 3.2 MB/s | 162 kB 00:00 2024-03-21T12:50:20.853 INFO:teuthology.orchestra.run.smithi163.stdout:(10/32): python3-rados-19.0.0-2267.g960fbbe4.el 5.4 MB/s | 320 kB 00:00 2024-03-21T12:50:20.879 INFO:teuthology.orchestra.run.smithi163.stdout:(11/32): python3-rbd-19.0.0-2267.g960fbbe4.el9. 5.0 MB/s | 298 kB 00:00 2024-03-21T12:50:20.904 INFO:teuthology.orchestra.run.smithi163.stdout:(12/32): python3-rgw-19.0.0-2267.g960fbbe4.el9. 1.9 MB/s | 99 kB 00:00 2024-03-21T12:50:21.013 INFO:teuthology.orchestra.run.smithi163.stdout:(13/32): ceph-radosgw-19.0.0-2267.g960fbbe4.el9 4.2 MB/s | 9.9 MB 00:02 2024-03-21T12:50:21.146 INFO:teuthology.orchestra.run.smithi163.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 396 kB/s | 106 kB 00:00 2024-03-21T12:50:21.171 INFO:teuthology.orchestra.run.smithi163.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 401 kB/s | 44 kB 00:00 2024-03-21T12:50:21.197 INFO:teuthology.orchestra.run.smithi163.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 544 kB/s | 159 kB 00:00 2024-03-21T12:50:21.322 INFO:teuthology.orchestra.run.smithi163.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 661 kB 00:00 2024-03-21T12:50:21.347 INFO:teuthology.orchestra.run.smithi163.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 282 kB 00:00 2024-03-21T12:50:21.372 INFO:teuthology.orchestra.run.smithi163.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 235 kB/s | 41 kB 00:00 2024-03-21T12:50:21.414 INFO:teuthology.orchestra.run.smithi163.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 523 kB/s | 35 kB 00:00 2024-03-21T12:50:21.440 INFO:teuthology.orchestra.run.smithi163.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 665 kB/s | 44 kB 00:00 2024-03-21T12:50:21.465 INFO:teuthology.orchestra.run.smithi163.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.5 MB/s | 216 kB 00:00 2024-03-21T12:50:21.615 INFO:teuthology.orchestra.run.smithi163.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 173 kB/s | 26 kB 00:00 2024-03-21T12:50:21.724 INFO:teuthology.orchestra.run.smithi163.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-03-21T12:50:21.757 INFO:teuthology.orchestra.run.smithi163.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 901 kB/s | 309 kB 00:00 2024-03-21T12:50:21.791 INFO:teuthology.orchestra.run.smithi163.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-21T12:50:21.900 INFO:teuthology.orchestra.run.smithi163.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-21T12:50:21.925 INFO:teuthology.orchestra.run.smithi163.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-21T12:50:22.025 INFO:teuthology.orchestra.run.smithi163.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 4.4 MB 00:00 2024-03-21T12:50:22.067 INFO:teuthology.orchestra.run.smithi163.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-03-21T12:50:22.166 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:50:22.235 INFO:teuthology.orchestra.run.smithi163.stdout:(31/32): librados2-19.0.0-2267.g960fbbe4.el9.x8 11 MB/s | 3.4 MB 00:00 2024-03-21T12:50:22.239 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-21T12:50:22.271 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-21T12:50:22.303 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-21T12:50:22.327 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-21T12:50:22.344 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-21T12:50:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-21T12:50:22.501 INFO:teuthology.orchestra.run.smithi163.stdout:(32/32): librbd1-19.0.0-2267.g960fbbe4.el9.x86_ 6.6 MB/s | 3.1 MB 00:00 2024-03-21T12:50:22.505 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:50:22.505 INFO:teuthology.orchestra.run.smithi163.stdout:Total 15 MB/s | 59 MB 00:04 2024-03-21T12:50:22.535 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:50:22.541 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-21T12:50:22.569 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:50:22.569 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:50:22.882 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-21T12:50:22.965 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 7/34 2024-03-21T12:50:23.066 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 7/34 2024-03-21T12:50:23.073 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:50:23.074 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:50:23.152 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 8/34 2024-03-21T12:50:23.175 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 9/34 2024-03-21T12:50:23.237 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 9/34 2024-03-21T12:50:23.266 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-21T12:50:23.309 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-argparse-2:19.0.0-2267.g960fbbe4.el9. 11/34 2024-03-21T12:50:23.377 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 12/34 2024-03-21T12:50:23.402 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_6 13/34 2024-03-21T12:50:23.465 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_6 13/34 2024-03-21T12:50:23.492 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-21T12:50:23.602 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-21T12:50:23.627 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-21T12:50:23.667 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-21T12:50:23.708 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-21T12:50:23.736 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-21T12:50:23.769 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-21T12:50:24.082 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-21T12:50:24.116 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 22/34 2024-03-21T12:50:24.206 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 22/34 2024-03-21T12:50:24.237 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 23/34 2024-03-21T12:50:24.284 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-21T12:50:24.548 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:50:24.640 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-21T12:50:24.689 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-21T12:50:24.715 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-21T12:50:24.737 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-21T12:50:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-21T12:50:24.756 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-21T12:50:24.777 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 26/34 2024-03-21T12:50:24.848 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 26/34 2024-03-21T12:50:24.917 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 27/34 2024-03-21T12:50:24.946 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-common-2:19.0.0-2267.g960fbbe4.el9.x8 28/34 2024-03-21T12:50:24.990 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-21T12:50:25.044 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-21T12:50:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-21T12:50:25.466 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 7/34 2024-03-21T12:50:25.565 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 7/34 2024-03-21T12:50:25.655 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 8/34 2024-03-21T12:50:25.684 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 9/34 2024-03-21T12:50:25.730 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 9/34 2024-03-21T12:50:25.758 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-21T12:50:25.802 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:19.0.0-2267.g960fbbe4.el9. 11/34 2024-03-21T12:50:25.879 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 12/34 2024-03-21T12:50:25.912 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_6 13/34 2024-03-21T12:50:25.966 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_6 13/34 2024-03-21T12:50:25.993 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-21T12:50:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-21T12:50:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-21T12:50:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-21T12:50:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-21T12:50:26.238 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-21T12:50:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-21T12:50:26.587 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-21T12:50:26.618 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 22/34 2024-03-21T12:50:26.707 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 22/34 2024-03-21T12:50:26.739 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 23/34 2024-03-21T12:50:26.789 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-21T12:50:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 29/34 2024-03-21T12:50:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 29/34 2024-03-21T12:50:27.250 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-21T12:50:27.287 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 26/34 2024-03-21T12:50:27.358 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 26/34 2024-03-21T12:50:27.425 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 27/34 2024-03-21T12:50:27.456 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:19.0.0-2267.g960fbbe4.el9.x8 28/34 2024-03-21T12:50:27.754 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 29/34 2024-03-21T12:50:27.791 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 30/34 2024-03-21T12:50:27.863 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 30/34 2024-03-21T12:50:27.863 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-21T12:50:27.863 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-21T12:50:27.863 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:50:27.906 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 31/34 2024-03-21T12:50:29.691 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 29/34 2024-03-21T12:50:29.727 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 29/34 2024-03-21T12:50:30.287 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 29/34 2024-03-21T12:50:30.363 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 30/34 2024-03-21T12:50:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 30/34 2024-03-21T12:50:30.455 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-21T12:50:30.456 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-21T12:50:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:50:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 31/34 2024-03-21T12:50:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 31/34 2024-03-21T12:50:37.677 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /sys 2024-03-21T12:50:37.677 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /proc 2024-03-21T12:50:37.677 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /mnt 2024-03-21T12:50:37.677 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /var/tmp 2024-03-21T12:50:37.677 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /home 2024-03-21T12:50:37.677 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /root 2024-03-21T12:50:37.678 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /tmp 2024-03-21T12:50:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:50:38.609 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/34 2024-03-21T12:50:38.634 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/34 2024-03-21T12:50:38.634 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:50:38.634 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-21T12:50:38.635 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T12:50:38.635 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T12:50:38.635 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:50:38.636 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-21T12:50:38.720 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-21T12:50:38.721 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T12:50:39.824 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T12:50:39.824 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/34 2024-03-21T12:50:39.824 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/34 2024-03-21T12:50:39.824 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 3/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 4/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_6 6/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 7/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2267.g960fbbe4.el9. 8/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-common-2:19.0.0-2267.g960fbbe4.el9.x8 9/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 10/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 11/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 12/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 13/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-21T12:50:39.825 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-21T12:50:39.826 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-21T12:50:39.826 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-21T12:50:39.826 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-21T12:50:39.826 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-21T12:50:39.826 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-21T12:50:39.826 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-21T12:50:39.826 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-21T12:50:39.826 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-21T12:50:39.826 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-21T12:50:39.826 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-21T12:50:39.826 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-21T12:50:39.826 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-21T12:50:39.826 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-21T12:50:39.827 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 31/34 2024-03-21T12:50:39.827 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-21T12:50:39.827 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 33/34 2024-03-21T12:50:40.042 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 31/34 2024-03-21T12:50:40.043 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2024-03-21T12:50:40.043 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2024-03-21T12:50:40.043 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2024-03-21T12:50:40.043 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2024-03-21T12:50:40.043 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2024-03-21T12:50:40.043 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /root 2024-03-21T12:50:40.043 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2024-03-21T12:50:40.043 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:50:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T12:50:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:50:40.900 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2024-03-21T12:50:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.900 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:50:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-21T12:50:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-21T12:50:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-21T12:50:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-21T12:50:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-21T12:50:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: re2-1:20211101-3.el9.x86_64 2024-03-21T12:50:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-21T12:50:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:50:40.902 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:50:40.981 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/34 2024-03-21T12:50:41.006 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/34 2024-03-21T12:50:41.006 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:50:41.006 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-21T12:50:41.006 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T12:50:41.006 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T12:50:41.006 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:50:41.008 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-21T12:50:41.076 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-21T12:50:41.076 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T12:50:41.247 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-test 2024-03-21T12:50:41.813 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:28 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:50:41.963 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:50:41.964 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:50:41.964 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-21T12:50:41.964 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:50:41.964 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:50:41.964 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 48 M 2024-03-21T12:50:41.964 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-21T12:50:41.964 INFO:teuthology.orchestra.run.smithi204.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-21T12:50:41.964 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 163 k 2024-03-21T12:50:41.964 INFO:teuthology.orchestra.run.smithi204.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-21T12:50:41.964 INFO:teuthology.orchestra.run.smithi204.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-21T12:50:41.964 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-21T12:50:41.965 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:50:41.965 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:50:41.965 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:50:41.965 INFO:teuthology.orchestra.run.smithi204.stdout:Install 6 Packages 2024-03-21T12:50:41.965 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:50:41.965 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 49 M 2024-03-21T12:50:41.965 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 206 M 2024-03-21T12:50:41.966 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:50:42.159 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T12:50:42.159 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/34 2024-03-21T12:50:42.159 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/34 2024-03-21T12:50:42.159 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 3/34 2024-03-21T12:50:42.159 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 4/34 2024-03-21T12:50:42.159 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/34 2024-03-21T12:50:42.159 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_6 6/34 2024-03-21T12:50:42.159 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 7/34 2024-03-21T12:50:42.159 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2267.g960fbbe4.el9. 8/34 2024-03-21T12:50:42.159 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:19.0.0-2267.g960fbbe4.el9.x8 9/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 10/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 11/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 12/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 13/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-21T12:50:42.160 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-21T12:50:42.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-21T12:50:42.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-21T12:50:42.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-21T12:50:42.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-21T12:50:42.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-21T12:50:42.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-21T12:50:42.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-21T12:50:42.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 31/34 2024-03-21T12:50:42.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-21T12:50:42.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 33/34 2024-03-21T12:50:42.214 INFO:teuthology.orchestra.run.smithi204.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-21T12:50:42.256 INFO:teuthology.orchestra.run.smithi204.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-21T12:50:42.281 INFO:teuthology.orchestra.run.smithi204.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-21T12:50:42.382 INFO:teuthology.orchestra.run.smithi204.stdout:(4/6): libcephsqlite-19.0.0-2267.g960fbbe4.el9. 467 kB/s | 163 kB 00:00 2024-03-21T12:50:42.415 INFO:teuthology.orchestra.run.smithi204.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-21T12:50:43.385 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T12:50:43.385 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:50:43.385 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2024-03-21T12:50:43.385 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.385 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.386 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:50:43.386 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-21T12:50:43.386 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.386 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.386 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.386 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.386 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-21T12:50:43.386 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-21T12:50:43.386 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-21T12:50:43.386 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.386 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-21T12:50:43.386 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-21T12:50:43.386 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-21T12:50:43.387 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-21T12:50:43.388 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-21T12:50:43.388 INFO:teuthology.orchestra.run.smithi163.stdout: re2-1:20211101-3.el9.x86_64 2024-03-21T12:50:43.388 INFO:teuthology.orchestra.run.smithi163.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-21T12:50:43.388 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:50:43.388 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:50:43.746 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-test 2024-03-21T12:50:44.303 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:28 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:50:44.456 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:50:44.456 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 48 M 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 163 k 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:50:44.457 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2024-03-21T12:50:44.458 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:50:44.458 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 49 M 2024-03-21T12:50:44.458 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 206 M 2024-03-21T12:50:44.458 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:50:44.599 INFO:teuthology.orchestra.run.smithi204.stdout:(6/6): ceph-test-19.0.0-2267.g960fbbe4.el9.x86_ 19 MB/s | 48 MB 00:02 2024-03-21T12:50:44.602 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:50:44.602 INFO:teuthology.orchestra.run.smithi204.stdout:Total 18 MB/s | 49 MB 00:02 2024-03-21T12:50:44.617 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:50:44.638 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:50:44.638 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:50:44.680 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-21T12:50:44.723 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-21T12:50:44.756 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-21T12:50:44.890 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): libcephsqlite-19.0.0-2267.g960fbbe4.el9. 422 kB/s | 163 kB 00:00 2024-03-21T12:50:45.021 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:50:45.021 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:50:45.257 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 374 kB/s | 187 kB 00:00 2024-03-21T12:50:45.612 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:50:45.691 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-21T12:50:45.733 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-21T12:50:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-21T12:50:45.816 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-21T12:50:45.843 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/6 2024-03-21T12:50:47.457 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): ceph-test-19.0.0-2267.g960fbbe4.el9.x86_ 16 MB/s | 48 MB 00:02 2024-03-21T12:50:47.459 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:50:47.460 INFO:teuthology.orchestra.run.smithi163.stdout:Total 16 MB/s | 49 MB 00:03 2024-03-21T12:50:47.472 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:50:47.490 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:50:47.491 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:50:47.880 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:50:47.880 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:50:48.461 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:50:48.541 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-21T12:50:48.584 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-21T12:50:48.624 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-21T12:50:48.666 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-21T12:50:48.694 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/6 2024-03-21T12:50:50.390 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/6 2024-03-21T12:50:50.470 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 6/6 2024-03-21T12:50:52.958 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 6/6 2024-03-21T12:50:52.959 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/6 2024-03-21T12:50:52.959 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/6 2024-03-21T12:50:52.959 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-21T12:50:52.959 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-21T12:50:52.959 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-21T12:50:53.230 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/6 2024-03-21T12:50:53.311 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 6/6 2024-03-21T12:50:53.372 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-21T12:50:53.372 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:50:53.372 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:50:53.372 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:53.372 INFO:teuthology.orchestra.run.smithi204.stdout: jq-1.6-16.el9.x86_64 2024-03-21T12:50:53.372 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:53.372 INFO:teuthology.orchestra.run.smithi204.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-21T12:50:53.372 INFO:teuthology.orchestra.run.smithi204.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-21T12:50:53.372 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-21T12:50:53.372 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:50:53.372 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:50:53.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph 2024-03-21T12:50:54.231 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:41 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:50:54.389 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:50:54.391 INFO:teuthology.orchestra.run.smithi204.stdout:========================================================================================= 2024-03-21T12:50:54.391 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-21T12:50:54.391 INFO:teuthology.orchestra.run.smithi204.stdout:========================================================================================= 2024-03-21T12:50:54.391 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:50:54.391 INFO:teuthology.orchestra.run.smithi204.stdout: ceph x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 6.6 k 2024-03-21T12:50:54.391 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-21T12:50:54.392 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 2.3 M 2024-03-21T12:50:54.392 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 1.5 M 2024-03-21T12:50:54.392 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 248 k 2024-03-21T12:50:54.392 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 4.7 M 2024-03-21T12:50:54.392 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 17 M 2024-03-21T12:50:54.392 INFO:teuthology.orchestra.run.smithi204.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-21T12:50:54.392 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-21T12:50:54.392 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-21T12:50:54.392 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-21T12:50:54.392 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-21T12:50:54.392 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-21T12:50:54.392 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-21T12:50:54.392 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-21T12:50:54.393 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-21T12:50:54.394 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-21T12:50:54.394 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-21T12:50:54.394 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-21T12:50:54.394 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-21T12:50:54.394 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-21T12:50:54.394 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-21T12:50:54.394 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-21T12:50:54.394 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-21T12:50:54.394 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:50:54.394 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:50:54.394 INFO:teuthology.orchestra.run.smithi204.stdout:========================================================================================= 2024-03-21T12:50:54.394 INFO:teuthology.orchestra.run.smithi204.stdout:Install 36 Packages 2024-03-21T12:50:54.394 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:50:54.395 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 30 M 2024-03-21T12:50:54.395 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 107 M 2024-03-21T12:50:54.395 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:50:55.069 INFO:teuthology.orchestra.run.smithi204.stdout:(1/36): ceph-19.0.0-2267.g960fbbe4.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-03-21T12:50:55.302 INFO:teuthology.orchestra.run.smithi204.stdout:(2/36): ceph-mds-19.0.0-2267.g960fbbe4.el9.x86_ 5.7 MB/s | 2.3 MB 00:00 2024-03-21T12:50:55.344 INFO:teuthology.orchestra.run.smithi204.stdout:(3/36): ceph-mgr-19.0.0-2267.g960fbbe4.el9.x86_ 3.3 MB/s | 1.5 MB 00:00 2024-03-21T12:50:55.436 INFO:teuthology.orchestra.run.smithi204.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2267.g960f 2.6 MB/s | 248 kB 00:00 2024-03-21T12:50:55.553 INFO:teuthology.orchestra.run.smithi204.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-21T12:50:55.595 INFO:teuthology.orchestra.run.smithi204.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-21T12:50:55.620 INFO:teuthology.orchestra.run.smithi204.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-21T12:50:55.646 INFO:teuthology.orchestra.run.smithi204.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-21T12:50:55.671 INFO:teuthology.orchestra.run.smithi204.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-21T12:50:55.713 INFO:teuthology.orchestra.run.smithi204.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-21T12:50:55.780 INFO:teuthology.orchestra.run.smithi204.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-21T12:50:55.814 INFO:teuthology.orchestra.run.smithi204.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-21T12:50:55.847 INFO:teuthology.orchestra.run.smithi204.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-21T12:50:55.881 INFO:teuthology.orchestra.run.smithi204.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-21T12:50:55.885 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 6/6 2024-03-21T12:50:55.885 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/6 2024-03-21T12:50:55.886 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/6 2024-03-21T12:50:55.886 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-21T12:50:55.886 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-21T12:50:55.886 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-21T12:50:56.165 INFO:teuthology.orchestra.run.smithi204.stdout:(15/36): ceph-osd-19.0.0-2267.g960fbbe4.el9.x86 20 MB/s | 17 MB 00:00 2024-03-21T12:50:56.249 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-21T12:50:56.250 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:50:56.250 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:50:56.250 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:56.250 INFO:teuthology.orchestra.run.smithi163.stdout: jq-1.6-16.el9.x86_64 2024-03-21T12:50:56.250 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:50:56.250 INFO:teuthology.orchestra.run.smithi163.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-21T12:50:56.250 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-21T12:50:56.250 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-21T12:50:56.250 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:50:56.250 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:50:56.256 INFO:teuthology.orchestra.run.smithi204.stdout:(16/36): ceph-mon-19.0.0-2267.g960fbbe4.el9.x86 4.0 MB/s | 4.7 MB 00:01 2024-03-21T12:50:56.282 INFO:teuthology.orchestra.run.smithi204.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 312 kB/s | 125 kB 00:00 2024-03-21T12:50:56.407 INFO:teuthology.orchestra.run.smithi204.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 92 kB/s | 22 kB 00:00 2024-03-21T12:50:56.581 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph 2024-03-21T12:50:56.582 INFO:teuthology.orchestra.run.smithi204.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 145 kB/s | 43 kB 00:00 2024-03-21T12:50:56.724 INFO:teuthology.orchestra.run.smithi204.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 323 kB/s | 151 kB 00:00 2024-03-21T12:50:56.833 INFO:teuthology.orchestra.run.smithi204.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 99 kB/s | 11 kB 00:00 2024-03-21T12:50:56.858 INFO:teuthology.orchestra.run.smithi204.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 381 kB/s | 172 kB 00:00 2024-03-21T12:50:56.925 INFO:teuthology.orchestra.run.smithi204.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-03-21T12:50:56.951 INFO:teuthology.orchestra.run.smithi204.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-03-21T12:50:56.976 INFO:teuthology.orchestra.run.smithi204.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 197 kB/s | 23 kB 00:00 2024-03-21T12:50:57.018 INFO:teuthology.orchestra.run.smithi204.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-03-21T12:50:57.051 INFO:teuthology.orchestra.run.smithi204.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-21T12:50:57.085 INFO:teuthology.orchestra.run.smithi204.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 426 kB/s | 46 kB 00:00 2024-03-21T12:50:57.118 INFO:teuthology.orchestra.run.smithi204.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 784 kB/s | 79 kB 00:00 2024-03-21T12:50:57.178 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:41 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:50:57.185 INFO:teuthology.orchestra.run.smithi204.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-21T12:50:57.219 INFO:teuthology.orchestra.run.smithi204.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 897 kB/s | 90 kB 00:00 2024-03-21T12:50:57.244 INFO:teuthology.orchestra.run.smithi204.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-21T12:50:57.286 INFO:teuthology.orchestra.run.smithi204.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-03-21T12:50:57.345 INFO:teuthology.orchestra.run.smithi204.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-21T12:50:57.361 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:50:57.363 INFO:teuthology.orchestra.run.smithi163.stdout:========================================================================================= 2024-03-21T12:50:57.364 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-21T12:50:57.364 INFO:teuthology.orchestra.run.smithi163.stdout:========================================================================================= 2024-03-21T12:50:57.364 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:50:57.364 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 6.6 k 2024-03-21T12:50:57.364 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-21T12:50:57.364 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 2.3 M 2024-03-21T12:50:57.364 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 1.5 M 2024-03-21T12:50:57.364 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 248 k 2024-03-21T12:50:57.364 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 4.7 M 2024-03-21T12:50:57.364 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 17 M 2024-03-21T12:50:57.364 INFO:teuthology.orchestra.run.smithi163.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-21T12:50:57.364 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-21T12:50:57.365 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-21T12:50:57.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-21T12:50:57.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-21T12:50:57.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-21T12:50:57.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-21T12:50:57.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-21T12:50:57.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-21T12:50:57.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-21T12:50:57.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-21T12:50:57.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-21T12:50:57.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-21T12:50:57.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-21T12:50:57.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-21T12:50:57.366 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-21T12:50:57.366 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-21T12:50:57.366 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-21T12:50:57.366 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-21T12:50:57.366 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-21T12:50:57.366 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-21T12:50:57.366 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-21T12:50:57.366 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-21T12:50:57.366 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-21T12:50:57.366 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-21T12:50:57.366 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-21T12:50:57.366 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-21T12:50:57.367 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-21T12:50:57.367 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-21T12:50:57.367 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-21T12:50:57.367 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:50:57.367 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:50:57.367 INFO:teuthology.orchestra.run.smithi163.stdout:========================================================================================= 2024-03-21T12:50:57.367 INFO:teuthology.orchestra.run.smithi163.stdout:Install 36 Packages 2024-03-21T12:50:57.367 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:50:57.368 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 30 M 2024-03-21T12:50:57.368 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 107 M 2024-03-21T12:50:57.368 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:50:57.379 INFO:teuthology.orchestra.run.smithi204.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-03-21T12:50:57.412 INFO:teuthology.orchestra.run.smithi204.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.5 MB/s | 427 kB 00:00 2024-03-21T12:50:57.416 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:50:57.416 INFO:teuthology.orchestra.run.smithi204.stdout:Total 9.9 MB/s | 30 MB 00:03 2024-03-21T12:50:57.440 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:50:57.484 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:50:57.484 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:50:57.879 INFO:teuthology.orchestra.run.smithi163.stdout:(1/36): ceph-19.0.0-2267.g960fbbe4.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-03-21T12:50:57.903 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:50:57.903 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:50:58.004 INFO:teuthology.orchestra.run.smithi163.stdout:(2/36): ceph-mds-19.0.0-2267.g960fbbe4.el9.x86_ 6.5 MB/s | 2.3 MB 00:00 2024-03-21T12:50:58.597 INFO:teuthology.orchestra.run.smithi163.stdout:(3/36): ceph-mgr-19.0.0-2267.g960fbbe4.el9.x86_ 1.6 MB/s | 1.5 MB 00:00 2024-03-21T12:50:58.847 INFO:teuthology.orchestra.run.smithi163.stdout:(4/36): ceph-osd-19.0.0-2267.g960fbbe4.el9.x86_ 20 MB/s | 17 MB 00:00 2024-03-21T12:50:58.872 INFO:teuthology.orchestra.run.smithi163.stdout:(5/36): ceph-mgr-modules-core-19.0.0-2267.g960f 903 kB/s | 248 kB 00:00 2024-03-21T12:50:58.998 INFO:teuthology.orchestra.run.smithi163.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-21T12:50:59.023 INFO:teuthology.orchestra.run.smithi163.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 961 kB/s | 144 kB 00:00 2024-03-21T12:50:59.048 INFO:teuthology.orchestra.run.smithi163.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 630 kB/s | 32 kB 00:00 2024-03-21T12:50:59.073 INFO:teuthology.orchestra.run.smithi163.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 942 kB/s | 47 kB 00:00 2024-03-21T12:50:59.098 INFO:teuthology.orchestra.run.smithi163.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-03-21T12:50:59.124 INFO:teuthology.orchestra.run.smithi163.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-21T12:50:59.182 INFO:teuthology.orchestra.run.smithi163.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-21T12:50:59.208 INFO:teuthology.orchestra.run.smithi163.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-21T12:50:59.231 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:50:59.233 INFO:teuthology.orchestra.run.smithi163.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-21T12:50:59.259 INFO:teuthology.orchestra.run.smithi163.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-21T12:50:59.298 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-21T12:50:59.317 INFO:teuthology.orchestra.run.smithi163.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 384 kB/s | 22 kB 00:00 2024-03-21T12:50:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-21T12:50:59.351 INFO:teuthology.orchestra.run.smithi163.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-03-21T12:50:59.359 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-21T12:50:59.438 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-21T12:50:59.474 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-21T12:50:59.507 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-21T12:50:59.585 INFO:teuthology.orchestra.run.smithi163.stdout:(18/36): ceph-mon-19.0.0-2267.g960fbbe4.el9.x86 2.8 MB/s | 4.7 MB 00:01 2024-03-21T12:50:59.598 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-21T12:50:59.694 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-21T12:50:59.734 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-21T12:50:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-21T12:50:59.826 INFO:teuthology.orchestra.run.smithi163.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 91 kB/s | 43 kB 00:00 2024-03-21T12:50:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-21T12:50:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-21T12:50:59.956 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-21T12:50:59.977 INFO:teuthology.orchestra.run.smithi163.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 229 kB/s | 151 kB 00:00 2024-03-21T12:51:00.019 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-21T12:51:00.077 INFO:teuthology.orchestra.run.smithi163.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-21T12:51:00.186 INFO:teuthology.orchestra.run.smithi163.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-21T12:51:00.204 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-21T12:51:00.220 INFO:teuthology.orchestra.run.smithi163.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 271 kB/s | 172 kB 00:00 2024-03-21T12:51:00.253 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-21T12:51:00.253 INFO:teuthology.orchestra.run.smithi163.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 840 kB/s | 358 kB 00:00 2024-03-21T12:51:00.286 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-21T12:51:00.295 INFO:teuthology.orchestra.run.smithi163.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 212 kB/s | 23 kB 00:00 2024-03-21T12:51:00.326 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-21T12:51:00.345 INFO:teuthology.orchestra.run.smithi163.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2024-03-21T12:51:00.357 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-21T12:51:00.379 INFO:teuthology.orchestra.run.smithi163.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 156 kB/s | 20 kB 00:00 2024-03-21T12:51:00.429 INFO:teuthology.orchestra.run.smithi163.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 346 kB/s | 46 kB 00:00 2024-03-21T12:51:00.433 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-21T12:51:00.467 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-21T12:51:00.488 INFO:teuthology.orchestra.run.smithi163.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 554 kB/s | 79 kB 00:00 2024-03-21T12:51:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-21T12:51:00.547 INFO:teuthology.orchestra.run.smithi163.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-03-21T12:51:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-21T12:51:00.564 INFO:teuthology.orchestra.run.smithi163.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 122 kB/s | 16 kB 00:00 2024-03-21T12:51:00.584 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-21T12:51:00.619 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-21T12:51:00.647 INFO:teuthology.orchestra.run.smithi163.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 565 kB/s | 90 kB 00:00 2024-03-21T12:51:00.673 INFO:teuthology.orchestra.run.smithi163.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 284 kB/s | 36 kB 00:00 2024-03-21T12:51:00.708 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-21T12:51:00.750 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-21T12:51:00.806 INFO:teuthology.orchestra.run.smithi163.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 948 kB/s | 230 kB 00:00 2024-03-21T12:51:00.822 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-21T12:51:00.823 INFO:teuthology.orchestra.run.smithi163.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 133 kB/s | 20 kB 00:00 2024-03-21T12:51:00.957 INFO:teuthology.orchestra.run.smithi163.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.3 MB/s | 427 kB 00:00 2024-03-21T12:51:00.958 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-21T12:51:00.965 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:00.965 INFO:teuthology.orchestra.run.smithi163.stdout:Total 8.3 MB/s | 30 MB 00:03 2024-03-21T12:51:00.989 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:51:01.032 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:51:01.032 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:51:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-21T12:51:01.244 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2267.g960fbbe4.el9. 31/36 2024-03-21T12:51:01.272 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/36 2024-03-21T12:51:01.297 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/36 2024-03-21T12:51:01.297 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:51:01.297 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-21T12:51:01.297 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T12:51:01.298 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T12:51:01.298 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:01.452 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:51:01.452 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:51:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 33/36 2024-03-21T12:51:02.757 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 33/36 2024-03-21T12:51:02.758 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:51:02.758 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-21T12:51:02.758 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T12:51:02.758 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T12:51:02.758 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:02.776 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:51:02.843 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-21T12:51:02.884 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-21T12:51:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-21T12:51:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-21T12:51:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-21T12:51:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-21T12:51:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-21T12:51:03.199 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 34/36 2024-03-21T12:51:03.221 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 34/36 2024-03-21T12:51:03.222 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:51:03.222 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-21T12:51:03.222 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T12:51:03.222 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T12:51:03.222 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-21T12:51:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-21T12:51:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-21T12:51:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-21T12:51:03.449 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 35/36 2024-03-21T12:51:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-21T12:51:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 35/36 2024-03-21T12:51:03.472 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:51:03.473 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-21T12:51:03.473 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T12:51:03.473 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T12:51:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-21T12:51:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 36/36 2024-03-21T12:51:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-21T12:51:03.767 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-21T12:51:03.824 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-21T12:51:03.868 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-21T12:51:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-21T12:51:03.936 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-21T12:51:04.020 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-21T12:51:04.063 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-21T12:51:04.096 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-21T12:51:04.137 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-21T12:51:04.172 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-21T12:51:04.207 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-21T12:51:04.296 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-21T12:51:04.330 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-21T12:51:04.402 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-21T12:51:04.547 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-21T12:51:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-21T12:51:04.830 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2267.g960fbbe4.el9. 31/36 2024-03-21T12:51:04.861 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/36 2024-03-21T12:51:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/36 2024-03-21T12:51:04.887 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:51:04.887 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-21T12:51:04.887 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T12:51:04.887 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T12:51:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:05.312 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 36/36 2024-03-21T12:51:05.312 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/36 2024-03-21T12:51:05.312 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/36 2024-03-21T12:51:05.312 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 3/36 2024-03-21T12:51:05.312 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 4/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2267.g960fbbe4.el9. 6/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-21T12:51:05.313 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-21T12:51:05.314 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-21T12:51:05.314 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-21T12:51:05.314 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-21T12:51:05.314 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-21T12:51:05.314 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-21T12:51:05.314 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-21T12:51:05.314 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-21T12:51:05.314 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-21T12:51:05.314 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-21T12:51:05.315 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-21T12:51:05.315 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-21T12:51:05.316 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-21T12:51:05.316 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-21T12:51:05.316 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-21T12:51:05.316 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-21T12:51:05.316 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-21T12:51:06.250 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-21T12:51:06.250 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout: lua-5.4.4-4.el9.x86_64 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-21T12:51:06.251 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply-3.11-14.el9.noarch 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-21T12:51:06.252 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-21T12:51:06.253 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-21T12:51:06.253 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-21T12:51:06.253 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-21T12:51:06.253 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-21T12:51:06.253 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-21T12:51:06.253 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-21T12:51:06.253 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-21T12:51:06.253 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-21T12:51:06.253 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:06.253 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:06.337 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 33/36 2024-03-21T12:51:06.363 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 33/36 2024-03-21T12:51:06.364 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:51:06.364 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-21T12:51:06.364 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T12:51:06.364 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T12:51:06.364 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:06.593 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-base 2024-03-21T12:51:06.804 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 34/36 2024-03-21T12:51:06.827 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 34/36 2024-03-21T12:51:06.827 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:51:06.827 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-21T12:51:06.828 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T12:51:06.828 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T12:51:06.828 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:07.046 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 35/36 2024-03-21T12:51:07.070 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 35/36 2024-03-21T12:51:07.070 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:51:07.070 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-21T12:51:07.070 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T12:51:07.070 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T12:51:07.070 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:07.136 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 36/36 2024-03-21T12:51:07.158 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:54 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:07.255 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:07.308 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:07.309 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-21T12:51:07.309 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:07.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install cephadm 2024-03-21T12:51:07.931 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:54 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:08.078 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:08.078 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:08.079 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-21T12:51:08.079 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:08.079 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:51:08.079 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 750 k 2024-03-21T12:51:08.079 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:08.079 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:51:08.079 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:08.079 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-21T12:51:08.079 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:08.079 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 750 k 2024-03-21T12:51:08.079 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 757 k 2024-03-21T12:51:08.079 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:51:08.585 INFO:teuthology.orchestra.run.smithi204.stdout:cephadm-19.0.0-2267.g960fbbe4.el9.noarch.rpm 1.4 MB/s | 750 kB 00:00 2024-03-21T12:51:08.586 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:08.586 INFO:teuthology.orchestra.run.smithi204.stdout:Total 1.4 MB/s | 750 kB 00:00 2024-03-21T12:51:08.586 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:51:08.591 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:51:08.591 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:51:08.597 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:51:08.598 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:51:08.693 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:51:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 36/36 2024-03-21T12:51:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/36 2024-03-21T12:51:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/36 2024-03-21T12:51:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 3/36 2024-03-21T12:51:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 4/36 2024-03-21T12:51:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/36 2024-03-21T12:51:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2267.g960fbbe4.el9. 6/36 2024-03-21T12:51:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-21T12:51:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-21T12:51:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-21T12:51:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-21T12:51:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-21T12:51:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-21T12:51:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-21T12:51:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-21T12:51:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-21T12:51:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-21T12:51:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-21T12:51:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-21T12:51:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-21T12:51:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-21T12:51:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-21T12:51:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-21T12:51:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-21T12:51:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-21T12:51:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-21T12:51:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-21T12:51:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-21T12:51:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-21T12:51:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-21T12:51:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-21T12:51:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-21T12:51:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-21T12:51:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-21T12:51:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-21T12:51:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-21T12:51:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T12:51:09.206 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T12:51:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T12:51:09.732 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-21T12:51:09.732 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:09.732 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:51:09.732 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:09.732 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:09.732 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:09.732 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:09.732 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:09.732 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:09.732 INFO:teuthology.orchestra.run.smithi163.stdout: lua-5.4.4-4.el9.x86_64 2024-03-21T12:51:09.732 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-21T12:51:09.733 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply-3.11-14.el9.noarch 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-21T12:51:09.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-21T12:51:09.735 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:09.735 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:09.887 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T12:51:09.887 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:09.887 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:51:09.887 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:09.887 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:09.888 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:10.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-immutable-object-cache 2024-03-21T12:51:10.126 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-base 2024-03-21T12:51:10.690 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:54 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:10.697 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:57 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:10.787 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:10.840 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:10.841 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-21T12:51:10.841 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:10.852 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:10.852 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:10.852 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-21T12:51:10.853 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:10.853 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:51:10.853 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 145 k 2024-03-21T12:51:10.853 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:10.853 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:51:10.853 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:10.853 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-21T12:51:10.853 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:10.854 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 145 k 2024-03-21T12:51:10.854 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 436 k 2024-03-21T12:51:10.854 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:51:10.914 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install cephadm 2024-03-21T12:51:11.084 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-immutable-object-cache-19.0.0-2267.g960fbb 633 kB/s | 145 kB 00:00 2024-03-21T12:51:11.084 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:11.084 INFO:teuthology.orchestra.run.smithi204.stdout:Total 629 kB/s | 145 kB 00:00 2024-03-21T12:51:11.085 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:51:11.092 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:51:11.092 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:51:11.146 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:51:11.147 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:51:11.283 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:51:11.334 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4. 1/1 2024-03-21T12:51:11.359 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4. 1/1 2024-03-21T12:51:11.360 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:51:11.360 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-21T12:51:11.360 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:11.473 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:55 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:11.623 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:11.624 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:11.624 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-21T12:51:11.624 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:11.624 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:51:11.624 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 750 k 2024-03-21T12:51:11.625 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:11.625 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:51:11.625 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:11.625 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-21T12:51:11.625 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:11.625 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 750 k 2024-03-21T12:51:11.625 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 757 k 2024-03-21T12:51:11.625 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:51:12.071 INFO:teuthology.orchestra.run.smithi163.stdout:cephadm-19.0.0-2267.g960fbbe4.el9.noarch.rpm 1.6 MB/s | 750 kB 00:00 2024-03-21T12:51:12.072 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:12.073 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.6 MB/s | 750 kB 00:00 2024-03-21T12:51:12.073 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:51:12.078 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:51:12.078 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:51:12.083 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:51:12.084 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:51:12.156 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:51:12.429 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4. 1/1 2024-03-21T12:51:12.430 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:12.430 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:51:12.430 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:12.430 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:12.430 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:12.613 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T12:51:12.689 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr 2024-03-21T12:51:12.701 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T12:51:13.058 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T12:51:13.262 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:00 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:13.333 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T12:51:13.334 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:13.334 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:51:13.334 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:13.334 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:13.334 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:13.357 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:13.410 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:13.411 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-21T12:51:13.411 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:13.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-dashboard 2024-03-21T12:51:13.578 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-immutable-object-cache 2024-03-21T12:51:14.037 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:01 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:14.152 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:58 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:14.187 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:14.188 INFO:teuthology.orchestra.run.smithi204.stdout:====================================================================================== 2024-03-21T12:51:14.188 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-21T12:51:14.188 INFO:teuthology.orchestra.run.smithi204.stdout:====================================================================================== 2024-03-21T12:51:14.188 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:51:14.188 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 3.5 M 2024-03-21T12:51:14.188 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-21T12:51:14.188 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-21T12:51:14.188 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 34 k 2024-03-21T12:51:14.188 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 16 k 2024-03-21T12:51:14.189 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-21T12:51:14.189 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-21T12:51:14.189 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-21T12:51:14.189 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-21T12:51:14.189 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-21T12:51:14.189 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-21T12:51:14.189 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-21T12:51:14.189 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-21T12:51:14.189 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:14.189 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:51:14.189 INFO:teuthology.orchestra.run.smithi204.stdout:====================================================================================== 2024-03-21T12:51:14.189 INFO:teuthology.orchestra.run.smithi204.stdout:Install 12 Packages 2024-03-21T12:51:14.189 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:14.190 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 8.6 M 2024-03-21T12:51:14.190 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 103 M 2024-03-21T12:51:14.190 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:51:14.304 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:14.304 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:14.304 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-21T12:51:14.305 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:14.305 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:51:14.305 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 145 k 2024-03-21T12:51:14.305 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:14.305 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:51:14.305 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:14.305 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-21T12:51:14.305 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:14.306 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 145 k 2024-03-21T12:51:14.306 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 436 k 2024-03-21T12:51:14.306 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:51:14.612 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-immutable-object-cache-19.0.0-2267.g960fbb 474 kB/s | 145 kB 00:00 2024-03-21T12:51:14.612 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:14.613 INFO:teuthology.orchestra.run.smithi163.stdout:Total 472 kB/s | 145 kB 00:00 2024-03-21T12:51:14.613 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:51:14.620 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:51:14.621 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:51:14.674 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:51:14.674 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:51:14.780 INFO:teuthology.orchestra.run.smithi204.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2267.g96 141 kB/s | 34 kB 00:00 2024-03-21T12:51:14.811 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:51:14.830 INFO:teuthology.orchestra.run.smithi204.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2267.g960 57 kB/s | 16 kB 00:00 2024-03-21T12:51:14.880 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4. 1/1 2024-03-21T12:51:14.904 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4. 1/1 2024-03-21T12:51:14.904 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:51:14.904 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-21T12:51:14.904 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:14.914 INFO:teuthology.orchestra.run.smithi204.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-21T12:51:14.947 INFO:teuthology.orchestra.run.smithi204.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-21T12:51:15.123 INFO:teuthology.orchestra.run.smithi204.stdout:(5/12): ceph-mgr-dashboard-19.0.0-2267.g960fbbe 6.0 MB/s | 3.5 MB 00:00 2024-03-21T12:51:15.248 INFO:teuthology.orchestra.run.smithi204.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.5 MB/s | 863 kB 00:00 2024-03-21T12:51:15.357 INFO:teuthology.orchestra.run.smithi204.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-03-21T12:51:15.390 INFO:teuthology.orchestra.run.smithi204.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-21T12:51:15.449 INFO:teuthology.orchestra.run.smithi204.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 535 kB/s | 31 kB 00:00 2024-03-21T12:51:15.532 INFO:teuthology.orchestra.run.smithi204.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-21T12:51:15.557 INFO:teuthology.orchestra.run.smithi204.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 720 kB/s | 144 kB 00:00 2024-03-21T12:51:15.799 INFO:teuthology.orchestra.run.smithi204.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.7 MB/s | 2.0 MB 00:00 2024-03-21T12:51:15.805 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:15.805 INFO:teuthology.orchestra.run.smithi204.stdout:Total 5.3 MB/s | 8.6 MB 00:01 2024-03-21T12:51:15.822 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:51:15.839 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:51:15.839 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:51:15.892 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4. 1/1 2024-03-21T12:51:15.892 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:15.892 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:51:15.892 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:15.892 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:15.892 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:16.024 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:51:16.024 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:51:16.202 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr 2024-03-21T12:51:16.552 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:51:16.633 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-21T12:51:16.668 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-21T12:51:16.715 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-21T12:51:16.754 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-21T12:51:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:00 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:16.861 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-21T12:51:16.865 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:16.917 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:16.918 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-21T12:51:16.918 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:16.997 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-21T12:51:17.001 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-dashboard 2024-03-21T12:51:17.100 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-21T12:51:17.150 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-21T12:51:17.179 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-21T12:51:17.211 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2267.g960fbbe4.el9 10/12 2024-03-21T12:51:17.562 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:01 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:17.711 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:17.712 INFO:teuthology.orchestra.run.smithi163.stdout:====================================================================================== 2024-03-21T12:51:17.713 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-21T12:51:17.713 INFO:teuthology.orchestra.run.smithi163.stdout:====================================================================================== 2024-03-21T12:51:17.713 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:51:17.713 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 3.5 M 2024-03-21T12:51:17.713 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-21T12:51:17.713 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-21T12:51:17.713 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 34 k 2024-03-21T12:51:17.713 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 16 k 2024-03-21T12:51:17.713 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-21T12:51:17.713 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-21T12:51:17.713 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-21T12:51:17.713 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-21T12:51:17.713 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-21T12:51:17.714 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-21T12:51:17.714 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-21T12:51:17.714 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-21T12:51:17.714 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:17.714 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:51:17.714 INFO:teuthology.orchestra.run.smithi163.stdout:====================================================================================== 2024-03-21T12:51:17.714 INFO:teuthology.orchestra.run.smithi163.stdout:Install 12 Packages 2024-03-21T12:51:17.714 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:17.715 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 8.6 M 2024-03-21T12:51:17.715 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 103 M 2024-03-21T12:51:17.715 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:51:18.043 INFO:teuthology.orchestra.run.smithi163.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2267.g96 180 kB/s | 34 kB 00:00 2024-03-21T12:51:18.143 INFO:teuthology.orchestra.run.smithi163.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2267.g960 57 kB/s | 16 kB 00:00 2024-03-21T12:51:18.227 INFO:teuthology.orchestra.run.smithi163.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-21T12:51:18.252 INFO:teuthology.orchestra.run.smithi163.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 238 kB 00:00 2024-03-21T12:51:18.268 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2267.g960fbbe4.el 11/12 2024-03-21T12:51:18.303 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noa 12/12 2024-03-21T12:51:18.444 INFO:teuthology.orchestra.run.smithi163.stdout:(5/12): ceph-mgr-dashboard-19.0.0-2267.g960fbbe 6.0 MB/s | 3.5 MB 00:00 2024-03-21T12:51:18.777 INFO:teuthology.orchestra.run.smithi163.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 58 kB/s | 19 kB 00:00 2024-03-21T12:51:18.961 INFO:teuthology.orchestra.run.smithi163.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.1 MB/s | 863 kB 00:00 2024-03-21T12:51:18.986 INFO:teuthology.orchestra.run.smithi163.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 747 kB/s | 548 kB 00:00 2024-03-21T12:51:19.087 INFO:teuthology.orchestra.run.smithi163.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 307 kB/s | 31 kB 00:00 2024-03-21T12:51:19.212 INFO:teuthology.orchestra.run.smithi163.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-21T12:51:19.379 INFO:teuthology.orchestra.run.smithi163.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.4 MB/s | 2.0 MB 00:00 2024-03-21T12:51:19.487 INFO:teuthology.orchestra.run.smithi163.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 275 kB/s | 144 kB 00:00 2024-03-21T12:51:19.492 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:19.492 INFO:teuthology.orchestra.run.smithi163.stdout:Total 4.8 MB/s | 8.6 MB 00:01 2024-03-21T12:51:19.510 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:51:19.527 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:51:19.527 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:51:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noa 12/12 2024-03-21T12:51:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2267.g960fbbe4.el 1/12 2024-03-21T12:51:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noa 2/12 2024-03-21T12:51:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2267.g960fbbe4.el9 3/12 2024-03-21T12:51:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-21T12:51:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-21T12:51:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-21T12:51:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-21T12:51:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-21T12:51:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-21T12:51:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-21T12:51:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-21T12:51:19.731 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:51:19.732 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:51:20.038 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-21T12:51:20.038 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:20.038 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:51:20.038 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-21T12:51:20.038 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:20.038 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:20.038 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:20.038 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-21T12:51:20.038 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-21T12:51:20.038 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-21T12:51:20.038 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-21T12:51:20.039 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-21T12:51:20.039 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-21T12:51:20.039 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-21T12:51:20.039 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-21T12:51:20.039 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:20.039 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:20.266 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:51:20.299 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-21T12:51:20.346 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-21T12:51:20.382 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-21T12:51:20.428 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-21T12:51:20.459 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-21T12:51:20.582 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-21T12:51:20.710 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-21T12:51:20.804 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-21T12:51:20.848 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-21T12:51:20.865 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:07 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:20.876 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-21T12:51:20.908 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2267.g960fbbe4.el9 10/12 2024-03-21T12:51:21.019 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:21.020 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================================ 2024-03-21T12:51:21.020 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-21T12:51:21.020 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================================ 2024-03-21T12:51:21.021 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:51:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 7.4 M 2024-03-21T12:51:21.021 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-21T12:51:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-21T12:51:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-21T12:51:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-21T12:51:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-21T12:51:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-21T12:51:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-21T12:51:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-21T12:51:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-21T12:51:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-21T12:51:21.022 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-21T12:51:21.022 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-21T12:51:21.022 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:21.022 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:51:21.022 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================================ 2024-03-21T12:51:21.022 INFO:teuthology.orchestra.run.smithi204.stdout:Install 12 Packages 2024-03-21T12:51:21.022 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:21.023 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 37 M 2024-03-21T12:51:21.023 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 209 M 2024-03-21T12:51:21.023 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:51:21.275 INFO:teuthology.orchestra.run.smithi204.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 213 kB/s | 32 kB 00:00 2024-03-21T12:51:21.317 INFO:teuthology.orchestra.run.smithi204.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 442 kB/s | 17 kB 00:00 2024-03-21T12:51:21.367 INFO:teuthology.orchestra.run.smithi204.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-21T12:51:21.392 INFO:teuthology.orchestra.run.smithi204.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 460 kB/s | 34 kB 00:00 2024-03-21T12:51:21.550 INFO:teuthology.orchestra.run.smithi204.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-21T12:51:21.626 INFO:teuthology.orchestra.run.smithi204.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-21T12:51:21.651 INFO:teuthology.orchestra.run.smithi204.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-21T12:51:21.959 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2267.g960fbbe4.el 11/12 2024-03-21T12:51:21.960 INFO:teuthology.orchestra.run.smithi204.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-21T12:51:21.992 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noa 12/12 2024-03-21T12:51:22.085 INFO:teuthology.orchestra.run.smithi204.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-03-21T12:51:22.169 INFO:teuthology.orchestra.run.smithi204.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 937 kB/s | 195 kB 00:00 2024-03-21T12:51:22.452 INFO:teuthology.orchestra.run.smithi204.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 669 kB/s | 245 kB 00:00 2024-03-21T12:51:22.802 INFO:teuthology.orchestra.run.smithi204.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 4.4 MB/s | 7.4 MB 00:01 2024-03-21T12:51:22.804 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:22.804 INFO:teuthology.orchestra.run.smithi204.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-21T12:51:22.845 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:51:22.868 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:51:22.868 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:51:23.220 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:51:23.220 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:51:23.433 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noa 12/12 2024-03-21T12:51:23.433 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2267.g960fbbe4.el 1/12 2024-03-21T12:51:23.433 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noa 2/12 2024-03-21T12:51:23.433 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2267.g960fbbe4.el9 3/12 2024-03-21T12:51:23.433 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-21T12:51:23.433 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-21T12:51:23.433 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-21T12:51:23.433 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-21T12:51:23.433 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-21T12:51:23.433 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-21T12:51:23.433 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-21T12:51:23.434 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-21T12:51:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-21T12:51:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:23.786 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:23.921 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:51:23.984 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-21T12:51:24.015 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-21T12:51:24.068 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-21T12:51:24.116 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-21T12:51:24.168 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-21T12:51:24.329 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-21T12:51:24.359 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-21T12:51:24.444 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-21T12:51:24.638 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:08 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:24.791 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:24.792 INFO:teuthology.orchestra.run.smithi163.stdout:============================================================================================ 2024-03-21T12:51:24.792 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-21T12:51:24.792 INFO:teuthology.orchestra.run.smithi163.stdout:============================================================================================ 2024-03-21T12:51:24.792 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:51:24.792 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 7.4 M 2024-03-21T12:51:24.792 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-21T12:51:24.792 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout:============================================================================================ 2024-03-21T12:51:24.793 INFO:teuthology.orchestra.run.smithi163.stdout:Install 12 Packages 2024-03-21T12:51:24.794 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:24.794 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 37 M 2024-03-21T12:51:24.794 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 209 M 2024-03-21T12:51:24.794 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:51:25.096 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-21T12:51:25.188 INFO:teuthology.orchestra.run.smithi163.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 148 kB/s | 32 kB 00:00 2024-03-21T12:51:25.204 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-21T12:51:25.247 INFO:teuthology.orchestra.run.smithi163.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 294 kB/s | 17 kB 00:00 2024-03-21T12:51:25.314 INFO:teuthology.orchestra.run.smithi163.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 512 kB/s | 34 kB 00:00 2024-03-21T12:51:25.464 INFO:teuthology.orchestra.run.smithi163.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.1 MB/s | 3.0 MB 00:00 2024-03-21T12:51:25.698 INFO:teuthology.orchestra.run.smithi163.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-03-21T12:51:25.756 INFO:teuthology.orchestra.run.smithi163.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-21T12:51:25.782 INFO:teuthology.orchestra.run.smithi163.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-21T12:51:25.874 INFO:teuthology.orchestra.run.smithi163.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-22 8.2 MB/s | 7.4 MB 00:00 2024-03-21T12:51:26.191 INFO:teuthology.orchestra.run.smithi163.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-21T12:51:26.424 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-21T12:51:26.491 INFO:teuthology.orchestra.run.smithi163.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 318 kB/s | 195 kB 00:00 2024-03-21T12:51:26.616 INFO:teuthology.orchestra.run.smithi163.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 965 kB/s | 803 kB 00:00 2024-03-21T12:51:26.692 INFO:teuthology.orchestra.run.smithi163.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 490 kB/s | 245 kB 00:00 2024-03-21T12:51:26.694 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:26.694 INFO:teuthology.orchestra.run.smithi163.stdout:Total 20 MB/s | 37 MB 00:01 2024-03-21T12:51:26.738 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:51:26.763 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:51:26.764 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:51:27.122 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:51:27.122 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:51:27.438 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-21T12:51:27.468 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fb 12/12 2024-03-21T12:51:27.834 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:51:27.906 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-21T12:51:27.937 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-21T12:51:28.032 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-21T12:51:28.066 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-21T12:51:28.241 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-21T12:51:28.264 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-21T12:51:28.353 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-21T12:51:29.031 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-21T12:51:29.137 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-21T12:51:29.956 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fb 12/12 2024-03-21T12:51:29.956 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fb 1/12 2024-03-21T12:51:29.956 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-21T12:51:29.956 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-21T12:51:29.956 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-21T12:51:29.956 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-21T12:51:29.956 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-21T12:51:29.956 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-21T12:51:29.956 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-21T12:51:29.956 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-21T12:51:29.957 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-21T12:51:29.957 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-21T12:51:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-21T12:51:30.414 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-21T12:51:30.414 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:30.414 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:51:30.414 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:30.414 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-21T12:51:30.414 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-21T12:51:30.414 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-21T12:51:30.414 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-21T12:51:30.414 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-21T12:51:30.414 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-21T12:51:30.415 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-21T12:51:30.415 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-21T12:51:30.415 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-21T12:51:30.415 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-21T12:51:30.415 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-21T12:51:30.415 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:30.415 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:30.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-rook 2024-03-21T12:51:31.392 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:18 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:31.421 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-21T12:51:31.451 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fb 12/12 2024-03-21T12:51:31.543 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:31.544 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-21T12:51:31.544 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-21T12:51:31.544 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-21T12:51:31.544 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:51:31.544 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 49 k 2024-03-21T12:51:31.545 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-21T12:51:31.545 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-21T12:51:31.545 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-21T12:51:31.545 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-21T12:51:31.545 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-21T12:51:31.545 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-21T12:51:31.545 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-21T12:51:31.545 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-21T12:51:31.545 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-21T12:51:31.545 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-21T12:51:31.545 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-21T12:51:31.545 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-21T12:51:31.545 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-21T12:51:31.546 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:31.546 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:51:31.546 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-21T12:51:31.546 INFO:teuthology.orchestra.run.smithi204.stdout:Install 13 Packages 2024-03-21T12:51:31.546 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:31.546 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 2.0 M 2024-03-21T12:51:31.547 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 26 M 2024-03-21T12:51:31.547 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:51:31.753 INFO:teuthology.orchestra.run.smithi204.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 150 kB/s | 18 kB 00:00 2024-03-21T12:51:31.770 INFO:teuthology.orchestra.run.smithi204.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 178 kB/s | 24 kB 00:00 2024-03-21T12:51:31.820 INFO:teuthology.orchestra.run.smithi204.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-21T12:51:31.845 INFO:teuthology.orchestra.run.smithi204.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-21T12:51:31.870 INFO:teuthology.orchestra.run.smithi204.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-21T12:51:31.896 INFO:teuthology.orchestra.run.smithi204.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-03-21T12:51:31.921 INFO:teuthology.orchestra.run.smithi204.stdout:(7/13): ceph-mgr-rook-19.0.0-2267.g960fbbe4.el9 171 kB/s | 49 kB 00:00 2024-03-21T12:51:32.005 INFO:teuthology.orchestra.run.smithi204.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 137 kB/s | 15 kB 00:00 2024-03-21T12:51:32.031 INFO:teuthology.orchestra.run.smithi204.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-03-21T12:51:32.139 INFO:teuthology.orchestra.run.smithi204.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 931 kB/s | 203 kB 00:00 2024-03-21T12:51:32.164 INFO:teuthology.orchestra.run.smithi204.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 442 kB/s | 59 kB 00:00 2024-03-21T12:51:32.215 INFO:teuthology.orchestra.run.smithi204.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-21T12:51:32.290 INFO:teuthology.orchestra.run.smithi204.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2024-03-21T12:51:32.291 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:32.291 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-03-21T12:51:32.292 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:51:32.304 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:51:32.304 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:51:32.394 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:51:32.394 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:51:32.902 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:51:33.077 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-21T12:51:33.132 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-21T12:51:33.196 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-21T12:51:33.233 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-21T12:51:33.275 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-21T12:51:33.376 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-21T12:51:33.485 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-21T12:51:33.534 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-21T12:51:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fb 12/12 2024-03-21T12:51:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fb 1/12 2024-03-21T12:51:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-21T12:51:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-21T12:51:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-21T12:51:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-21T12:51:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-21T12:51:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-21T12:51:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-21T12:51:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-21T12:51:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-21T12:51:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-21T12:51:34.258 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-21T12:51:34.307 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-21T12:51:34.341 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-21T12:51:34.372 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-21T12:51:34.373 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:34.373 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:51:34.373 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:34.373 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-21T12:51:34.373 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-21T12:51:34.373 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-21T12:51:34.373 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-21T12:51:34.373 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-21T12:51:34.373 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-21T12:51:34.373 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-21T12:51:34.373 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-21T12:51:34.373 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-21T12:51:34.373 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-21T12:51:34.374 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-21T12:51:34.374 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:34.374 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:34.389 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-21T12:51:34.411 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 13/13 2024-03-21T12:51:34.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-rook 2024-03-21T12:51:35.277 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 13/13 2024-03-21T12:51:35.277 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 1/13 2024-03-21T12:51:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-21T12:51:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-21T12:51:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-21T12:51:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-21T12:51:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-21T12:51:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-21T12:51:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-21T12:51:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-21T12:51:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-21T12:51:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-21T12:51:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-21T12:51:35.405 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:19 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:35.555 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:35.556 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-21T12:51:35.556 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-21T12:51:35.556 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-21T12:51:35.556 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:51:35.556 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 49 k 2024-03-21T12:51:35.557 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-21T12:51:35.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-21T12:51:35.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-21T12:51:35.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-21T12:51:35.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-21T12:51:35.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-21T12:51:35.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-21T12:51:35.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-21T12:51:35.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-21T12:51:35.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-21T12:51:35.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-21T12:51:35.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-21T12:51:35.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-21T12:51:35.558 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:35.558 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:51:35.558 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-21T12:51:35.558 INFO:teuthology.orchestra.run.smithi163.stdout:Install 13 Packages 2024-03-21T12:51:35.558 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:35.558 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 2.0 M 2024-03-21T12:51:35.559 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 26 M 2024-03-21T12:51:35.559 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:51:35.637 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-21T12:51:35.637 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:35.637 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:51:35.637 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:35.637 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-21T12:51:35.637 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-21T12:51:35.637 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-21T12:51:35.637 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-21T12:51:35.637 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-21T12:51:35.638 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-21T12:51:35.638 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-21T12:51:35.638 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-21T12:51:35.638 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-21T12:51:35.638 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-21T12:51:35.638 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-21T12:51:35.638 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-21T12:51:35.638 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:35.638 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:35.835 INFO:teuthology.orchestra.run.smithi163.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 193 kB/s | 24 kB 00:00 2024-03-21T12:51:35.860 INFO:teuthology.orchestra.run.smithi163.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-03-21T12:51:35.902 INFO:teuthology.orchestra.run.smithi163.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-21T12:51:35.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-cephadm 2024-03-21T12:51:35.927 INFO:teuthology.orchestra.run.smithi163.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-21T12:51:35.953 INFO:teuthology.orchestra.run.smithi163.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-21T12:51:35.978 INFO:teuthology.orchestra.run.smithi163.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-21T12:51:36.004 INFO:teuthology.orchestra.run.smithi163.stdout:(7/13): ceph-mgr-rook-19.0.0-2267.g960fbbe4.el9 167 kB/s | 49 kB 00:00 2024-03-21T12:51:36.221 INFO:teuthology.orchestra.run.smithi163.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-03-21T12:51:36.246 INFO:teuthology.orchestra.run.smithi163.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-03-21T12:51:36.380 INFO:teuthology.orchestra.run.smithi163.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 539 kB/s | 203 kB 00:00 2024-03-21T12:51:36.405 INFO:teuthology.orchestra.run.smithi163.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 372 kB/s | 59 kB 00:00 2024-03-21T12:51:36.468 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:23 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:36.488 INFO:teuthology.orchestra.run.smithi163.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 825 kB/s | 90 kB 00:00 2024-03-21T12:51:36.530 INFO:teuthology.orchestra.run.smithi163.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2024-03-21T12:51:36.534 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:36.535 INFO:teuthology.orchestra.run.smithi163.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-21T12:51:36.535 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:51:36.548 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:51:36.548 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:51:36.619 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:36.620 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-21T12:51:36.620 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-21T12:51:36.620 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-21T12:51:36.620 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:51:36.620 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 140 k 2024-03-21T12:51:36.620 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-21T12:51:36.620 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-21T12:51:36.620 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-21T12:51:36.620 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-21T12:51:36.620 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-21T12:51:36.621 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-21T12:51:36.621 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:36.621 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:51:36.621 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-21T12:51:36.621 INFO:teuthology.orchestra.run.smithi204.stdout:Install 6 Packages 2024-03-21T12:51:36.621 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:36.621 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 6.8 M 2024-03-21T12:51:36.622 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 33 M 2024-03-21T12:51:36.622 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:51:36.638 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:51:36.638 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:51:37.084 INFO:teuthology.orchestra.run.smithi204.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-21T12:51:37.126 INFO:teuthology.orchestra.run.smithi204.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2267.g960fbbe4.e 444 kB/s | 140 kB 00:00 2024-03-21T12:51:37.146 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:51:37.151 INFO:teuthology.orchestra.run.smithi204.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 728 kB/s | 248 kB 00:00 2024-03-21T12:51:37.285 INFO:teuthology.orchestra.run.smithi204.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 364 kB/s | 58 kB 00:00 2024-03-21T12:51:37.310 INFO:teuthology.orchestra.run.smithi204.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 322 kB/s | 51 kB 00:00 2024-03-21T12:51:37.312 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-21T12:51:37.364 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-21T12:51:37.402 INFO:teuthology.orchestra.run.smithi204.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-03-21T12:51:37.405 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:37.405 INFO:teuthology.orchestra.run.smithi204.stdout:Total 8.7 MB/s | 6.8 MB 00:00 2024-03-21T12:51:37.417 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:51:37.419 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-21T12:51:37.428 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:51:37.428 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:51:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-21T12:51:37.490 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-21T12:51:37.515 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:51:37.515 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:51:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-21T12:51:37.693 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-21T12:51:37.742 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-21T12:51:37.815 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:51:37.974 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-21T12:51:38.029 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-21T12:51:38.466 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-21T12:51:38.515 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-21T12:51:38.545 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-21T12:51:38.549 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-21T12:51:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-21T12:51:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 13/13 2024-03-21T12:51:38.625 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-21T12:51:38.687 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-21T12:51:38.710 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 6/6 2024-03-21T12:51:39.416 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 13/13 2024-03-21T12:51:39.416 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 1/13 2024-03-21T12:51:39.416 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-21T12:51:39.416 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-21T12:51:39.416 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-21T12:51:39.416 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-21T12:51:39.416 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-21T12:51:39.416 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-21T12:51:39.416 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-21T12:51:39.416 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-21T12:51:39.416 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-21T12:51:39.417 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-21T12:51:39.417 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-21T12:51:39.512 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 6/6 2024-03-21T12:51:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/6 2024-03-21T12:51:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-21T12:51:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-21T12:51:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-21T12:51:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-21T12:51:39.772 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-21T12:51:39.772 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:39.772 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:51:39.772 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:39.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-21T12:51:39.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-21T12:51:39.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-21T12:51:39.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-21T12:51:39.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-21T12:51:39.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-21T12:51:39.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-21T12:51:39.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-21T12:51:39.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-21T12:51:39.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-21T12:51:39.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-21T12:51:39.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-21T12:51:39.773 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:39.773 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:39.817 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-21T12:51:39.818 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:39.818 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:51:39.818 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:39.818 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-21T12:51:39.818 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-21T12:51:39.818 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-21T12:51:39.818 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-21T12:51:39.818 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-21T12:51:39.818 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:39.818 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:40.056 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-cephadm 2024-03-21T12:51:40.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-fuse 2024-03-21T12:51:40.624 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:24 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:40.646 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:27 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:40.774 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:40.774 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-21T12:51:40.775 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-21T12:51:40.775 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-21T12:51:40.775 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:51:40.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 140 k 2024-03-21T12:51:40.775 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-21T12:51:40.775 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-21T12:51:40.775 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-21T12:51:40.775 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-21T12:51:40.775 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-21T12:51:40.775 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-21T12:51:40.775 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:40.776 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:51:40.776 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-21T12:51:40.776 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2024-03-21T12:51:40.776 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:40.776 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 6.8 M 2024-03-21T12:51:40.776 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 33 M 2024-03-21T12:51:40.776 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:51:40.797 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:40.798 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:40.798 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-21T12:51:40.799 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:40.799 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:51:40.799 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 890 k 2024-03-21T12:51:40.799 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-21T12:51:40.799 INFO:teuthology.orchestra.run.smithi204.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-21T12:51:40.799 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:40.799 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:51:40.799 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:40.799 INFO:teuthology.orchestra.run.smithi204.stdout:Install 2 Packages 2024-03-21T12:51:40.799 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:40.800 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 969 k 2024-03-21T12:51:40.800 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 2.8 M 2024-03-21T12:51:40.800 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:51:40.931 INFO:teuthology.orchestra.run.smithi204.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 599 kB/s | 79 kB 00:00 2024-03-21T12:51:41.139 INFO:teuthology.orchestra.run.smithi204.stdout:(2/2): ceph-fuse-19.0.0-2267.g960fbbe4.el9.x86_ 2.6 MB/s | 890 kB 00:00 2024-03-21T12:51:41.141 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:41.141 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.8 MB/s | 969 kB 00:00 2024-03-21T12:51:41.141 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:51:41.161 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:51:41.161 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:51:41.186 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-21T12:51:41.235 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:51:41.235 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:51:41.253 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2267.g960fbbe4.e 575 kB/s | 140 kB 00:00 2024-03-21T12:51:41.328 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-21T12:51:41.410 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:51:41.554 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-21T12:51:41.612 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 161 kB/s | 58 kB 00:00 2024-03-21T12:51:41.623 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/2 2024-03-21T12:51:41.637 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 165 kB/s | 51 kB 00:00 2024-03-21T12:51:41.837 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 844 kB/s | 548 kB 00:00 2024-03-21T12:51:41.839 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:41.839 INFO:teuthology.orchestra.run.smithi163.stdout:Total 6.4 MB/s | 6.8 MB 00:01 2024-03-21T12:51:41.852 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:51:41.863 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:51:41.863 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:51:41.950 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:51:41.951 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:51:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:51:42.358 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/2 2024-03-21T12:51:42.358 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/2 2024-03-21T12:51:42.429 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-21T12:51:42.481 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-21T12:51:42.677 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-21T12:51:42.677 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:42.677 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:51:42.677 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-21T12:51:42.678 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:42.678 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:42.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-volume 2024-03-21T12:51:43.004 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-21T12:51:43.092 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-21T12:51:43.155 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-21T12:51:43.185 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 6/6 2024-03-21T12:51:43.554 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:30 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:43.707 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:43.708 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:43.708 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-21T12:51:43.708 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:43.708 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:51:43.708 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 273 k 2024-03-21T12:51:43.708 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-21T12:51:43.708 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-21T12:51:43.708 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:43.708 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:51:43.709 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:43.709 INFO:teuthology.orchestra.run.smithi204.stdout:Install 2 Packages 2024-03-21T12:51:43.709 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:43.709 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 342 k 2024-03-21T12:51:43.709 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 1.5 M 2024-03-21T12:51:43.709 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:51:43.836 INFO:teuthology.orchestra.run.smithi204.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 543 kB/s | 69 kB 00:00 2024-03-21T12:51:43.983 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 6/6 2024-03-21T12:51:43.983 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/6 2024-03-21T12:51:43.983 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-21T12:51:43.983 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-21T12:51:43.984 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-21T12:51:43.984 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-21T12:51:44.078 INFO:teuthology.orchestra.run.smithi204.stdout:(2/2): ceph-volume-19.0.0-2267.g960fbbe4.el9.no 742 kB/s | 273 kB 00:00 2024-03-21T12:51:44.079 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:44.080 INFO:teuthology.orchestra.run.smithi204.stdout:Total 922 kB/s | 342 kB 00:00 2024-03-21T12:51:44.080 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:51:44.089 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:51:44.089 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:51:44.121 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:51:44.122 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:51:44.266 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:51:44.287 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-21T12:51:44.287 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:44.289 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:51:44.289 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:44.289 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-21T12:51:44.289 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-21T12:51:44.289 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-21T12:51:44.289 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-21T12:51:44.289 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-21T12:51:44.289 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:44.289 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:44.442 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-21T12:51:44.489 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 2/2 2024-03-21T12:51:44.506 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 2/2 2024-03-21T12:51:44.506 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:51:44.506 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-21T12:51:44.506 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:44.562 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-fuse 2024-03-21T12:51:45.144 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:29 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:45.244 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 1/2 2024-03-21T12:51:45.297 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:45.297 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:45.297 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-21T12:51:45.297 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:45.297 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:51:45.297 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 890 k 2024-03-21T12:51:45.298 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-21T12:51:45.298 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-21T12:51:45.298 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:45.298 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:51:45.298 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:45.298 INFO:teuthology.orchestra.run.smithi163.stdout:Install 2 Packages 2024-03-21T12:51:45.298 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:45.299 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 969 k 2024-03-21T12:51:45.299 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 2.8 M 2024-03-21T12:51:45.299 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:51:45.515 INFO:teuthology.orchestra.run.smithi163.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 364 kB/s | 79 kB 00:00 2024-03-21T12:51:45.576 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-21T12:51:45.576 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:45.576 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:51:45.576 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:45.576 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-21T12:51:45.576 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:45.577 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:45.815 INFO:teuthology.orchestra.run.smithi163.stdout:(2/2): ceph-fuse-19.0.0-2267.g960fbbe4.el9.x86_ 1.7 MB/s | 890 kB 00:00 2024-03-21T12:51:45.816 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:45.817 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.8 MB/s | 969 kB 00:00 2024-03-21T12:51:45.817 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:51:45.837 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:51:45.837 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:51:45.910 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:51:45.911 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:51:45.938 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados-devel 2024-03-21T12:51:46.091 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:51:46.244 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-21T12:51:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/2 2024-03-21T12:51:46.521 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:33 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:46.672 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:46.672 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:46.673 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-21T12:51:46.673 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:46.673 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:51:46.673 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 127 k 2024-03-21T12:51:46.673 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:46.673 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:51:46.673 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:46.673 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-21T12:51:46.673 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:46.673 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 127 k 2024-03-21T12:51:46.673 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 456 k 2024-03-21T12:51:46.673 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:51:46.984 INFO:teuthology.orchestra.run.smithi204.stdout:librados-devel-19.0.0-2267.g960fbbe4.el9.x86_64 408 kB/s | 127 kB 00:00 2024-03-21T12:51:46.985 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:46.985 INFO:teuthology.orchestra.run.smithi204.stdout:Total 405 kB/s | 127 kB 00:00 2024-03-21T12:51:46.986 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:51:46.991 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:51:46.991 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:51:47.012 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/2 2024-03-21T12:51:47.012 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/2 2024-03-21T12:51:47.052 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:51:47.052 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:51:47.204 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:51:47.308 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-21T12:51:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:47.328 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:51:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-21T12:51:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:47.328 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:47.628 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-volume 2024-03-21T12:51:47.772 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:48.047 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:48.048 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:48.048 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:51:48.048 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:48.048 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:48.048 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:48.204 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:32 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:48.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs2 2024-03-21T12:51:48.360 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:48.361 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:48.361 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-21T12:51:48.361 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:48.361 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:51:48.361 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 273 k 2024-03-21T12:51:48.361 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-21T12:51:48.361 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-21T12:51:48.361 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:48.361 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:51:48.361 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:48.361 INFO:teuthology.orchestra.run.smithi163.stdout:Install 2 Packages 2024-03-21T12:51:48.361 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:48.362 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 342 k 2024-03-21T12:51:48.362 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 1.5 M 2024-03-21T12:51:48.362 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:51:48.480 INFO:teuthology.orchestra.run.smithi163.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 583 kB/s | 69 kB 00:00 2024-03-21T12:51:48.630 INFO:teuthology.orchestra.run.smithi163.stdout:(2/2): ceph-volume-19.0.0-2267.g960fbbe4.el9.no 1.0 MB/s | 273 kB 00:00 2024-03-21T12:51:48.631 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:48.632 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.2 MB/s | 342 kB 00:00 2024-03-21T12:51:48.632 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:51:48.640 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:51:48.640 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:51:48.673 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:51:48.673 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:51:48.819 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:51:48.869 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:35 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:48.968 INFO:teuthology.orchestra.run.smithi204.stdout:Package libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:49.010 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-21T12:51:49.024 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:49.025 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-21T12:51:49.026 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:49.041 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 2/2 2024-03-21T12:51:49.058 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 2/2 2024-03-21T12:51:49.058 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:51:49.058 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-21T12:51:49.059 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:49.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs-devel 2024-03-21T12:51:49.661 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:36 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:49.776 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 1/2 2024-03-21T12:51:49.812 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:49.813 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:49.813 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-21T12:51:49.813 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:49.814 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:51:49.814 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 31 k 2024-03-21T12:51:49.814 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:49.814 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:51:49.814 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:49.814 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-21T12:51:49.815 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:49.815 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 31 k 2024-03-21T12:51:49.815 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 140 k 2024-03-21T12:51:49.815 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:51:49.980 INFO:teuthology.orchestra.run.smithi204.stdout:libcephfs-devel-19.0.0-2267.g960fbbe4.el9.x86_6 188 kB/s | 31 kB 00:00 2024-03-21T12:51:49.980 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:49.980 INFO:teuthology.orchestra.run.smithi204.stdout:Total 187 kB/s | 31 kB 00:00 2024-03-21T12:51:49.981 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:51:49.984 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:51:49.984 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:51:50.014 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:51:50.015 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:51:50.100 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-21T12:51:50.100 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:50.100 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:51:50.100 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T12:51:50.100 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-21T12:51:50.100 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:50.100 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:50.142 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:51:50.245 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:50.506 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados-devel 2024-03-21T12:51:50.646 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:50.908 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:50.908 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:50.908 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:51:50.908 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:50.908 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:50.908 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:51.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados2 2024-03-21T12:51:51.089 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:35 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:51.238 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:51.239 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:51.239 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-21T12:51:51.239 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:51.240 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:51:51.240 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 127 k 2024-03-21T12:51:51.240 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:51.240 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:51:51.240 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:51.240 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-21T12:51:51.240 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:51.240 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 127 k 2024-03-21T12:51:51.240 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 456 k 2024-03-21T12:51:51.240 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:51:51.546 INFO:teuthology.orchestra.run.smithi163.stdout:librados-devel-19.0.0-2267.g960fbbe4.el9.x86_64 415 kB/s | 127 kB 00:00 2024-03-21T12:51:51.547 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:51.547 INFO:teuthology.orchestra.run.smithi163.stdout:Total 412 kB/s | 127 kB 00:00 2024-03-21T12:51:51.547 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:51:51.552 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:51:51.553 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:51:51.612 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:51:51.613 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:51:51.635 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:38 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:51.735 INFO:teuthology.orchestra.run.smithi204.stdout:Package librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:51.758 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:51:51.786 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:51.787 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-21T12:51:51.787 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:51.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librbd1 2024-03-21T12:51:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:52.366 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:52.392 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:39 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:52.490 INFO:teuthology.orchestra.run.smithi204.stdout:Package librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:52.541 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:52.542 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-21T12:51:52.542 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:52.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rados 2024-03-21T12:51:52.636 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:52.636 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:52.636 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:51:52.636 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:52.636 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:52.636 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:52.937 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs2 2024-03-21T12:51:53.168 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:40 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:53.268 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:53.320 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:53.321 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-21T12:51:53.321 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:53.369 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rgw 2024-03-21T12:51:53.516 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:37 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:53.618 INFO:teuthology.orchestra.run.smithi163.stdout:Package libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:53.673 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:53.674 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-21T12:51:53.675 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:53.764 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs-devel 2024-03-21T12:51:53.931 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:40 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:54.028 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:54.079 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:54.080 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-21T12:51:54.080 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:54.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-cephfs 2024-03-21T12:51:54.331 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:38 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:54.482 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:54.483 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:54.484 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-21T12:51:54.484 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:54.484 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:51:54.484 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 31 k 2024-03-21T12:51:54.484 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:54.484 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:51:54.484 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:51:54.485 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-21T12:51:54.485 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:54.485 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 31 k 2024-03-21T12:51:54.485 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 140 k 2024-03-21T12:51:54.485 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:51:54.705 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:41 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:54.751 INFO:teuthology.orchestra.run.smithi163.stdout:libcephfs-devel-19.0.0-2267.g960fbbe4.el9.x86_6 117 kB/s | 31 kB 00:00 2024-03-21T12:51:54.752 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:54.752 INFO:teuthology.orchestra.run.smithi163.stdout:Total 116 kB/s | 31 kB 00:00 2024-03-21T12:51:54.752 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:51:54.755 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:51:54.755 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:51:54.786 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:51:54.787 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:51:54.807 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:54.860 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:54.861 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-21T12:51:54.861 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:54.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rbd 2024-03-21T12:51:54.935 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:51:55.033 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:55.409 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:55.482 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:42 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:55.582 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:55.634 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:55.635 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-21T12:51:55.635 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:55.675 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:55.675 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:55.675 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:51:55.675 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:55.675 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:51:55.675 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:55.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-fuse 2024-03-21T12:51:55.866 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados2 2024-03-21T12:51:56.247 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:43 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:56.396 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:56.397 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-03-21T12:51:56.397 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:56.398 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:51:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 85 k 2024-03-21T12:51:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:56.398 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:51:56.398 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:56.398 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-21T12:51:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:56.398 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 85 k 2024-03-21T12:51:56.398 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 231 k 2024-03-21T12:51:56.398 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:51:56.427 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:40 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:56.526 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:56.577 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:56.578 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-21T12:51:56.578 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:56.627 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-fuse-19.0.0-2267.g960fbbe4.el9.x86_64.rpm 372 kB/s | 85 kB 00:00 2024-03-21T12:51:56.628 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:56.628 INFO:teuthology.orchestra.run.smithi204.stdout:Total 369 kB/s | 85 kB 00:00 2024-03-21T12:51:56.628 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:51:56.635 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:51:56.635 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:51:56.637 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librbd1 2024-03-21T12:51:56.689 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:51:56.690 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:51:56.829 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:51:56.940 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:57.204 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:41 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:57.303 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:57.329 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:57.355 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:57.356 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-21T12:51:57.356 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:57.414 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rados 2024-03-21T12:51:57.609 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:57.609 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:57.610 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:51:57.610 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:51:57.610 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:57.610 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:51:57.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-mirror 2024-03-21T12:51:57.981 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:41 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:58.080 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:58.131 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:58.132 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-21T12:51:58.133 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:58.186 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rgw 2024-03-21T12:51:58.464 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:45 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:51:58.623 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:51:58.623 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:58.623 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-21T12:51:58.624 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:58.624 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:51:58.624 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 3.1 M 2024-03-21T12:51:58.624 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:58.624 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:51:58.624 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:51:58.624 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-21T12:51:58.624 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:58.624 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 3.1 M 2024-03-21T12:51:58.624 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 13 M 2024-03-21T12:51:58.624 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:51:58.768 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:42 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:58.873 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:58.929 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:58.930 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-21T12:51:58.930 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:58.988 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-cephfs 2024-03-21T12:51:59.029 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-mirror-19.0.0-2267.g960fbbe4.el9.x86_64.rpm 7.6 MB/s | 3.1 MB 00:00 2024-03-21T12:51:59.030 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:51:59.030 INFO:teuthology.orchestra.run.smithi204.stdout:Total 7.6 MB/s | 3.1 MB 00:00 2024-03-21T12:51:59.030 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:51:59.040 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:51:59.041 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:51:59.115 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:51:59.115 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:51:59.535 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:51:59.581 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:43 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:51:59.605 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:59.630 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:51:59.630 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:51:59.631 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-21T12:51:59.631 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T12:51:59.631 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T12:51:59.631 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:51:59.682 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:51:59.736 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:51:59.737 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-21T12:51:59.738 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:51:59.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rbd 2024-03-21T12:52:00.351 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:44 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:52:00.448 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T12:52:00.500 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:52:00.500 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-21T12:52:00.501 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:52:00.551 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-fuse 2024-03-21T12:52:00.735 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:52:00.736 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:52:00.736 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:52:00.736 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:52:00.736 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:52:00.736 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:52:00.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-nbd 2024-03-21T12:52:01.117 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:45 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:52:01.267 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:52:01.268 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:52:01.268 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-03-21T12:52:01.268 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:52:01.268 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:52:01.268 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 85 k 2024-03-21T12:52:01.269 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:01.269 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:52:01.269 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:52:01.269 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-21T12:52:01.269 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:01.269 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 85 k 2024-03-21T12:52:01.269 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 231 k 2024-03-21T12:52:01.269 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:52:01.480 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-fuse-19.0.0-2267.g960fbbe4.el9.x86_64.rpm 404 kB/s | 85 kB 00:00 2024-03-21T12:52:01.481 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:52:01.481 INFO:teuthology.orchestra.run.smithi163.stdout:Total 402 kB/s | 85 kB 00:00 2024-03-21T12:52:01.481 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:52:01.488 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:52:01.488 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:52:01.543 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:52:01.543 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:52:01.550 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:48 ago on Thu 21 Mar 2024 12:50:13 PM UTC. 2024-03-21T12:52:01.681 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:52:01.700 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-21T12:52:01.701 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:52:01.701 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-03-21T12:52:01.701 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:52:01.701 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-21T12:52:01.701 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 170 k 2024-03-21T12:52:01.701 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:52:01.702 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-21T12:52:01.702 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-21T12:52:01.702 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-21T12:52:01.702 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:52:01.702 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 170 k 2024-03-21T12:52:01.702 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 490 k 2024-03-21T12:52:01.702 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-21T12:52:01.778 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:52:02.084 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-nbd-19.0.0-2267.g960fbbe4.el9.x86_64.rpm 447 kB/s | 170 kB 00:00 2024-03-21T12:52:02.084 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:52:02.084 INFO:teuthology.orchestra.run.smithi204.stdout:Total 445 kB/s | 170 kB 00:00 2024-03-21T12:52:02.085 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-21T12:52:02.092 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-21T12:52:02.093 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-21T12:52:02.149 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-21T12:52:02.149 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-21T12:52:02.242 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:52:02.291 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-21T12:52:02.391 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:52:02.527 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:52:02.529 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:02.529 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:52:02.529 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:52:02.529 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:02.529 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:52:02.807 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-mirror 2024-03-21T12:52:02.811 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:52:03.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:52:03.107 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:52:03.108 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-21T12:52:03.108 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:52:03.108 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:52:03.108 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-21T12:52:03.321 DEBUG:teuthology.parallel:result is None 2024-03-21T12:52:03.382 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:47 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:52:03.539 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:52:03.540 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:52:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-21T12:52:03.540 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:52:03.540 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:52:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 3.1 M 2024-03-21T12:52:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:03.541 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:52:03.541 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:52:03.541 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-21T12:52:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:03.541 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 3.1 M 2024-03-21T12:52:03.541 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 13 M 2024-03-21T12:52:03.541 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:52:04.601 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-mirror-19.0.0-2267.g960fbbe4.el9.x86_64.rpm 2.9 MB/s | 3.1 MB 00:01 2024-03-21T12:52:04.602 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:52:04.602 INFO:teuthology.orchestra.run.smithi163.stdout:Total 2.9 MB/s | 3.1 MB 00:01 2024-03-21T12:52:04.602 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:52:04.612 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:52:04.612 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:52:04.687 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:52:04.687 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:52:05.109 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:52:05.177 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:52:05.201 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:52:05.201 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T12:52:05.201 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-21T12:52:05.201 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T12:52:05.201 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T12:52:05.201 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:06.366 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:52:06.366 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:06.366 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:52:06.366 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:52:06.367 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:06.367 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:52:06.662 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-nbd 2024-03-21T12:52:07.235 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:51 ago on Thu 21 Mar 2024 12:50:16 PM UTC. 2024-03-21T12:52:07.387 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-21T12:52:07.388 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:52:07.388 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-03-21T12:52:07.388 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:52:07.388 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-21T12:52:07.388 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 170 k 2024-03-21T12:52:07.388 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:07.388 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-21T12:52:07.388 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-21T12:52:07.388 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-21T12:52:07.389 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:07.389 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 170 k 2024-03-21T12:52:07.389 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 490 k 2024-03-21T12:52:07.389 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-21T12:52:07.731 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-nbd-19.0.0-2267.g960fbbe4.el9.x86_64.rpm 497 kB/s | 170 kB 00:00 2024-03-21T12:52:07.732 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:52:07.732 INFO:teuthology.orchestra.run.smithi163.stdout:Total 496 kB/s | 170 kB 00:00 2024-03-21T12:52:07.732 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-21T12:52:07.740 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-21T12:52:07.740 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-21T12:52:07.795 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-21T12:52:07.796 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-21T12:52:07.942 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-21T12:52:08.047 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:52:08.478 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:52:08.784 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T12:52:08.784 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:08.784 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-21T12:52:08.784 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T12:52:08.784 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:08.785 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-21T12:52:09.018 DEBUG:teuthology.parallel:result is None 2024-03-21T12:52:09.019 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:52:09.167 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-21T12:52:09.188 INFO:teuthology.orchestra.run.smithi163.stdout:19.0.0-2267.g960fbbe4.el9 2024-03-21T12:52:09.189 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2267.g960fbbe4.el9 2024-03-21T12:52:09.189 INFO:teuthology.task.install:The correct ceph version 19.0.0-2267.g960fbbe4 is installed. 2024-03-21T12:52:09.191 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:52:09.327 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-21T12:52:09.351 INFO:teuthology.orchestra.run.smithi204.stdout:19.0.0-2267.g960fbbe4.el9 2024-03-21T12:52:09.351 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2267.g960fbbe4.el9 2024-03-21T12:52:09.351 INFO:teuthology.task.install:The correct ceph version 19.0.0-2267.g960fbbe4 is installed. 2024-03-21T12:52:09.353 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-21T12:52:09.353 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:52:09.353 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-21T12:52:09.381 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-21T12:52:09.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-21T12:52:09.422 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-21T12:52:09.422 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:52:09.422 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2024-03-21T12:52:09.449 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-21T12:52:09.516 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-21T12:52:09.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-03-21T12:52:09.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-21T12:52:09.615 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-21T12:52:09.615 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:52:09.615 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-21T12:52:09.642 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-21T12:52:09.709 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-21T12:52:09.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-21T12:52:09.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-21T12:52:09.807 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-21T12:52:09.808 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:52:09.808 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2024-03-21T12:52:09.835 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-21T12:52:09.902 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-21T12:52:09.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-03-21T12:52:09.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-21T12:52:10.003 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-21T12:52:10.117 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '960fbbe4645c2becc8084669be194c95183c7753', 'cephadm_mode': 'root'} 2024-03-21T12:52:10.117 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:52:10.118 INFO:tasks.cephadm:Cluster fsid is d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:52:10.118 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-21T12:52:10.118 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.163', 'mon.c': '[v2:172.21.15.163:3301,v1:172.21.15.163:6790]', 'mon.b': '172.21.15.204'} 2024-03-21T12:52:10.118 INFO:tasks.cephadm:First mon is mon.a on smithi163 2024-03-21T12:52:10.118 INFO:tasks.cephadm:First mgr is y 2024-03-21T12:52:10.118 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-21T12:52:10.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo hostname $(hostname -s) 2024-03-21T12:52:10.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo hostname $(hostname -s) 2024-03-21T12:52:10.175 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-21T12:52:10.175 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:52:10.391 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '960fbbe4645c2becc8084669be194c95183c7753', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77830/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2267-g960fbbe4', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2267.g960fbbe4'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-21 09:55:14.969817', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-testing-2024-03-21-0818', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-21T12:52:10.605 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-guits-testing-2024-03-21-0818, sha1 960fbbe4645c2becc8084669be194c95183c7753 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:52:10.607 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/x86_64/flavors/default/cephadm 2024-03-21T12:52:10.607 DEBUG:teuthology.orchestra.run.smithi163:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-21T12:52:11.317 INFO:teuthology.orchestra.run.smithi163.stdout:-rw-r--r--. 1 ubuntu ubuntu 769397 Mar 21 12:52 /home/ubuntu/cephtest/cephadm 2024-03-21T12:52:11.317 DEBUG:teuthology.orchestra.run.smithi204:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-21T12:52:11.988 INFO:teuthology.orchestra.run.smithi204.stdout:-rw-r--r--. 1 ubuntu ubuntu 769397 Mar 21 12:52 /home/ubuntu/cephtest/cephadm 2024-03-21T12:52:11.988 DEBUG:teuthology.orchestra.run.smithi163:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-21T12:52:12.008 DEBUG:teuthology.orchestra.run.smithi204:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-21T12:52:12.041 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 on all hosts... 2024-03-21T12:52:12.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 pull 2024-03-21T12:52:12.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 pull 2024-03-21T12:52:12.902 INFO:teuthology.orchestra.run.smithi163.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753... 2024-03-21T12:52:12.919 INFO:teuthology.orchestra.run.smithi204.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753... 2024-03-21T12:52:41.242 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T12:52:41.243 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_version": "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)", 2024-03-21T12:52:41.243 INFO:teuthology.orchestra.run.smithi204.stdout: "image_id": "19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e", 2024-03-21T12:52:41.243 INFO:teuthology.orchestra.run.smithi204.stdout: "repo_digests": [ 2024-03-21T12:52:41.243 INFO:teuthology.orchestra.run.smithi204.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93" 2024-03-21T12:52:41.243 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T12:52:41.243 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T12:52:47.090 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T12:52:47.090 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)", 2024-03-21T12:52:47.090 INFO:teuthology.orchestra.run.smithi163.stdout: "image_id": "19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e", 2024-03-21T12:52:47.090 INFO:teuthology.orchestra.run.smithi163.stdout: "repo_digests": [ 2024-03-21T12:52:47.090 INFO:teuthology.orchestra.run.smithi163.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93" 2024-03-21T12:52:47.090 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T12:52:47.090 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T12:52:47.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph 2024-03-21T12:52:47.181 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph 2024-03-21T12:52:47.215 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /etc/ceph 2024-03-21T12:52:47.245 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /etc/ceph 2024-03-21T12:52:47.285 INFO:tasks.cephadm:Writing seed config... 2024-03-21T12:52:47.286 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-21T12:52:47.287 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-21T12:52:47.288 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-21T12:52:47.288 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-21T12:52:47.288 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-21T12:52:47.288 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-21T12:52:47.288 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-21T12:52:47.288 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-21T12:52:47.288 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-21T12:52:47.288 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-21T12:52:47.288 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-21T12:52:47.288 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-21T12:52:47.289 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:52:47.289 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-21T12:52:47.304 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d4bf79c0-e781-11ee-95cd-87774f69a715 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-21T12:52:47.304 DEBUG:teuthology.orchestra.run.smithi163:mon.a> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.a.service 2024-03-21T12:52:47.347 DEBUG:teuthology.orchestra.run.smithi163:mgr.y> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mgr.y.service 2024-03-21T12:52:47.389 INFO:tasks.cephadm:Bootstrapping... 2024-03-21T12:52:47.389 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 -v bootstrap --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.163 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-21T12:52:47.580 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:52:47.580 INFO:teuthology.orchestra.run.smithi163.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753', '-v', 'bootstrap', '--fsid', 'd4bf79c0-e781-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.163', '--skip-admin-label'] 2024-03-21T12:52:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-21T12:52:47.580 INFO:teuthology.orchestra.run.smithi163.stdout:Verifying podman|docker is present... 2024-03-21T12:52:47.606 INFO:teuthology.orchestra.run.smithi163.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-21T12:52:47.606 INFO:teuthology.orchestra.run.smithi163.stdout:Verifying lvm2 is present... 2024-03-21T12:52:47.606 INFO:teuthology.orchestra.run.smithi163.stdout:Verifying time synchronization is in place... 2024-03-21T12:52:47.613 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-21T12:52:47.613 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-21T12:52:47.619 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-21T12:52:47.619 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout inactive 2024-03-21T12:52:47.625 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout enabled 2024-03-21T12:52:47.631 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout active 2024-03-21T12:52:47.631 INFO:teuthology.orchestra.run.smithi163.stdout:Unit chronyd.service is enabled and running 2024-03-21T12:52:47.631 INFO:teuthology.orchestra.run.smithi163.stdout:Repeating the final host check... 2024-03-21T12:52:47.655 INFO:teuthology.orchestra.run.smithi163.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-21T12:52:47.655 INFO:teuthology.orchestra.run.smithi163.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-21T12:52:47.655 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl is present 2024-03-21T12:52:47.655 INFO:teuthology.orchestra.run.smithi163.stdout:lvcreate is present 2024-03-21T12:52:47.661 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-21T12:52:47.661 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-21T12:52:47.667 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-21T12:52:47.667 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout inactive 2024-03-21T12:52:47.673 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout enabled 2024-03-21T12:52:47.679 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout active 2024-03-21T12:52:47.679 INFO:teuthology.orchestra.run.smithi163.stdout:Unit chronyd.service is enabled and running 2024-03-21T12:52:47.679 INFO:teuthology.orchestra.run.smithi163.stdout:Host looks OK 2024-03-21T12:52:47.679 INFO:teuthology.orchestra.run.smithi163.stdout:Cluster fsid: d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:52:47.679 INFO:teuthology.orchestra.run.smithi163.stdout:Acquiring lock 140530945733584 on /run/cephadm/d4bf79c0-e781-11ee-95cd-87774f69a715.lock 2024-03-21T12:52:47.679 INFO:teuthology.orchestra.run.smithi163.stdout:Lock 140530945733584 acquired on /run/cephadm/d4bf79c0-e781-11ee-95cd-87774f69a715.lock 2024-03-21T12:52:47.680 INFO:teuthology.orchestra.run.smithi163.stdout:Verifying IP 172.21.15.163 port 3300 ... 2024-03-21T12:52:47.680 INFO:teuthology.orchestra.run.smithi163.stdout:Verifying IP 172.21.15.163 port 6789 ... 2024-03-21T12:52:47.680 INFO:teuthology.orchestra.run.smithi163.stdout:Base mon IP(s) is [172.21.15.163:3300, 172.21.15.163:6789], mon addrv is [v2:172.21.15.163:3300,v1:172.21.15.163:6789] 2024-03-21T12:52:47.683 INFO:teuthology.orchestra.run.smithi163.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.163 metric 100 2024-03-21T12:52:47.683 INFO:teuthology.orchestra.run.smithi163.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.163 metric 100 2024-03-21T12:52:47.686 INFO:teuthology.orchestra.run.smithi163.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-21T12:52:47.686 INFO:teuthology.orchestra.run.smithi163.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-21T12:52:47.686 INFO:teuthology.orchestra.run.smithi163.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2024-03-21T12:52:47.688 INFO:teuthology.orchestra.run.smithi163.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-21T12:52:47.688 INFO:teuthology.orchestra.run.smithi163.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-21T12:52:47.688 INFO:teuthology.orchestra.run.smithi163.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-21T12:52:47.688 INFO:teuthology.orchestra.run.smithi163.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-21T12:52:47.689 INFO:teuthology.orchestra.run.smithi163.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7d5b/64 scope link 2024-03-21T12:52:47.689 INFO:teuthology.orchestra.run.smithi163.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-21T12:52:47.689 INFO:teuthology.orchestra.run.smithi163.stdout:Mon IP `172.21.15.163` is in CIDR network `172.21.0.0/20` 2024-03-21T12:52:47.690 INFO:teuthology.orchestra.run.smithi163.stdout:Mon IP `172.21.15.163` is in CIDR network `172.21.0.0/20` 2024-03-21T12:52:47.690 INFO:teuthology.orchestra.run.smithi163.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-21T12:52:47.690 INFO:teuthology.orchestra.run.smithi163.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-21T12:52:47.690 INFO:teuthology.orchestra.run.smithi163.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753... 2024-03-21T12:52:48.152 INFO:teuthology.orchestra.run.smithi163.stdout:/bin/podman: stdout 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e 2024-03-21T12:52:48.152 INFO:teuthology.orchestra.run.smithi163.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753... 2024-03-21T12:52:48.152 INFO:teuthology.orchestra.run.smithi163.stdout:/bin/podman: stderr Getting image source signatures 2024-03-21T12:52:48.152 INFO:teuthology.orchestra.run.smithi163.stdout:/bin/podman: stderr Copying blob sha256:42ed92a376f0fc24f900533b2c051b7c28b467ca85a1aa456c3a11e39d805f31 2024-03-21T12:52:48.152 INFO:teuthology.orchestra.run.smithi163.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-21T12:52:48.153 INFO:teuthology.orchestra.run.smithi163.stdout:/bin/podman: stderr Copying config sha256:19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e 2024-03-21T12:52:48.153 INFO:teuthology.orchestra.run.smithi163.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-21T12:52:49.032 INFO:teuthology.orchestra.run.smithi163.stdout:ceph: stdout ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev) 2024-03-21T12:52:49.032 INFO:teuthology.orchestra.run.smithi163.stdout:Ceph version: ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev) 2024-03-21T12:52:49.032 INFO:teuthology.orchestra.run.smithi163.stdout:Extracting ceph user uid/gid from container image... 2024-03-21T12:52:49.996 INFO:teuthology.orchestra.run.smithi163.stdout:stat: stdout 167 167 2024-03-21T12:52:49.996 INFO:teuthology.orchestra.run.smithi163.stdout:Creating initial keys... 2024-03-21T12:52:50.884 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph-authtool: stdout AQCiLfxlPVo9EhAAXrlZzT/403y+PMo5TpriNQ== 2024-03-21T12:52:51.781 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph-authtool: stdout AQCjLfxlwAqdDBAAFvTcIi3TVgzXzuRLsp7IQg== 2024-03-21T12:52:52.745 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph-authtool: stdout AQCkLfxlvqzdBRAAkSYVwa/+lWcpR/usPBIErw== 2024-03-21T12:52:52.745 INFO:teuthology.orchestra.run.smithi163.stdout:Creating initial monmap... 2024-03-21T12:52:53.751 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-21T12:52:53.751 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-21T12:52:53.751 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:52:53.751 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-21T12:52:53.751 INFO:teuthology.orchestra.run.smithi163.stdout:monmaptool for a [v2:172.21.15.163:3300,v1:172.21.15.163:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-21T12:52:53.751 INFO:teuthology.orchestra.run.smithi163.stdout:setting min_mon_release = quincy 2024-03-21T12:52:53.751 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/monmaptool: set fsid to d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:52:53.752 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-21T12:52:53.752 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:52:53.752 INFO:teuthology.orchestra.run.smithi163.stdout:Creating mon... 2024-03-21T12:52:54.556 INFO:teuthology.orchestra.run.smithi163.stdout:create mon.a on 2024-03-21T12:52:54.756 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-21T12:52:54.891 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-21T12:52:55.060 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d4bf79c0-e781-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-d4bf79c0-e781-11ee-95cd-87774f69a715.target. 2024-03-21T12:52:55.060 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d4bf79c0-e781-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-d4bf79c0-e781-11ee-95cd-87774f69a715.target. 2024-03-21T12:52:55.309 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.a 2024-03-21T12:52:55.309 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-21T12:52:55.480 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d4bf79c0-e781-11ee-95cd-87774f69a715.target.wants/ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@.service. 2024-03-21T12:52:56.064 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T12:52:56.064 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout disabled 2024-03-21T12:52:56.071 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T12:52:56.071 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout inactive 2024-03-21T12:52:56.071 INFO:teuthology.orchestra.run.smithi163.stdout:firewalld.service is not enabled 2024-03-21T12:52:56.071 INFO:teuthology.orchestra.run.smithi163.stdout:Not possible to enable service . firewalld.service is not available 2024-03-21T12:52:56.071 INFO:teuthology.orchestra.run.smithi163.stdout:Waiting for mon to start... 2024-03-21T12:52:56.071 INFO:teuthology.orchestra.run.smithi163.stdout:Waiting for mon... 2024-03-21T12:52:57.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:52:57 smithi163 ceph-mon[29240]: from='client.? 172.21.15.163:0/4116789585' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:53:02.172 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout cluster: 2024-03-21T12:53:02.172 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout id: d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:53:02.172 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-21T12:53:02.172 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 2024-03-21T12:53:02.172 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout services: 2024-03-21T12:53:02.172 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.515425s) 2024-03-21T12:53:02.172 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-21T12:53:02.172 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-21T12:53:02.172 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 2024-03-21T12:53:02.173 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout data: 2024-03-21T12:53:02.173 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-21T12:53:02.173 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-21T12:53:02.173 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-21T12:53:02.173 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout pgs: 2024-03-21T12:53:02.173 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 2024-03-21T12:53:02.173 INFO:teuthology.orchestra.run.smithi163.stdout:mon is available 2024-03-21T12:53:02.173 INFO:teuthology.orchestra.run.smithi163.stdout:Assimilating anything we can from ceph.conf... 2024-03-21T12:53:05.273 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:05 smithi163 ceph-mon[29240]: from='client.? 172.21.15.163:0/3873782610' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-21T12:53:05.273 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:05 smithi163 ceph-mon[29240]: from='client.? 172.21.15.163:0/3873782610' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-21T12:53:08.427 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 2024-03-21T12:53:08.427 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout [global] 2024-03-21T12:53:08.427 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout fsid = d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:53:08.427 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.163:3300,v1:172.21.15.163:6789] 2024-03-21T12:53:08.427 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-21T12:53:08.427 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-21T12:53:08.428 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-21T12:53:08.428 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-21T12:53:08.428 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 2024-03-21T12:53:08.428 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-21T12:53:08.428 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-21T12:53:08.428 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 2024-03-21T12:53:08.428 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout [osd] 2024-03-21T12:53:08.428 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-21T12:53:08.428 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-21T12:53:08.428 INFO:teuthology.orchestra.run.smithi163.stdout:Generating new minimal ceph.conf... 2024-03-21T12:53:09.187 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:08 smithi163 ceph-mon[29240]: from='client.? 172.21.15.163:0/649665403' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:53:09.448 INFO:teuthology.orchestra.run.smithi163.stdout:Restarting the monitor... 2024-03-21T12:53:09.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:09 smithi163 systemd[1]: Stopping Ceph mon.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T12:53:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:09 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a[29236]: 2024-03-21T12:53:09.722+0000 7f935dec2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T12:53:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:09 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a[29236]: 2024-03-21T12:53:09.722+0000 7f935dec2700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-21T12:53:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:09 smithi163 podman[29497]: 2024-03-21 12:53:09.846266466 +0000 UTC m=+0.230654053 container died 8584db0ca31109b66a437fb16b91f19d60f3ea222001e3447cf611571bb58ff8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-21T12:53:10.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:10 smithi163 podman[29497]: 2024-03-21 12:53:10.198356115 +0000 UTC m=+0.582743752 container cleanup 8584db0ca31109b66a437fb16b91f19d60f3ea222001e3447cf611571bb58ff8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-21T12:53:10.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:10 smithi163 bash[29497]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a 2024-03-21T12:53:10.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:10 smithi163 podman[29512]: 2024-03-21 12:53:10.473969151 +0000 UTC m=+0.621190098 container remove 8584db0ca31109b66a437fb16b91f19d60f3ea222001e3447cf611571bb58ff8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T12:53:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:10 smithi163 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-21T12:53:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:10 smithi163 systemd[1]: Stopped Ceph mon.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T12:53:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:10 smithi163 systemd[1]: Starting Ceph mon.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T12:53:11.077 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:10 smithi163 podman[29603]: 2024-03-21 12:53:10.722409875 +0000 UTC m=+0.017743569 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:53:11.077 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:10 smithi163 podman[29603]: 2024-03-21 12:53:10.833233119 +0000 UTC m=+0.128566804 container create 46f326bcd8aaa2147b8c7d2a503206e204feb8959dc77dc2aed060e667598c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-21T12:53:11.077 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 podman[29603]: 2024-03-21 12:53:11.051993668 +0000 UTC m=+0.347327357 container init 46f326bcd8aaa2147b8c7d2a503206e204feb8959dc77dc2aed060e667598c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-21T12:53:11.078 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 podman[29603]: 2024-03-21 12:53:11.05495661 +0000 UTC m=+0.350290294 container start 46f326bcd8aaa2147b8c7d2a503206e204feb8959dc77dc2aed060e667598c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-21T12:53:11.078 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T12:53:11.139 INFO:teuthology.orchestra.run.smithi163.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-21T12:53:11.333 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev), process ceph-mon, pid 2 2024-03-21T12:53:11.333 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: pidfile_write: ignore empty --pid-file 2024-03-21T12:53:11.333 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: load: jerasure load: lrc 2024-03-21T12:53:11.333 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: RocksDB version: 7.9.2 2024-03-21T12:53:11.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Git sha 0 2024-03-21T12:53:11.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Compile date 2024-03-21 08:53:26 2024-03-21T12:53:11.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: DB SUMMARY 2024-03-21T12:53:11.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: DB Session ID: ECMN2Z7462MD12VVAQQO 2024-03-21T12:53:11.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: CURRENT file: CURRENT 2024-03-21T12:53:11.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T12:53:11.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-21T12:53:11.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-21T12:53:11.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 92061 ; 2024-03-21T12:53:11.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.error_if_exists: 0 2024-03-21T12:53:11.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.create_if_missing: 0 2024-03-21T12:53:11.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.paranoid_checks: 1 2024-03-21T12:53:11.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T12:53:11.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T12:53:11.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T12:53:11.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.env: 0x55dcdf52eb20 2024-03-21T12:53:11.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T12:53:11.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.info_log: 0x55dcdffdd920 2024-03-21T12:53:11.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T12:53:11.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.statistics: (nil) 2024-03-21T12:53:11.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.use_fsync: 0 2024-03-21T12:53:11.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_log_file_size: 0 2024-03-21T12:53:11.336 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T12:53:11.336 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T12:53:11.336 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T12:53:11.336 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T12:53:11.336 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.allow_fallocate: 1 2024-03-21T12:53:11.336 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T12:53:11.336 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T12:53:11.336 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.use_direct_reads: 0 2024-03-21T12:53:11.336 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T12:53:11.336 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T12:53:11.336 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.db_log_dir: 2024-03-21T12:53:11.337 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.wal_dir: 2024-03-21T12:53:11.337 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T12:53:11.337 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T12:53:11.337 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T12:53:11.337 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T12:53:11.337 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T12:53:11.337 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T12:53:11.337 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T12:53:11.337 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T12:53:11.337 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.write_buffer_manager: 0x55dcdffae1e0 2024-03-21T12:53:11.337 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T12:53:11.337 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T12:53:11.337 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T12:53:11.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T12:53:11.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T12:53:11.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T12:53:11.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T12:53:11.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T12:53:11.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.unordered_write: 0 2024-03-21T12:53:11.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T12:53:11.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T12:53:11.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T12:53:11.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T12:53:11.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.row_cache: None 2024-03-21T12:53:11.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.wal_filter: None 2024-03-21T12:53:11.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.two_write_queues: 0 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.wal_compression: 0 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.atomic_flush: 0 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.log_readahead_size: 0 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T12:53:11.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T12:53:11.341 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_background_jobs: 2 2024-03-21T12:53:11.341 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_background_compactions: -1 2024-03-21T12:53:11.341 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_subcompactions: 1 2024-03-21T12:53:11.341 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T12:53:11.341 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T12:53:11.341 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T12:53:11.341 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T12:53:11.341 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T12:53:11.341 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T12:53:11.341 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T12:53:11.342 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T12:53:11.342 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_open_files: -1 2024-03-21T12:53:11.342 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T12:53:11.342 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T12:53:11.342 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T12:53:11.342 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T12:53:11.342 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_background_flushes: -1 2024-03-21T12:53:11.342 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Compression algorithms supported: 2024-03-21T12:53:11.342 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T12:53:11.342 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: kZSTD supported: 0 2024-03-21T12:53:11.342 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: kXpressCompression supported: 0 2024-03-21T12:53:11.342 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T12:53:11.342 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: kZlibCompression supported: 1 2024-03-21T12:53:11.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: kSnappyCompression supported: 1 2024-03-21T12:53:11.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: kLZ4Compression supported: 1 2024-03-21T12:53:11.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: kBZip2Compression supported: 0 2024-03-21T12:53:11.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T12:53:11.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T12:53:11.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-21T12:53:11.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T12:53:11.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T12:53:11.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.merge_operator: 2024-03-21T12:53:11.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compaction_filter: None 2024-03-21T12:53:11.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dcdffdda20) 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout: cache_index_and_filter_blocks: 1 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout: pin_top_level_index_and_filter: 1 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout: index_type: 0 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout: data_block_index_type: 0 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout: index_shortening: 1 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout: checksum: 4 2024-03-21T12:53:11.344 INFO:journalctl@ceph.mon.a.smithi163.stdout: no_block_cache: 0 2024-03-21T12:53:11.345 INFO:journalctl@ceph.mon.a.smithi163.stdout: block_cache: 0x55dcdffe1090 2024-03-21T12:53:11.345 INFO:journalctl@ceph.mon.a.smithi163.stdout: block_cache_name: BinnedLRUCache 2024-03-21T12:53:11.345 INFO:journalctl@ceph.mon.a.smithi163.stdout: block_cache_options: 2024-03-21T12:53:11.345 INFO:journalctl@ceph.mon.a.smithi163.stdout: capacity : 536870912 2024-03-21T12:53:11.345 INFO:journalctl@ceph.mon.a.smithi163.stdout: num_shard_bits : 4 2024-03-21T12:53:11.345 INFO:journalctl@ceph.mon.a.smithi163.stdout: strict_capacity_limit : 0 2024-03-21T12:53:11.345 INFO:journalctl@ceph.mon.a.smithi163.stdout: high_pri_pool_ratio: 0.000 2024-03-21T12:53:11.345 INFO:journalctl@ceph.mon.a.smithi163.stdout: block_cache_compressed: (nil) 2024-03-21T12:53:11.345 INFO:journalctl@ceph.mon.a.smithi163.stdout: persistent_cache: (nil) 2024-03-21T12:53:11.345 INFO:journalctl@ceph.mon.a.smithi163.stdout: block_size: 4096 2024-03-21T12:53:11.345 INFO:journalctl@ceph.mon.a.smithi163.stdout: block_size_deviation: 10 2024-03-21T12:53:11.345 INFO:journalctl@ceph.mon.a.smithi163.stdout: block_restart_interval: 16 2024-03-21T12:53:11.345 INFO:journalctl@ceph.mon.a.smithi163.stdout: index_block_restart_interval: 1 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: metadata_block_size: 4096 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: partition_filters: 0 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: use_delta_encoding: 1 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: filter_policy: bloomfilter 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: whole_key_filtering: 1 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: verify_compression: 0 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: read_amp_bytes_per_bit: 0 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: format_version: 5 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: enable_index_compression: 1 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: block_align: 0 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: max_auto_readahead_size: 262144 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: prepopulate_block_cache: 0 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: initial_auto_readahead_size: 8192 2024-03-21T12:53:11.346 INFO:journalctl@ceph.mon.a.smithi163.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compression: NoCompression 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.num_levels: 7 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T12:53:11.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T12:53:11.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T12:53:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T12:53:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T12:53:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T12:53:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T12:53:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T12:53:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T12:53:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T12:53:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T12:53:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T12:53:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T12:53:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T12:53:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T12:53:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T12:53:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T12:53:11.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T12:53:11.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T12:53:11.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.table_properties_collectors: 2024-03-21T12:53:11.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.inplace_update_support: 0 2024-03-21T12:53:11.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T12:53:11.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T12:53:11.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T12:53:11.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T12:53:11.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.bloom_locality: 0 2024-03-21T12:53:11.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.max_successive_merges: 0 2024-03-21T12:53:11.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T12:53:11.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T12:53:11.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.ttl: 2592000 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.enable_blob_files: false 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.min_blob_size: 0 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T12:53:11.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T12:53:11.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T12:53:11.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-21T12:53:11.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-21T12:53:11.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fd3fb57e-4938-4238-985c-020ca466c4a6 2024-03-21T12:53:11.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025591083508, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-21T12:53:11.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-21T12:53:11.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025591084419, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 87934, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 282, "table_properties": {"data_size": 86022, "index_size": 258, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11802, "raw_average_key_size": 48, "raw_value_size": 79557, "raw_average_value_size": 330, "num_data_blocks": 11, "num_entries": 241, "num_filter_entries": 241, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711025591, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fd3fb57e-4938-4238-985c-020ca466c4a6", "db_session_id": "ECMN2Z7462MD12VVAQQO", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-21T12:53:11.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025591084505, "job": 1, "event": "recovery_finished"} 2024-03-21T12:53:11.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-21T12:53:11.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T12:53:11.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55dce00d0000 2024-03-21T12:53:11.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: DB pointer 0x55dce00ba000 2024-03-21T12:53:11.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T12:53:11.354 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T12:53:11.354 INFO:journalctl@ceph.mon.a.smithi163.stdout: ** DB Stats ** 2024-03-21T12:53:11.354 INFO:journalctl@ceph.mon.a.smithi163.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:53:11.354 INFO:journalctl@ceph.mon.a.smithi163.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T12:53:11.354 INFO:journalctl@ceph.mon.a.smithi163.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T12:53:11.354 INFO:journalctl@ceph.mon.a.smithi163.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:53:11.354 INFO:journalctl@ceph.mon.a.smithi163.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T12:53:11.354 INFO:journalctl@ceph.mon.a.smithi163.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T12:53:11.354 INFO:journalctl@ceph.mon.a.smithi163.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:53:11.354 INFO:journalctl@ceph.mon.a.smithi163.stdout: 2024-03-21T12:53:11.354 INFO:journalctl@ceph.mon.a.smithi163.stdout: ** Compaction Stats [default] ** 2024-03-21T12:53:11.354 INFO:journalctl@ceph.mon.a.smithi163.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T12:53:11.354 INFO:journalctl@ceph.mon.a.smithi163.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T12:53:11.355 INFO:journalctl@ceph.mon.a.smithi163.stdout: L0 2/0 87.71 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T12:53:11.355 INFO:journalctl@ceph.mon.a.smithi163.stdout: Sum 2/0 87.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T12:53:11.355 INFO:journalctl@ceph.mon.a.smithi163.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T12:53:11.355 INFO:journalctl@ceph.mon.a.smithi163.stdout: 2024-03-21T12:53:11.355 INFO:journalctl@ceph.mon.a.smithi163.stdout: ** Compaction Stats [default] ** 2024-03-21T12:53:11.355 INFO:journalctl@ceph.mon.a.smithi163.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T12:53:11.355 INFO:journalctl@ceph.mon.a.smithi163.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T12:53:11.355 INFO:journalctl@ceph.mon.a.smithi163.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T12:53:11.355 INFO:journalctl@ceph.mon.a.smithi163.stdout: 2024-03-21T12:53:11.355 INFO:journalctl@ceph.mon.a.smithi163.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T12:53:11.355 INFO:journalctl@ceph.mon.a.smithi163.stdout: 2024-03-21T12:53:11.355 INFO:journalctl@ceph.mon.a.smithi163.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:53:11.355 INFO:journalctl@ceph.mon.a.smithi163.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout: Cumulative compaction: 0.00 GB write, 29.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout: Interval compaction: 0.00 GB write, 29.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout: Block cache BinnedLRUCache@0x55dcdffe1090#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout: 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: starting mon.a rank 0 at public addrs [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] at bind addrs [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: mon.a@-1(???) e1 preinit fsid d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: mon.a@-1(???).mds e1 new map 2024-03-21T12:53:11.356 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: mon.a@-1(???).mds e1 print_map 2024-03-21T12:53:11.357 INFO:journalctl@ceph.mon.a.smithi163.stdout: e1 2024-03-21T12:53:11.357 INFO:journalctl@ceph.mon.a.smithi163.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T12:53:11.357 INFO:journalctl@ceph.mon.a.smithi163.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-21T12:53:11.357 INFO:journalctl@ceph.mon.a.smithi163.stdout: legacy client fscid: -1 2024-03-21T12:53:11.357 INFO:journalctl@ceph.mon.a.smithi163.stdout: 2024-03-21T12:53:11.357 INFO:journalctl@ceph.mon.a.smithi163.stdout: No filesystems configured 2024-03-21T12:53:11.357 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-21T12:53:11.357 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:53:11.357 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:53:11.357 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:53:11.357 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-21T12:53:11.357 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 bash[29603]: 46f326bcd8aaa2147b8c7d2a503206e204feb8959dc77dc2aed060e667598c9a 2024-03-21T12:53:11.357 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 systemd[1]: Started Ceph mon.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T12:53:11.358 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T12:53:11.358 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: monmap e1: 1 mons at {a=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:53:11.358 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: fsmap 2024-03-21T12:53:11.358 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: osdmap e1: 0 total, 0 up, 0 in 2024-03-21T12:53:11.358 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:11 smithi163 ceph-mon[29621]: mgrmap e1: no daemons active 2024-03-21T12:53:12.161 INFO:teuthology.orchestra.run.smithi163.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-21T12:53:12.182 INFO:teuthology.orchestra.run.smithi163.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-21T12:53:12.182 INFO:teuthology.orchestra.run.smithi163.stdout:Creating mgr... 2024-03-21T12:53:12.182 INFO:teuthology.orchestra.run.smithi163.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-21T12:53:12.183 INFO:teuthology.orchestra.run.smithi163.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-21T12:53:12.435 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mgr.y 2024-03-21T12:53:12.436 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-21T12:53:12.579 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d4bf79c0-e781-11ee-95cd-87774f69a715.target.wants/ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@.service. 2024-03-21T12:53:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:12 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3539467291' entity='client.admin' 2024-03-21T12:53:12.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:12 smithi163 systemd[1]: Starting Ceph mgr.y for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T12:53:13.261 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:12 smithi163 podman[29838]: 2024-03-21 12:53:12.809888002 +0000 UTC m=+0.110265998 container create 3e827e3422520045d7a0789121ca254510860ac74ea8672debd1c48414cdcec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T12:53:13.261 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:12 smithi163 podman[29838]: 2024-03-21 12:53:12.717386462 +0000 UTC m=+0.017764477 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:53:14.261 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:13 smithi163 podman[29838]: 2024-03-21 12:53:13.901591552 +0000 UTC m=+1.201969560 container init 3e827e3422520045d7a0789121ca254510860ac74ea8672debd1c48414cdcec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-21T12:53:14.261 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:13 smithi163 podman[29838]: 2024-03-21 12:53:13.904907102 +0000 UTC m=+1.205285100 container start 3e827e3422520045d7a0789121ca254510860ac74ea8672debd1c48414cdcec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-21T12:53:14.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:14 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:14.454+0000 7f89b212b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:53:14.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:14 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:14.548+0000 7f89b212b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:53:14.947 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T12:53:14.947 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout disabled 2024-03-21T12:53:14.954 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T12:53:14.954 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout inactive 2024-03-21T12:53:14.954 INFO:teuthology.orchestra.run.smithi163.stdout:firewalld.service is not enabled 2024-03-21T12:53:14.954 INFO:teuthology.orchestra.run.smithi163.stdout:Not possible to enable service . firewalld.service is not available 2024-03-21T12:53:14.962 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T12:53:14.962 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout disabled 2024-03-21T12:53:14.970 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T12:53:14.970 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout inactive 2024-03-21T12:53:14.970 INFO:teuthology.orchestra.run.smithi163.stdout:firewalld.service is not enabled 2024-03-21T12:53:14.970 INFO:teuthology.orchestra.run.smithi163.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-21T12:53:14.970 INFO:teuthology.orchestra.run.smithi163.stdout:Waiting for mgr to start... 2024-03-21T12:53:14.970 INFO:teuthology.orchestra.run.smithi163.stdout:Waiting for mgr... 2024-03-21T12:53:15.207 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:14 smithi163 bash[29838]: 3e827e3422520045d7a0789121ca254510860ac74ea8672debd1c48414cdcec5 2024-03-21T12:53:15.207 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:14 smithi163 systemd[1]: Started Ceph mgr.y for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T12:53:15.207 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:15.206+0000 7f89b212b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:53:15.510 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:15.299+0000 7f89b212b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:53:15.510 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:15.440+0000 7f89b212b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:53:15.884 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:15.536+0000 7f89b212b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:53:15.884 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:15.619+0000 7f89b212b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:53:15.884 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:15.883+0000 7f89b212b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:53:16.172 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:15.991+0000 7f89b212b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:53:16.172 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:16 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:16.078+0000 7f89b212b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:53:16.172 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:16 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:16.171+0000 7f89b212b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:53:16.510 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:16 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:16.351+0000 7f89b212b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:53:17.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:17 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1004629127' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T12:53:17.480 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:17 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:17.346+0000 7f89b212b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:53:17.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:17 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:17.479+0000 7f89b212b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:53:17.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 2024-03-21T12:53:17.767 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout { 2024-03-21T12:53:17.767 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "fsid": "d4bf79c0-e781-11ee-95cd-87774f69a715", 2024-03-21T12:53:17.767 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T12:53:17.767 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T12:53:17.767 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T12:53:17.767 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T12:53:17.767 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:17.767 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T12:53:17.767 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T12:53:17.767 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 0 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout ], 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout ], 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T12:53:17.768 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T12:53:17.769 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T12:53:17.769 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T12:53:17.769 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T12:53:17.772 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:17.772 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T12:53:17.772 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T12:53:17.772 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T12:53:17.772 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T12:53:17.772 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T12:53:17.772 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T12:53:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T12:53:17.774 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T12:53:17.774 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout ], 2024-03-21T12:53:17.774 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T12:53:17.774 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:17.774 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T12:53:17.774 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T12:53:17.774 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T12:52:56.031860+0000", 2024-03-21T12:53:17.774 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T12:53:17.774 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:17.774 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T12:53:17.774 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout } 2024-03-21T12:53:17.774 INFO:teuthology.orchestra.run.smithi163.stdout:mgr not available, waiting (1/15)... 2024-03-21T12:53:18.099 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:17 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:17.818+0000 7f89b212b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:53:18.099 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:17 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:17.906+0000 7f89b212b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:53:18.099 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:18.098+0000 7f89b212b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:53:18.510 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:18.286+0000 7f89b212b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:53:18.839 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:18.570+0000 7f89b212b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:53:18.839 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:18.838+0000 7f89b212b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:53:19.260 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:19 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:19.072+0000 7f89b212b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:53:20.180 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:19 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:19.875+0000 7f89b212b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:53:20.467 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:20 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3981920736' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T12:53:20.758 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:20 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:20.466+0000 7f89b212b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:53:20.759 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:20 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:20.565+0000 7f89b212b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:53:20.763 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 2024-03-21T12:53:20.763 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout { 2024-03-21T12:53:20.763 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "fsid": "d4bf79c0-e781-11ee-95cd-87774f69a715", 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 0 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout ], 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout ], 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-21T12:53:20.764 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T12:53:20.765 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T12:53:20.765 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T12:53:20.765 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T12:53:20.765 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:20.765 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T12:53:20.765 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T12:53:20.765 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T12:53:20.765 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T12:53:20.765 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T12:53:20.765 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T12:53:20.765 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T12:53:20.765 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T12:53:20.769 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:20.769 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T12:53:20.769 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T12:53:20.769 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T12:53:20.769 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T12:53:20.769 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T12:53:20.769 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T12:53:20.769 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T12:53:20.769 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T12:53:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T12:53:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T12:53:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T12:53:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T12:53:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T12:53:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T12:53:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-21T12:53:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T12:53:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T12:53:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T12:53:20.770 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T12:53:20.771 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T12:53:20.771 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout ], 2024-03-21T12:53:20.771 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T12:53:20.771 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:20.771 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T12:53:20.771 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T12:53:20.771 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T12:52:56.031860+0000", 2024-03-21T12:53:20.771 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T12:53:20.771 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:20.771 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T12:53:20.771 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout } 2024-03-21T12:53:20.771 INFO:teuthology.orchestra.run.smithi163.stdout:mgr not available, waiting (2/15)... 2024-03-21T12:53:21.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:20 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:20.785+0000 7f89b212b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:53:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:21 smithi163 ceph-mon[29621]: Activating manager daemon y 2024-03-21T12:53:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:21 smithi163 ceph-mon[29621]: mgrmap e2: y(active, starting, since 0.00298347s) 2024-03-21T12:53:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:21 smithi163 ceph-mon[29621]: from='mgr.14100 172.21.15.163:0/54512847' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:53:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:21 smithi163 ceph-mon[29621]: from='mgr.14100 172.21.15.163:0/54512847' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:53:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:21 smithi163 ceph-mon[29621]: from='mgr.14100 172.21.15.163:0/54512847' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:53:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:21 smithi163 ceph-mon[29621]: from='mgr.14100 172.21.15.163:0/54512847' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:53:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:21 smithi163 ceph-mon[29621]: from='mgr.14100 172.21.15.163:0/54512847' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:53:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:21 smithi163 ceph-mon[29621]: Manager daemon y is now available 2024-03-21T12:53:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:21 smithi163 ceph-mon[29621]: from='mgr.14100 172.21.15.163:0/54512847' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:53:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:21 smithi163 ceph-mon[29621]: from='mgr.14100 172.21.15.163:0/54512847' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:53:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:21 smithi163 ceph-mon[29621]: from='mgr.14100 172.21.15.163:0/54512847' entity='mgr.y' 2024-03-21T12:53:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:21 smithi163 ceph-mon[29621]: from='mgr.14100 172.21.15.163:0/54512847' entity='mgr.y' 2024-03-21T12:53:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:21 smithi163 ceph-mon[29621]: from='mgr.14100 172.21.15.163:0/54512847' entity='mgr.y' 2024-03-21T12:53:23.094 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:22 smithi163 ceph-mon[29621]: mgrmap e3: y(active, since 1.00705s) 2024-03-21T12:53:24.019 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 2024-03-21T12:53:24.020 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout { 2024-03-21T12:53:24.020 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "fsid": "d4bf79c0-e781-11ee-95cd-87774f69a715", 2024-03-21T12:53:24.020 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T12:53:24.020 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T12:53:24.020 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T12:53:24.020 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T12:53:24.020 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:24.020 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T12:53:24.020 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T12:53:24.020 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 0 2024-03-21T12:53:24.020 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout ], 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout ], 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T12:53:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T12:53:24.022 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T12:53:24.022 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T12:53:24.025 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:24.025 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T12:53:24.025 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T12:53:24.025 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T12:53:24.025 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T12:53:24.025 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T12:53:24.025 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T12:53:24.025 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T12:53:24.026 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T12:53:24.026 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T12:53:24.026 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:24.026 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T12:53:24.026 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T12:53:24.027 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T12:53:24.027 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T12:53:24.027 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:24.027 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T12:53:24.027 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-21T12:53:24.027 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T12:53:24.027 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T12:53:24.027 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T12:53:24.028 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T12:53:24.028 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T12:53:24.028 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout ], 2024-03-21T12:53:24.028 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T12:53:24.028 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:24.028 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T12:53:24.028 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T12:53:24.028 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T12:52:56.031860+0000", 2024-03-21T12:53:24.028 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T12:53:24.028 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout }, 2024-03-21T12:53:24.028 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T12:53:24.028 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout } 2024-03-21T12:53:24.028 INFO:teuthology.orchestra.run.smithi163.stdout:mgr is available 2024-03-21T12:53:24.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:23 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/938394948' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T12:53:25.212 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:24 smithi163 ceph-mon[29621]: mgrmap e4: y(active, since 3s) 2024-03-21T12:53:25.212 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:24 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2426561845' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-21T12:53:25.215 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 2024-03-21T12:53:25.215 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout [global] 2024-03-21T12:53:25.215 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout fsid = d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:53:25.215 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-21T12:53:25.215 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-21T12:53:25.216 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-21T12:53:25.216 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-21T12:53:25.216 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 2024-03-21T12:53:25.216 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-21T12:53:25.216 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-21T12:53:25.216 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 2024-03-21T12:53:25.216 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout [osd] 2024-03-21T12:53:25.216 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-21T12:53:25.216 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-21T12:53:25.216 INFO:teuthology.orchestra.run.smithi163.stdout:Enabling cephadm module... 2024-03-21T12:53:26.218 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:25 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1916465542' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-21T12:53:26.218 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ignoring --setuser ceph since I am not root 2024-03-21T12:53:26.219 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ignoring --setgroup ceph since I am not root 2024-03-21T12:53:26.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:26 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:26.469+0000 7f5f97087200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:53:26.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:26 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:26.567+0000 7f5f97087200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:53:27.226 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:26 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1916465542' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-21T12:53:27.226 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:26 smithi163 ceph-mon[29621]: mgrmap e5: y(active, since 5s) 2024-03-21T12:53:27.510 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:27 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:27.225+0000 7f5f97087200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:53:27.511 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:27 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:27.323+0000 7f5f97087200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:53:27.511 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:27 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:27.467+0000 7f5f97087200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:53:27.858 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:27 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:27.564+0000 7f5f97087200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:53:27.858 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:27 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:27.647+0000 7f5f97087200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:53:27.860 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout { 2024-03-21T12:53:27.861 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-21T12:53:27.861 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-21T12:53:27.861 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-21T12:53:27.861 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-21T12:53:27.861 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout } 2024-03-21T12:53:27.861 INFO:teuthology.orchestra.run.smithi163.stdout:Waiting for the mgr to restart... 2024-03-21T12:53:27.861 INFO:teuthology.orchestra.run.smithi163.stdout:Waiting for mgr epoch 5... 2024-03-21T12:53:28.123 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:27 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/974086107' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T12:53:28.123 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:27 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:27.923+0000 7f5f97087200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:53:28.123 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:28 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:28.033+0000 7f5f97087200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:53:28.123 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:28 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:28.122+0000 7f5f97087200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:53:28.402 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:28 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:28.219+0000 7f5f97087200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:53:28.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:28 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:28.401+0000 7f5f97087200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:53:29.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:29 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:29.359+0000 7f5f97087200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:53:29.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:29 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:29.491+0000 7f5f97087200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:53:30.127 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:29 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:29.840+0000 7f5f97087200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:53:30.127 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:29 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:29.929+0000 7f5f97087200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:53:30.127 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:30 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:30.126+0000 7f5f97087200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:53:30.510 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:30 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:30.312+0000 7f5f97087200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:53:30.857 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:30 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:30.597+0000 7f5f97087200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:53:30.857 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:30 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:30.856+0000 7f5f97087200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:53:31.260 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:31 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:31.093+0000 7f5f97087200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:53:32.260 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:31 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:31.891+0000 7f5f97087200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:53:32.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:32.462+0000 7f5f97087200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:53:32.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:32.560+0000 7f5f97087200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:53:33.260 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:32.776+0000 7f5f97087200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:53:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-mon[29621]: Active manager daemon y restarted 2024-03-21T12:53:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-mon[29621]: Activating manager daemon y 2024-03-21T12:53:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-mon[29621]: osdmap e2: 0 total, 0 up, 0 in 2024-03-21T12:53:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-mon[29621]: mgrmap e6: y(active, starting, since 0.00334886s) 2024-03-21T12:53:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:53:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:53:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:53:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:53:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:53:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-mon[29621]: Manager daemon y is now available 2024-03-21T12:53:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:32 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:53:34.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:33 smithi163 ceph-mon[29621]: Found migration_current of "None". Setting to last migration. 2024-03-21T12:53:34.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:33 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:53:34.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:33 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:53:34.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:33 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:53:34.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:33 smithi163 ceph-mon[29621]: mgrmap e7: y(active, since 1.00538s) 2024-03-21T12:53:34.433 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout { 2024-03-21T12:53:34.433 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-21T12:53:34.433 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-21T12:53:34.434 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout } 2024-03-21T12:53:34.434 INFO:teuthology.orchestra.run.smithi163.stdout:mgr epoch 5 is available 2024-03-21T12:53:34.434 INFO:teuthology.orchestra.run.smithi163.stdout:Setting orchestrator backend to cephadm... 2024-03-21T12:53:35.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:34 smithi163 ceph-mon[29621]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-21T12:53:35.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:34 smithi163 ceph-mon[29621]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-21T12:53:35.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:35 smithi163 ceph-mon[29621]: mgrmap e8: y(active, since 2s) 2024-03-21T12:53:35.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:35 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:35.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:35 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:35.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:35 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:35.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:35 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:53:35.957 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:35 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:53:36.862 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-21T12:53:36.862 INFO:teuthology.orchestra.run.smithi163.stdout:Generating ssh key... 2024-03-21T12:53:37.101 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:36 smithi163 ceph-mon[29621]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:37.101 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:36 smithi163 ceph-mon[29621]: [21/Mar/2024:12:53:35] ENGINE Bus STARTING 2024-03-21T12:53:37.101 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:36 smithi163 ceph-mon[29621]: [21/Mar/2024:12:53:35] ENGINE Serving on https://172.21.15.163:7150 2024-03-21T12:53:37.101 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:36 smithi163 ceph-mon[29621]: [21/Mar/2024:12:53:35] ENGINE Serving on http://172.21.15.163:8765 2024-03-21T12:53:37.101 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:36 smithi163 ceph-mon[29621]: [21/Mar/2024:12:53:35] ENGINE Bus STARTED 2024-03-21T12:53:38.085 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-mon[29621]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:38.085 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:38.085 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:38.085 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: Generating public/private rsa key pair. 2024-03-21T12:53:38.085 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: Your identification has been saved in /tmp/tmpnqd_5dy9/key. 2024-03-21T12:53:38.085 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: Your public key has been saved in /tmp/tmpnqd_5dy9/key.pub. 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: The key fingerprint is: 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: SHA256:dFiZIU7wIUjKsetso/H9cL7v3V1Pef6bZrT36tCYZRM ceph-d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: The key's randomart image is: 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: +---[RSA 3072]----+ 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: | ....o.+ o+ | 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: | . +. = =o | 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: | + = . E | 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: | . . . . | 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: | . S + | 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: | o * o.| 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: |. = . . + o.=| 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: | = o + . . o B=| 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: |. . ..++o . ..*+X| 2024-03-21T12:53:38.086 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:37 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: +----[SHA256]-----+ 2024-03-21T12:53:38.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:38 smithi163 ceph-mon[29621]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:38.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:38 smithi163 ceph-mon[29621]: Generating ssh key... 2024-03-21T12:53:39.608 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi+inm7TjL18rx+QLDeJ5r/FVNGnjaELnuhMIzeqb3rRxtzIEQGqem2Eil+vKl4tZry1LlMkXFLS45Z2K9nEOZ0ByjCONB9KC0jmNjJ3THjMU2mjYlTJHZ4qQsPskfXFQxvH0OavCiC21dcYIdztrU9VIkUXEuxVUMmB6op1YunnVik8aA7tlGwnYjbz3sKBg/vjEibnYVzh9WEilHoXECnvUsylgoPWzX0LS40Q3gPNGeRaN/H1zOXqXBX6gz4YZWXD+BMTkwJd4b/itrqbDK860e91huUbtbRvEZpFquP/ODhGh1qdfjf80EVTjewZ0Gene3JSpNbAsPemD8f6Zd3UxlAaUjQvwEaDACUD7sjSyq1P2t6+utE163YywwwzPpNWvoo0xw2poaesV3kY6v99V5kE4zC8D25Lb4yUM8AhDwk+JqD5OA7jGCQ0u2qa0rMdeDw+BtkhaZBVrKabE+NXMo/SLocBE7MaHIwVBp7eaQyWW2o+MW0Bg2o594/Rs= ceph-d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:53:39.608 INFO:teuthology.orchestra.run.smithi163.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-21T12:53:39.608 INFO:teuthology.orchestra.run.smithi163.stdout:Adding key to root@localhost authorized_keys... 2024-03-21T12:53:39.609 INFO:teuthology.orchestra.run.smithi163.stdout:Adding host smithi163... 2024-03-21T12:53:40.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:40 smithi163 ceph-mon[29621]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:41.206 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:41 smithi163 ceph-mon[29621]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi163", "addr": "172.21.15.163", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:41.207 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:41 smithi163 ceph-mon[29621]: Deploying cephadm binary to smithi163 2024-03-21T12:53:45.073 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout Added host 'smithi163' with addr '172.21.15.163' 2024-03-21T12:53:45.074 INFO:teuthology.orchestra.run.smithi163.stdout:Deploying unmanaged mon service... 2024-03-21T12:53:45.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:45 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:45.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:45 smithi163 ceph-mon[29621]: Added host smithi163 2024-03-21T12:53:45.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:45 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:53:46.515 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-21T12:53:46.515 INFO:teuthology.orchestra.run.smithi163.stdout:Deploying unmanaged mgr service... 2024-03-21T12:53:47.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:46 smithi163 ceph-mon[29621]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:47.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:46 smithi163 ceph-mon[29621]: Saving service mon spec with placement count:5 2024-03-21T12:53:47.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:46 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:47.730 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-21T12:53:48.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:48 smithi163 ceph-mon[29621]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:48.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:48 smithi163 ceph-mon[29621]: Saving service mgr spec with placement count:2 2024-03-21T12:53:48.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:48 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:49.888 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:49 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:49.888 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:49 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:49.889 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:49 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3833946838' entity='client.admin' 2024-03-21T12:53:51.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:51 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1111207774' entity='client.admin' 2024-03-21T12:53:51.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:51 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:51.608 INFO:teuthology.orchestra.run.smithi163.stdout:Enabling the dashboard module... 2024-03-21T12:53:53.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:53 smithi163 ceph-mon[29621]: from='mgr.14118 172.21.15.163:0/2816227928' entity='mgr.y' 2024-03-21T12:53:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:53 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2524916278' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-21T12:53:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:53 smithi163 ceph-mon[29621]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:53:53.511 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:53 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ignoring --setuser ceph since I am not root 2024-03-21T12:53:53.511 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:53 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ignoring --setgroup ceph since I am not root 2024-03-21T12:53:54.152 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:53 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:53.825+0000 7fc7c9ffd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:53:54.152 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:53 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:53.920+0000 7fc7c9ffd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:53:54.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:54 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2524916278' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-21T12:53:54.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:54 smithi163 ceph-mon[29621]: mgrmap e9: y(active, since 20s) 2024-03-21T12:53:54.706 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:54 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:54.583+0000 7fc7c9ffd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:53:54.706 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:54 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:54.679+0000 7fc7c9ffd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:53:55.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:54 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:54.830+0000 7fc7c9ffd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:53:55.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:54 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:54.931+0000 7fc7c9ffd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:53:55.286 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:53:55 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2084884096' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T12:53:55.287 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:55.015+0000 7fc7c9ffd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:53:55.287 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:55.285+0000 7fc7c9ffd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:53:55.582 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:55.395+0000 7fc7c9ffd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:53:55.582 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:55.484+0000 7fc7c9ffd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:53:55.582 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:55.581+0000 7fc7c9ffd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:53:55.805 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout { 2024-03-21T12:53:55.805 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-21T12:53:55.805 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-21T12:53:55.805 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-21T12:53:55.805 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-21T12:53:55.805 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout } 2024-03-21T12:53:55.805 INFO:teuthology.orchestra.run.smithi163.stdout:Waiting for the mgr to restart... 2024-03-21T12:53:55.805 INFO:teuthology.orchestra.run.smithi163.stdout:Waiting for mgr epoch 9... 2024-03-21T12:53:55.927 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:55.760+0000 7fc7c9ffd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:53:57.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:56 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:56.742+0000 7fc7c9ffd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:53:57.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:56 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:56.873+0000 7fc7c9ffd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:53:57.498 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:57 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:57.209+0000 7fc7c9ffd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:53:57.498 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:57 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:57.300+0000 7fc7c9ffd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:53:57.498 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:57 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:57.497+0000 7fc7c9ffd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:53:57.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:57 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:57.684+0000 7fc7c9ffd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:53:58.234 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:57 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:57.972+0000 7fc7c9ffd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:53:58.234 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:58 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:58.233+0000 7fc7c9ffd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:53:58.510 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:58 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:58.474+0000 7fc7c9ffd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:53:59.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:59 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:59.287+0000 7fc7c9ffd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:54:00.211 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:59 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:59.888+0000 7fc7c9ffd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:54:00.211 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:53:59 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:53:59.988+0000 7fc7c9ffd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:54:00.211 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:54:00 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:54:00.210+0000 7fc7c9ffd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:54:00.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:00 smithi163 ceph-mon[29621]: Active manager daemon y restarted 2024-03-21T12:54:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:00 smithi163 ceph-mon[29621]: Activating manager daemon y 2024-03-21T12:54:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:00 smithi163 ceph-mon[29621]: osdmap e3: 0 total, 0 up, 0 in 2024-03-21T12:54:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:00 smithi163 ceph-mon[29621]: mgrmap e10: y(active, starting, since 0.00371163s) 2024-03-21T12:54:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:54:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:54:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:54:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:54:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:54:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:00 smithi163 ceph-mon[29621]: Manager daemon y is now available 2024-03-21T12:54:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:01.503 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:01 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:54:01.503 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:01 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:54:01.503 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:01 smithi163 ceph-mon[29621]: mgrmap e11: y(active, since 1.00618s) 2024-03-21T12:54:02.222 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout { 2024-03-21T12:54:02.222 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-21T12:54:02.222 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-21T12:54:02.222 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout } 2024-03-21T12:54:02.222 INFO:teuthology.orchestra.run.smithi163.stdout:mgr epoch 9 is available 2024-03-21T12:54:02.222 INFO:teuthology.orchestra.run.smithi163.stdout:Generating a dashboard self-signed certificate... 2024-03-21T12:54:02.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:02 smithi163 ceph-mon[29621]: [21/Mar/2024:12:54:01] ENGINE Bus STARTING 2024-03-21T12:54:02.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:02 smithi163 ceph-mon[29621]: [21/Mar/2024:12:54:01] ENGINE Serving on http://172.21.15.163:8765 2024-03-21T12:54:02.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:02 smithi163 ceph-mon[29621]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-21T12:54:02.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:02 smithi163 ceph-mon[29621]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-21T12:54:02.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:02 smithi163 ceph-mon[29621]: [21/Mar/2024:12:54:01] ENGINE Serving on https://172.21.15.163:7150 2024-03-21T12:54:02.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:02 smithi163 ceph-mon[29621]: [21/Mar/2024:12:54:01] ENGINE Bus STARTED 2024-03-21T12:54:02.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:02 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:03.653 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-21T12:54:03.653 INFO:teuthology.orchestra.run.smithi163.stdout:Creating initial admin user... 2024-03-21T12:54:03.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:03 smithi163 ceph-mon[29621]: mgrmap e12: y(active, since 2s) 2024-03-21T12:54:03.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:03 smithi163 ceph-mon[29621]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:03.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:03.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:03.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:05.774 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$FbDRXqLWhb5XNhKvTvD.NOumvhfkwFE/1MQIg17mq.HpfR9gp2kla", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711025644, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-21T12:54:05.774 INFO:teuthology.orchestra.run.smithi163.stdout:Fetching dashboard port number... 2024-03-21T12:54:05.880 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:05 smithi163 ceph-mon[29621]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:05.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:05 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:06.832 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:06 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4215445045' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-21T12:54:07.098 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stdout 8443 2024-03-21T12:54:07.107 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T12:54:07.108 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout disabled 2024-03-21T12:54:07.114 INFO:teuthology.orchestra.run.smithi163.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T12:54:07.114 INFO:teuthology.orchestra.run.smithi163.stdout:systemctl: stdout inactive 2024-03-21T12:54:07.114 INFO:teuthology.orchestra.run.smithi163.stdout:firewalld.service is not enabled 2024-03-21T12:54:07.115 INFO:teuthology.orchestra.run.smithi163.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-21T12:54:07.116 INFO:teuthology.orchestra.run.smithi163.stdout:Ceph Dashboard is now available at: 2024-03-21T12:54:07.116 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:54:07.116 INFO:teuthology.orchestra.run.smithi163.stdout: URL: https://smithi163.front.sepia.ceph.com:8443/ 2024-03-21T12:54:07.116 INFO:teuthology.orchestra.run.smithi163.stdout: User: admin 2024-03-21T12:54:07.116 INFO:teuthology.orchestra.run.smithi163.stdout: Password: ae0w49i4nm 2024-03-21T12:54:07.117 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:54:07.117 INFO:teuthology.orchestra.run.smithi163.stdout:Saving cluster configuration to /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config directory 2024-03-21T12:54:07.117 INFO:teuthology.orchestra.run.smithi163.stdout:Enabling autotune for osd_memory_target 2024-03-21T12:54:07.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:07 smithi163 ceph-mon[29621]: mgrmap e13: y(active, since 6s) 2024-03-21T12:54:09.701 INFO:teuthology.orchestra.run.smithi163.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-21T12:54:09.701 INFO:teuthology.orchestra.run.smithi163.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-21T12:54:09.701 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:54:09.701 INFO:teuthology.orchestra.run.smithi163.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:09.701 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:54:09.702 INFO:teuthology.orchestra.run.smithi163.stdout:Or, if you are only running a single cluster on this host: 2024-03-21T12:54:09.702 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:54:09.702 INFO:teuthology.orchestra.run.smithi163.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-21T12:54:09.702 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:54:09.702 INFO:teuthology.orchestra.run.smithi163.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-21T12:54:09.702 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:54:09.702 INFO:teuthology.orchestra.run.smithi163.stdout: ceph telemetry on 2024-03-21T12:54:09.702 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:54:09.702 INFO:teuthology.orchestra.run.smithi163.stdout:For more information see: 2024-03-21T12:54:09.702 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:54:09.702 INFO:teuthology.orchestra.run.smithi163.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-21T12:54:09.702 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:54:09.705 INFO:teuthology.orchestra.run.smithi163.stdout:Bootstrap complete. 2024-03-21T12:54:09.744 INFO:tasks.cephadm:Fetching config... 2024-03-21T12:54:09.745 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:54:09.745 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-21T12:54:09.760 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-21T12:54:09.761 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:54:09.761 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-21T12:54:09.815 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-21T12:54:09.815 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:54:09.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-21T12:54:09.883 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-21T12:54:09.883 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:54:09.883 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-21T12:54:09.938 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-21T12:54:09.939 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi+inm7TjL18rx+QLDeJ5r/FVNGnjaELnuhMIzeqb3rRxtzIEQGqem2Eil+vKl4tZry1LlMkXFLS45Z2K9nEOZ0ByjCONB9KC0jmNjJ3THjMU2mjYlTJHZ4qQsPskfXFQxvH0OavCiC21dcYIdztrU9VIkUXEuxVUMmB6op1YunnVik8aA7tlGwnYjbz3sKBg/vjEibnYVzh9WEilHoXECnvUsylgoPWzX0LS40Q3gPNGeRaN/H1zOXqXBX6gz4YZWXD+BMTkwJd4b/itrqbDK860e91huUbtbRvEZpFquP/ODhGh1qdfjf80EVTjewZ0Gene3JSpNbAsPemD8f6Zd3UxlAaUjQvwEaDACUD7sjSyq1P2t6+utE163YywwwzPpNWvoo0xw2poaesV3kY6v99V5kE4zC8D25Lb4yUM8AhDwk+JqD5OA7jGCQ0u2qa0rMdeDw+BtkhaZBVrKabE+NXMo/SLocBE7MaHIwVBp7eaQyWW2o+MW0Bg2o594/Rs= ceph-d4bf79c0-e781-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-21T12:54:10.017 INFO:teuthology.orchestra.run.smithi163.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi+inm7TjL18rx+QLDeJ5r/FVNGnjaELnuhMIzeqb3rRxtzIEQGqem2Eil+vKl4tZry1LlMkXFLS45Z2K9nEOZ0ByjCONB9KC0jmNjJ3THjMU2mjYlTJHZ4qQsPskfXFQxvH0OavCiC21dcYIdztrU9VIkUXEuxVUMmB6op1YunnVik8aA7tlGwnYjbz3sKBg/vjEibnYVzh9WEilHoXECnvUsylgoPWzX0LS40Q3gPNGeRaN/H1zOXqXBX6gz4YZWXD+BMTkwJd4b/itrqbDK860e91huUbtbRvEZpFquP/ODhGh1qdfjf80EVTjewZ0Gene3JSpNbAsPemD8f6Zd3UxlAaUjQvwEaDACUD7sjSyq1P2t6+utE163YywwwzPpNWvoo0xw2poaesV3kY6v99V5kE4zC8D25Lb4yUM8AhDwk+JqD5OA7jGCQ0u2qa0rMdeDw+BtkhaZBVrKabE+NXMo/SLocBE7MaHIwVBp7eaQyWW2o+MW0Bg2o594/Rs= ceph-d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:54:10.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi+inm7TjL18rx+QLDeJ5r/FVNGnjaELnuhMIzeqb3rRxtzIEQGqem2Eil+vKl4tZry1LlMkXFLS45Z2K9nEOZ0ByjCONB9KC0jmNjJ3THjMU2mjYlTJHZ4qQsPskfXFQxvH0OavCiC21dcYIdztrU9VIkUXEuxVUMmB6op1YunnVik8aA7tlGwnYjbz3sKBg/vjEibnYVzh9WEilHoXECnvUsylgoPWzX0LS40Q3gPNGeRaN/H1zOXqXBX6gz4YZWXD+BMTkwJd4b/itrqbDK860e91huUbtbRvEZpFquP/ODhGh1qdfjf80EVTjewZ0Gene3JSpNbAsPemD8f6Zd3UxlAaUjQvwEaDACUD7sjSyq1P2t6+utE163YywwwzPpNWvoo0xw2poaesV3kY6v99V5kE4zC8D25Lb4yUM8AhDwk+JqD5OA7jGCQ0u2qa0rMdeDw+BtkhaZBVrKabE+NXMo/SLocBE7MaHIwVBp7eaQyWW2o+MW0Bg2o594/Rs= ceph-d4bf79c0-e781-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-21T12:54:10.088 INFO:teuthology.orchestra.run.smithi204.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi+inm7TjL18rx+QLDeJ5r/FVNGnjaELnuhMIzeqb3rRxtzIEQGqem2Eil+vKl4tZry1LlMkXFLS45Z2K9nEOZ0ByjCONB9KC0jmNjJ3THjMU2mjYlTJHZ4qQsPskfXFQxvH0OavCiC21dcYIdztrU9VIkUXEuxVUMmB6op1YunnVik8aA7tlGwnYjbz3sKBg/vjEibnYVzh9WEilHoXECnvUsylgoPWzX0LS40Q3gPNGeRaN/H1zOXqXBX6gz4YZWXD+BMTkwJd4b/itrqbDK860e91huUbtbRvEZpFquP/ODhGh1qdfjf80EVTjewZ0Gene3JSpNbAsPemD8f6Zd3UxlAaUjQvwEaDACUD7sjSyq1P2t6+utE163YywwwzPpNWvoo0xw2poaesV3kY6v99V5kE4zC8D25Lb4yUM8AhDwk+JqD5OA7jGCQ0u2qa0rMdeDw+BtkhaZBVrKabE+NXMo/SLocBE7MaHIwVBp7eaQyWW2o+MW0Bg2o594/Rs= ceph-d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:54:10.103 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-21T12:54:10.344 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:54:10.466 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:10 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2273136413' entity='client.admin' 2024-03-21T12:54:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:12 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:12 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:12 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi163", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:54:13.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:12 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:13.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:12 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:13.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:12 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:13.950 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-21T12:54:13.951 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-21T12:54:14.218 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:54:14.559 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:14 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3739239245' entity='client.admin' 2024-03-21T12:54:14.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:14 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:14.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:14 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:16.958 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi204 2024-03-21T12:54:16.958 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-21T12:54:16.958 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.conf 2024-03-21T12:54:16.977 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-21T12:54:16.977 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:17.035 INFO:tasks.cephadm:Adding host smithi204 to orchestrator... 2024-03-21T12:54:17.035 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch host add smithi204 2024-03-21T12:54:17.272 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:54:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:17 smithi163 ceph-mon[29621]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:17 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:17 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:17 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:17 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:17 smithi163 ceph-mon[29621]: Updating smithi163:/etc/ceph/ceph.conf 2024-03-21T12:54:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:17 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:17 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:17.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:17 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:18.399 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:18 smithi163 ceph-mon[29621]: Updating smithi163:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:18.399 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:18 smithi163 ceph-mon[29621]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:18.399 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:18 smithi163 ceph-mon[29621]: Updating smithi163:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:54:19.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:19 smithi163 ceph-mon[29621]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:20.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:20 smithi163 ceph-mon[29621]: Deploying cephadm binary to smithi204 2024-03-21T12:54:20.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:20 smithi163 ceph-mon[29621]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:23.368 INFO:teuthology.orchestra.run.smithi163.stdout:Added host 'smithi204' with addr '172.21.15.204' 2024-03-21T12:54:23.679 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:23 smithi163 ceph-mon[29621]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:23.954 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-21T12:54:24.187 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:54:24.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:24 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:24.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:24 smithi163 ceph-mon[29621]: Added host smithi204 2024-03-21T12:54:24.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:24 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:24.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:24 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:24.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:24 smithi163 ceph-mon[29621]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:25.508 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:25.929 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:54:25.930 INFO:teuthology.orchestra.run.smithi163.stdout:[{"addr": "172.21.15.163", "hostname": "smithi163", "labels": [], "status": ""}, {"addr": "172.21.15.204", "hostname": "smithi204", "labels": [], "status": ""}] 2024-03-21T12:54:26.553 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-21T12:54:26.553 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-21T12:54:26.785 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:54:27.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:26 smithi163 ceph-mon[29621]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T12:54:27.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:26 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:27.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:26 smithi163 ceph-mon[29621]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:29.289 INFO:teuthology.orchestra.run.smithi163.stderr:adjusted tunables profile to default 2024-03-21T12:54:29.662 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:29 smithi163 ceph-mon[29621]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:29.662 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:29 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3058284462' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-21T12:54:30.003 INFO:tasks.cephadm:Adding mon.a on smithi163 2024-03-21T12:54:30.003 INFO:tasks.cephadm:Adding mon.c on smithi163 2024-03-21T12:54:30.003 INFO:tasks.cephadm:Adding mon.b on smithi204 2024-03-21T12:54:30.004 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi163:172.21.15.163=a;smithi163:[v2:172.21.15.163:3301,v1:172.21.15.163:6790]=c;smithi204:172.21.15.204=b' 2024-03-21T12:54:30.238 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:30 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3058284462' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-21T12:54:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:30 smithi163 ceph-mon[29621]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:54:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:30.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:30.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:54:30.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:30.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:30.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:30.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:30.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:31 smithi163 ceph-mon[29621]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-21T12:54:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:31 smithi163 ceph-mon[29621]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:31 smithi163 ceph-mon[29621]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:31 smithi163 ceph-mon[29621]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:54:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:31 smithi163 ceph-mon[29621]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:31.889 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mon update... 2024-03-21T12:54:32.447 DEBUG:teuthology.orchestra.run.smithi163:mon.c> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.c.service 2024-03-21T12:54:32.450 DEBUG:teuthology.orchestra.run.smithi204:mon.b> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.b.service 2024-03-21T12:54:32.452 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T12:54:32.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T12:54:33.211 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:32 smithi163 ceph-mon[29621]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi163:172.21.15.163=a;smithi163:[v2:172.21.15.163:3301,v1:172.21.15.163:6790]=c;smithi204:172.21.15.204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:32 smithi163 ceph-mon[29621]: Saving service mon spec with placement smithi163:172.21.15.163=a;smithi163:[v2:172.21.15.163:3301,v1:172.21.15.163:6790]=c;smithi204:172.21.15.204=b;count:3 2024-03-21T12:54:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:32 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:32 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:32 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:32 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:32 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:32 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:32 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:32 smithi163 ceph-mon[29621]: Deploying daemon mon.b on smithi204 2024-03-21T12:54:33.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:32 smithi163 ceph-mon[29621]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:35.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:35 smithi163 ceph-mon[29621]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:36.716 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:54:36.716 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":1,"fsid":"d4bf79c0-e781-11ee-95cd-87774f69a715","modified":"2024-03-21T12:52:53.111495Z","created":"2024-03-21T12:52:53.111495Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-21T12:54:36.718 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 1 2024-03-21T12:54:36.852 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-21T12:54:36.852 INFO:journalctl@ceph.mon.b.smithi204.stdout: e1 2024-03-21T12:54:36.852 INFO:journalctl@ceph.mon.b.smithi204.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T12:54:36.852 INFO:journalctl@ceph.mon.b.smithi204.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-21T12:54:36.853 INFO:journalctl@ceph.mon.b.smithi204.stdout: legacy client fscid: -1 2024-03-21T12:54:36.853 INFO:journalctl@ceph.mon.b.smithi204.stdout: 2024-03-21T12:54:36.853 INFO:journalctl@ceph.mon.b.smithi204.stdout: No filesystems configured 2024-03-21T12:54:36.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-21T12:54:36.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-21T12:54:36.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-21T12:54:36.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-21T12:54:36.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-21T12:54:36.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-21T12:54:36.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-21T12:54:36.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:54:36.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:54:36.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:54:37.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:37.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3058284462' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-21T12:54:37.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3058284462' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-21T12:54:37.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:54:37.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:37.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:37.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:37.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:37.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:54:37.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:37.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:37.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:37.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:37.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:37.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-21T12:54:37.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:37.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:37.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:54:37.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi163:172.21.15.163=a;smithi163:[v2:172.21.15.163:3301,v1:172.21.15.163:6790]=c;smithi204:172.21.15.204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: Saving service mon spec with placement smithi163:172.21.15.163=a;smithi163:[v2:172.21.15.163:3301,v1:172.21.15.163:6790]=c;smithi204:172.21.15.204=b;count:3 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: Deploying daemon mon.b on smithi204 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 ceph-mon[33153]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 bash[33122]: defe8b351948e7ae80e2ec0459d01a01a81d49c88aed69b47f142dca27d3925d 2024-03-21T12:54:37.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:36 smithi204 systemd[1]: Started Ceph mon.b for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T12:54:38.537 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T12:54:38.537 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T12:54:38.782 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:54:42.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:54:42.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: mon.a calling monitor election 2024-03-21T12:54:42.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:42.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:42.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:42.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:42.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: mon.b calling monitor election 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: monmap e2: 2 mons at {a=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: fsmap 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: mgrmap e13: y(active, since 41s) 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: overall HEALTH_OK 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:42.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:42.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:41 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:42.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:54:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: mon.a calling monitor election 2024-03-21T12:54:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: mon.b calling monitor election 2024-03-21T12:54:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-21T12:54:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: monmap e2: 2 mons at {a=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:54:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: fsmap 2024-03-21T12:54:42.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:54:42.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: mgrmap e13: y(active, since 41s) 2024-03-21T12:54:42.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: overall HEALTH_OK 2024-03-21T12:54:42.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:42.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:42.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:42.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:42.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:41 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:43.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:42 smithi204 ceph-mon[33153]: Deploying daemon mon.c on smithi163 2024-03-21T12:54:43.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:42 smithi204 ceph-mon[33153]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:43.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:42 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:43.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:42 smithi163 ceph-mon[29621]: Deploying daemon mon.c on smithi163 2024-03-21T12:54:43.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:42 smithi163 ceph-mon[29621]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:43.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:42 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:43.587 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:54:43.588 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":2,"fsid":"d4bf79c0-e781-11ee-95cd-87774f69a715","modified":"2024-03-21T12:54:36.857678Z","created":"2024-03-21T12:52:53.111495Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-21T12:54:43.590 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 2 2024-03-21T12:54:44.136 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:43 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/2379241171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:54:44.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:43 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/2379241171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:54:45.166 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T12:54:45.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T12:54:45.219 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:44 smithi163 ceph-mon[29621]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:45.219 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 systemd[1]: Starting Ceph mon.c for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T12:54:45.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:44 smithi204 ceph-mon[33153]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:45.414 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:54:45.477 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 podman[40656]: 2024-03-21 12:54:45.12264377 +0000 UTC m=+0.018843867 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T12:54:45.477 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 podman[40656]: 2024-03-21 12:54:45.244368876 +0000 UTC m=+0.140569014 container create 4cd4267d4f7d15ff70a972c35873ce847efaaa263b241c1f1c0312f4b5a0297d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T12:54:45.477 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 podman[40656]: 2024-03-21 12:54:45.430229525 +0000 UTC m=+0.326429624 container init 4cd4267d4f7d15ff70a972c35873ce847efaaa263b241c1f1c0312f4b5a0297d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-c, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-21T12:54:45.477 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 podman[40656]: 2024-03-21 12:54:45.433263012 +0000 UTC m=+0.329463105 container start 4cd4267d4f7d15ff70a972c35873ce847efaaa263b241c1f1c0312f4b5a0297d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-c, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-21T12:54:45.478 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T12:54:45.478 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev), process ceph-mon, pid 2 2024-03-21T12:54:45.478 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: pidfile_write: ignore empty --pid-file 2024-03-21T12:54:45.478 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: load: jerasure load: lrc 2024-03-21T12:54:45.478 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: RocksDB version: 7.9.2 2024-03-21T12:54:45.478 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Git sha 0 2024-03-21T12:54:45.478 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Compile date 2024-03-21 08:53:26 2024-03-21T12:54:45.478 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: DB SUMMARY 2024-03-21T12:54:45.478 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: DB Session ID: ULHD2RCL1QZCRT3EL5I6 2024-03-21T12:54:45.478 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: CURRENT file: CURRENT 2024-03-21T12:54:45.478 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T12:54:45.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-21T12:54:45.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-21T12:54:45.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-21T12:54:45.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.error_if_exists: 0 2024-03-21T12:54:45.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.create_if_missing: 0 2024-03-21T12:54:45.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.paranoid_checks: 1 2024-03-21T12:54:45.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T12:54:45.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T12:54:45.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T12:54:45.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.env: 0x560d7ab62b20 2024-03-21T12:54:45.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T12:54:45.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.info_log: 0x560d7b60f920 2024-03-21T12:54:45.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T12:54:45.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.statistics: (nil) 2024-03-21T12:54:45.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.use_fsync: 0 2024-03-21T12:54:45.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_log_file_size: 0 2024-03-21T12:54:45.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T12:54:45.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T12:54:45.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T12:54:45.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T12:54:45.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.allow_fallocate: 1 2024-03-21T12:54:45.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T12:54:45.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T12:54:45.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.use_direct_reads: 0 2024-03-21T12:54:45.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T12:54:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T12:54:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.db_log_dir: 2024-03-21T12:54:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.wal_dir: 2024-03-21T12:54:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T12:54:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T12:54:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T12:54:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T12:54:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T12:54:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T12:54:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T12:54:45.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T12:54:45.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.write_buffer_manager: 0x560d7b5e01e0 2024-03-21T12:54:45.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T12:54:45.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T12:54:45.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T12:54:45.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T12:54:45.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T12:54:45.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T12:54:45.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T12:54:45.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T12:54:45.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.unordered_write: 0 2024-03-21T12:54:45.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T12:54:45.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T12:54:45.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T12:54:45.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T12:54:45.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.row_cache: None 2024-03-21T12:54:45.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.wal_filter: None 2024-03-21T12:54:45.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T12:54:45.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T12:54:45.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.two_write_queues: 0 2024-03-21T12:54:45.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T12:54:45.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.wal_compression: 0 2024-03-21T12:54:45.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.atomic_flush: 0 2024-03-21T12:54:45.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T12:54:45.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T12:54:45.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T12:54:45.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.log_readahead_size: 0 2024-03-21T12:54:45.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T12:54:45.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T12:54:45.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T12:54:45.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T12:54:45.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T12:54:45.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T12:54:45.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T12:54:45.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_background_jobs: 2 2024-03-21T12:54:45.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_background_compactions: -1 2024-03-21T12:54:45.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_subcompactions: 1 2024-03-21T12:54:45.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T12:54:45.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T12:54:45.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T12:54:45.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T12:54:45.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T12:54:45.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T12:54:45.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T12:54:45.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T12:54:45.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_open_files: -1 2024-03-21T12:54:45.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T12:54:45.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T12:54:45.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T12:54:45.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T12:54:45.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_background_flushes: -1 2024-03-21T12:54:45.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Compression algorithms supported: 2024-03-21T12:54:45.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T12:54:45.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: kZSTD supported: 0 2024-03-21T12:54:45.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: kXpressCompression supported: 0 2024-03-21T12:54:45.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T12:54:45.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: kZlibCompression supported: 1 2024-03-21T12:54:45.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: kSnappyCompression supported: 1 2024-03-21T12:54:45.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: kLZ4Compression supported: 1 2024-03-21T12:54:45.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: kBZip2Compression supported: 0 2024-03-21T12:54:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T12:54:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T12:54:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-21T12:54:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T12:54:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T12:54:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.merge_operator: 2024-03-21T12:54:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compaction_filter: None 2024-03-21T12:54:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T12:54:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T12:54:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T12:54:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T12:54:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560d7b60fa20) 2024-03-21T12:54:45.489 INFO:journalctl@ceph.mon.c.smithi163.stdout: cache_index_and_filter_blocks: 1 2024-03-21T12:54:45.489 INFO:journalctl@ceph.mon.c.smithi163.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T12:54:45.489 INFO:journalctl@ceph.mon.c.smithi163.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T12:54:45.489 INFO:journalctl@ceph.mon.c.smithi163.stdout: pin_top_level_index_and_filter: 1 2024-03-21T12:54:45.489 INFO:journalctl@ceph.mon.c.smithi163.stdout: index_type: 0 2024-03-21T12:54:45.489 INFO:journalctl@ceph.mon.c.smithi163.stdout: data_block_index_type: 0 2024-03-21T12:54:45.489 INFO:journalctl@ceph.mon.c.smithi163.stdout: index_shortening: 1 2024-03-21T12:54:45.489 INFO:journalctl@ceph.mon.c.smithi163.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T12:54:45.489 INFO:journalctl@ceph.mon.c.smithi163.stdout: checksum: 4 2024-03-21T12:54:45.489 INFO:journalctl@ceph.mon.c.smithi163.stdout: no_block_cache: 0 2024-03-21T12:54:45.489 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_cache: 0x560d7b613090 2024-03-21T12:54:45.489 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_cache_name: BinnedLRUCache 2024-03-21T12:54:45.489 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_cache_options: 2024-03-21T12:54:45.490 INFO:journalctl@ceph.mon.c.smithi163.stdout: capacity : 536870912 2024-03-21T12:54:45.490 INFO:journalctl@ceph.mon.c.smithi163.stdout: num_shard_bits : 4 2024-03-21T12:54:45.490 INFO:journalctl@ceph.mon.c.smithi163.stdout: strict_capacity_limit : 0 2024-03-21T12:54:45.490 INFO:journalctl@ceph.mon.c.smithi163.stdout: high_pri_pool_ratio: 0.000 2024-03-21T12:54:45.490 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_cache_compressed: (nil) 2024-03-21T12:54:45.490 INFO:journalctl@ceph.mon.c.smithi163.stdout: persistent_cache: (nil) 2024-03-21T12:54:45.490 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_size: 4096 2024-03-21T12:54:45.492 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_size_deviation: 10 2024-03-21T12:54:45.492 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_restart_interval: 16 2024-03-21T12:54:45.492 INFO:journalctl@ceph.mon.c.smithi163.stdout: index_block_restart_interval: 1 2024-03-21T12:54:45.492 INFO:journalctl@ceph.mon.c.smithi163.stdout: metadata_block_size: 4096 2024-03-21T12:54:45.492 INFO:journalctl@ceph.mon.c.smithi163.stdout: partition_filters: 0 2024-03-21T12:54:45.492 INFO:journalctl@ceph.mon.c.smithi163.stdout: use_delta_encoding: 1 2024-03-21T12:54:45.492 INFO:journalctl@ceph.mon.c.smithi163.stdout: filter_policy: bloomfilter 2024-03-21T12:54:45.492 INFO:journalctl@ceph.mon.c.smithi163.stdout: whole_key_filtering: 1 2024-03-21T12:54:45.492 INFO:journalctl@ceph.mon.c.smithi163.stdout: verify_compression: 0 2024-03-21T12:54:45.492 INFO:journalctl@ceph.mon.c.smithi163.stdout: read_amp_bytes_per_bit: 0 2024-03-21T12:54:45.492 INFO:journalctl@ceph.mon.c.smithi163.stdout: format_version: 5 2024-03-21T12:54:45.492 INFO:journalctl@ceph.mon.c.smithi163.stdout: enable_index_compression: 1 2024-03-21T12:54:45.493 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_align: 0 2024-03-21T12:54:45.493 INFO:journalctl@ceph.mon.c.smithi163.stdout: max_auto_readahead_size: 262144 2024-03-21T12:54:45.493 INFO:journalctl@ceph.mon.c.smithi163.stdout: prepopulate_block_cache: 0 2024-03-21T12:54:45.493 INFO:journalctl@ceph.mon.c.smithi163.stdout: initial_auto_readahead_size: 8192 2024-03-21T12:54:45.493 INFO:journalctl@ceph.mon.c.smithi163.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T12:54:45.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T12:54:45.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T12:54:45.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compression: NoCompression 2024-03-21T12:54:45.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T12:54:45.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T12:54:45.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T12:54:45.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.num_levels: 7 2024-03-21T12:54:45.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T12:54:45.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T12:54:45.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T12:54:45.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T12:54:45.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T12:54:45.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T12:54:45.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T12:54:45.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:54:45.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T12:54:45.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T12:54:45.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T12:54:45.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T12:54:45.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T12:54:45.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T12:54:45.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T12:54:45.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T12:54:45.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:54:45.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T12:54:45.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T12:54:45.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T12:54:45.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T12:54:45.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T12:54:45.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T12:54:45.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T12:54:45.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T12:54:45.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T12:54:45.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T12:54:45.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T12:54:45.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T12:54:45.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T12:54:45.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T12:54:45.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T12:54:45.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T12:54:45.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T12:54:45.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T12:54:45.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T12:54:45.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T12:54:45.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T12:54:45.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T12:54:45.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T12:54:45.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T12:54:45.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T12:54:45.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T12:54:45.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T12:54:45.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T12:54:45.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T12:54:45.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T12:54:45.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T12:54:45.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T12:54:45.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T12:54:45.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T12:54:45.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T12:54:45.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T12:54:45.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.table_properties_collectors: 2024-03-21T12:54:45.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.inplace_update_support: 0 2024-03-21T12:54:45.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T12:54:45.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T12:54:45.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T12:54:45.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T12:54:45.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.bloom_locality: 0 2024-03-21T12:54:45.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.max_successive_merges: 0 2024-03-21T12:54:45.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T12:54:45.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T12:54:45.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T12:54:45.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T12:54:45.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.ttl: 2592000 2024-03-21T12:54:45.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T12:54:45.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T12:54:45.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T12:54:45.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.enable_blob_files: false 2024-03-21T12:54:45.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.min_blob_size: 0 2024-03-21T12:54:45.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T12:54:45.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T12:54:45.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T12:54:45.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T12:54:45.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T12:54:45.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T12:54:45.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T12:54:45.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T12:54:45.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T12:54:45.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-21T12:54:45.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5ac29755-264a-452f-bdee-50244ffc6381 2024-03-21T12:54:45.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025685462229, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-21T12:54:45.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-21T12:54:45.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025685462629, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711025685, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5ac29755-264a-452f-bdee-50244ffc6381", "db_session_id": "ULHD2RCL1QZCRT3EL5I6", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-21T12:54:45.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025685462710, "job": 1, "event": "recovery_finished"} 2024-03-21T12:54:45.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-21T12:54:45.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T12:54:45.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560d7b702000 2024-03-21T12:54:45.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: DB pointer 0x560d7b6ec000 2024-03-21T12:54:45.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T12:54:45.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T12:54:45.501 INFO:journalctl@ceph.mon.c.smithi163.stdout: ** DB Stats ** 2024-03-21T12:54:45.501 INFO:journalctl@ceph.mon.c.smithi163.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:54:45.501 INFO:journalctl@ceph.mon.c.smithi163.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T12:54:45.501 INFO:journalctl@ceph.mon.c.smithi163.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T12:54:45.502 INFO:journalctl@ceph.mon.c.smithi163.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:54:45.502 INFO:journalctl@ceph.mon.c.smithi163.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T12:54:45.502 INFO:journalctl@ceph.mon.c.smithi163.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T12:54:45.502 INFO:journalctl@ceph.mon.c.smithi163.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:54:45.502 INFO:journalctl@ceph.mon.c.smithi163.stdout: 2024-03-21T12:54:45.502 INFO:journalctl@ceph.mon.c.smithi163.stdout: ** Compaction Stats [default] ** 2024-03-21T12:54:45.502 INFO:journalctl@ceph.mon.c.smithi163.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T12:54:45.502 INFO:journalctl@ceph.mon.c.smithi163.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T12:54:45.502 INFO:journalctl@ceph.mon.c.smithi163.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T12:54:45.502 INFO:journalctl@ceph.mon.c.smithi163.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T12:54:45.502 INFO:journalctl@ceph.mon.c.smithi163.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T12:54:45.502 INFO:journalctl@ceph.mon.c.smithi163.stdout: 2024-03-21T12:54:45.503 INFO:journalctl@ceph.mon.c.smithi163.stdout: ** Compaction Stats [default] ** 2024-03-21T12:54:45.503 INFO:journalctl@ceph.mon.c.smithi163.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T12:54:45.503 INFO:journalctl@ceph.mon.c.smithi163.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T12:54:45.503 INFO:journalctl@ceph.mon.c.smithi163.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T12:54:45.503 INFO:journalctl@ceph.mon.c.smithi163.stdout: 2024-03-21T12:54:45.503 INFO:journalctl@ceph.mon.c.smithi163.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T12:54:45.503 INFO:journalctl@ceph.mon.c.smithi163.stdout: 2024-03-21T12:54:45.503 INFO:journalctl@ceph.mon.c.smithi163.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:54:45.503 INFO:journalctl@ceph.mon.c.smithi163.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T12:54:45.503 INFO:journalctl@ceph.mon.c.smithi163.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T12:54:45.503 INFO:journalctl@ceph.mon.c.smithi163.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T12:54:45.503 INFO:journalctl@ceph.mon.c.smithi163.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T12:54:45.504 INFO:journalctl@ceph.mon.c.smithi163.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T12:54:45.504 INFO:journalctl@ceph.mon.c.smithi163.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:54:45.504 INFO:journalctl@ceph.mon.c.smithi163.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:54:45.504 INFO:journalctl@ceph.mon.c.smithi163.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T12:54:45.504 INFO:journalctl@ceph.mon.c.smithi163.stdout: Block cache BinnedLRUCache@0x560d7b613090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 7e-06 secs_since: 0 2024-03-21T12:54:45.504 INFO:journalctl@ceph.mon.c.smithi163.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-21T12:54:45.504 INFO:journalctl@ceph.mon.c.smithi163.stdout: 2024-03-21T12:54:45.504 INFO:journalctl@ceph.mon.c.smithi163.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T12:54:45.504 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-21T12:54:45.504 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: using public_addrv [v2:172.21.15.163:3301/0,v1:172.21.15.163:6790/0] 2024-03-21T12:54:45.504 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: starting mon.c rank -1 at public addrs [v2:172.21.15.163:3301/0,v1:172.21.15.163:6790/0] at bind addrs [v2:172.21.15.163:3301/0,v1:172.21.15.163:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:54:45.504 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(???) e0 preinit fsid d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:54:45.505 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(synchronizing).mds e1 new map 2024-03-21T12:54:45.505 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-21T12:54:45.505 INFO:journalctl@ceph.mon.c.smithi163.stdout: e1 2024-03-21T12:54:45.505 INFO:journalctl@ceph.mon.c.smithi163.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T12:54:45.505 INFO:journalctl@ceph.mon.c.smithi163.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-21T12:54:45.505 INFO:journalctl@ceph.mon.c.smithi163.stdout: legacy client fscid: -1 2024-03-21T12:54:45.505 INFO:journalctl@ceph.mon.c.smithi163.stdout: 2024-03-21T12:54:45.505 INFO:journalctl@ceph.mon.c.smithi163.stdout: No filesystems configured 2024-03-21T12:54:45.505 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-21T12:54:45.505 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-21T12:54:45.505 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-21T12:54:45.505 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-21T12:54:45.505 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-21T12:54:45.506 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-21T12:54:45.506 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-21T12:54:45.506 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:54:45.506 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:54:45.506 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:54:45.506 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:45.506 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3058284462' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-21T12:54:45.506 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3058284462' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-21T12:54:45.506 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:54:45.506 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:45.506 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:45.506 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:45.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:45.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:54:45.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:45.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:45.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:45.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:45.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:45.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-21T12:54:45.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:45.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:45.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:54:45.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:45.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi163:172.21.15.163=a;smithi163:[v2:172.21.15.163:3301,v1:172.21.15.163:6790]=c;smithi204:172.21.15.204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:45.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: Saving service mon spec with placement smithi163:172.21.15.163=a;smithi163:[v2:172.21.15.163:3301,v1:172.21.15.163:6790]=c;smithi204:172.21.15.204=b;count:3 2024-03-21T12:54:45.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:45.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:45.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:45.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:45.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:45.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:45.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:45.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: Deploying daemon mon.b on smithi204 2024-03-21T12:54:45.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:45.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:45.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:54:45.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.a calling monitor election 2024-03-21T12:54:45.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:45.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:45.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:45.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:45.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.b calling monitor election 2024-03-21T12:54:45.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:45.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:45.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:45.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:45.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-21T12:54:45.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: monmap e2: 2 mons at {a=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:54:45.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: fsmap 2024-03-21T12:54:45.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:54:45.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mgrmap e13: y(active, since 41s) 2024-03-21T12:54:45.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: overall HEALTH_OK 2024-03-21T12:54:45.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:45.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:45.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:45.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:45.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:45.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: Deploying daemon mon.c on smithi163 2024-03-21T12:54:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/2379241171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:54:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 ceph-mon[40675]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-21T12:54:45.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 bash[40656]: 4cd4267d4f7d15ff70a972c35873ce847efaaa263b241c1f1c0312f4b5a0297d 2024-03-21T12:54:45.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:45 smithi163 systemd[1]: Started Ceph mon.c for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T12:54:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:54:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: mon.a calling monitor election 2024-03-21T12:54:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: mon.b calling monitor election 2024-03-21T12:54:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: mon.c calling monitor election 2024-03-21T12:54:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:51.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-21T12:54:51.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: monmap e3: 3 mons at {a=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0],c=[v2:172.21.15.163:3301/0,v1:172.21.15.163:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:54:51.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: fsmap 2024-03-21T12:54:51.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:54:51.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: mgrmap e13: y(active, since 50s) 2024-03-21T12:54:51.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: overall HEALTH_OK 2024-03-21T12:54:51.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:51.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:51.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:51.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:50 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:51.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:54:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: mon.a calling monitor election 2024-03-21T12:54:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: mon.b calling monitor election 2024-03-21T12:54:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: mon.c calling monitor election 2024-03-21T12:54:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-21T12:54:51.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: monmap e3: 3 mons at {a=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0],c=[v2:172.21.15.163:3301/0,v1:172.21.15.163:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:54:51.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: fsmap 2024-03-21T12:54:51.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:54:51.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: mgrmap e13: y(active, since 50s) 2024-03-21T12:54:51.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: overall HEALTH_OK 2024-03-21T12:54:51.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:51.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:51.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:51.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:51.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:54:51.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: mon.a calling monitor election 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: mon.b calling monitor election 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: mon.c calling monitor election 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: monmap e3: 3 mons at {a=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0],c=[v2:172.21.15.163:3301/0,v1:172.21.15.163:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: fsmap 2024-03-21T12:54:51.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:54:51.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: mgrmap e13: y(active, since 50s) 2024-03-21T12:54:51.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: overall HEALTH_OK 2024-03-21T12:54:51.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:51.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:51.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:51.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:50 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:51.571 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-21T12:54:51.572 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":3,"fsid":"d4bf79c0-e781-11ee-95cd-87774f69a715","modified":"2024-03-21T12:54:45.484385Z","created":"2024-03-21T12:52:53.111495Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3301","nonce":0},{"type":"v1","addr":"172.21.15.163:6790","nonce":0}]},"addr":"172.21.15.163:6790/0","public_addr":"172.21.15.163:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-21T12:54:51.574 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 3 2024-03-21T12:54:51.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:51 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:51 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:51 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/295923450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:54:51.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:51 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:51.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:51 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:51 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/295923450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:54:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:51 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:51 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:51 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/295923450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:54:52.502 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-21T12:54:52.503 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-21T12:54:52.748 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:54:53.171 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.171 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.171 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.171 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.171 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:53.171 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:53.171 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: Updating smithi163:/etc/ceph/ceph.conf 2024-03-21T12:54:53.171 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-21T12:54:53.171 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: Updating smithi163:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:54:53.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:53.173 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.173 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.173 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.173 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.173 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:53.173 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:53.173 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: Updating smithi163:/etc/ceph/ceph.conf 2024-03-21T12:54:53.173 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-21T12:54:53.173 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:53.173 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: Updating smithi163:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:53.173 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:53.173 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.173 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:53.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:54:53.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: Updating smithi163:/etc/ceph/ceph.conf 2024-03-21T12:54:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-21T12:54:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: Updating smithi163:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:53.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:53.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:54:53.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:54.121 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:54 smithi163 ceph-mon[29621]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T12:54:54.122 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:54 smithi163 ceph-mon[29621]: Reconfiguring daemon mon.a on smithi163 2024-03-21T12:54:54.122 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:54 smithi163 ceph-mon[40675]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T12:54:54.122 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:54 smithi163 ceph-mon[40675]: Reconfiguring daemon mon.a on smithi163 2024-03-21T12:54:54.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:54 smithi204 ceph-mon[33153]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T12:54:54.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:54 smithi204 ceph-mon[33153]: Reconfiguring daemon mon.a on smithi163 2024-03-21T12:54:55.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:55 smithi163 ceph-mon[29621]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:55.141 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:55 smithi163 ceph-mon[40675]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:55.239 INFO:teuthology.orchestra.run.smithi163.stdout:# minimal ceph.conf for d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:54:55.240 INFO:teuthology.orchestra.run.smithi163.stdout:[global] 2024-03-21T12:54:55.240 INFO:teuthology.orchestra.run.smithi163.stdout: fsid = d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T12:54:55.240 INFO:teuthology.orchestra.run.smithi163.stdout: mon_host = [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] [v2:172.21.15.163:3301/0,v1:172.21.15.163:6790/0] 2024-03-21T12:54:55.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:55 smithi204 ceph-mon[33153]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:56.178 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/86559370' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:56.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:56.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:56.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:56.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:56.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:54:56.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:56.179 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/86559370' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:56.179 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:56.179 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:56.179 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:56.180 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:56.180 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:54:56.180 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:56.206 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-21T12:54:56.206 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:54:56.206 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.conf 2024-03-21T12:54:56.234 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:54:56.235 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:56.303 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-21T12:54:56.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.conf 2024-03-21T12:54:56.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:56 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/86559370' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:56.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:56.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:56.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:56.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:56.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:54:56.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:56.339 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-21T12:54:56.339 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:56.411 INFO:tasks.cephadm:Adding mgr.y on smithi163 2024-03-21T12:54:56.411 INFO:tasks.cephadm:Adding mgr.x on smithi204 2024-03-21T12:54:56.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch apply mgr '2;smithi163=y;smithi204=x' 2024-03-21T12:54:56.683 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:54:57.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:57 smithi163 ceph-mon[29621]: Reconfiguring mon.c (monmap changed)... 2024-03-21T12:54:57.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:57 smithi163 ceph-mon[29621]: Reconfiguring daemon mon.c on smithi163 2024-03-21T12:54:57.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:57 smithi163 ceph-mon[29621]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:57.214 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:57 smithi163 ceph-mon[40675]: Reconfiguring mon.c (monmap changed)... 2024-03-21T12:54:57.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:57 smithi163 ceph-mon[40675]: Reconfiguring daemon mon.c on smithi163 2024-03-21T12:54:57.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:57 smithi163 ceph-mon[40675]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:57.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:57 smithi204 ceph-mon[33153]: Reconfiguring mon.c (monmap changed)... 2024-03-21T12:54:57.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:57 smithi204 ceph-mon[33153]: Reconfiguring daemon mon.c on smithi163 2024-03-21T12:54:57.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:57 smithi204 ceph-mon[33153]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:58.650 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:58.651 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:58.651 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:58 smithi204 ceph-mon[33153]: Reconfiguring mon.b (monmap changed)... 2024-03-21T12:54:58.651 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:58.651 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:54:58.651 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:58.651 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:58 smithi204 ceph-mon[33153]: Reconfiguring daemon mon.b on smithi204 2024-03-21T12:54:58.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[29621]: Reconfiguring mon.b (monmap changed)... 2024-03-21T12:54:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:54:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[29621]: Reconfiguring daemon mon.b on smithi204 2024-03-21T12:54:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[40675]: Reconfiguring mon.b (monmap changed)... 2024-03-21T12:54:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:54:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:54:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:58 smithi163 ceph-mon[40675]: Reconfiguring daemon mon.b on smithi204 2024-03-21T12:54:58.978 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mgr update... 2024-03-21T12:54:59.495 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:59 smithi204 ceph-mon[33153]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:59.495 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:59 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:59.495 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:59 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:59.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:59 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:59.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:59 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:59.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:59 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:59.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:59 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:59.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:59 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:59.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:59 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:54:59.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:59 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T12:54:59.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:59 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:54:59.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:54:59 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[29621]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:54:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T12:54:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:54:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[40675]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:54:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:54:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:54:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T12:54:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:54:59.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:54:59 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:59.800 DEBUG:teuthology.orchestra.run.smithi204:mgr.x> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mgr.x.service 2024-03-21T12:54:59.802 INFO:tasks.cephadm:Deploying OSDs... 2024-03-21T12:54:59.802 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:54:59.802 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T12:54:59.821 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T12:54:59.822 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_1 2024-03-21T12:54:59.879 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T12:54:59.879 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:54:59.879 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-21T12:54:59.879 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:54:59.879 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:54:59.879 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-03-21 12:54:44.942839939 +0000 2024-03-21T12:54:59.880 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-03-21 12:49:30.479231686 +0000 2024-03-21T12:54:59.880 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-03-21 12:49:30.479231686 +0000 2024-03-21T12:54:59.880 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-03-21 12:49:30.479231686 +0000 2024-03-21T12:54:59.880 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T12:54:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-03-21T12:54:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-03-21T12:54:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000279073 s, 1.8 MB/s 2024-03-21T12:54:59.950 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T12:55:00.008 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_2 2024-03-21T12:55:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T12:55:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:55:00.065 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-21T12:55:00.065 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:55:00.065 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:55:00.065 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-03-21 12:54:44.942839939 +0000 2024-03-21T12:55:00.065 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-03-21 12:49:30.804224241 +0000 2024-03-21T12:55:00.065 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-03-21 12:49:30.804224241 +0000 2024-03-21T12:55:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-03-21 12:49:30.804224241 +0000 2024-03-21T12:55:00.066 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T12:55:00.133 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-03-21T12:55:00.133 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-03-21T12:55:00.133 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000279195 s, 1.8 MB/s 2024-03-21T12:55:00.134 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T12:55:00.191 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_3 2024-03-21T12:55:00.247 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T12:55:00.247 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:55:00.247 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-21T12:55:00.247 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:55:00.248 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:55:00.248 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-03-21 12:54:44.943839916 +0000 2024-03-21T12:55:00.248 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-03-21 12:49:31.154216224 +0000 2024-03-21T12:55:00.248 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-03-21 12:49:31.154216224 +0000 2024-03-21T12:55:00.248 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-03-21 12:49:31.154216224 +0000 2024-03-21T12:55:00.248 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T12:55:00.314 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-03-21T12:55:00.314 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-03-21T12:55:00.314 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000227838 s, 2.2 MB/s 2024-03-21T12:55:00.315 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T12:55:00.372 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_4 2024-03-21T12:55:00.428 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T12:55:00.428 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:55:00.428 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-21T12:55:00.428 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:55:00.428 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:55:00.428 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-03-21 12:54:44.943839916 +0000 2024-03-21T12:55:00.428 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-03-21 12:49:31.500208299 +0000 2024-03-21T12:55:00.428 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-03-21 12:49:31.500208299 +0000 2024-03-21T12:55:00.428 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-03-21 12:49:31.500208299 +0000 2024-03-21T12:55:00.429 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T12:55:00.495 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-03-21T12:55:00.495 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-03-21T12:55:00.495 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000185954 s, 2.8 MB/s 2024-03-21T12:55:00.496 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T12:55:00.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:00 smithi204 ceph-mon[33153]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi163=y;smithi204=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:00.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:00 smithi204 ceph-mon[33153]: Saving service mgr spec with placement smithi163=y;smithi204=x;count:2 2024-03-21T12:55:00.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:00 smithi204 ceph-mon[33153]: Deploying daemon mgr.x on smithi204 2024-03-21T12:55:00.552 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-21T12:55:00.552 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T12:55:00.570 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T12:55:00.570 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-03-21T12:55:00.627 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T12:55:00.627 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:55:00.627 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-21T12:55:00.627 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:55:00.627 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:55:00.627 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-21 12:54:36.116739296 +0000 2024-03-21T12:55:00.627 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-21 12:49:28.976661343 +0000 2024-03-21T12:55:00.627 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-21 12:49:28.976661343 +0000 2024-03-21T12:55:00.627 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-21 12:49:28.976661343 +0000 2024-03-21T12:55:00.628 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T12:55:00.696 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-21T12:55:00.696 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-21T12:55:00.697 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000203152 s, 2.5 MB/s 2024-03-21T12:55:00.698 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T12:55:00.756 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-03-21T12:55:00.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:00 smithi163 ceph-mon[29621]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi163=y;smithi204=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:00.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:00 smithi163 ceph-mon[29621]: Saving service mgr spec with placement smithi163=y;smithi204=x;count:2 2024-03-21T12:55:00.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:00 smithi163 ceph-mon[29621]: Deploying daemon mgr.x on smithi204 2024-03-21T12:55:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:00 smithi163 ceph-mon[40675]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi163=y;smithi204=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:00 smithi163 ceph-mon[40675]: Saving service mgr spec with placement smithi163=y;smithi204=x;count:2 2024-03-21T12:55:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:00 smithi163 ceph-mon[40675]: Deploying daemon mgr.x on smithi204 2024-03-21T12:55:00.812 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T12:55:00.812 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:55:00.812 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-21T12:55:00.812 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:55:00.812 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:55:00.812 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-21 12:54:36.116739296 +0000 2024-03-21T12:55:00.813 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-21 12:49:29.305656718 +0000 2024-03-21T12:55:00.813 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-21 12:49:29.305656718 +0000 2024-03-21T12:55:00.813 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-21 12:49:29.305656718 +0000 2024-03-21T12:55:00.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T12:55:01.162 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-21T12:55:01.162 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-21T12:55:01.162 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000194397 s, 2.6 MB/s 2024-03-21T12:55:01.163 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T12:55:01.178 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-03-21T12:55:01.238 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T12:55:01.238 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:55:01.238 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-21T12:55:01.238 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:55:01.238 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:55:01.238 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-21 12:55:01.116392444 +0000 2024-03-21T12:55:01.238 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-21 12:49:29.640652008 +0000 2024-03-21T12:55:01.238 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-21 12:49:29.640652008 +0000 2024-03-21T12:55:01.239 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-21 12:49:29.640652008 +0000 2024-03-21T12:55:01.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T12:55:01.342 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-21T12:55:01.342 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-21T12:55:01.343 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000189922 s, 2.7 MB/s 2024-03-21T12:55:01.344 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T12:55:01.358 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-03-21T12:55:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T12:55:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:55:01.416 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-21T12:55:01.416 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:55:01.416 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:55:01.416 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-21 12:55:01.300389891 +0000 2024-03-21T12:55:01.416 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-21 12:49:29.978647256 +0000 2024-03-21T12:55:01.416 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-21 12:49:29.978647256 +0000 2024-03-21T12:55:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-21 12:49:29.978647256 +0000 2024-03-21T12:55:01.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T12:55:01.481 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:01 smithi204 ceph-mon[33153]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:01.481 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:01 smithi204 systemd[1]: Starting Ceph mgr.x for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T12:55:01.484 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-21T12:55:01.484 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-21T12:55:01.484 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000189252 s, 2.7 MB/s 2024-03-21T12:55:01.485 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T12:55:01.542 INFO:tasks.cephadm:Deploying osd.0 on smithi163 with /dev/vg_nvme/lv_4... 2024-03-21T12:55:01.542 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-21T12:55:01.570 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:01 smithi163 ceph-mon[29621]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:01.571 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:01 smithi163 ceph-mon[40675]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:01.752 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:01 smithi204 podman[34962]: 2024-03-21 12:55:01.486303152 +0000 UTC m=+0.018126520 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T12:55:01.752 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:01 smithi204 podman[34962]: 2024-03-21 12:55:01.646527148 +0000 UTC m=+0.178350491 container create 0d0e832d97240c0ba84b7863d5e665e9164584e8fae32e49b81b2eaccb93c346 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-03-21T12:55:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:55:02.252 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:01 smithi204 podman[34962]: 2024-03-21 12:55:01.848834767 +0000 UTC m=+0.380658122 container init 0d0e832d97240c0ba84b7863d5e665e9164584e8fae32e49b81b2eaccb93c346 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-21T12:55:02.253 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:01 smithi204 podman[34962]: 2024-03-21 12:55:01.851333146 +0000 UTC m=+0.383156497 container start 0d0e832d97240c0ba84b7863d5e665e9164584e8fae32e49b81b2eaccb93c346 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T12:55:02.253 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:01 smithi204 ceph-mgr[35013]: -- 172.21.15.204:0/949736241 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a3c68345a0 con 0x55a3c6808c00 2024-03-21T12:55:02.253 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:01 smithi204 bash[34962]: 0d0e832d97240c0ba84b7863d5e665e9164584e8fae32e49b81b2eaccb93c346 2024-03-21T12:55:02.253 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:01 smithi204 systemd[1]: Started Ceph mgr.x for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T12:55:02.520 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:02 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:02.421+0000 7f7ef11a6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:55:02.862 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:02 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:02.520+0000 7f7ef11a6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:55:03.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:03.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:03.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:03.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:03.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:03.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:03 smithi163 ceph-mon[29621]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:03.151 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:03 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:03.151 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:03 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:03.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:03 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:03.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:03 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:03.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:03 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:03.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:03 smithi163 ceph-mon[40675]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:03.205 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:03 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:03.206 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:03 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:03.206 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:03 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:03.206 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:03 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:03.206 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:03 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:03.206 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:03 smithi204 ceph-mon[33153]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:03.456 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:03 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:03.204+0000 7f7ef11a6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:55:03.456 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:03 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:03.300+0000 7f7ef11a6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:55:03.724 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:03 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:03.455+0000 7f7ef11a6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:55:03.724 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:03 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:03.556+0000 7f7ef11a6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:55:03.725 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:03 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:03.643+0000 7f7ef11a6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:55:03.767 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-21T12:55:03.768 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-21T12:55:03.768 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-03-21T12:55:03.768 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-03-21T12:55:03.768 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130593 s, 803 MB/s 2024-03-21T12:55:03.768 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-03-21T12:55:04.002 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:03 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:03.920+0000 7f7ef11a6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:55:04.380 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_4 2024-03-21T12:55:04.405 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:04.034+0000 7f7ef11a6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:55:04.406 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:04.122+0000 7f7ef11a6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:55:04.406 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:04.218+0000 7f7ef11a6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:55:04.406 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:04.400+0000 7f7ef11a6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:55:04.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:04.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:04.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:04.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:04.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:04.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[29621]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T12:55:04.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:55:04.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:55:04.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:04.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[29621]: Reconfiguring daemon mgr.y on smithi163 2024-03-21T12:55:04.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[29621]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:04.622 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:04.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:04.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:04.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:04.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:04.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[40675]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T12:55:04.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:55:04.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:55:04.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:04.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[40675]: Reconfiguring daemon mgr.y on smithi163 2024-03-21T12:55:04.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:04 smithi163 ceph-mon[40675]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:04.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-mon[33153]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T12:55:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:55:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:55:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-mon[33153]: Reconfiguring daemon mgr.y on smithi163 2024-03-21T12:55:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:04 smithi204 ceph-mon[33153]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:05.429 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:55:05.752 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:05 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:05.375+0000 7f7ef11a6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:55:05.752 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:05 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:05.507+0000 7f7ef11a6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:55:06.131 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:05 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:05.844+0000 7f7ef11a6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:55:06.131 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:05 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:05.934+0000 7f7ef11a6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:55:06.132 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:06 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:06.131+0000 7f7ef11a6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:55:06.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:06.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:06.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:06.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:06.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:06.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:06.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:06.469 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:06.469 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:06.470 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:06.470 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:06.470 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:06.470 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:06.470 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:06 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:06.502 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:06 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:06.319+0000 7f7ef11a6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:55:06.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:06 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:06.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:06 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:06.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:06 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:06.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:06 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:06.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:06 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:06.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:06 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:06.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:06 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:06.868 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:06 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:06.604+0000 7f7ef11a6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:55:06.868 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:06 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:06.867+0000 7f7ef11a6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:55:07.252 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:07.104+0000 7f7ef11a6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:55:07.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:07 smithi204 ceph-mon[33153]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:07.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:07 smithi163 ceph-mon[29621]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:07.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:07 smithi163 ceph-mon[40675]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:08.252 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:07.907+0000 7f7ef11a6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:55:08.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:08 smithi163 ceph-mon[29621]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:08.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:08 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:55:08.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:08 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:55:08.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:08 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:08.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:08 smithi163 ceph-mon[40675]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:08.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:08 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:55:08.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:08 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:55:08.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:08 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:08.577 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:08.480+0000 7f7ef11a6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:55:08.577 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:08.576+0000 7f7ef11a6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:55:08.577 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:08 smithi204 ceph-mon[33153]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:08.578 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:08 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:55:08.578 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:08 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:55:08.578 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:08 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:09.002 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:55:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:55:08.794+0000 7f7ef11a6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:55:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:09 smithi163 ceph-mon[40675]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:09 smithi163 ceph-mon[40675]: Standby manager daemon x started 2024-03-21T12:55:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:09 smithi163 ceph-mon[40675]: from='mgr.? 172.21.15.204:0/245433252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:55:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:09 smithi163 ceph-mon[40675]: from='mgr.? 172.21.15.204:0/245433252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:55:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:09 smithi163 ceph-mon[40675]: from='mgr.? 172.21.15.204:0/245433252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:55:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:09 smithi163 ceph-mon[40675]: from='mgr.? 172.21.15.204:0/245433252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:55:09.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:09 smithi163 ceph-mon[29621]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:09.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:09 smithi163 ceph-mon[29621]: Standby manager daemon x started 2024-03-21T12:55:09.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:09 smithi163 ceph-mon[29621]: from='mgr.? 172.21.15.204:0/245433252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:55:09.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:09 smithi163 ceph-mon[29621]: from='mgr.? 172.21.15.204:0/245433252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:55:09.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:09 smithi163 ceph-mon[29621]: from='mgr.? 172.21.15.204:0/245433252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:55:09.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:09 smithi163 ceph-mon[29621]: from='mgr.? 172.21.15.204:0/245433252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:55:09.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:09 smithi204 ceph-mon[33153]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:09.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:09 smithi204 ceph-mon[33153]: Standby manager daemon x started 2024-03-21T12:55:09.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:09 smithi204 ceph-mon[33153]: from='mgr.? 172.21.15.204:0/245433252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:55:09.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:09 smithi204 ceph-mon[33153]: from='mgr.? 172.21.15.204:0/245433252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:55:09.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:09 smithi204 ceph-mon[33153]: from='mgr.? 172.21.15.204:0/245433252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:55:09.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:09 smithi204 ceph-mon[33153]: from='mgr.? 172.21.15.204:0/245433252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:55:10.654 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:10 smithi204 ceph-mon[33153]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-21T12:55:10.654 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:10 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:55:10.654 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:10 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/846020371' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f"}]: dispatch 2024-03-21T12:55:10.654 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:10 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f"}]: dispatch 2024-03-21T12:55:10.654 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:10 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f"}]': finished 2024-03-21T12:55:10.654 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:10 smithi204 ceph-mon[33153]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T12:55:10.654 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:10 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[29621]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-21T12:55:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:55:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/846020371' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f"}]: dispatch 2024-03-21T12:55:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f"}]: dispatch 2024-03-21T12:55:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f"}]': finished 2024-03-21T12:55:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[29621]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T12:55:10.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[40675]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-21T12:55:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:55:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/846020371' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f"}]: dispatch 2024-03-21T12:55:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f"}]: dispatch 2024-03-21T12:55:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f"}]': finished 2024-03-21T12:55:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[40675]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T12:55:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:10 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:11.665 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:11 smithi204 ceph-mon[33153]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:11.665 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:11 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3797866262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:55:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:11 smithi163 ceph-mon[29621]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:11 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3797866262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:55:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:11 smithi163 ceph-mon[40675]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:11 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3797866262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:55:13.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:13 smithi163 ceph-mon[29621]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:13.453 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:13 smithi163 ceph-mon[40675]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:13.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:13 smithi204 ceph-mon[33153]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:15.360 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:15 smithi163 ceph-mon[29621]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:15.360 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:15 smithi163 ceph-mon[40675]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:15.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:15 smithi204 ceph-mon[33153]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:17.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:17 smithi163 ceph-mon[29621]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:17.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:17 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T12:55:17.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:17 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:17.418 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:17 smithi163 ceph-mon[40675]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:17.419 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:17 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T12:55:17.419 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:17 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:17.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:17 smithi204 ceph-mon[33153]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:17.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:17 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T12:55:17.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:17 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:18.363 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:18 smithi163 ceph-mon[29621]: Deploying daemon osd.0 on smithi163 2024-03-21T12:55:18.363 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:18 smithi163 ceph-mon[40675]: Deploying daemon osd.0 on smithi163 2024-03-21T12:55:18.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:18 smithi204 ceph-mon[33153]: Deploying daemon osd.0 on smithi163 2024-03-21T12:55:19.388 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:19 smithi163 ceph-mon[29621]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:19.389 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:19 smithi163 ceph-mon[40675]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:19.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:19 smithi204 ceph-mon[33153]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:21.463 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:21 smithi163 ceph-mon[29621]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:21.464 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:21 smithi163 ceph-mon[40675]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:21.706 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:21 smithi204 ceph-mon[33153]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:23.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:23.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:23.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:23.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:22 smithi163 ceph-mon[29621]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:23.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:23.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:23.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:23.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:22 smithi163 ceph-mon[40675]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:23.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:22 smithi204 ceph-mon[33153]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:25.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[40675]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:25.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:25.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:25.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:25.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:25.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:25.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[40675]: from='osd.0 v2:172.21.15.163:6801/3086410981' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:55:25.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[29621]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:25.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:25.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:25.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:25.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:25.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:25.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:25 smithi163 ceph-mon[29621]: from='osd.0 v2:172.21.15.163:6801/3086410981' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:55:25.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:25 smithi204 ceph-mon[33153]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:25 smithi204 ceph-mon[33153]: from='osd.0 v2:172.21.15.163:6801/3086410981' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:55:25.896 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 0 on host 'smithi163' 2024-03-21T12:55:26.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:26.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:26.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:26.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[29621]: from='osd.0 v2:172.21.15.163:6801/3086410981' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T12:55:26.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[29621]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T12:55:26.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[29621]: from='osd.0 v2:172.21.15.163:6801/3086410981' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:55:26.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:26.478 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:26.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:26.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:26.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[40675]: from='osd.0 v2:172.21.15.163:6801/3086410981' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T12:55:26.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[40675]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T12:55:26.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[40675]: from='osd.0 v2:172.21.15.163:6801/3086410981' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:55:26.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:26.508 DEBUG:teuthology.orchestra.run.smithi163:osd.0> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.0.service 2024-03-21T12:55:26.511 INFO:tasks.cephadm:Deploying osd.1 on smithi163 with /dev/vg_nvme/lv_3... 2024-03-21T12:55:26.511 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-21T12:55:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:26 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:26 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:26 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:26 smithi204 ceph-mon[33153]: from='osd.0 v2:172.21.15.163:6801/3086410981' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T12:55:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:26 smithi204 ceph-mon[33153]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T12:55:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:26 smithi204 ceph-mon[33153]: from='osd.0 v2:172.21.15.163:6801/3086410981' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:55:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:26 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:27.261 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0[44656]: 2024-03-21T12:55:26.911+0000 7fb7d15f9700 -1 osd.0 0 waiting for initial osdmap 2024-03-21T12:55:27.261 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 12:55:26 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0[44656]: 2024-03-21T12:55:26.915+0000 7fb7cd3ec700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:55:27.634 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:27 smithi163 ceph-mon[29621]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:27.634 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:27 smithi163 ceph-mon[29621]: from='osd.0 v2:172.21.15.163:6801/3086410981' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-21T12:55:27.634 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:27 smithi163 ceph-mon[29621]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T12:55:27.634 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:27 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:27.634 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:27 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:27.634 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:27 smithi163 ceph-mon[40675]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:27.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:27 smithi163 ceph-mon[40675]: from='osd.0 v2:172.21.15.163:6801/3086410981' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-21T12:55:27.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:27 smithi163 ceph-mon[40675]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T12:55:27.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:27 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:27.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:27 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:27.706 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:55:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:27 smithi204 ceph-mon[33153]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:27 smithi204 ceph-mon[33153]: from='osd.0 v2:172.21.15.163:6801/3086410981' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-21T12:55:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:27 smithi204 ceph-mon[33153]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T12:55:27.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:27 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:27.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:27 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:28.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:28 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:55:28.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:28 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:55:28.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:28 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:28.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:28 smithi163 ceph-mon[29621]: osd.0 v2:172.21.15.163:6801/3086410981 boot 2024-03-21T12:55:28.616 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:28 smithi163 ceph-mon[29621]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T12:55:28.616 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:28 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:28.616 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:28 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:55:28.616 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:28 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:55:28.616 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:28 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:28.616 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:28 smithi163 ceph-mon[40675]: osd.0 v2:172.21.15.163:6801/3086410981 boot 2024-03-21T12:55:28.616 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:28 smithi163 ceph-mon[40675]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T12:55:28.616 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:28 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:28 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:55:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:28 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:55:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:28 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:28 smithi204 ceph-mon[33153]: osd.0 v2:172.21.15.163:6801/3086410981 boot 2024-03-21T12:55:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:28 smithi204 ceph-mon[33153]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T12:55:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:28 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:55:29.402 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:29 smithi163 ceph-mon[29621]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:29.403 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:29 smithi163 ceph-mon[29621]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T12:55:29.403 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:29 smithi163 ceph-mon[40675]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:29.403 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:29 smithi163 ceph-mon[40675]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T12:55:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:29 smithi204 ceph-mon[33153]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:55:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:29 smithi204 ceph-mon[33153]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T12:55:30.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[29621]: Detected new or changed devices on smithi163 2024-03-21T12:55:30.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:30.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:30.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:55:30.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[29621]: Adjusting osd_memory_target on smithi163 to 16064M 2024-03-21T12:55:30.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:30.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:30.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:30.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:30.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[29621]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:30.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[40675]: Detected new or changed devices on smithi163 2024-03-21T12:55:30.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:30.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:30.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:55:30.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[40675]: Adjusting osd_memory_target on smithi163 to 16064M 2024-03-21T12:55:30.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:30.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:30.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:30.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:30.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:30 smithi163 ceph-mon[40675]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:30.748 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-21T12:55:30.748 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-21T12:55:30.748 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-03-21T12:55:30.748 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-03-21T12:55:30.748 INFO:teuthology.orchestra.run.smithi163.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114003 s, 920 MB/s 2024-03-21T12:55:30.748 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-03-21T12:55:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:30 smithi204 ceph-mon[33153]: Detected new or changed devices on smithi163 2024-03-21T12:55:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:30 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:30 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:30 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:55:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:30 smithi204 ceph-mon[33153]: Adjusting osd_memory_target on smithi163 to 16064M 2024-03-21T12:55:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:30 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:30 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:30 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:30 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:30 smithi204 ceph-mon[33153]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:31.430 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_3 2024-03-21T12:55:31.666 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:55:33.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:33 smithi204 ceph-mon[33153]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:33.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:33 smithi163 ceph-mon[29621]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:33.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:33 smithi163 ceph-mon[40675]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:34 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:55:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:34 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:55:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:34 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:34.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:34 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:55:34.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:34 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:55:34.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:34 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:34 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:55:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:34 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:55:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:34 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:35.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:35 smithi163 ceph-mon[29621]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:35.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:35 smithi163 ceph-mon[29621]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:35.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:35 smithi163 ceph-mon[40675]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:35.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:35 smithi163 ceph-mon[40675]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:35.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:35 smithi204 ceph-mon[33153]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:35.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:35 smithi204 ceph-mon[33153]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:36.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:36 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3669078881' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36e05f89-c238-41f7-be65-1553b529e8ce"}]: dispatch 2024-03-21T12:55:36.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:36 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3669078881' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "36e05f89-c238-41f7-be65-1553b529e8ce"}]': finished 2024-03-21T12:55:36.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:36 smithi163 ceph-mon[29621]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T12:55:36.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:36 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:36.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:36 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3669078881' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36e05f89-c238-41f7-be65-1553b529e8ce"}]: dispatch 2024-03-21T12:55:36.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:36 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3669078881' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "36e05f89-c238-41f7-be65-1553b529e8ce"}]': finished 2024-03-21T12:55:36.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:36 smithi163 ceph-mon[40675]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T12:55:36.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:36 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:36 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3669078881' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36e05f89-c238-41f7-be65-1553b529e8ce"}]: dispatch 2024-03-21T12:55:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:36 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3669078881' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "36e05f89-c238-41f7-be65-1553b529e8ce"}]': finished 2024-03-21T12:55:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:36 smithi204 ceph-mon[33153]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T12:55:36.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:36 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:37.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:37 smithi204 ceph-mon[33153]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:37.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:37 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1398304853' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:55:37.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:37 smithi163 ceph-mon[29621]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:37.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:37 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1398304853' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:55:37.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:37 smithi163 ceph-mon[40675]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:37.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:37 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1398304853' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:55:39.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:39 smithi163 ceph-mon[29621]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:39.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:39 smithi163 ceph-mon[40675]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:39.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:39 smithi204 ceph-mon[33153]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:41.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:41 smithi163 ceph-mon[40675]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:41.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:41 smithi163 ceph-mon[29621]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:41.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:41 smithi204 ceph-mon[33153]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:43.457 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:43 smithi163 ceph-mon[29621]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:43.457 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:43 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T12:55:43.458 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:43 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:43.458 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:43 smithi163 ceph-mon[40675]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:43.458 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:43 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T12:55:43.458 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:43 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:43 smithi204 ceph-mon[33153]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:43 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T12:55:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:43 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:44.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:44 smithi163 ceph-mon[29621]: Deploying daemon osd.1 on smithi163 2024-03-21T12:55:44.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:44 smithi163 ceph-mon[40675]: Deploying daemon osd.1 on smithi163 2024-03-21T12:55:44.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:44 smithi204 ceph-mon[33153]: Deploying daemon osd.1 on smithi163 2024-03-21T12:55:45.333 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:45 smithi163 ceph-mon[29621]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:45.334 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:45 smithi163 ceph-mon[40675]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:45.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:45 smithi204 ceph-mon[33153]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:47.366 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:47 smithi163 ceph-mon[29621]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:47.367 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:47 smithi163 ceph-mon[40675]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:47.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:47 smithi204 ceph-mon[33153]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:49.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:48 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:49.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:48 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:49.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:48 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:49.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:48 smithi204 ceph-mon[33153]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:49.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:48 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:49.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:48 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:49.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:48 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:49.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:48 smithi163 ceph-mon[40675]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:49.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:48 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:49.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:48 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:49.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:48 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:49.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:48 smithi163 ceph-mon[29621]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:51.394 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:51 smithi163 ceph-mon[29621]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:51.395 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:51 smithi163 ceph-mon[29621]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T12:55:51.395 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:51 smithi163 ceph-mon[40675]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:51.395 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:51 smithi163 ceph-mon[40675]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T12:55:51.453 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 1 on host 'smithi163' 2024-03-21T12:55:51.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:51 smithi204 ceph-mon[33153]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:51.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:51 smithi204 ceph-mon[33153]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T12:55:52.050 DEBUG:teuthology.orchestra.run.smithi163:osd.1> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.1.service 2024-03-21T12:55:52.052 INFO:tasks.cephadm:Deploying osd.2 on smithi163 with /dev/vg_nvme/lv_2... 2024-03-21T12:55:52.053 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-21T12:55:52.362 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:55:52.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[29621]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T12:55:52.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[29621]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T12:55:52.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[29621]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:55:52.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:52.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:52.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:52.706 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:52.706 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:52.706 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[29621]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-21T12:55:52.706 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[29621]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T12:55:52.706 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:52.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[40675]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T12:55:52.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[40675]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T12:55:52.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[40675]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:55:52.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:52.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:52.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:52.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:52.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:52.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[40675]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-21T12:55:52.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[40675]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T12:55:52.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:52 smithi204 ceph-mon[33153]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T12:55:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:52 smithi204 ceph-mon[33153]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T12:55:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:52 smithi204 ceph-mon[33153]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:55:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:52 smithi204 ceph-mon[33153]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-21T12:55:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:52 smithi204 ceph-mon[33153]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T12:55:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:53.010 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1[50228]: 2024-03-21T12:55:52.721+0000 7f48910ce700 -1 osd.1 0 waiting for initial osdmap 2024-03-21T12:55:53.011 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 12:55:52 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1[50228]: 2024-03-21T12:55:52.725+0000 7f4888eb9700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:55:53.437 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:53 smithi163 ceph-mon[29621]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:53.437 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:53 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:53.437 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:53 smithi163 ceph-mon[29621]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' 2024-03-21T12:55:53.437 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:53 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:53.437 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:53 smithi163 ceph-mon[40675]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:53.437 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:53 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:53.438 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:53 smithi163 ceph-mon[40675]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' 2024-03-21T12:55:53.438 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:53 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:53 smithi204 ceph-mon[33153]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:53 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:53 smithi204 ceph-mon[33153]: from='osd.1 v2:172.21.15.163:6805/3666062784' entity='osd.1' 2024-03-21T12:55:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:53 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:54.299 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-21T12:55:54.299 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-21T12:55:54.299 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-03-21T12:55:54.300 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-03-21T12:55:54.300 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148752 s, 705 MB/s 2024-03-21T12:55:54.300 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-03-21T12:55:54.938 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:54 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:55:54.939 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:54 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:55:54.939 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:54 smithi163 ceph-mon[29621]: osd.1 v2:172.21.15.163:6805/3666062784 boot 2024-03-21T12:55:54.939 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:54 smithi163 ceph-mon[29621]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T12:55:54.939 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:54 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:54.939 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:54 smithi163 ceph-mon[29621]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:54.939 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:54 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:55:54.939 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:54 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:55:54.939 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:54 smithi163 ceph-mon[40675]: osd.1 v2:172.21.15.163:6805/3666062784 boot 2024-03-21T12:55:54.939 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:54 smithi163 ceph-mon[40675]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T12:55:54.939 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:54 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:54.940 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:54 smithi163 ceph-mon[40675]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:54 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:55:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:54 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:55:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:54 smithi204 ceph-mon[33153]: osd.1 v2:172.21.15.163:6805/3666062784 boot 2024-03-21T12:55:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:54 smithi204 ceph-mon[33153]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T12:55:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:54 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:55:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:54 smithi204 ceph-mon[33153]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:55:55.686 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_2 2024-03-21T12:55:55.961 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:55 smithi163 ceph-mon[29621]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T12:55:55.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:55 smithi163 ceph-mon[40675]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T12:55:56.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:55 smithi204 ceph-mon[33153]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T12:55:56.200 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:55:57.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:57 smithi163 ceph-mon[29621]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:55:57.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:57 smithi163 ceph-mon[40675]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:55:57.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:57 smithi204 ceph-mon[33153]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:55:59.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: Detected new or changed devices on smithi163 2024-03-21T12:55:59.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: Adjusting osd_memory_target on smithi163 to 8032M 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:55:59.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:55:59.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:55:59.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:55:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:59.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: Detected new or changed devices on smithi163 2024-03-21T12:55:59.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:55:59.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:55:59.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: Adjusting osd_memory_target on smithi163 to 8032M 2024-03-21T12:55:59.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:59.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:59.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:59.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:59.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:59.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:55:59.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:55:59.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:55:59.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:59.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: Detected new or changed devices on smithi163 2024-03-21T12:55:59.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:55:59.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:55:59.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: Adjusting osd_memory_target on smithi163 to 8032M 2024-03-21T12:55:59.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:59.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:59.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:59.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:59.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:59.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:55:59.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:55:59.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:55:59.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:55:59.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:55:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:00.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:00 smithi163 ceph-mon[29621]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:00.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:00 smithi163 ceph-mon[40675]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:00.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:00 smithi204 ceph-mon[33153]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:01.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:01 smithi204 ceph-mon[33153]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:01.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:01 smithi163 ceph-mon[29621]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:01.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:01 smithi163 ceph-mon[40675]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:02.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:02 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/787231999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f"}]: dispatch 2024-03-21T12:56:02.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:02 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f"}]: dispatch 2024-03-21T12:56:02.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:02 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f"}]': finished 2024-03-21T12:56:02.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:02 smithi204 ceph-mon[33153]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T12:56:02.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:02 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:02.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:02 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/239135461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:56:02.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:02 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/787231999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f"}]: dispatch 2024-03-21T12:56:02.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:02 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f"}]: dispatch 2024-03-21T12:56:02.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:02 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f"}]': finished 2024-03-21T12:56:02.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:02 smithi163 ceph-mon[40675]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T12:56:02.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:02 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:02.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:02 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/239135461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:56:02.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:02 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/787231999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f"}]: dispatch 2024-03-21T12:56:02.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:02 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f"}]: dispatch 2024-03-21T12:56:02.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:02 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f"}]': finished 2024-03-21T12:56:02.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:02 smithi163 ceph-mon[29621]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T12:56:02.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:02 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:02.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:02 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/239135461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:56:03.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:03 smithi204 ceph-mon[33153]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:03.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:03 smithi163 ceph-mon[40675]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:03.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:03 smithi163 ceph-mon[29621]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:05.551 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:05 smithi163 ceph-mon[29621]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:05.551 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:05 smithi163 ceph-mon[40675]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:05 smithi204 ceph-mon[33153]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:07.634 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:07 smithi163 ceph-mon[29621]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:07.634 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:07 smithi163 ceph-mon[40675]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:07.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:07 smithi204 ceph-mon[33153]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:08.640 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:08 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T12:56:08.640 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:08 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:08.640 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:08 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T12:56:08.640 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:08 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:08 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T12:56:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:08 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:09.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:09 smithi163 ceph-mon[29621]: Deploying daemon osd.2 on smithi163 2024-03-21T12:56:09.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:09 smithi163 ceph-mon[29621]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:09.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:09 smithi163 ceph-mon[40675]: Deploying daemon osd.2 on smithi163 2024-03-21T12:56:09.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:09 smithi163 ceph-mon[40675]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:09.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:09 smithi204 ceph-mon[33153]: Deploying daemon osd.2 on smithi163 2024-03-21T12:56:09.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:09 smithi204 ceph-mon[33153]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:11.564 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:11 smithi163 ceph-mon[29621]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:11.565 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:11 smithi163 ceph-mon[40675]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:11.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:11 smithi204 ceph-mon[33153]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:13.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:13 smithi204 ceph-mon[33153]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:13.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:13 smithi163 ceph-mon[29621]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:13.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:13 smithi163 ceph-mon[40675]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:14.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:14 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:14.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:14 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:14 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:14 smithi163 ceph-mon[29621]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:14 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:14 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:14 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:14 smithi163 ceph-mon[40675]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:14 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:14 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:14 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:14 smithi204 ceph-mon[33153]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:17.204 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 2 on host 'smithi163' 2024-03-21T12:56:17.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:17 smithi163 ceph-mon[29621]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:17.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:17 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:17.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:17 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:17.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:17 smithi163 ceph-mon[29621]: from='osd.2 v2:172.21.15.163:6809/4009213507' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T12:56:17.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:17 smithi163 ceph-mon[29621]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T12:56:17.453 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:17 smithi163 ceph-mon[40675]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:17.454 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:17 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:17.454 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:17 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:17.454 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:17 smithi163 ceph-mon[40675]: from='osd.2 v2:172.21.15.163:6809/4009213507' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T12:56:17.454 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:17 smithi163 ceph-mon[40675]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T12:56:17.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:17 smithi204 ceph-mon[33153]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:17.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:17 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:17.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:17 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:17.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:17 smithi204 ceph-mon[33153]: from='osd.2 v2:172.21.15.163:6809/4009213507' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T12:56:17.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:17 smithi204 ceph-mon[33153]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T12:56:17.888 DEBUG:teuthology.orchestra.run.smithi163:osd.2> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.2.service 2024-03-21T12:56:17.890 INFO:tasks.cephadm:Deploying osd.3 on smithi163 with /dev/vg_nvme/lv_1... 2024-03-21T12:56:17.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-21T12:56:18.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:18 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:18.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:18 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:18.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:18 smithi204 ceph-mon[33153]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T12:56:18.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:18 smithi204 ceph-mon[33153]: from='osd.2 v2:172.21.15.163:6809/4009213507' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:56:18.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:18 smithi204 ceph-mon[33153]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T12:56:18.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:18 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:18.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:18 smithi204 ceph-mon[33153]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:56:18.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[40675]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T12:56:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[40675]: from='osd.2 v2:172.21.15.163:6809/4009213507' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:56:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[40675]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T12:56:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[40675]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:56:18.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:18.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:18.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[29621]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T12:56:18.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[29621]: from='osd.2 v2:172.21.15.163:6809/4009213507' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:56:18.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[29621]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T12:56:18.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:18.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-mon[29621]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:56:19.000 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2[56029]: 2024-03-21T12:56:18.890+0000 7fd061482700 -1 osd.2 0 waiting for initial osdmap 2024-03-21T12:56:19.000 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 12:56:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2[56029]: 2024-03-21T12:56:18.893+0000 7fd05926d700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:56:19.018 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:56:19.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:19 smithi163 ceph-mon[40675]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:19.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:19 smithi163 ceph-mon[40675]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-21T12:56:19.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:19 smithi163 ceph-mon[40675]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T12:56:19.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:19 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:19.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:19 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:19.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:19 smithi163 ceph-mon[40675]: from='osd.2 ' entity='osd.2' 2024-03-21T12:56:19.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:19 smithi163 ceph-mon[29621]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:19.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:19 smithi163 ceph-mon[29621]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-21T12:56:19.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:19 smithi163 ceph-mon[29621]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T12:56:19.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:19 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:19.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:19 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:19.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:19 smithi163 ceph-mon[29621]: from='osd.2 ' entity='osd.2' 2024-03-21T12:56:19.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:19 smithi204 ceph-mon[33153]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:19.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:19 smithi204 ceph-mon[33153]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-21T12:56:19.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:19 smithi204 ceph-mon[33153]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T12:56:19.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:19 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:19.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:19 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:19.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:19 smithi204 ceph-mon[33153]: from='osd.2 ' entity='osd.2' 2024-03-21T12:56:20.308 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:20 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:56:20.308 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:20 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:56:20.308 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:20 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:20.308 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:20 smithi163 ceph-mon[29621]: osd.2 v2:172.21.15.163:6809/4009213507 boot 2024-03-21T12:56:20.309 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:20 smithi163 ceph-mon[29621]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T12:56:20.309 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:20 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:20.309 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:20 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:56:20.309 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:20 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:56:20.309 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:20 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:20.309 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:20 smithi163 ceph-mon[40675]: osd.2 v2:172.21.15.163:6809/4009213507 boot 2024-03-21T12:56:20.309 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:20 smithi163 ceph-mon[40675]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T12:56:20.309 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:20 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:20.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:20 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:56:20.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:20 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:56:20.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:20 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:20.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:20 smithi204 ceph-mon[33153]: osd.2 v2:172.21.15.163:6809/4009213507 boot 2024-03-21T12:56:20.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:20 smithi204 ceph-mon[33153]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T12:56:20.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:20 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:56:21.337 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-21T12:56:21.337 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-21T12:56:21.337 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-03-21T12:56:21.338 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-03-21T12:56:21.338 INFO:teuthology.orchestra.run.smithi163.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0129638 s, 809 MB/s 2024-03-21T12:56:21.338 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-03-21T12:56:21.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:21 smithi163 ceph-mon[29621]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:21.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:21 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-21T12:56:21.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:21 smithi163 ceph-mon[40675]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:21.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:21 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-21T12:56:21.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:21 smithi204 ceph-mon[33153]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:56:21.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:21 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-21T12:56:22.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-21T12:56:22.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T12:56:22.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T12:56:22.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:22.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:22.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:22.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.331 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:22.331 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:22.331 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.331 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:22.331 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:22.331 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:22.331 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.331 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-21T12:56:22.331 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T12:56:22.331 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T12:56:22.331 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.331 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.331 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:22.332 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:22.332 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:22.332 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.332 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:22.332 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:22.332 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.332 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:22.332 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:22.332 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:22.332 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.358 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_1 2024-03-21T12:56:22.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-21T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:22.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:22.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:22.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:22.606 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:56:22.731 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 12:56:22 smithi163 sudo[59976]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T12:56:22.731 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 12:56:22 smithi163 sudo[59976]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T12:56:22.731 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 12:56:22 smithi163 sudo[59976]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T12:56:22.732 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 12:56:22 smithi163 sudo[59976]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:56:23.011 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 12:56:22 smithi163 sudo[60022]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T12:56:23.011 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 12:56:22 smithi163 sudo[60022]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T12:56:23.011 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 12:56:22 smithi163 sudo[60022]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T12:56:23.011 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 12:56:22 smithi163 sudo[60022]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:56:23.319 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60043]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T12:56:23.319 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60043]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T12:56:23.320 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60043]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T12:56:23.320 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60043]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:56:23.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:23 smithi163 ceph-mon[29621]: Detected new or changed devices on smithi163 2024-03-21T12:56:23.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:23 smithi163 ceph-mon[29621]: Adjusting osd_memory_target on smithi163 to 5354M 2024-03-21T12:56:23.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:23 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T12:56:23.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:23 smithi163 ceph-mon[29621]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T12:56:23.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:23 smithi163 ceph-mon[29621]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:23.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:23 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:23.321 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:23 smithi163 ceph-mon[40675]: Detected new or changed devices on smithi163 2024-03-21T12:56:23.321 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:23 smithi163 ceph-mon[40675]: Adjusting osd_memory_target on smithi163 to 5354M 2024-03-21T12:56:23.321 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:23 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T12:56:23.321 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:23 smithi163 ceph-mon[40675]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T12:56:23.321 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:23 smithi163 ceph-mon[40675]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:23.321 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:23 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:23.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:23 smithi204 ceph-mon[33153]: Detected new or changed devices on smithi163 2024-03-21T12:56:23.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:23 smithi204 ceph-mon[33153]: Adjusting osd_memory_target on smithi163 to 5354M 2024-03-21T12:56:23.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:23 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T12:56:23.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:23 smithi204 ceph-mon[33153]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T12:56:23.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:23 smithi204 ceph-mon[33153]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:23.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:23 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:23.594 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60048]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T12:56:23.594 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60048]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T12:56:23.594 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60048]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T12:56:23.594 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60048]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:56:23.860 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60051]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T12:56:23.860 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60051]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T12:56:23.860 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60051]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T12:56:23.860 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60051]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:56:24.110 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:24 smithi163 sudo[60081]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T12:56:24.110 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:24 smithi163 sudo[60081]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T12:56:24.110 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60074]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T12:56:24.110 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60074]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T12:56:24.110 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60074]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T12:56:24.110 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 12:56:23 smithi163 sudo[60074]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:56:24.373 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:24 smithi163 sudo[60081]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T12:56:24.373 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:24 smithi163 sudo[60081]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:56:24.636 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:24 smithi204 ceph-mon[33153]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T12:56:24.636 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:24 smithi204 ceph-mon[33153]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T12:56:24.636 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:24 smithi204 sudo[35712]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T12:56:24.636 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:24 smithi204 sudo[35712]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T12:56:24.637 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:24 smithi204 sudo[35712]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T12:56:24.637 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:24 smithi204 sudo[35712]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:56:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:24 smithi163 ceph-mon[40675]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T12:56:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:24 smithi163 ceph-mon[40675]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T12:56:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:24 smithi163 sudo[60108]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T12:56:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:24 smithi163 sudo[60108]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T12:56:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:24 smithi163 sudo[60108]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T12:56:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:24 smithi163 sudo[60108]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:56:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:24 smithi163 ceph-mon[29621]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T12:56:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:24 smithi163 ceph-mon[29621]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T12:56:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:24 smithi204 sudo[35715]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T12:56:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:24 smithi204 sudo[35715]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T12:56:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:24 smithi204 sudo[35715]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T12:56:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:24 smithi204 sudo[35715]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:56:25.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 sudo[60234]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T12:56:25.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 sudo[60234]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T12:56:25.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 sudo[60234]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T12:56:25.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 sudo[60234]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:56:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:56:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:56:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:56:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:56:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:56:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:56:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:56:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:56:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:56:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:56:25.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:56:25.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:25.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:56:25.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:25.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:56:25.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:56:25.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:56:25.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 sudo[60254]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T12:56:25.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 sudo[60254]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T12:56:25.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 sudo[60254]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T12:56:25.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 sudo[60254]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:56:25.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:56:25.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:25.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:56:25.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:56:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:56:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:56:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:56:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:56:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:56:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:56:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:56:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:56:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:56:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:56:25.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:56:25.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:56:25.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:56:25.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:25.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:56:25.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:56:25.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:56:25.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:56:25.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:56:25.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:56:25.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:56:25.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:56:25.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:25.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:56:25.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:56:25.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:25.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:56:25.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:25.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:56:25.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:56:25.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:56:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:26 smithi204 ceph-mon[33153]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:56:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:26 smithi204 ceph-mon[33153]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:56:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:26 smithi204 ceph-mon[33153]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:26 smithi163 ceph-mon[29621]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:56:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:26 smithi163 ceph-mon[29621]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:56:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:26 smithi163 ceph-mon[29621]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:26 smithi163 ceph-mon[40675]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:56:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:26 smithi163 ceph-mon[40675]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:56:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:26 smithi163 ceph-mon[40675]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:27.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:27 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/265051516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d"}]: dispatch 2024-03-21T12:56:27.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:27 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d"}]: dispatch 2024-03-21T12:56:27.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:27 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d"}]': finished 2024-03-21T12:56:27.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:27 smithi204 ceph-mon[33153]: osdmap e22: 4 total, 3 up, 4 in 2024-03-21T12:56:27.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:27 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:27.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:27 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/265051516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d"}]: dispatch 2024-03-21T12:56:27.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:27 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d"}]: dispatch 2024-03-21T12:56:27.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:27 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d"}]': finished 2024-03-21T12:56:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:27 smithi163 ceph-mon[29621]: osdmap e22: 4 total, 3 up, 4 in 2024-03-21T12:56:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:27 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:27 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/265051516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d"}]: dispatch 2024-03-21T12:56:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:27 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d"}]: dispatch 2024-03-21T12:56:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:27 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d"}]': finished 2024-03-21T12:56:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:27 smithi163 ceph-mon[40675]: osdmap e22: 4 total, 3 up, 4 in 2024-03-21T12:56:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:27 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:28 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2774065885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:56:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:28 smithi204 ceph-mon[33153]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:28.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:28 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2774065885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:56:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:28 smithi163 ceph-mon[29621]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:28 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2774065885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:56:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:28 smithi163 ceph-mon[40675]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:31.590 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:31 smithi163 ceph-mon[29621]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-21T12:56:31.591 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:31 smithi163 ceph-mon[40675]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-21T12:56:31.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:31 smithi204 ceph-mon[33153]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-21T12:56:33.400 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:33 smithi163 ceph-mon[29621]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-21T12:56:33.400 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:33 smithi163 ceph-mon[40675]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-21T12:56:33.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:33 smithi204 ceph-mon[33153]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-21T12:56:34.561 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:34 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T12:56:34.561 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:34 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:34.561 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:34 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T12:56:34.561 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:34 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:34 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T12:56:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:34 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:35.601 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:35 smithi163 ceph-mon[29621]: Deploying daemon osd.3 on smithi163 2024-03-21T12:56:35.601 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:35 smithi163 ceph-mon[29621]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-21T12:56:35.601 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:35 smithi163 ceph-mon[40675]: Deploying daemon osd.3 on smithi163 2024-03-21T12:56:35.602 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:35 smithi163 ceph-mon[40675]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-21T12:56:35.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:35 smithi204 ceph-mon[33153]: Deploying daemon osd.3 on smithi163 2024-03-21T12:56:35.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:35 smithi204 ceph-mon[33153]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-21T12:56:37.363 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:37 smithi163 ceph-mon[29621]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-21T12:56:37.364 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:37 smithi163 ceph-mon[40675]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-21T12:56:37.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:37 smithi204 ceph-mon[33153]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-21T12:56:39.721 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:39 smithi163 ceph-mon[29621]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-03-21T12:56:39.721 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:39 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:39.721 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:39 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:39.721 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:39 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:39.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:39 smithi163 ceph-mon[40675]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-03-21T12:56:39.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:39 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:39.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:39 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:39.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:39 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:39.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:39 smithi204 ceph-mon[33153]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-03-21T12:56:39.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:39 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:39.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:39 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:39.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:39 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:41.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:41 smithi163 ceph-mon[29621]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T12:56:41.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:41 smithi163 ceph-mon[40675]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T12:56:41.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:41 smithi204 ceph-mon[33153]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T12:56:42.815 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 3 on host 'smithi163' 2024-03-21T12:56:43.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:43.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:43.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[40675]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T12:56:43.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[40675]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:43.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:43.428 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.428 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.428 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:43.428 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:43.428 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.428 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[29621]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T12:56:43.428 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[29621]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:43.428 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.428 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:43 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:43.451 DEBUG:teuthology.orchestra.run.smithi163:osd.3> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.3.service 2024-03-21T12:56:43.453 INFO:tasks.cephadm:Deploying osd.4 on smithi204 with /dev/vg_nvme/lv_4... 2024-03-21T12:56:43.453 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-21T12:56:43.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:43 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:43 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:43 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:43.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:43 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:43.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:43 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:43 smithi204 ceph-mon[33153]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T12:56:43.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:43 smithi204 ceph-mon[33153]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:43.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:43 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:43 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:43.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:43 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:43.700 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:56:44.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:44 smithi163 ceph-mon[29621]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T12:56:44.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:44 smithi163 ceph-mon[29621]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T12:56:44.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:44 smithi163 ceph-mon[29621]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:56:44.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:44 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:44.350 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:44 smithi163 ceph-mon[40675]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T12:56:44.350 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:44 smithi163 ceph-mon[40675]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T12:56:44.350 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:44 smithi163 ceph-mon[40675]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:56:44.350 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:44 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:44.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:44 smithi204 ceph-mon[33153]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T12:56:44.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:44 smithi204 ceph-mon[33153]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T12:56:44.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:44 smithi204 ceph-mon[33153]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-21T12:56:44.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:44 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:44.658 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 12:56:44 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3[62023]: 2024-03-21T12:56:44.471+0000 7f8ba19eb700 -1 osd.3 0 waiting for initial osdmap 2024-03-21T12:56:44.658 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 12:56:44 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3[62023]: 2024-03-21T12:56:44.475+0000 7f8b98fd5700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:56:45.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[29621]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-21T12:56:45.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[29621]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T12:56:45.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:45.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:45.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[29621]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:45.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[29621]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' 2024-03-21T12:56:45.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:45.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[40675]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-21T12:56:45.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[40675]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T12:56:45.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:45.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:45.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[40675]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:45.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[40675]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' 2024-03-21T12:56:45.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:45 smithi204 ceph-mon[33153]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-21T12:56:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:45 smithi204 ceph-mon[33153]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T12:56:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:45 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:45.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:45 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:45.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:45 smithi204 ceph-mon[33153]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:56:45.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:45 smithi204 ceph-mon[33153]: from='osd.3 v2:172.21.15.163:6813/3981878189' entity='osd.3' 2024-03-21T12:56:45.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:45 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:45.645 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-21T12:56:45.645 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-21T12:56:45.645 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-03-21T12:56:45.645 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-03-21T12:56:45.646 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112223 s, 934 MB/s 2024-03-21T12:56:45.646 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-03-21T12:56:46.225 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:46 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:56:46.226 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:46 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:56:46.226 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:46 smithi204 ceph-mon[33153]: osd.3 v2:172.21.15.163:6813/3981878189 boot 2024-03-21T12:56:46.226 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:46 smithi204 ceph-mon[33153]: osdmap e25: 4 total, 4 up, 4 in 2024-03-21T12:56:46.226 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:46 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:46.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi204:vg_nvme/lv_4 2024-03-21T12:56:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:46 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:56:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:46 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:56:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:46 smithi163 ceph-mon[29621]: osd.3 v2:172.21.15.163:6813/3981878189 boot 2024-03-21T12:56:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:46 smithi163 ceph-mon[29621]: osdmap e25: 4 total, 4 up, 4 in 2024-03-21T12:56:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:46 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:46 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:56:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:46 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:56:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:46 smithi163 ceph-mon[40675]: osd.3 v2:172.21.15.163:6813/3981878189 boot 2024-03-21T12:56:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:46 smithi163 ceph-mon[40675]: osdmap e25: 4 total, 4 up, 4 in 2024-03-21T12:56:46.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:46 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:56:46.539 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:56:47.482 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:47 smithi204 ceph-mon[33153]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:47.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:47 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:47.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:47 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:47.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:47 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:47.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:47 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:47.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:47 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:47.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:47 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:47.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:47 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:47.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:47 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:47.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:47 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:47.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:47 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:47.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[29621]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:47.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:47.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:47.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:47.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:47.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:47.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:47.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:47.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:47.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:47.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:47.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[40675]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:47.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:47.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:47.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:47.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:47.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:47.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:56:47.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:47.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:47.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:47.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:47 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:56:48.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:48 smithi204 ceph-mon[33153]: Detected new or changed devices on smithi163 2024-03-21T12:56:48.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:48 smithi204 ceph-mon[33153]: Adjusting osd_memory_target on smithi163 to 4016M 2024-03-21T12:56:48.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:48 smithi204 ceph-mon[33153]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T12:56:48.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:48 smithi163 ceph-mon[40675]: Detected new or changed devices on smithi163 2024-03-21T12:56:48.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:48 smithi163 ceph-mon[40675]: Adjusting osd_memory_target on smithi163 to 4016M 2024-03-21T12:56:48.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:48 smithi163 ceph-mon[40675]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T12:56:48.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:48 smithi163 ceph-mon[29621]: Detected new or changed devices on smithi163 2024-03-21T12:56:48.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:48 smithi163 ceph-mon[29621]: Adjusting osd_memory_target on smithi163 to 4016M 2024-03-21T12:56:48.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:48 smithi163 ceph-mon[29621]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T12:56:49.458 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:49 smithi204 ceph-mon[33153]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:49.458 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:49 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:56:49.458 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:49 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:56:49.458 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:49 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:49.458 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:49 smithi204 ceph-mon[33153]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:49.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:49 smithi163 ceph-mon[40675]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:49.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:49 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:56:49.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:49 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:56:49.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:49 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:49.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:49 smithi163 ceph-mon[40675]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:49.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:49 smithi163 ceph-mon[29621]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:49.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:49 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:56:49.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:49 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:56:49.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:49 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:49.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:49 smithi163 ceph-mon[29621]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:51 smithi204 ceph-mon[33153]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:51 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:56:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:51 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:56:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:51 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/4247796846' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471"}]: dispatch 2024-03-21T12:56:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:51 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471"}]: dispatch 2024-03-21T12:56:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:51 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471"}]': finished 2024-03-21T12:56:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:51 smithi204 ceph-mon[33153]: osdmap e27: 5 total, 4 up, 5 in 2024-03-21T12:56:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:51 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:56:51.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[40675]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:51.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:56:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:56:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/4247796846' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471"}]: dispatch 2024-03-21T12:56:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471"}]: dispatch 2024-03-21T12:56:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471"}]': finished 2024-03-21T12:56:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[40675]: osdmap e27: 5 total, 4 up, 5 in 2024-03-21T12:56:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:56:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[29621]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:56:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:56:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/4247796846' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471"}]: dispatch 2024-03-21T12:56:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471"}]: dispatch 2024-03-21T12:56:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471"}]': finished 2024-03-21T12:56:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[29621]: osdmap e27: 5 total, 4 up, 5 in 2024-03-21T12:56:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:51 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:56:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:52 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/1982158916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:56:52.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:52 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/1982158916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:56:52.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:52 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/1982158916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:56:53.734 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:53 smithi204 ceph-mon[33153]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:53.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:53 smithi163 ceph-mon[40675]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:53 smithi163 ceph-mon[29621]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:55.640 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:55 smithi204 ceph-mon[33153]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:55.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:55 smithi163 ceph-mon[29621]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:55.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:55 smithi163 ceph-mon[40675]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:57.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:57 smithi204 ceph-mon[33153]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:57.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:57 smithi163 ceph-mon[29621]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:57.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:57 smithi163 ceph-mon[40675]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:58.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T12:56:58.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:58.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:58 smithi204 ceph-mon[33153]: Deploying daemon osd.4 on smithi204 2024-03-21T12:56:58.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T12:56:58.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:58.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:58 smithi163 ceph-mon[29621]: Deploying daemon osd.4 on smithi204 2024-03-21T12:56:58.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T12:56:58.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:58.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:58 smithi163 ceph-mon[40675]: Deploying daemon osd.4 on smithi204 2024-03-21T12:56:59.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:56:59 smithi204 ceph-mon[33153]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:59.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:56:59 smithi163 ceph-mon[29621]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:56:59.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:56:59 smithi163 ceph-mon[40675]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:01.440 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:01 smithi204 ceph-mon[33153]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:01 smithi163 ceph-mon[29621]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:01 smithi163 ceph-mon[40675]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:03.591 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:03 smithi204 ceph-mon[33153]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:03.591 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:03 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:03.591 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:03 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:03.591 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:03 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:03 smithi163 ceph-mon[29621]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:03 smithi163 ceph-mon[40675]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:03 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:03 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:03 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:05.471 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:05 smithi204 ceph-mon[33153]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:05.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:05 smithi163 ceph-mon[29621]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:05.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:05 smithi163 ceph-mon[40675]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:05.814 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 4 on host 'smithi204' 2024-03-21T12:57:06.433 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:06 smithi204 ceph-mon[33153]: from='osd.4 v2:172.21.15.204:6800/719276054' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T12:57:06.433 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:06 smithi204 ceph-mon[33153]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T12:57:06.433 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:06 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:06.433 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:06 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:06.433 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:06 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:06.433 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:06 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:06.469 DEBUG:teuthology.orchestra.run.smithi204:osd.4> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.4.service 2024-03-21T12:57:06.471 INFO:tasks.cephadm:Deploying osd.5 on smithi204 with /dev/vg_nvme/lv_3... 2024-03-21T12:57:06.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-21T12:57:06.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:06 smithi163 ceph-mon[40675]: from='osd.4 v2:172.21.15.204:6800/719276054' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T12:57:06.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:06 smithi163 ceph-mon[40675]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T12:57:06.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:06 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:06.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:06 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:06.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:06 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:06.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:06 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:06.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:06 smithi163 ceph-mon[29621]: from='osd.4 v2:172.21.15.204:6800/719276054' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T12:57:06.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:06 smithi163 ceph-mon[29621]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T12:57:06.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:06 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:06.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:06 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:06.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:06 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:06.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:06 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:07.294 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:57:07.705 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:07 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:57:07.706 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:07 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:57:07.706 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:07 smithi204 ceph-mon[33153]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:07.706 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:07 smithi204 ceph-mon[33153]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T12:57:07.706 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:07 smithi204 ceph-mon[33153]: from='osd.4 v2:172.21.15.204:6800/719276054' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:07.706 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:07 smithi204 ceph-mon[33153]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T12:57:07.706 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:07 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:57:07.706 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:07 smithi204 ceph-mon[33153]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:07.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:57:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:57:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[40675]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[40675]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T12:57:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[40675]: from='osd.4 v2:172.21.15.204:6800/719276054' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[40675]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T12:57:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:57:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[40675]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:57:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:57:07.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[29621]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:07.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[29621]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T12:57:07.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[29621]: from='osd.4 v2:172.21.15.204:6800/719276054' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:07.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[29621]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T12:57:07.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:57:07.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:07 smithi163 ceph-mon[29621]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:07.999 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 12:57:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T12:57:07.762+0000 7fcb7cdca700 -1 osd.4 0 waiting for initial osdmap 2024-03-21T12:57:07.999 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 12:57:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T12:57:07.766+0000 7fcb74bb5700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:57:08.687 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:08 smithi204 ceph-mon[33153]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-21T12:57:08.687 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:08 smithi204 ceph-mon[33153]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T12:57:08.687 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:08 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:57:08.687 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:08 smithi204 ceph-mon[33153]: from='osd.4 ' entity='osd.4' 2024-03-21T12:57:08.687 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:08 smithi204 ceph-mon[33153]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:08.688 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:08 smithi204 ceph-mon[33153]: osd.4 v2:172.21.15.204:6800/719276054 boot 2024-03-21T12:57:08.688 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:08 smithi204 ceph-mon[33153]: osdmap e30: 5 total, 5 up, 5 in 2024-03-21T12:57:08.688 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:08 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:57:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[29621]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-21T12:57:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[29621]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T12:57:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:57:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[29621]: from='osd.4 ' entity='osd.4' 2024-03-21T12:57:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[29621]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[29621]: osd.4 v2:172.21.15.204:6800/719276054 boot 2024-03-21T12:57:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[29621]: osdmap e30: 5 total, 5 up, 5 in 2024-03-21T12:57:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:57:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[40675]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-21T12:57:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[40675]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T12:57:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:57:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[40675]: from='osd.4 ' entity='osd.4' 2024-03-21T12:57:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[40675]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:57:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[40675]: osd.4 v2:172.21.15.204:6800/719276054 boot 2024-03-21T12:57:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[40675]: osdmap e30: 5 total, 5 up, 5 in 2024-03-21T12:57:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:08 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:57:09.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:09 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:57:09.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:09 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:57:09.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:09 smithi204 ceph-mon[33153]: osdmap e31: 5 total, 5 up, 5 in 2024-03-21T12:57:09.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:09 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:57:09.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:09 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:57:09.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:09 smithi163 ceph-mon[29621]: osdmap e31: 5 total, 5 up, 5 in 2024-03-21T12:57:09.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:09 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:57:09.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:09 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:57:09.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:09 smithi163 ceph-mon[40675]: osdmap e31: 5 total, 5 up, 5 in 2024-03-21T12:57:09.952 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-21T12:57:09.952 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-21T12:57:09.952 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-03-21T12:57:09.952 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-03-21T12:57:09.952 INFO:teuthology.orchestra.run.smithi204.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0119928 s, 874 MB/s 2024-03-21T12:57:09.953 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-03-21T12:57:10.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi204:vg_nvme/lv_3 2024-03-21T12:57:11.216 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:57:11.371 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:11.372 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T12:57:11.372 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T12:57:11.372 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.372 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.372 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:57:11.372 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.372 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:11.372 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:11.372 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.372 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:11.372 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:11.372 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:11.372 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T12:57:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T12:57:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:57:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:11.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:11.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T12:57:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T12:57:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:57:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:11.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:11.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:11.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:11.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:12.641 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:12 smithi204 ceph-mon[33153]: Detected new or changed devices on smithi204 2024-03-21T12:57:12.641 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:12 smithi204 ceph-mon[33153]: Adjusting osd_memory_target on smithi204 to 17088M 2024-03-21T12:57:12.641 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:12 smithi204 ceph-mon[33153]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T12:57:12.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:12 smithi163 ceph-mon[29621]: Detected new or changed devices on smithi204 2024-03-21T12:57:12.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:12 smithi163 ceph-mon[29621]: Adjusting osd_memory_target on smithi204 to 17088M 2024-03-21T12:57:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:12 smithi163 ceph-mon[29621]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T12:57:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:12 smithi163 ceph-mon[40675]: Detected new or changed devices on smithi204 2024-03-21T12:57:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:12 smithi163 ceph-mon[40675]: Adjusting osd_memory_target on smithi204 to 17088M 2024-03-21T12:57:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:12 smithi163 ceph-mon[40675]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T12:57:13.679 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:13 smithi204 ceph-mon[33153]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:13.679 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:13 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:57:13.680 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:13 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:57:13.680 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:13 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:13.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:13 smithi163 ceph-mon[29621]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:13.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:13 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:57:13.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:13 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:57:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:13 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:13.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:13 smithi163 ceph-mon[40675]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:13.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:13 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:57:13.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:13 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:57:13.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:13 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:14.628 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:14 smithi204 ceph-mon[33153]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:14 smithi163 ceph-mon[29621]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:14 smithi163 ceph-mon[40675]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:15.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:15 smithi204 ceph-mon[33153]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:15.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:15 smithi163 ceph-mon[29621]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:15.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:15 smithi163 ceph-mon[40675]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:16 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/13320035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1"}]: dispatch 2024-03-21T12:57:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:16 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1"}]: dispatch 2024-03-21T12:57:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:16 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1"}]': finished 2024-03-21T12:57:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:16 smithi204 ceph-mon[33153]: osdmap e35: 6 total, 5 up, 6 in 2024-03-21T12:57:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:16 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:16 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/2002099747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:57:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:16 smithi204 ceph-mon[33153]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-21T12:57:16.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/13320035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1"}]: dispatch 2024-03-21T12:57:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1"}]: dispatch 2024-03-21T12:57:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1"}]': finished 2024-03-21T12:57:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[29621]: osdmap e35: 6 total, 5 up, 6 in 2024-03-21T12:57:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/2002099747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:57:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[29621]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-21T12:57:16.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/13320035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1"}]: dispatch 2024-03-21T12:57:16.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1"}]: dispatch 2024-03-21T12:57:16.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1"}]': finished 2024-03-21T12:57:16.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[40675]: osdmap e35: 6 total, 5 up, 6 in 2024-03-21T12:57:16.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:16.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/2002099747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:57:16.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:16 smithi163 ceph-mon[40675]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-21T12:57:19.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:19 smithi204 ceph-mon[33153]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-21T12:57:19.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:19 smithi163 ceph-mon[29621]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-21T12:57:19.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:19 smithi163 ceph-mon[40675]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-21T12:57:21.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:21 smithi204 ceph-mon[33153]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-21T12:57:21.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:21 smithi163 ceph-mon[29621]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-21T12:57:21.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:21 smithi163 ceph-mon[40675]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-21T12:57:22.573 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T12:57:22.573 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:22.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T12:57:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T12:57:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:23.578 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:23 smithi204 ceph-mon[33153]: Deploying daemon osd.5 on smithi204 2024-03-21T12:57:23.579 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:23 smithi204 ceph-mon[33153]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T12:57:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:23 smithi163 ceph-mon[29621]: Deploying daemon osd.5 on smithi204 2024-03-21T12:57:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:23 smithi163 ceph-mon[29621]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T12:57:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:23 smithi163 ceph-mon[40675]: Deploying daemon osd.5 on smithi204 2024-03-21T12:57:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:23 smithi163 ceph-mon[40675]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T12:57:24.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:24 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:57:24.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:24 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:57:24.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:24 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:57:24.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:24 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:57:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:24 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:57:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:24 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:57:25.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:25 smithi204 ceph-mon[33153]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T12:57:25.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:25 smithi204 ceph-mon[33153]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T12:57:25.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:25.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:25 smithi163 ceph-mon[29621]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T12:57:25.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:25 smithi163 ceph-mon[29621]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T12:57:25.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:25.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:25 smithi163 ceph-mon[40675]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T12:57:25.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:25 smithi163 ceph-mon[40675]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T12:57:25.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:27.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:27 smithi204 ceph-mon[33153]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:27 smithi163 ceph-mon[29621]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:27 smithi163 ceph-mon[40675]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:28 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:28 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:28 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:28 smithi204 ceph-mon[33153]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:28.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:28 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:28.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:28 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:28.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:28 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:28.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:28 smithi163 ceph-mon[29621]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:28 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:28 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:28 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:28 smithi163 ceph-mon[40675]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:30.837 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 5 on host 'smithi204' 2024-03-21T12:57:31.463 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:31 smithi204 ceph-mon[33153]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:31.463 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:31 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.463 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:31 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.463 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:31 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:31.463 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:31 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:31.463 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:31 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.463 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:31 smithi204 ceph-mon[33153]: from='osd.5 v2:172.21.15.204:6804/1346947668' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T12:57:31.463 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:31 smithi204 ceph-mon[33153]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T12:57:31.463 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:31 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.463 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:31 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.463 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:31 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:31.488 DEBUG:teuthology.orchestra.run.smithi204:osd.5> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.5.service 2024-03-21T12:57:31.491 INFO:tasks.cephadm:Deploying osd.6 on smithi204 with /dev/vg_nvme/lv_2... 2024-03-21T12:57:31.491 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-21T12:57:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[29621]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[29621]: from='osd.5 v2:172.21.15.204:6804/1346947668' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T12:57:31.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[29621]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T12:57:31.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[40675]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[40675]: from='osd.5 v2:172.21.15.204:6804/1346947668' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T12:57:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[40675]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T12:57:31.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:31.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:31 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:32.198 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:57:32.602 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:32 smithi204 ceph-mon[33153]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T12:57:32.602 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:32 smithi204 ceph-mon[33153]: from='osd.5 v2:172.21.15.204:6804/1346947668' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:32.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:32 smithi204 ceph-mon[33153]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T12:57:32.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:32 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:32.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:32 smithi204 ceph-mon[33153]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:32.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:32 smithi204 ceph-mon[33153]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:32.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:32 smithi163 ceph-mon[29621]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T12:57:32.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:32 smithi163 ceph-mon[29621]: from='osd.5 v2:172.21.15.204:6804/1346947668' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:32.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:32 smithi163 ceph-mon[29621]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T12:57:32.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:32 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:32.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:32 smithi163 ceph-mon[29621]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:32.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:32 smithi163 ceph-mon[29621]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:32.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:32 smithi163 ceph-mon[40675]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T12:57:32.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:32 smithi163 ceph-mon[40675]: from='osd.5 v2:172.21.15.204:6804/1346947668' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:32.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:32 smithi163 ceph-mon[40675]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T12:57:32.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:32 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:32.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:32 smithi163 ceph-mon[40675]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:32.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:32 smithi163 ceph-mon[40675]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:57:33.002 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 12:57:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T12:57:32.781+0000 7f38ab9ce700 -1 osd.5 0 waiting for initial osdmap 2024-03-21T12:57:33.002 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 12:57:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T12:57:32.787+0000 7f38a37b9700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:57:33.676 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:33 smithi204 ceph-mon[33153]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-21T12:57:33.676 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:33 smithi204 ceph-mon[33153]: osdmap e38: 6 total, 5 up, 6 in 2024-03-21T12:57:33.676 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:33 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:33.676 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:33 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:33.676 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:33 smithi204 ceph-mon[33153]: from='osd.5 ' entity='osd.5' 2024-03-21T12:57:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:33 smithi163 ceph-mon[29621]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-21T12:57:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:33 smithi163 ceph-mon[29621]: osdmap e38: 6 total, 5 up, 6 in 2024-03-21T12:57:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:33 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:33 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:33 smithi163 ceph-mon[29621]: from='osd.5 ' entity='osd.5' 2024-03-21T12:57:33.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:33 smithi163 ceph-mon[40675]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-21T12:57:33.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:33 smithi163 ceph-mon[40675]: osdmap e38: 6 total, 5 up, 6 in 2024-03-21T12:57:33.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:33 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:33.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:33 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:33.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:33 smithi163 ceph-mon[40675]: from='osd.5 ' entity='osd.5' 2024-03-21T12:57:34.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:34 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:57:34.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:34 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:57:34.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:34 smithi204 ceph-mon[33153]: osd.5 v2:172.21.15.204:6804/1346947668 boot 2024-03-21T12:57:34.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:34 smithi204 ceph-mon[33153]: osdmap e39: 6 total, 6 up, 6 in 2024-03-21T12:57:34.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:34 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:34.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:34 smithi204 ceph-mon[33153]: pgmap v136: 1 pgs: 1 remapped; 577 KiB data, 462 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:34 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:57:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:34 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:57:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:34 smithi163 ceph-mon[29621]: osd.5 v2:172.21.15.204:6804/1346947668 boot 2024-03-21T12:57:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:34 smithi163 ceph-mon[29621]: osdmap e39: 6 total, 6 up, 6 in 2024-03-21T12:57:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:34 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:34 smithi163 ceph-mon[29621]: pgmap v136: 1 pgs: 1 remapped; 577 KiB data, 462 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:34 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:57:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:34 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:57:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:34 smithi163 ceph-mon[40675]: osd.5 v2:172.21.15.204:6804/1346947668 boot 2024-03-21T12:57:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:34 smithi163 ceph-mon[40675]: osdmap e39: 6 total, 6 up, 6 in 2024-03-21T12:57:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:34 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:34 smithi163 ceph-mon[40675]: pgmap v136: 1 pgs: 1 remapped; 577 KiB data, 462 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:34.763 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-21T12:57:34.764 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-21T12:57:34.764 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-03-21T12:57:34.764 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-03-21T12:57:34.764 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129369 s, 811 MB/s 2024-03-21T12:57:34.764 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-03-21T12:57:35.420 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:35 smithi204 ceph-mon[33153]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T12:57:35.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi204:vg_nvme/lv_2 2024-03-21T12:57:35.662 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:35 smithi163 ceph-mon[29621]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T12:57:35.662 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:35 smithi163 ceph-mon[40675]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T12:57:35.716 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:57:36.735 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:36 smithi204 ceph-mon[33153]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T12:57:36.735 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:36 smithi204 ceph-mon[33153]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T12:57:36.735 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:36 smithi204 ceph-mon[33153]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 462 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:36.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:36 smithi163 ceph-mon[29621]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T12:57:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:36 smithi163 ceph-mon[29621]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T12:57:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:36 smithi163 ceph-mon[29621]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 462 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:36 smithi163 ceph-mon[40675]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T12:57:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:36 smithi163 ceph-mon[40675]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T12:57:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:36 smithi163 ceph-mon[40675]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 462 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:38.407 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: osdmap e43: 6 total, 6 up, 6 in 2024-03-21T12:57:38.407 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: Detected new or changed devices on smithi204 2024-03-21T12:57:38.407 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:38.407 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:38.408 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:57:38.408 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:57:38.408 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: Adjusting osd_memory_target on smithi204 to 8544M 2024-03-21T12:57:38.408 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:38.408 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:38.408 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:38.408 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:38.408 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:57:38.408 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:57:38.408 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:38.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: osdmap e43: 6 total, 6 up, 6 in 2024-03-21T12:57:38.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: Detected new or changed devices on smithi204 2024-03-21T12:57:38.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:38.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:38.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:57:38.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:57:38.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: Adjusting osd_memory_target on smithi204 to 8544M 2024-03-21T12:57:38.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:38.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:38.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:38.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:38.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:57:38.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:57:38.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:38.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: osdmap e43: 6 total, 6 up, 6 in 2024-03-21T12:57:38.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: Detected new or changed devices on smithi204 2024-03-21T12:57:38.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:38.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:38.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:57:38.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:57:38.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: Adjusting osd_memory_target on smithi204 to 8544M 2024-03-21T12:57:38.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:38.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:38.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:38.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:38.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:57:38.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:57:38.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:39.436 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:39 smithi204 ceph-mon[33153]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:39.437 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:39 smithi204 ceph-mon[33153]: pgmap v142: 1 pgs: 1 remapped; 577 KiB data, 462 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:39.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:39 smithi163 ceph-mon[29621]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:39.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:39 smithi163 ceph-mon[29621]: pgmap v142: 1 pgs: 1 remapped; 577 KiB data, 462 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:39.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:39 smithi163 ceph-mon[40675]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:39.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:39 smithi163 ceph-mon[40675]: pgmap v142: 1 pgs: 1 remapped; 577 KiB data, 462 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:41.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:41 smithi204 ceph-mon[33153]: 1.0 scrub starts 2024-03-21T12:57:41.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:41 smithi204 ceph-mon[33153]: 1.0 scrub ok 2024-03-21T12:57:41.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:41 smithi204 ceph-mon[33153]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-21T12:57:41.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:41 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/1930878951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b"}]: dispatch 2024-03-21T12:57:41.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:41 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b"}]: dispatch 2024-03-21T12:57:41.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:41 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b"}]': finished 2024-03-21T12:57:41.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:41 smithi204 ceph-mon[33153]: osdmap e44: 7 total, 6 up, 7 in 2024-03-21T12:57:41.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:41 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:41.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:41 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/1094953493' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:57:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[40675]: 1.0 scrub starts 2024-03-21T12:57:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[40675]: 1.0 scrub ok 2024-03-21T12:57:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[40675]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-21T12:57:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/1930878951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b"}]: dispatch 2024-03-21T12:57:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b"}]: dispatch 2024-03-21T12:57:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b"}]': finished 2024-03-21T12:57:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[40675]: osdmap e44: 7 total, 6 up, 7 in 2024-03-21T12:57:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/1094953493' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:57:41.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[29621]: 1.0 scrub starts 2024-03-21T12:57:41.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[29621]: 1.0 scrub ok 2024-03-21T12:57:41.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[29621]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-21T12:57:41.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/1930878951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b"}]: dispatch 2024-03-21T12:57:41.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b"}]: dispatch 2024-03-21T12:57:41.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b"}]': finished 2024-03-21T12:57:41.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[29621]: osdmap e44: 7 total, 6 up, 7 in 2024-03-21T12:57:41.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:41.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:41 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/1094953493' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:57:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:43 smithi204 ceph-mon[33153]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-21T12:57:43.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:43 smithi163 ceph-mon[29621]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-21T12:57:43.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:43 smithi163 ceph-mon[40675]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-21T12:57:45.730 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:45 smithi204 ceph-mon[33153]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T12:57:45.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:45 smithi163 ceph-mon[29621]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T12:57:45.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:45 smithi163 ceph-mon[40675]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T12:57:47.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:47 smithi204 ceph-mon[33153]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-21T12:57:47.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:47 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T12:57:47.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:47 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:47.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:47 smithi163 ceph-mon[40675]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-21T12:57:47.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:47 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T12:57:47.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:47 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:47.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:47 smithi163 ceph-mon[29621]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-21T12:57:47.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:47 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T12:57:47.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:47 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:48.652 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:48 smithi204 ceph-mon[33153]: Deploying daemon osd.6 on smithi204 2024-03-21T12:57:48.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:48 smithi163 ceph-mon[29621]: Deploying daemon osd.6 on smithi204 2024-03-21T12:57:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:48 smithi163 ceph-mon[40675]: Deploying daemon osd.6 on smithi204 2024-03-21T12:57:49.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:49 smithi204 ceph-mon[33153]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T12:57:49.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:49 smithi163 ceph-mon[29621]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T12:57:49.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:49 smithi163 ceph-mon[40675]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T12:57:51.633 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:51 smithi204 ceph-mon[33153]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:51.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:51 smithi163 ceph-mon[29621]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:51 smithi163 ceph-mon[40675]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:53.481 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:53 smithi204 ceph-mon[33153]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:53.481 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:53 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:53.481 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:53 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:53.481 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:53 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:53 smithi163 ceph-mon[29621]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:53 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:53 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:53 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:53 smithi163 ceph-mon[40675]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:53 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:53 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:53 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:55.619 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:55 smithi204 ceph-mon[33153]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:55.619 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:55 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:55.619 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:55 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:55.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:55 smithi163 ceph-mon[29621]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:55.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:55 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:55.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:55 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:55.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:55 smithi163 ceph-mon[40675]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:55.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:55 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:55.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:55 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:56.214 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 6 on host 'smithi204' 2024-03-21T12:57:56.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:56.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:56.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:56.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:56 smithi204 ceph-mon[33153]: from='osd.6 v2:172.21.15.204:6808/200077988' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T12:57:56.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:56 smithi204 ceph-mon[33153]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T12:57:56.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:56.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:56.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:56.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:56.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:56.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[29621]: from='osd.6 v2:172.21.15.204:6808/200077988' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T12:57:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[29621]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T12:57:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[40675]: from='osd.6 v2:172.21.15.204:6808/200077988' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T12:57:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[40675]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T12:57:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:57:56.807 DEBUG:teuthology.orchestra.run.smithi204:osd.6> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.6.service 2024-03-21T12:57:56.810 INFO:tasks.cephadm:Deploying osd.7 on smithi204 with /dev/vg_nvme/lv_1... 2024-03-21T12:57:56.810 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-21T12:57:57.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:57 smithi204 ceph-mon[33153]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:57.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:57 smithi204 ceph-mon[33153]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T12:57:57.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:57 smithi204 ceph-mon[33153]: from='osd.6 v2:172.21.15.204:6808/200077988' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:57.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:57 smithi204 ceph-mon[33153]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T12:57:57.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:57 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:57.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:57 smithi204 ceph-mon[33153]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:57 smithi163 ceph-mon[29621]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:57 smithi163 ceph-mon[29621]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T12:57:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:57 smithi163 ceph-mon[29621]: from='osd.6 v2:172.21.15.204:6808/200077988' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:57 smithi163 ceph-mon[29621]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T12:57:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:57 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:57 smithi163 ceph-mon[29621]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:57.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:57 smithi163 ceph-mon[40675]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:57.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:57 smithi163 ceph-mon[40675]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T12:57:57.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:57 smithi163 ceph-mon[40675]: from='osd.6 v2:172.21.15.204:6808/200077988' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:57.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:57 smithi163 ceph-mon[40675]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T12:57:57.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:57 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:57.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:57 smithi163 ceph-mon[40675]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:57:57.962 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:57:58.002 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 12:57:57 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T12:57:57.837+0000 7f326f4d8700 -1 osd.6 0 waiting for initial osdmap 2024-03-21T12:57:58.002 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 12:57:57 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T12:57:57.844+0000 7f32672c3700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:57:58.642 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:58 smithi204 ceph-mon[33153]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-21T12:57:58.642 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:58 smithi204 ceph-mon[33153]: osdmap e46: 7 total, 6 up, 7 in 2024-03-21T12:57:58.642 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:58.642 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:58 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:58.642 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:58 smithi204 ceph-mon[33153]: from='osd.6 ' entity='osd.6' 2024-03-21T12:57:58.642 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:58 smithi204 ceph-mon[33153]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:58 smithi163 ceph-mon[29621]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-21T12:57:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:58 smithi163 ceph-mon[29621]: osdmap e46: 7 total, 6 up, 7 in 2024-03-21T12:57:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:58 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:58 smithi163 ceph-mon[29621]: from='osd.6 ' entity='osd.6' 2024-03-21T12:57:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:58 smithi163 ceph-mon[29621]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:58 smithi163 ceph-mon[40675]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-21T12:57:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:58 smithi163 ceph-mon[40675]: osdmap e46: 7 total, 6 up, 7 in 2024-03-21T12:57:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:58 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:58 smithi163 ceph-mon[40675]: from='osd.6 ' entity='osd.6' 2024-03-21T12:57:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:58 smithi163 ceph-mon[40675]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:57:59.434 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:59 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:57:59.435 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:59 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:57:59.435 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:59 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:59.435 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:59 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:57:59.435 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:59 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:57:59.435 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:59 smithi204 ceph-mon[33153]: osd.6 v2:172.21.15.204:6808/200077988 boot 2024-03-21T12:57:59.435 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:59 smithi204 ceph-mon[33153]: osdmap e47: 7 total, 7 up, 7 in 2024-03-21T12:57:59.435 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:57:59 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:57:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:57:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:57:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:57:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[29621]: osd.6 v2:172.21.15.204:6808/200077988 boot 2024-03-21T12:57:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[29621]: osdmap e47: 7 total, 7 up, 7 in 2024-03-21T12:57:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:57:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:57:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:57:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:57:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[40675]: osd.6 v2:172.21.15.204:6808/200077988 boot 2024-03-21T12:57:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[40675]: osdmap e47: 7 total, 7 up, 7 in 2024-03-21T12:57:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:57:59 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:58:00.423 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-21T12:58:00.423 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-21T12:58:00.423 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-03-21T12:58:00.423 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-03-21T12:58:00.423 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121731 s, 861 MB/s 2024-03-21T12:58:00.423 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-03-21T12:58:01.020 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:00 smithi204 ceph-mon[33153]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T12:58:01.020 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:00 smithi204 ceph-mon[33153]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:01.020 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:00 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:01.020 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:00 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:01.020 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:00 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:01.020 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:00 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:01.021 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:00 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:01.021 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:00 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:01.021 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:00 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:01.021 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:00 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:01.021 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:00 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:01.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi204:vg_nvme/lv_1 2024-03-21T12:58:01.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[40675]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T12:58:01.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[40675]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:01.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:01.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:01.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:01.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:01.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:01.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:01.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:01.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:01.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:01.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[29621]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T12:58:01.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[29621]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:01.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:01.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:01.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:01.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:01.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:01.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:01.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:01.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:01.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:01.285 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:58:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:01 smithi204 ceph-mon[33153]: Detected new or changed devices on smithi204 2024-03-21T12:58:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:01 smithi204 ceph-mon[33153]: Adjusting osd_memory_target on smithi204 to 5696M 2024-03-21T12:58:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:01 smithi204 ceph-mon[33153]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T12:58:02.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:01 smithi163 ceph-mon[29621]: Detected new or changed devices on smithi204 2024-03-21T12:58:02.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:01 smithi163 ceph-mon[29621]: Adjusting osd_memory_target on smithi204 to 5696M 2024-03-21T12:58:02.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:01 smithi163 ceph-mon[29621]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T12:58:02.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:01 smithi163 ceph-mon[40675]: Detected new or changed devices on smithi204 2024-03-21T12:58:02.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:01 smithi163 ceph-mon[40675]: Adjusting osd_memory_target on smithi204 to 5696M 2024-03-21T12:58:02.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:01 smithi163 ceph-mon[40675]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T12:58:03.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:03 smithi204 ceph-mon[33153]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T12:58:03.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:03 smithi204 ceph-mon[33153]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:03.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:03 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:58:03.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:03 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:58:03.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:03 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:03.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:03 smithi163 ceph-mon[29621]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T12:58:03.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:03 smithi163 ceph-mon[29621]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:03.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:58:03.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:58:03.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:03 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:03.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:03 smithi163 ceph-mon[40675]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T12:58:03.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:03 smithi163 ceph-mon[40675]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:03.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:03 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:58:03.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:03 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:58:03.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:03 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:04.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:04 smithi204 ceph-mon[33153]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:04.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:04 smithi204 ceph-mon[33153]: osdmap e51: 7 total, 7 up, 7 in 2024-03-21T12:58:04.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:04 smithi163 ceph-mon[29621]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:04.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:04 smithi163 ceph-mon[29621]: osdmap e51: 7 total, 7 up, 7 in 2024-03-21T12:58:04.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:04 smithi163 ceph-mon[40675]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:04.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:04 smithi163 ceph-mon[40675]: osdmap e51: 7 total, 7 up, 7 in 2024-03-21T12:58:05.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:05 smithi204 ceph-mon[33153]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:05.412 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:05 smithi163 ceph-mon[29621]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:05.412 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:05 smithi163 ceph-mon[40675]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:06.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:06 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:58:06.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:06 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:58:06.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:06 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/1665441380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49"}]: dispatch 2024-03-21T12:58:06.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:06 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49"}]: dispatch 2024-03-21T12:58:06.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:06 smithi204 ceph-mon[33153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49"}]': finished 2024-03-21T12:58:06.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:06 smithi204 ceph-mon[33153]: osdmap e52: 8 total, 7 up, 8 in 2024-03-21T12:58:06.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:06 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:06.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:58:06.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:58:06.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/1665441380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49"}]: dispatch 2024-03-21T12:58:06.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49"}]: dispatch 2024-03-21T12:58:06.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[29621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49"}]': finished 2024-03-21T12:58:06.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[29621]: osdmap e52: 8 total, 7 up, 8 in 2024-03-21T12:58:06.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:06.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:58:06.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:58:06.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/1665441380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49"}]: dispatch 2024-03-21T12:58:06.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49"}]: dispatch 2024-03-21T12:58:06.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[40675]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49"}]': finished 2024-03-21T12:58:06.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[40675]: osdmap e52: 8 total, 7 up, 8 in 2024-03-21T12:58:06.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:06 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:07.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:07 smithi204 ceph-mon[33153]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-21T12:58:07.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:07 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/3408507252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:58:07.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:07 smithi163 ceph-mon[29621]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-21T12:58:07.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:07 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/3408507252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:58:07.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:07 smithi163 ceph-mon[40675]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-21T12:58:07.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:07 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/3408507252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:58:09.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:09 smithi204 ceph-mon[33153]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-21T12:58:09.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:09 smithi163 ceph-mon[29621]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-21T12:58:09.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:09 smithi163 ceph-mon[40675]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-21T12:58:11.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:11 smithi204 ceph-mon[33153]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T12:58:11.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:11 smithi163 ceph-mon[29621]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T12:58:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:11 smithi163 ceph-mon[40675]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T12:58:13.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:13 smithi204 ceph-mon[33153]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-21T12:58:13.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:13 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T12:58:13.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:13 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:13.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:13 smithi204 ceph-mon[33153]: Deploying daemon osd.7 on smithi204 2024-03-21T12:58:13.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:13 smithi163 ceph-mon[40675]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-21T12:58:13.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:13 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T12:58:13.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:13 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:13.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:13 smithi163 ceph-mon[40675]: Deploying daemon osd.7 on smithi204 2024-03-21T12:58:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:13 smithi163 ceph-mon[29621]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-21T12:58:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:13 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T12:58:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:13 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:13 smithi163 ceph-mon[29621]: Deploying daemon osd.7 on smithi204 2024-03-21T12:58:15.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:15 smithi204 ceph-mon[33153]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T12:58:15.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:15 smithi163 ceph-mon[29621]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T12:58:15.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:15 smithi163 ceph-mon[40675]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T12:58:17.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:17 smithi204 ceph-mon[33153]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:17.342 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:17 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:58:17.342 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:17 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:58:17.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:17 smithi163 ceph-mon[29621]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:17 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:58:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:17 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:58:17.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:17 smithi163 ceph-mon[40675]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:17.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:17 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:58:17.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:17 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:58:18.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:18 smithi204 ceph-mon[33153]: osdmap e53: 8 total, 7 up, 8 in 2024-03-21T12:58:18.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:18 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:18.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:18 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:18.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:18 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:18.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:18 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:18.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:18 smithi163 ceph-mon[29621]: osdmap e53: 8 total, 7 up, 8 in 2024-03-21T12:58:18.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:18 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:18.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:18 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:18.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:18 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:18.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:18 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:18 smithi163 ceph-mon[40675]: osdmap e53: 8 total, 7 up, 8 in 2024-03-21T12:58:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:18 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:18 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:18 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:18 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:19.434 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:19 smithi204 ceph-mon[33153]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:19.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:19 smithi163 ceph-mon[29621]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:19.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:19 smithi163 ceph-mon[40675]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:21.486 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:21 smithi204 ceph-mon[33153]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:21.487 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:21 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:21.487 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:21 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:21.487 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:21 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:21.487 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:21 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:21.487 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:21 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:21.487 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:21 smithi204 ceph-mon[33153]: from='osd.7 v2:172.21.15.204:6812/315664888' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T12:58:21.487 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:21 smithi204 ceph-mon[33153]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T12:58:21.575 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 7 on host 'smithi204' 2024-03-21T12:58:21.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[40675]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:21.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:21.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:21.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:21.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:21.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:21.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[40675]: from='osd.7 v2:172.21.15.204:6812/315664888' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T12:58:21.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[40675]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T12:58:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[29621]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[29621]: from='osd.7 v2:172.21.15.204:6812/315664888' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T12:58:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:21 smithi163 ceph-mon[29621]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T12:58:22.266 DEBUG:teuthology.orchestra.run.smithi204:osd.7> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.7.service 2024-03-21T12:58:22.268 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-21T12:58:22.268 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-21T12:58:22.522 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:58:22.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:22.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:22.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:22.645 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[29621]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T12:58:22.645 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[29621]: from='osd.7 v2:172.21.15.204:6812/315664888' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:58:22.645 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[29621]: osdmap e54: 8 total, 7 up, 8 in 2024-03-21T12:58:22.645 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:22.645 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[29621]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:58:22.645 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[29621]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:22.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:22.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:22.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:22.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[40675]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T12:58:22.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[40675]: from='osd.7 v2:172.21.15.204:6812/315664888' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:58:22.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[40675]: osdmap e54: 8 total, 7 up, 8 in 2024-03-21T12:58:22.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:22.646 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[40675]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:58:22.646 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:22 smithi163 ceph-mon[40675]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:22.781 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:22.782 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:22.782 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:22.782 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:22 smithi204 ceph-mon[33153]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T12:58:22.782 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:22 smithi204 ceph-mon[33153]: from='osd.7 v2:172.21.15.204:6812/315664888' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:58:22.782 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:22 smithi204 ceph-mon[33153]: osdmap e54: 8 total, 7 up, 8 in 2024-03-21T12:58:22.782 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:22 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:22.782 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:22 smithi204 ceph-mon[33153]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-21T12:58:22.782 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:22 smithi204 ceph-mon[33153]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:58:22.782 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 12:58:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T12:58:22.780+0000 7ffa4fb0f700 -1 osd.7 0 waiting for initial osdmap 2024-03-21T12:58:23.051 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 12:58:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T12:58:22.786+0000 7ffa4b902700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:58:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:23 smithi204 ceph-mon[33153]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-21T12:58:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:23 smithi204 ceph-mon[33153]: osdmap e55: 8 total, 7 up, 8 in 2024-03-21T12:58:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:23 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:23 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:23 smithi163 ceph-mon[29621]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-21T12:58:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:23 smithi163 ceph-mon[29621]: osdmap e55: 8 total, 7 up, 8 in 2024-03-21T12:58:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:23 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:23 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:23 smithi163 ceph-mon[40675]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-21T12:58:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:23 smithi163 ceph-mon[40675]: osdmap e55: 8 total, 7 up, 8 in 2024-03-21T12:58:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:23 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:23 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:24.444 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:58:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:24 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:58:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:24 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:58:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:24 smithi204 ceph-mon[33153]: osd.7 v2:172.21.15.204:6812/315664888 boot 2024-03-21T12:58:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:24 smithi204 ceph-mon[33153]: osdmap e56: 8 total, 8 up, 8 in 2024-03-21T12:58:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:24 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:24 smithi204 ceph-mon[33153]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:58:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:24 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3235173675' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T12:58:25.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:58:25.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:58:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[40675]: osd.7 v2:172.21.15.204:6812/315664888 boot 2024-03-21T12:58:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[40675]: osdmap e56: 8 total, 8 up, 8 in 2024-03-21T12:58:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[40675]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:58:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3235173675' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T12:58:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:58:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:58:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[29621]: osd.7 v2:172.21.15.204:6812/315664888 boot 2024-03-21T12:58:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[29621]: osdmap e56: 8 total, 8 up, 8 in 2024-03-21T12:58:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:58:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[29621]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:58:25.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:24 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3235173675' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T12:58:25.162 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1711025903,"num_in_osds":8,"osd_in_since":1711025885,"num_remapped_pgs":1} 2024-03-21T12:58:25.163 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T12:58:25.416 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:58:26.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: osdmap e57: 8 total, 8 up, 8 in 2024-03-21T12:58:26.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: Detected new or changed devices on smithi204 2024-03-21T12:58:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: Adjusting osd_memory_target on smithi204 to 4272M 2024-03-21T12:58:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:25 smithi204 ceph-mon[33153]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T12:58:26.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: osdmap e57: 8 total, 8 up, 8 in 2024-03-21T12:58:26.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: Detected new or changed devices on smithi204 2024-03-21T12:58:26.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: Adjusting osd_memory_target on smithi204 to 4272M 2024-03-21T12:58:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[29621]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T12:58:26.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: osdmap e57: 8 total, 8 up, 8 in 2024-03-21T12:58:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: Detected new or changed devices on smithi204 2024-03-21T12:58:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:58:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: Adjusting osd_memory_target on smithi204 to 4272M 2024-03-21T12:58:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:25 smithi163 ceph-mon[40675]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T12:58:27.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:27 smithi163 ceph-mon[29621]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T12:58:27.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:27 smithi163 ceph-mon[29621]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:58:27.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:27 smithi163 ceph-mon[40675]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T12:58:27.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:27 smithi163 ceph-mon[40675]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:58:27.373 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:58:27.373 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":60,"fsid":"d4bf79c0-e781-11ee-95cd-87774f69a715","created":"2024-03-21T12:52:56.031419+0000","modified":"2024-03-21T12:58:27.151430+0000","last_up_change":"2024-03-21T12:58:23.775302+0000","last_in_change":"2024-03-21T12:58:05.763916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T12:56:20.557490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c34a3b25-9e02-46e6-8f9a-d0af2615258f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6801","nonce":3086410981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3086410981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3086410981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6803","nonce":3086410981}]},"public_addr":"172.21.15.163:6801/3086410981","cluster_addr":"172.21.15.163:6802/3086410981","heartbeat_back_addr":"172.21.15.163:6804/3086410981","heartbeat_front_addr":"172.21.15.163:6803/3086410981","state":["exists","up"]},{"osd":1,"uuid":"36e05f89-c238-41f7-be65-1553b529e8ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6805","nonce":3666062784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3666062784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3666062784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6807","nonce":3666062784}]},"public_addr":"172.21.15.163:6805/3666062784","cluster_addr":"172.21.15.163:6806/3666062784","heartbeat_back_addr":"172.21.15.163:6808/3666062784","heartbeat_front_addr":"172.21.15.163:6807/3666062784","state":["exists","up"]},{"osd":2,"uuid":"968ec78d-25b8-4905-b328-d490c38ca85f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6809","nonce":4009213507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4009213507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4009213507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":4009213507}]},"public_addr":"172.21.15.163:6809/4009213507","cluster_addr":"172.21.15.163:6810/4009213507","heartbeat_back_addr":"172.21.15.163:6812/4009213507","heartbeat_front_addr":"172.21.15.163:6811/4009213507","state":["exists","up"]},{"osd":3,"uuid":"67dfa4f6-1709-48f6-971d-c6287233492d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6813","nonce":3981878189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3981878189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":3981878189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":3981878189}]},"public_addr":"172.21.15.163:6813/3981878189","cluster_addr":"172.21.15.163:6814/3981878189","heartbeat_back_addr":"172.21.15.163:6816/3981878189","heartbeat_front_addr":"172.21.15.163:6815/3981878189","state":["exists","up"]},{"osd":4,"uuid":"7fd311a9-f3c2-456d-8ab1-341cf5b33471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":719276054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":719276054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":719276054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":719276054}]},"public_addr":"172.21.15.204:6800/719276054","cluster_addr":"172.21.15.204:6801/719276054","heartbeat_back_addr":"172.21.15.204:6803/719276054","heartbeat_front_addr":"172.21.15.204:6802/719276054","state":["exists","up"]},{"osd":5,"uuid":"a87759c8-251c-415d-9387-a49f7d2f36d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1346947668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":1346947668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":1346947668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1346947668}]},"public_addr":"172.21.15.204:6804/1346947668","cluster_addr":"172.21.15.204:6805/1346947668","heartbeat_back_addr":"172.21.15.204:6807/1346947668","heartbeat_front_addr":"172.21.15.204:6806/1346947668","state":["exists","up"]},{"osd":6,"uuid":"5eb135cd-f677-4d0c-b23a-738f31615b6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":200077988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":200077988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":200077988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":200077988}]},"public_addr":"172.21.15.204:6808/200077988","cluster_addr":"172.21.15.204:6809/200077988","heartbeat_back_addr":"172.21.15.204:6811/200077988","heartbeat_front_addr":"172.21.15.204:6810/200077988","state":["exists","up"]},{"osd":7,"uuid":"4f528e10-2c93-4bc2-a54c-06ca4132ed49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":315664888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":315664888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":315664888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":315664888}]},"public_addr":"172.21.15.204:6812/315664888","cluster_addr":"172.21.15.204:6813/315664888","heartbeat_back_addr":"172.21.15.204:6815/315664888","heartbeat_front_addr":"172.21.15.204:6814/315664888","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:05.391289+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:16.427153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:57:23.622611+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:57:58.813617+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:57:06.808924+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:57:31.532016+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:57:56.590024+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:21.966326+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:0/116934309":"2024-03-22T12:54:00.224135+0000","172.21.15.163:0/1783921524":"2024-03-22T12:54:00.224135+0000","172.21.15.163:6800/2553440211":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/1812694077":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/1070540283":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/2354157786":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/4152198951":"2024-03-22T12:54:00.224135+0000","172.21.15.163:6800/1798115633":"2024-03-22T12:54:00.224135+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T12:58:27.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:27 smithi204 ceph-mon[33153]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T12:58:27.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:27 smithi204 ceph-mon[33153]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:58:28.130 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T12:56:20.557490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T12:58:28.130 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-21T12:58:28.372 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:58:28.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:28 smithi163 ceph-mon[29621]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T12:58:28.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:28 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3876847876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:58:28.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:28 smithi163 ceph-mon[40675]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T12:58:28.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:28 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3876847876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:58:28.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:28 smithi204 ceph-mon[33153]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T12:58:28.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:28 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3876847876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:58:29.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:29 smithi163 ceph-mon[29621]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:58:29.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:29 smithi163 ceph-mon[40675]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:58:29.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:29 smithi204 ceph-mon[33153]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:58:30.202 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 1 2024-03-21T12:58:30.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:30 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2930032300' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T12:58:30.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:30 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2930032300' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T12:58:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:30 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2930032300' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T12:58:30.851 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi163 2024-03-21T12:58:30.851 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi163=foo.a' 2024-03-21T12:58:31.103 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:58:31.482 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:31 smithi204 ceph-mon[33153]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-21T12:58:31.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:31 smithi163 ceph-mon[29621]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-21T12:58:31.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:31 smithi163 ceph-mon[40675]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-21T12:58:32.804 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled rgw.foo.a update... 2024-03-21T12:58:33.439 DEBUG:teuthology.orchestra.run.smithi163:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@rgw.foo.a.service 2024-03-21T12:58:33.442 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi204 2024-03-21T12:58:33.442 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-21T12:58:33.468 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:33 smithi204 ceph-mon[33153]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-21T12:58:33.468 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:33 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:33.468 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:33 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:33.682 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:58:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:33 smithi163 ceph-mon[29621]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-21T12:58:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:33 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:33 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:33.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:33 smithi163 ceph-mon[40675]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-21T12:58:33.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:33 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:33.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:33 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:34.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[40675]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi163=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:34.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[40675]: Saving service rgw.foo.a spec with placement smithi163=foo.a;count:1 2024-03-21T12:58:34.418 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:34.418 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:34.418 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:34.418 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:58:34.418 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T12:58:34.418 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:34.418 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:34.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[29621]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi163=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:34.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[29621]: Saving service rgw.foo.a spec with placement smithi163=foo.a;count:1 2024-03-21T12:58:34.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:34.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:34.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:34.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:58:34.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T12:58:34.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:34.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:34 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:34.457 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:34 smithi204 ceph-mon[33153]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi163=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:34.457 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:34 smithi204 ceph-mon[33153]: Saving service rgw.foo.a spec with placement smithi163=foo.a;count:1 2024-03-21T12:58:34.457 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:34 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:34.457 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:34 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:34.457 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:34 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:34.457 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:34 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:58:34.458 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:34 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T12:58:34.458 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:34 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:34.458 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:34 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:35.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:35 smithi163 ceph-mon[29621]: Deploying daemon rgw.foo.a on smithi163 2024-03-21T12:58:35.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:35 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:58:35.491 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:35 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:58:35.491 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:35 smithi163 ceph-mon[29621]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-21T12:58:35.491 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:35 smithi163 ceph-mon[29621]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T12:58:35.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:35 smithi163 ceph-mon[40675]: Deploying daemon rgw.foo.a on smithi163 2024-03-21T12:58:35.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:35 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:58:35.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:35 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:58:35.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:35 smithi163 ceph-mon[40675]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-21T12:58:35.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:35 smithi163 ceph-mon[40675]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T12:58:35.720 INFO:teuthology.orchestra.run.smithi204.stderr:pool 'datapool' created 2024-03-21T12:58:35.721 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:35 smithi204 ceph-mon[33153]: Deploying daemon rgw.foo.a on smithi163 2024-03-21T12:58:35.721 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:35 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:58:35.721 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:35 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:58:35.721 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:35 smithi204 ceph-mon[33153]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-21T12:58:35.721 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:35 smithi204 ceph-mon[33153]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T12:58:35.760 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 12:58:35 smithi163 systemd[1]: Starting Ceph rgw.foo.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T12:58:36.032 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 12:58:35 smithi163 podman[67111]: 2024-03-21 12:58:35.801291412 +0000 UTC m=+0.018537758 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T12:58:36.032 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 12:58:35 smithi163 podman[67111]: 2024-03-21 12:58:35.898685718 +0000 UTC m=+0.115932112 container create f432467663204361a881d26a340a14ac5eb73de04addb629c08dbb685ec9b8ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-03-21T12:58:36.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- rbd pool init datapool 2024-03-21T12:58:36.438 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 12:58:36 smithi163 podman[67111]: 2024-03-21 12:58:36.159952755 +0000 UTC m=+0.377199103 container init f432467663204361a881d26a340a14ac5eb73de04addb629c08dbb685ec9b8ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-21T12:58:36.438 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 12:58:36 smithi163 podman[67111]: 2024-03-21 12:58:36.163122148 +0000 UTC m=+0.380368495 container start f432467663204361a881d26a340a14ac5eb73de04addb629c08dbb685ec9b8ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True) 2024-03-21T12:58:36.438 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 12:58:36 smithi163 bash[67111]: f432467663204361a881d26a340a14ac5eb73de04addb629c08dbb685ec9b8ab 2024-03-21T12:58:36.438 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 12:58:36 smithi163 systemd[1]: Started Ceph rgw.foo.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T12:58:36.653 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:58:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:36 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:58:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:36 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:58:36.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:36 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/3792524227' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T12:58:36.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:36 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T12:58:36.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:36 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-21T12:58:36.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:36 smithi204 ceph-mon[33153]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T12:58:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:36 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:58:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:36 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:58:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:36 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/3792524227' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T12:58:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:36 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T12:58:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:36 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-21T12:58:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:36 smithi163 ceph-mon[29621]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T12:58:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:36 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:58:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:36 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:58:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:36 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/3792524227' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T12:58:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:36 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T12:58:36.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:36 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-21T12:58:36.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:36 smithi163 ceph-mon[40675]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T12:58:37.623 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[29621]: pgmap v190: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T12:58:37.623 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.624 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.624 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.624 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[29621]: Saving service rgw.foo.a spec with placement smithi163=foo.a;count:1 2024-03-21T12:58:37.624 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.624 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.624 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:37.624 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[29621]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T12:58:37.624 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3025743487' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T12:58:37.624 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[29621]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T12:58:37.624 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[40675]: pgmap v190: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T12:58:37.624 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.624 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.625 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.625 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[40675]: Saving service rgw.foo.a spec with placement smithi163=foo.a;count:1 2024-03-21T12:58:37.625 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.625 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.625 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:37.625 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[40675]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T12:58:37.625 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3025743487' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T12:58:37.625 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:37 smithi163 ceph-mon[40675]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T12:58:37.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:37 smithi204 ceph-mon[33153]: pgmap v190: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T12:58:37.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:37 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:37 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:37 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:37 smithi204 ceph-mon[33153]: Saving service rgw.foo.a spec with placement smithi163=foo.a;count:1 2024-03-21T12:58:37.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:37 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:37 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:37.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:37 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:37.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:37 smithi204 ceph-mon[33153]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T12:58:37.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:37 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3025743487' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T12:58:37.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:37 smithi204 ceph-mon[33153]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T12:58:39.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:38 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3025743487' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T12:58:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:38 smithi204 ceph-mon[33153]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T12:58:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:38 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/898757128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T12:58:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:38 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T12:58:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:38 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:38 smithi204 ceph-mon[33153]: Checking dashboard <-> RGW credentials 2024-03-21T12:58:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:38 smithi204 ceph-mon[33153]: pgmap v193: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:58:39.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3025743487' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T12:58:39.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[29621]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T12:58:39.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/898757128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T12:58:39.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T12:58:39.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:39.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:39.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:39.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:39.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[29621]: Checking dashboard <-> RGW credentials 2024-03-21T12:58:39.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[29621]: pgmap v193: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:58:39.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3025743487' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T12:58:39.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[40675]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T12:58:39.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/898757128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T12:58:39.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T12:58:39.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:39.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:39.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:39.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:39.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[40675]: Checking dashboard <-> RGW credentials 2024-03-21T12:58:39.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:38 smithi163 ceph-mon[40675]: pgmap v193: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:58:40.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:39 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-21T12:58:40.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:39 smithi204 ceph-mon[33153]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T12:58:40.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:39 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1134746574' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:58:40.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:39 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1345925145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:58:40.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:39 smithi204 ceph-mon[33153]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:58:40.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:39 smithi204 ceph-mon[33153]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:58:40.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:39 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-21T12:58:40.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:39 smithi163 ceph-mon[40675]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T12:58:40.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:39 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1134746574' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:58:40.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:39 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1345925145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:58:40.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:39 smithi163 ceph-mon[40675]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:58:40.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:39 smithi163 ceph-mon[40675]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:58:40.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:39 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-21T12:58:40.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:39 smithi163 ceph-mon[29621]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T12:58:40.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:39 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1134746574' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:58:40.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:39 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1345925145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:58:40.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:39 smithi163 ceph-mon[29621]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:58:40.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:39 smithi163 ceph-mon[29621]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:58:41.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:40 smithi204 ceph-mon[33153]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T12:58:41.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:40 smithi204 ceph-mon[33153]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T12:58:41.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:40 smithi204 ceph-mon[33153]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T12:58:41.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:40 smithi204 ceph-mon[33153]: pgmap v196: 68 pgs: 13 creating+peering, 29 active+clean, 26 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-21T12:58:41.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:40 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:41.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:40 smithi163 ceph-mon[29621]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T12:58:41.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:40 smithi163 ceph-mon[29621]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T12:58:41.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:40 smithi163 ceph-mon[29621]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T12:58:41.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:40 smithi163 ceph-mon[29621]: pgmap v196: 68 pgs: 13 creating+peering, 29 active+clean, 26 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-21T12:58:41.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:40 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:41.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:40 smithi163 ceph-mon[40675]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T12:58:41.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:40 smithi163 ceph-mon[40675]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T12:58:41.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:40 smithi163 ceph-mon[40675]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T12:58:41.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:40 smithi163 ceph-mon[40675]: pgmap v196: 68 pgs: 13 creating+peering, 29 active+clean, 26 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-21T12:58:41.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:40 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:41.417 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.204 --placement '1;smithi204=iscsi.a' 2024-03-21T12:58:41.656 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:58:41.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:41 smithi204 ceph-mon[33153]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T12:58:41.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:41 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T12:58:41.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:41 smithi204 ceph-mon[33153]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T12:58:41.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:41 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1134746574' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:58:41.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:41 smithi204 ceph-mon[33153]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:58:41.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:41 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1345925145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:58:41.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:41 smithi204 ceph-mon[33153]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:58:41.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:41 smithi204 ceph-mon[33153]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T12:58:41.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:41 smithi204 ceph-mon[33153]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T12:58:41.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:41 smithi204 ceph-mon[33153]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T12:58:42.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[29621]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T12:58:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T12:58:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[29621]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T12:58:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1134746574' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:58:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[29621]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:58:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1345925145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:58:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[29621]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:58:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[29621]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T12:58:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[29621]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T12:58:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[29621]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T12:58:42.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[40675]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T12:58:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T12:58:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[40675]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T12:58:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1134746574' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:58:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[40675]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:58:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1345925145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:58:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[40675]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:58:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[40675]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T12:58:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[40675]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T12:58:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:41 smithi163 ceph-mon[40675]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T12:58:43.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:43 smithi204 ceph-mon[33153]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T12:58:43.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:43 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1134746574' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:58:43.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:43 smithi204 ceph-mon[33153]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:58:43.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:43 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1345925145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:58:43.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:43 smithi204 ceph-mon[33153]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:58:43.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:43 smithi204 ceph-mon[33153]: pgmap v200: 132 pgs: 17 creating+peering, 44 active+clean, 71 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-21T12:58:43.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:43 smithi163 ceph-mon[29621]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T12:58:43.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:43 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1134746574' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:58:43.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:43 smithi163 ceph-mon[29621]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:58:43.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:43 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1345925145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:58:43.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:43 smithi163 ceph-mon[29621]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:58:43.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:43 smithi163 ceph-mon[29621]: pgmap v200: 132 pgs: 17 creating+peering, 44 active+clean, 71 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-21T12:58:43.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:43 smithi163 ceph-mon[40675]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T12:58:43.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:43 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1134746574' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:58:43.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:43 smithi163 ceph-mon[40675]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:58:43.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:43 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1345925145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:58:43.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:43 smithi163 ceph-mon[40675]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:58:43.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:43 smithi163 ceph-mon[40675]: pgmap v200: 132 pgs: 17 creating+peering, 44 active+clean, 71 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-21T12:58:43.585 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled iscsi.datapool update... 2024-03-21T12:58:44.222 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-21T12:58:44.223 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:58:44.223 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-21T12:58:44.268 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-21T12:58:44.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-21T12:58:44.297 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:44 smithi204 ceph-mon[33153]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T12:58:44.297 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:44 smithi204 ceph-mon[33153]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T12:58:44.297 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:44 smithi204 ceph-mon[33153]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T12:58:44.297 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:44 smithi204 ceph-mon[33153]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T12:58:44.297 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1345925145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:58:44.297 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:44 smithi204 ceph-mon[33153]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:58:44.297 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1134746574' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:58:44.297 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:44 smithi204 ceph-mon[33153]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:58:44.298 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:44 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:44.302 DEBUG:teuthology.orchestra.run.smithi204:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@iscsi.iscsi.a.service 2024-03-21T12:58:44.345 INFO:tasks.cephadm:Adding prometheus.a on smithi204 2024-03-21T12:58:44.345 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi204=a' 2024-03-21T12:58:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[29621]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T12:58:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[29621]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T12:58:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[29621]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T12:58:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[29621]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T12:58:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1345925145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:58:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[29621]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:58:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1134746574' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:58:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[29621]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:58:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:44.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[40675]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T12:58:44.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[40675]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T12:58:44.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[40675]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T12:58:44.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[40675]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T12:58:44.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1345925145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:58:44.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[40675]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:58:44.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1134746574' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:58:44.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[40675]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:58:44.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:44 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:44.649 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:58:45.433 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:45 smithi204 ceph-mon[33153]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.204", "placement": "1;smithi204=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:45.433 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:45 smithi204 ceph-mon[33153]: Saving service iscsi.datapool spec with placement smithi204=iscsi.a;count:1 2024-03-21T12:58:45.433 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:45 smithi204 ceph-mon[33153]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T12:58:45.433 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:45 smithi204 ceph-mon[33153]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T12:58:45.434 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:45 smithi204 ceph-mon[33153]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T12:58:45.434 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:45 smithi204 ceph-mon[33153]: pgmap v203: 132 pgs: 12 creating+peering, 96 active+clean, 24 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 869 B/s wr, 2 op/s 2024-03-21T12:58:45.434 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:45 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:45.434 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:45 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:45.434 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:45 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:45.434 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:45 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:58:45.434 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:45 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T12:58:45.434 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:45 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[29621]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.204", "placement": "1;smithi204=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[29621]: Saving service iscsi.datapool spec with placement smithi204=iscsi.a;count:1 2024-03-21T12:58:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[29621]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T12:58:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[29621]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T12:58:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[29621]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T12:58:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[29621]: pgmap v203: 132 pgs: 12 creating+peering, 96 active+clean, 24 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 869 B/s wr, 2 op/s 2024-03-21T12:58:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:58:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T12:58:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:45.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[40675]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.204", "placement": "1;smithi204=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:45.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[40675]: Saving service iscsi.datapool spec with placement smithi204=iscsi.a;count:1 2024-03-21T12:58:45.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[40675]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T12:58:45.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[40675]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T12:58:45.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[40675]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T12:58:45.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[40675]: pgmap v203: 132 pgs: 12 creating+peering, 96 active+clean, 24 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 869 B/s wr, 2 op/s 2024-03-21T12:58:45.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:45.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:45.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:45.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:58:45.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T12:58:45.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:45 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:46.347 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:46 smithi204 ceph-mon[33153]: Deploying daemon iscsi.iscsi.a on smithi204 2024-03-21T12:58:46.347 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:46 smithi204 ceph-mon[33153]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T12:58:46.347 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:46 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T12:58:46.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:46 smithi204 ceph-mon[33153]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T12:58:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:46 smithi163 ceph-mon[29621]: Deploying daemon iscsi.iscsi.a on smithi204 2024-03-21T12:58:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:46 smithi163 ceph-mon[29621]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T12:58:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:46 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T12:58:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:46 smithi163 ceph-mon[29621]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T12:58:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:46 smithi163 ceph-mon[40675]: Deploying daemon iscsi.iscsi.a on smithi204 2024-03-21T12:58:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:46 smithi163 ceph-mon[40675]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T12:58:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:46 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T12:58:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:46 smithi163 ceph-mon[40675]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T12:58:46.899 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled prometheus update... 2024-03-21T12:58:47.487 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:47 smithi204 ceph-mon[33153]: pgmap v205: 132 pgs: 132 active+clean; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 5.3 KiB/s wr, 111 op/s 2024-03-21T12:58:47.487 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:47 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:47.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:47 smithi163 ceph-mon[29621]: pgmap v205: 132 pgs: 132 active+clean; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 5.3 KiB/s wr, 111 op/s 2024-03-21T12:58:47.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:47 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:47.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:47 smithi163 ceph-mon[40675]: pgmap v205: 132 pgs: 132 active+clean; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 5.3 KiB/s wr, 111 op/s 2024-03-21T12:58:47.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:47 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:47.519 DEBUG:teuthology.orchestra.run.smithi204:prometheus.a> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-21T12:58:47.521 INFO:tasks.cephadm:Adding node-exporter.a on smithi163 2024-03-21T12:58:47.522 INFO:tasks.cephadm:Adding node-exporter.b on smithi204 2024-03-21T12:58:47.522 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '2;smithi163=a;smithi204=b' 2024-03-21T12:58:48.136 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:58:48.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:48 smithi204 ceph-mon[33153]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:48.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:48 smithi204 ceph-mon[33153]: Saving service prometheus spec with placement smithi204=a;count:1 2024-03-21T12:58:48.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:48 smithi163 ceph-mon[29621]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:48.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:48 smithi163 ceph-mon[29621]: Saving service prometheus spec with placement smithi204=a;count:1 2024-03-21T12:58:48.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:48 smithi163 ceph-mon[40675]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:48.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:48 smithi163 ceph-mon[40675]: Saving service prometheus spec with placement smithi204=a;count:1 2024-03-21T12:58:49.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:49 smithi163 ceph-mon[29621]: pgmap v206: 132 pgs: 132 active+clean; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 3.7 KiB/s wr, 76 op/s 2024-03-21T12:58:49.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:49 smithi163 ceph-mon[40675]: pgmap v206: 132 pgs: 132 active+clean; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 3.7 KiB/s wr, 76 op/s 2024-03-21T12:58:49.373 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:49 smithi204 ceph-mon[33153]: pgmap v206: 132 pgs: 132 active+clean; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 3.7 KiB/s wr, 76 op/s 2024-03-21T12:58:50.067 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled node-exporter update... 2024-03-21T12:58:50.674 DEBUG:teuthology.orchestra.run.smithi163:node-exporter.a> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-21T12:58:50.677 DEBUG:teuthology.orchestra.run.smithi204:node-exporter.b> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-21T12:58:50.680 INFO:tasks.cephadm:Adding alertmanager.a on smithi163 2024-03-21T12:58:50.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi163=a' 2024-03-21T12:58:50.982 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:58:51.102 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:51 smithi204 ceph-mon[33153]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi163=a;smithi204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:51.102 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:51 smithi204 ceph-mon[33153]: Saving service node-exporter spec with placement smithi163=a;smithi204=b;count:2 2024-03-21T12:58:51.102 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:51 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.102 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:51 smithi204 ceph-mon[33153]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 5.3 KiB/s wr, 158 op/s 2024-03-21T12:58:51.102 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:51 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.102 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:51 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.102 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:51 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.102 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:51 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[29621]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi163=a;smithi204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:51.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[29621]: Saving service node-exporter spec with placement smithi163=a;smithi204=b;count:2 2024-03-21T12:58:51.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[29621]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 5.3 KiB/s wr, 158 op/s 2024-03-21T12:58:51.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[40675]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi163=a;smithi204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:51.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[40675]: Saving service node-exporter spec with placement smithi163=a;smithi204=b;count:2 2024-03-21T12:58:51.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[40675]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 5.3 KiB/s wr, 158 op/s 2024-03-21T12:58:51.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:51.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:51 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:52.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:52 smithi204 ceph-mon[33153]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-21T12:58:52.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:52 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:52.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:52 smithi204 ceph-mon[33153]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T12:58:52.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:52 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/2156121944' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:58:52.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:52 smithi163 ceph-mon[40675]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-21T12:58:52.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:52 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:52.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:52 smithi163 ceph-mon[40675]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T12:58:52.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:52 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/2156121944' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:58:52.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:52 smithi163 ceph-mon[29621]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-21T12:58:52.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:52 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:52.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:52 smithi163 ceph-mon[29621]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T12:58:52.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:52 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/2156121944' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:58:53.310 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled alertmanager update... 2024-03-21T12:58:53.485 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:53 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:58:53.485 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:53 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:58:53.485 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:53 smithi204 ceph-mon[33153]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T12:58:53.485 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:53 smithi204 ceph-mon[33153]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T12:58:53.485 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:53 smithi204 ceph-mon[33153]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 2.6 KiB/s wr, 109 op/s 2024-03-21T12:58:53.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:53 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:58:53.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:53 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:58:53.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:53 smithi163 ceph-mon[40675]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T12:58:53.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:53 smithi163 ceph-mon[40675]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T12:58:53.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:53 smithi163 ceph-mon[40675]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 2.6 KiB/s wr, 109 op/s 2024-03-21T12:58:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:53 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:58:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:53 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:58:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:53 smithi163 ceph-mon[29621]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T12:58:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:53 smithi163 ceph-mon[29621]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T12:58:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:53 smithi163 ceph-mon[29621]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 2.6 KiB/s wr, 109 op/s 2024-03-21T12:58:54.374 DEBUG:teuthology.orchestra.run.smithi163:alertmanager.a> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-21T12:58:54.377 INFO:tasks.cephadm:Adding grafana.a on smithi204 2024-03-21T12:58:54.377 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi204=a' 2024-03-21T12:58:54.647 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:58:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:54 smithi204 ceph-mon[33153]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi163=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:54 smithi204 ceph-mon[33153]: Saving service alertmanager spec with placement smithi163=a;count:1 2024-03-21T12:58:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:54 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:54.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:54 smithi163 ceph-mon[29621]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi163=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:54.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:54 smithi163 ceph-mon[29621]: Saving service alertmanager spec with placement smithi163=a;count:1 2024-03-21T12:58:54.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:54 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:54 smithi163 ceph-mon[40675]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi163=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:54 smithi163 ceph-mon[40675]: Saving service alertmanager spec with placement smithi163=a;count:1 2024-03-21T12:58:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:54 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:55.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:55 smithi204 ceph-mon[33153]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-03-21T12:58:55.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:55 smithi163 ceph-mon[29621]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-03-21T12:58:55.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:55 smithi163 ceph-mon[40675]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-03-21T12:58:56.634 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:56.635 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:56.635 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:56 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:56.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:56 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:56 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:57.062 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled grafana update... 2024-03-21T12:58:57.733 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:57 smithi204 ceph-mon[33153]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-03-21T12:58:57.733 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:57 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:57.733 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:57 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:57.733 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:57 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:57.733 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:57 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:57.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:57 smithi163 ceph-mon[29621]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-03-21T12:58:57.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:57 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:57 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:57 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:57 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:57.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:57 smithi163 ceph-mon[40675]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-03-21T12:58:57.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:57 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:57.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:57 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:57.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:57 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:57.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:57 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:58:57.764 DEBUG:teuthology.orchestra.run.smithi204:grafana.a> sudo journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@grafana.a.service 2024-03-21T12:58:57.766 INFO:tasks.cephadm:Setting up client nodes... 2024-03-21T12:58:57.767 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-21T12:58:58.018 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:58:58.415 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:58 smithi163 ceph-mon[29621]: Deploying daemon prometheus.a on smithi204 2024-03-21T12:58:58.415 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:58 smithi163 ceph-mon[29621]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:58.416 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:58 smithi163 ceph-mon[29621]: Saving service grafana spec with placement smithi204=a;count:1 2024-03-21T12:58:58.416 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:58 smithi163 ceph-mon[40675]: Deploying daemon prometheus.a on smithi204 2024-03-21T12:58:58.416 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:58 smithi163 ceph-mon[40675]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:58.416 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:58 smithi163 ceph-mon[40675]: Saving service grafana spec with placement smithi204=a;count:1 2024-03-21T12:58:58.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:58 smithi204 ceph-mon[33153]: Deploying daemon prometheus.a on smithi204 2024-03-21T12:58:58.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:58 smithi204 ceph-mon[33153]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:58.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:58 smithi204 ceph-mon[33153]: Saving service grafana spec with placement smithi204=a;count:1 2024-03-21T12:58:59.457 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:58:59 smithi163 ceph-mon[29621]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-03-21T12:58:59.457 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:58:59 smithi163 ceph-mon[40675]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-03-21T12:58:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:58:59 smithi204 ceph-mon[33153]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-03-21T12:58:59.971 INFO:teuthology.orchestra.run.smithi163.stdout:[client.0] 2024-03-21T12:58:59.971 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQATL/xlBh+gORAAl3hfIpB6hmKRqXwi/DcE0w== 2024-03-21T12:59:00.601 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:00 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1759861257' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:59:00.601 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:00 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:59:00.602 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:00 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T12:59:00.602 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-21T12:59:00.602 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-21T12:59:00.602 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:00 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-21T12:59:00.602 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:00 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1759861257' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:59:00.602 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:00 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:59:00.602 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:00 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T12:59:00.602 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:00 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-21T12:59:00.602 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:00 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-21T12:59:00.602 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:00 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-21T12:59:00.630 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-21T12:59:00.630 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-21T12:59:00.630 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-21T12:59:00.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-21T12:59:00.699 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:00 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1759861257' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:59:00.700 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:00 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:59:00.700 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:00 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T12:59:00.700 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:00 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-21T12:59:00.700 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:00 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-21T12:59:00.700 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:00 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-21T12:59:00.931 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T12:59:01.593 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:01 smithi204 ceph-mon[33153]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 112 B/s wr, 1 op/s 2024-03-21T12:59:01.593 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:01 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-21T12:59:01.593 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:01 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-21T12:59:01.593 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:01 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-21T12:59:01.593 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:01 smithi204 ceph-mon[33153]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T12:59:01.593 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:01 smithi204 ceph-mon[33153]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T12:59:01.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:01 smithi163 ceph-mon[29621]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 112 B/s wr, 1 op/s 2024-03-21T12:59:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:01 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-21T12:59:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:01 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-21T12:59:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:01 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-21T12:59:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:01 smithi163 ceph-mon[29621]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T12:59:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:01 smithi163 ceph-mon[29621]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T12:59:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:01 smithi163 ceph-mon[40675]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 112 B/s wr, 1 op/s 2024-03-21T12:59:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:01 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-21T12:59:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:01 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-21T12:59:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:01 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-21T12:59:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:01 smithi163 ceph-mon[40675]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T12:59:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:01 smithi163 ceph-mon[40675]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T12:59:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:02 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:02.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:02 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:02.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:02 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:03.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:03 smithi204 ceph-mon[33153]: 1.0 deep-scrub starts 2024-03-21T12:59:03.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:03 smithi204 ceph-mon[33153]: 1.0 deep-scrub ok 2024-03-21T12:59:03.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:03 smithi204 ceph-mon[33153]: pgmap v217: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:59:03.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:03 smithi163 ceph-mon[29621]: 1.0 deep-scrub starts 2024-03-21T12:59:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:03 smithi163 ceph-mon[29621]: 1.0 deep-scrub ok 2024-03-21T12:59:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:03 smithi163 ceph-mon[29621]: pgmap v217: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:59:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:03 smithi163 ceph-mon[40675]: 1.0 deep-scrub starts 2024-03-21T12:59:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:03 smithi163 ceph-mon[40675]: 1.0 deep-scrub ok 2024-03-21T12:59:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:03 smithi163 ceph-mon[40675]: pgmap v217: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:59:05.463 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:05 smithi204 ceph-mon[33153]: pgmap v218: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-03-21T12:59:05.464 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:05 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:59:05.464 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:05 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:59:05.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:05 smithi163 ceph-mon[29621]: pgmap v218: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-03-21T12:59:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:05 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:59:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:05 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:59:05.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:05 smithi163 ceph-mon[40675]: pgmap v218: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-03-21T12:59:05.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:05 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:59:05.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:05 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:59:06.057 INFO:teuthology.orchestra.run.smithi204.stdout:[client.1] 2024-03-21T12:59:06.057 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAaL/xlPy8vAxAAI7w5rb6CVNmBsFYUAWjSGg== 2024-03-21T12:59:06.412 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:06 smithi204 ceph-mon[33153]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T12:59:06.412 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:06 smithi204 ceph-mon[33153]: from='client.? 172.21.15.204:0/58458066' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:59:06.412 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:06 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:59:06.412 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:06 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T12:59:06.412 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:06 smithi204 ceph-mon[33153]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T12:59:06.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:06 smithi163 ceph-mon[29621]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T12:59:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:06 smithi163 ceph-mon[29621]: from='client.? 172.21.15.204:0/58458066' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:59:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:06 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:59:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:06 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T12:59:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:06 smithi163 ceph-mon[29621]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T12:59:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:06 smithi163 ceph-mon[40675]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T12:59:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:06 smithi163 ceph-mon[40675]: from='client.? 172.21.15.204:0/58458066' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:59:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:06 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:59:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:06 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T12:59:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:06 smithi163 ceph-mon[40675]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T12:59:07.444 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:07 smithi204 ceph-mon[33153]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s; 62 B/s, 2 objects/s recovering 2024-03-21T12:59:07.726 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-21T12:59:07.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-21T12:59:07.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-21T12:59:07.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:07 smithi163 ceph-mon[29621]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s; 62 B/s, 2 objects/s recovering 2024-03-21T12:59:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:07 smithi163 ceph-mon[40675]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s; 62 B/s, 2 objects/s recovering 2024-03-21T12:59:07.814 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-21T12:59:07.814 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-21T12:59:07.815 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-21T12:59:08.066 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:09.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:09 smithi163 ceph-mon[40675]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 717 B/s rd, 0 op/s; 46 B/s, 1 objects/s recovering 2024-03-21T12:59:09.476 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:09 smithi163 ceph-mon[29621]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 717 B/s rd, 0 op/s; 46 B/s, 1 objects/s recovering 2024-03-21T12:59:09.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:09 smithi204 ceph-mon[33153]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 717 B/s rd, 0 op/s; 46 B/s, 1 objects/s recovering 2024-03-21T12:59:09.753 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:09 smithi204 systemd[1]: Starting Ceph prometheus.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T12:59:10.026 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:09 smithi204 podman[62023]: 2024-03-21 12:59:09.672863283 +0000 UTC m=+0.019247179 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-21T12:59:10.026 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:09 smithi204 podman[62023]: 2024-03-21 12:59:09.823946536 +0000 UTC m=+0.170330433 container create 2f6d58b7a4bb63eb7ee5530b319a26e688a8f817921465a965f8557846e0f820 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:59:10.026 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 podman[62023]: 2024-03-21 12:59:10.026242972 +0000 UTC m=+0.372626874 container init 2f6d58b7a4bb63eb7ee5530b319a26e688a8f817921465a965f8557846e0f820 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:59:10.154 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:59:10.503 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 podman[62023]: 2024-03-21 12:59:10.028803095 +0000 UTC m=+0.375186991 container start 2f6d58b7a4bb63eb7ee5530b319a26e688a8f817921465a965f8557846e0f820 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:59:10.503 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.052Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-21T12:59:10.503 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.052Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.052Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi204 (none))" 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.052Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.052Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.054Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.054Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.055Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.056Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.057Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.057Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.388µs 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.057Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.057Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.057Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=15.749µs wal_replay_duration=360.474µs wbl_replay_duration=118ns total_replay_duration=392.633µs 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.058Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.058Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-21T12:59:10.504 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.058Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T12:59:10.505 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.078Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.959826ms db_storage=744ns remote_storage=1.268µs web_handler=283ns query_engine=521ns scrape=903.133µs scrape_sd=167.967µs notify=1.2µs notify_sd=2.521µs rules=18.613708ms tracing=5.318µs 2024-03-21T12:59:10.505 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.078Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-21T12:59:10.505 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T12:59:10.078Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-21T12:59:10.505 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 bash[62023]: 2f6d58b7a4bb63eb7ee5530b319a26e688a8f817921465a965f8557846e0f820 2024-03-21T12:59:10.505 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 12:59:10 smithi204 systemd[1]: Started Ceph prometheus.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T12:59:10.988 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":16,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3073378854}]},"active_addr":"172.21.15.163:6800/3073378854","active_change":"2024-03-21T12:54:00.224286+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.163:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.163:0","nonce":2571244897}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.163:0","nonce":3890775846}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.163:0","nonce":1591817392}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.163:0","nonce":1334591766}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.163:0","nonce":3453855427}]}]} 2024-03-21T12:59:10.992 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-21T12:59:10.992 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-21T12:59:10.992 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T12:59:11.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:59:11.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:59:11.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1201040775' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T12:59:11.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:59:11.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T12:59:11.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[40675]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-03-21T12:59:11.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:59:11.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:59:11.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:59:11.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1201040775' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T12:59:11.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:59:11.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T12:59:11.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[29621]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-03-21T12:59:11.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:59:11.250 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:11.407 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:59:11.407 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:59:11.407 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:11 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1201040775' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T12:59:11.407 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:59:11.407 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T12:59:11.407 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:11 smithi204 ceph-mon[33153]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-03-21T12:59:11.407 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:11 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' 2024-03-21T12:59:11.407 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: ignoring --setuser ceph since I am not root 2024-03-21T12:59:11.407 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: ignoring --setgroup ceph since I am not root 2024-03-21T12:59:11.408 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:11 smithi204 ceph-mgr[35013]: -- 172.21.15.204:0/916810372 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d27ad565a0 con 0x55d27ad2ac00 2024-03-21T12:59:11.510 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ignoring --setuser ceph since I am not root 2024-03-21T12:59:11.511 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ignoring --setgroup ceph since I am not root 2024-03-21T12:59:12.072 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:11.796+0000 7fa36a03b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:59:12.072 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:11 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:11.895+0000 7fa36a03b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:59:12.202 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:11.765+0000 7f81cd4fe200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:59:12.202 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:11.860+0000 7f81cd4fe200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:59:12.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:12 smithi204 ceph-mon[33153]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T12:59:12.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:12 smithi204 ceph-mon[33153]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-21T12:59:12.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:12 smithi163 ceph-mon[29621]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T12:59:12.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:12 smithi163 ceph-mon[29621]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-21T12:59:12.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:12 smithi163 ceph-mon[40675]: from='mgr.14150 172.21.15.163:0/1205785344' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T12:59:12.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:12 smithi163 ceph-mon[40675]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-21T12:59:12.850 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:12.505+0000 7f81cd4fe200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:59:12.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:12.601+0000 7f81cd4fe200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:59:12.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:12.751+0000 7f81cd4fe200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:59:12.898 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:12.554+0000 7fa36a03b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:59:12.898 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:12.654+0000 7fa36a03b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:59:12.898 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:12.800+0000 7fa36a03b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:59:12.898 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:12.897+0000 7fa36a03b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:59:13.187 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:12.987+0000 7fa36a03b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:59:13.215 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:12.854+0000 7f81cd4fe200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:59:13.215 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:12.942+0000 7f81cd4fe200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:59:13.215 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:13.213+0000 7f81cd4fe200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:59:13.406 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:59:13.406 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":78,"fsid":"d4bf79c0-e781-11ee-95cd-87774f69a715","created":"2024-03-21T12:52:56.031419+0000","modified":"2024-03-21T12:59:06.152537+0000","last_up_change":"2024-03-21T12:58:23.775302+0000","last_in_change":"2024-03-21T12:58:05.763916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T12:56:20.557490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-21T12:58:35.713383+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":67,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-21T12:58:36.436384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-21T12:58:38.049304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-21T12:58:39.990512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-21T12:58:41.463686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c34a3b25-9e02-46e6-8f9a-d0af2615258f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6801","nonce":3086410981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3086410981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3086410981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6803","nonce":3086410981}]},"public_addr":"172.21.15.163:6801/3086410981","cluster_addr":"172.21.15.163:6802/3086410981","heartbeat_back_addr":"172.21.15.163:6804/3086410981","heartbeat_front_addr":"172.21.15.163:6803/3086410981","state":["exists","up"]},{"osd":1,"uuid":"36e05f89-c238-41f7-be65-1553b529e8ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6805","nonce":3666062784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3666062784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3666062784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6807","nonce":3666062784}]},"public_addr":"172.21.15.163:6805/3666062784","cluster_addr":"172.21.15.163:6806/3666062784","heartbeat_back_addr":"172.21.15.163:6808/3666062784","heartbeat_front_addr":"172.21.15.163:6807/3666062784","state":["exists","up"]},{"osd":2,"uuid":"968ec78d-25b8-4905-b328-d490c38ca85f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6809","nonce":4009213507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4009213507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4009213507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":4009213507}]},"public_addr":"172.21.15.163:6809/4009213507","cluster_addr":"172.21.15.163:6810/4009213507","heartbeat_back_addr":"172.21.15.163:6812/4009213507","heartbeat_front_addr":"172.21.15.163:6811/4009213507","state":["exists","up"]},{"osd":3,"uuid":"67dfa4f6-1709-48f6-971d-c6287233492d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6813","nonce":3981878189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3981878189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":3981878189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":3981878189}]},"public_addr":"172.21.15.163:6813/3981878189","cluster_addr":"172.21.15.163:6814/3981878189","heartbeat_back_addr":"172.21.15.163:6816/3981878189","heartbeat_front_addr":"172.21.15.163:6815/3981878189","state":["exists","up"]},{"osd":4,"uuid":"7fd311a9-f3c2-456d-8ab1-341cf5b33471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":719276054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":719276054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":719276054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":719276054}]},"public_addr":"172.21.15.204:6800/719276054","cluster_addr":"172.21.15.204:6801/719276054","heartbeat_back_addr":"172.21.15.204:6803/719276054","heartbeat_front_addr":"172.21.15.204:6802/719276054","state":["exists","up"]},{"osd":5,"uuid":"a87759c8-251c-415d-9387-a49f7d2f36d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1346947668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":1346947668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":1346947668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1346947668}]},"public_addr":"172.21.15.204:6804/1346947668","cluster_addr":"172.21.15.204:6805/1346947668","heartbeat_back_addr":"172.21.15.204:6807/1346947668","heartbeat_front_addr":"172.21.15.204:6806/1346947668","state":["exists","up"]},{"osd":6,"uuid":"5eb135cd-f677-4d0c-b23a-738f31615b6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":200077988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":200077988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":200077988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":200077988}]},"public_addr":"172.21.15.204:6808/200077988","cluster_addr":"172.21.15.204:6809/200077988","heartbeat_back_addr":"172.21.15.204:6811/200077988","heartbeat_front_addr":"172.21.15.204:6810/200077988","state":["exists","up"]},{"osd":7,"uuid":"4f528e10-2c93-4bc2-a54c-06ca4132ed49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":315664888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":315664888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":315664888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":315664888}]},"public_addr":"172.21.15.204:6812/315664888","cluster_addr":"172.21.15.204:6813/315664888","heartbeat_back_addr":"172.21.15.204:6815/315664888","heartbeat_front_addr":"172.21.15.204:6814/315664888","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:05.391289+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:16.427153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:35.330043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:59:04.471096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:33.977577+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:51.890964+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:57:56.590024+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:21.966326+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:0/116934309":"2024-03-22T12:54:00.224135+0000","172.21.15.163:0/1783921524":"2024-03-22T12:54:00.224135+0000","172.21.15.163:6800/2553440211":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/1812694077":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/1070540283":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/2354157786":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/4152198951":"2024-03-22T12:54:00.224135+0000","172.21.15.163:6800/1798115633":"2024-03-22T12:54:00.224135+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T12:59:13.457 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:13 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:13.264+0000 7fa36a03b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:59:13.457 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:13 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:13.378+0000 7fa36a03b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:59:13.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:13 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3441649057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:59:13.502 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:13.326+0000 7f81cd4fe200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:59:13.502 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:13.414+0000 7f81cd4fe200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:59:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:13 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3441649057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:59:13.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:13 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3441649057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:59:13.750 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:13 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:13.471+0000 7fa36a03b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:59:13.750 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:13 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:13.568+0000 7fa36a03b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:59:14.002 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:13.513+0000 7f81cd4fe200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:59:14.002 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:13.694+0000 7f81cd4fe200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:59:14.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:13 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:13.748+0000 7fa36a03b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:59:14.096 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-21T12:59:14.097 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T12:59:14.368 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:15.002 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:14.656+0000 7f81cd4fe200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:59:15.002 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:14.788+0000 7f81cd4fe200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:59:15.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:14 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:14.771+0000 7fa36a03b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:59:15.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:14 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:14.905+0000 7fa36a03b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:59:15.418 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:15.126+0000 7f81cd4fe200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:59:15.418 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:15.218+0000 7f81cd4fe200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:59:15.418 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:15.416+0000 7f81cd4fe200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:59:15.511 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:15.251+0000 7fa36a03b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:59:15.511 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:15.344+0000 7fa36a03b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:59:15.752 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:15.602+0000 7f81cd4fe200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:59:15.895 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:15.547+0000 7fa36a03b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:59:15.895 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:15.750+0000 7fa36a03b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:59:16.146 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:16 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:16.049+0000 7fa36a03b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:59:16.153 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:15.889+0000 7f81cd4fe200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:59:16.383 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T12:59:16.383 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":78,"fsid":"d4bf79c0-e781-11ee-95cd-87774f69a715","created":"2024-03-21T12:52:56.031419+0000","modified":"2024-03-21T12:59:06.152537+0000","last_up_change":"2024-03-21T12:58:23.775302+0000","last_in_change":"2024-03-21T12:58:05.763916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T12:56:20.557490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-21T12:58:35.713383+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":67,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-21T12:58:36.436384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-21T12:58:38.049304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-21T12:58:39.990512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-21T12:58:41.463686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c34a3b25-9e02-46e6-8f9a-d0af2615258f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6801","nonce":3086410981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3086410981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3086410981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6803","nonce":3086410981}]},"public_addr":"172.21.15.163:6801/3086410981","cluster_addr":"172.21.15.163:6802/3086410981","heartbeat_back_addr":"172.21.15.163:6804/3086410981","heartbeat_front_addr":"172.21.15.163:6803/3086410981","state":["exists","up"]},{"osd":1,"uuid":"36e05f89-c238-41f7-be65-1553b529e8ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6805","nonce":3666062784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3666062784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3666062784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6807","nonce":3666062784}]},"public_addr":"172.21.15.163:6805/3666062784","cluster_addr":"172.21.15.163:6806/3666062784","heartbeat_back_addr":"172.21.15.163:6808/3666062784","heartbeat_front_addr":"172.21.15.163:6807/3666062784","state":["exists","up"]},{"osd":2,"uuid":"968ec78d-25b8-4905-b328-d490c38ca85f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6809","nonce":4009213507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4009213507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4009213507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":4009213507}]},"public_addr":"172.21.15.163:6809/4009213507","cluster_addr":"172.21.15.163:6810/4009213507","heartbeat_back_addr":"172.21.15.163:6812/4009213507","heartbeat_front_addr":"172.21.15.163:6811/4009213507","state":["exists","up"]},{"osd":3,"uuid":"67dfa4f6-1709-48f6-971d-c6287233492d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6813","nonce":3981878189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3981878189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":3981878189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":3981878189}]},"public_addr":"172.21.15.163:6813/3981878189","cluster_addr":"172.21.15.163:6814/3981878189","heartbeat_back_addr":"172.21.15.163:6816/3981878189","heartbeat_front_addr":"172.21.15.163:6815/3981878189","state":["exists","up"]},{"osd":4,"uuid":"7fd311a9-f3c2-456d-8ab1-341cf5b33471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":719276054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":719276054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":719276054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":719276054}]},"public_addr":"172.21.15.204:6800/719276054","cluster_addr":"172.21.15.204:6801/719276054","heartbeat_back_addr":"172.21.15.204:6803/719276054","heartbeat_front_addr":"172.21.15.204:6802/719276054","state":["exists","up"]},{"osd":5,"uuid":"a87759c8-251c-415d-9387-a49f7d2f36d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1346947668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":1346947668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":1346947668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1346947668}]},"public_addr":"172.21.15.204:6804/1346947668","cluster_addr":"172.21.15.204:6805/1346947668","heartbeat_back_addr":"172.21.15.204:6807/1346947668","heartbeat_front_addr":"172.21.15.204:6806/1346947668","state":["exists","up"]},{"osd":6,"uuid":"5eb135cd-f677-4d0c-b23a-738f31615b6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":200077988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":200077988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":200077988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":200077988}]},"public_addr":"172.21.15.204:6808/200077988","cluster_addr":"172.21.15.204:6809/200077988","heartbeat_back_addr":"172.21.15.204:6811/200077988","heartbeat_front_addr":"172.21.15.204:6810/200077988","state":["exists","up"]},{"osd":7,"uuid":"4f528e10-2c93-4bc2-a54c-06ca4132ed49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":315664888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":315664888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":315664888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":315664888}]},"public_addr":"172.21.15.204:6812/315664888","cluster_addr":"172.21.15.204:6813/315664888","heartbeat_back_addr":"172.21.15.204:6815/315664888","heartbeat_front_addr":"172.21.15.204:6814/315664888","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:05.391289+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:16.427153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:35.330043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:59:04.471096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:33.977577+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:51.890964+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:57:56.590024+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:58:21.966326+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:0/116934309":"2024-03-22T12:54:00.224135+0000","172.21.15.163:0/1783921524":"2024-03-22T12:54:00.224135+0000","172.21.15.163:6800/2553440211":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/1812694077":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/1070540283":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/2354157786":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/4152198951":"2024-03-22T12:54:00.224135+0000","172.21.15.163:6800/1798115633":"2024-03-22T12:54:00.224135+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T12:59:16.397 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:16 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:16.321+0000 7fa36a03b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:59:16.434 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:16 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:16.151+0000 7f81cd4fe200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:59:16.434 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:16 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:16.392+0000 7f81cd4fe200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:59:16.691 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:16 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1120557079' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:59:16.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:16 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1120557079' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:59:16.692 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:16 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:16.571+0000 7fa36a03b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:59:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:16 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1120557079' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:59:17.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-21T12:59:17.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-21T12:59:17.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-21T12:59:17.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-21T12:59:17.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-21T12:59:17.115 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-21T12:59:17.115 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-21T12:59:17.116 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-21T12:59:17.502 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:17 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:17.202+0000 7f81cd4fe200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:59:17.638 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:17.641 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:17.644 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:17.644 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:17.645 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:17.649 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:17.667 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:17.673 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:17.965 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:17 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:17.631+0000 7fa36a03b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:59:18.099 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:17 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:17.781+0000 7f81cd4fe200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:59:18.099 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:17 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:17.880+0000 7f81cd4fe200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:59:18.100 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:18 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:18.099+0000 7f81cd4fe200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:59:18.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-mon[40675]: Standby manager daemon x restarted 2024-03-21T12:59:18.245 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-mon[40675]: Standby manager daemon x started 2024-03-21T12:59:18.245 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-mon[40675]: from='mgr.? 172.21.15.204:0/3659181327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:59:18.245 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-mon[40675]: from='mgr.? 172.21.15.204:0/3659181327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:59:18.245 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-mon[40675]: from='mgr.? 172.21.15.204:0/3659181327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:59:18.245 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-mon[40675]: from='mgr.? 172.21.15.204:0/3659181327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:59:18.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-mon[29621]: Standby manager daemon x restarted 2024-03-21T12:59:18.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-mon[29621]: Standby manager daemon x started 2024-03-21T12:59:18.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-mon[29621]: from='mgr.? 172.21.15.204:0/3659181327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:59:18.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-mon[29621]: from='mgr.? 172.21.15.204:0/3659181327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:59:18.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-mon[29621]: from='mgr.? 172.21.15.204:0/3659181327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:59:18.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-mon[29621]: from='mgr.? 172.21.15.204:0/3659181327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:59:18.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:18 smithi204 ceph-mon[33153]: Standby manager daemon x restarted 2024-03-21T12:59:18.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:18 smithi204 ceph-mon[33153]: Standby manager daemon x started 2024-03-21T12:59:18.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:18 smithi204 ceph-mon[33153]: from='mgr.? 172.21.15.204:0/3659181327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:59:18.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:18 smithi204 ceph-mon[33153]: from='mgr.? 172.21.15.204:0/3659181327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:59:18.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:18 smithi204 ceph-mon[33153]: from='mgr.? 172.21.15.204:0/3659181327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:59:18.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:18 smithi204 ceph-mon[33153]: from='mgr.? 172.21.15.204:0/3659181327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:59:18.503 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:18 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: [21/Mar/2024:12:59:18] ENGINE Bus STARTING 2024-03-21T12:59:18.503 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:18 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: CherryPy Checker: 2024-03-21T12:59:18.503 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:18 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: The Application mounted at '' has an empty config. 2024-03-21T12:59:18.503 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:18 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: 2024-03-21T12:59:18.503 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:18 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: [21/Mar/2024:12:59:18] ENGINE Serving on http://:::9283 2024-03-21T12:59:18.503 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 12:59:18 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x[35009]: [21/Mar/2024:12:59:18] ENGINE Bus STARTED 2024-03-21T12:59:18.510 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:18.256+0000 7fa36a03b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:59:18.510 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:18.359+0000 7fa36a03b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:59:19.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:18.586+0000 7fa36a03b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:59:19.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:12:59:18] ENGINE Bus STARTING 2024-03-21T12:59:19.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: CherryPy Checker: 2024-03-21T12:59:19.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: The Application mounted at '' has an empty config. 2024-03-21T12:59:19.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T12:59:19.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:12:59:18] ENGINE Serving on http://:::9283 2024-03-21T12:59:19.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:12:59:18] ENGINE Bus STARTED 2024-03-21T12:59:19.426 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-21T12:59:19.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: Active manager daemon y restarted 2024-03-21T12:59:19.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: Activating manager daemon y 2024-03-21T12:59:19.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T12:59:19.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: mgrmap e19: y(active, starting, since 0.00903911s), standbys: x 2024-03-21T12:59:19.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:59:19.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:59:19.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:59:19.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:59:19.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:59:19.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:59:19.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:59:19.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:59:19.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:59:19.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:59:19.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:59:19.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:59:19.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:59:19.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:59:19.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:59:19.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:59:19.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: Manager daemon y is now available 2024-03-21T12:59:19.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:19.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:19.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:19.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:59:19.429 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:59:19.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-21T12:59:19.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: Active manager daemon y restarted 2024-03-21T12:59:19.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: Activating manager daemon y 2024-03-21T12:59:19.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T12:59:19.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: mgrmap e19: y(active, starting, since 0.00903911s), standbys: x 2024-03-21T12:59:19.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:59:19.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:59:19.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:59:19.430 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:59:19.430 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:59:19.430 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:59:19.430 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:59:19.430 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:59:19.430 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:59:19.430 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:59:19.430 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:59:19.430 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:59:19.430 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:59:19.430 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:59:19.430 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:59:19.431 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:59:19.431 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: Manager daemon y is now available 2024-03-21T12:59:19.431 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:19.431 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:19.431 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:19.431 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:59:19.431 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:59:19.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-21T12:59:19.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: Active manager daemon y restarted 2024-03-21T12:59:19.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: Activating manager daemon y 2024-03-21T12:59:19.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T12:59:19.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: mgrmap e19: y(active, starting, since 0.00903911s), standbys: x 2024-03-21T12:59:19.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:59:19.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:59:19.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:59:19.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:59:19.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: Manager daemon y is now available 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:19.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:19.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:59:19.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:59:20.895 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:20 smithi163 ceph-mon[29621]: [21/Mar/2024:12:59:19] ENGINE Bus STARTING 2024-03-21T12:59:20.895 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:20 smithi163 ceph-mon[29621]: [21/Mar/2024:12:59:19] ENGINE Serving on http://172.21.15.163:8765 2024-03-21T12:59:20.895 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:20 smithi163 ceph-mon[29621]: [21/Mar/2024:12:59:19] ENGINE Serving on https://172.21.15.163:7150 2024-03-21T12:59:20.895 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:20 smithi163 ceph-mon[29621]: [21/Mar/2024:12:59:19] ENGINE Bus STARTED 2024-03-21T12:59:20.895 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:20 smithi163 ceph-mon[29621]: mgrmap e20: y(active, since 1.01341s), standbys: x 2024-03-21T12:59:20.895 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:20 smithi163 ceph-mon[29621]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:59:20.895 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:20 smithi163 ceph-mon[40675]: [21/Mar/2024:12:59:19] ENGINE Bus STARTING 2024-03-21T12:59:20.895 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:20 smithi163 ceph-mon[40675]: [21/Mar/2024:12:59:19] ENGINE Serving on http://172.21.15.163:8765 2024-03-21T12:59:20.896 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:20 smithi163 ceph-mon[40675]: [21/Mar/2024:12:59:19] ENGINE Serving on https://172.21.15.163:7150 2024-03-21T12:59:20.896 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:20 smithi163 ceph-mon[40675]: [21/Mar/2024:12:59:19] ENGINE Bus STARTED 2024-03-21T12:59:20.896 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:20 smithi163 ceph-mon[40675]: mgrmap e20: y(active, since 1.01341s), standbys: x 2024-03-21T12:59:20.896 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:20 smithi163 ceph-mon[40675]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:59:21.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:20 smithi204 ceph-mon[33153]: [21/Mar/2024:12:59:19] ENGINE Bus STARTING 2024-03-21T12:59:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:20 smithi204 ceph-mon[33153]: [21/Mar/2024:12:59:19] ENGINE Serving on http://172.21.15.163:8765 2024-03-21T12:59:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:20 smithi204 ceph-mon[33153]: [21/Mar/2024:12:59:19] ENGINE Serving on https://172.21.15.163:7150 2024-03-21T12:59:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:20 smithi204 ceph-mon[33153]: [21/Mar/2024:12:59:19] ENGINE Bus STARTED 2024-03-21T12:59:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:20 smithi204 ceph-mon[33153]: mgrmap e20: y(active, since 1.01341s), standbys: x 2024-03-21T12:59:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:20 smithi204 ceph-mon[33153]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:59:21.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:21 smithi163 ceph-mon[29621]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:59:21.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:21 smithi163 ceph-mon[29621]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-21T12:59:21.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:21 smithi163 ceph-mon[29621]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T12:59:21.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:21 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:21.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:21 smithi163 ceph-mon[40675]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:59:21.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:21 smithi163 ceph-mon[40675]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-21T12:59:21.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:21 smithi163 ceph-mon[40675]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T12:59:21.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:21 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:22.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:21 smithi204 ceph-mon[33153]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:59:22.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:21 smithi204 ceph-mon[33153]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-21T12:59:22.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:21 smithi204 ceph-mon[33153]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T12:59:22.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:21 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:23.055 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:22 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:23.055 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:22 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:23.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:22 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:23.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:22 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:23.112 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:22 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:23.112 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:22 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:23 smithi204 ceph-mon[33153]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:59:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:23 smithi204 ceph-mon[33153]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-21T12:59:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:23 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:23 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:23 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:23 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:23 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:23 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:23 smithi204 ceph-mon[33153]: Adjusting osd_memory_target on smithi204 to 3760M 2024-03-21T12:59:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:23 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:24.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[29621]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:59:24.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[29621]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-21T12:59:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[29621]: Adjusting osd_memory_target on smithi204 to 3760M 2024-03-21T12:59:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[40675]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:59:24.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[40675]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-21T12:59:24.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:24.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:24.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:24.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:24.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:24.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:24.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[40675]: Adjusting osd_memory_target on smithi204 to 3760M 2024-03-21T12:59:24.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:23 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:24.644 INFO:teuthology.orchestra.run.smithi163.stdout:34359738417 2024-03-21T12:59:24.645 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-21T12:59:25.932 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:25 smithi163 ceph-mon[40675]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:59:25.932 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:25 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:59:25.932 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:25 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:59:25.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:25 smithi163 ceph-mon[29621]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:59:25.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:25 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:59:25.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:25 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:59:25.933 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:12:59:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T12:59:26.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:25 smithi204 ceph-mon[33153]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:59:26.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:25 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:59:26.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:25 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:59:27.005 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:26 smithi163 ceph-mon[40675]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T12:59:27.005 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:26 smithi163 ceph-mon[40675]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T12:59:27.005 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:26 smithi163 ceph-mon[29621]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T12:59:27.005 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:26 smithi163 ceph-mon[29621]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T12:59:27.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:26 smithi204 ceph-mon[33153]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T12:59:27.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:26 smithi204 ceph-mon[33153]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T12:59:28.436 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:28 smithi163 ceph-mon[29621]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-21T12:59:28.436 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:28 smithi163 ceph-mon[40675]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-21T12:59:28.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:28 smithi204 ceph-mon[33153]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-21T12:59:29.777 INFO:teuthology.orchestra.run.smithi163.stdout:128849018909 2024-03-21T12:59:29.777 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-21T12:59:30.240 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:30 smithi163 ceph-mon[29621]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-21T12:59:30.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:30 smithi163 ceph-mon[40675]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-21T12:59:30.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:30 smithi204 ceph-mon[33153]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-21T12:59:30.948 INFO:teuthology.orchestra.run.smithi163.stdout:240518168590 2024-03-21T12:59:30.949 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-21T12:59:31.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 12:59:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T12:59:31.843 INFO:teuthology.orchestra.run.smithi163.stdout:55834574893 2024-03-21T12:59:31.844 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-21T12:59:32.454 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:32 smithi163 ceph-mon[40675]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-21T12:59:32.454 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:32 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:59:32.454 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:32 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:59:32.454 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:32 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:32.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:32 smithi163 ceph-mon[40675]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T12:59:32.455 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:32 smithi163 ceph-mon[29621]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-21T12:59:32.455 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:32 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:59:32.455 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:32 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:59:32.455 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:32 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:32.455 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:32 smithi163 ceph-mon[29621]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T12:59:32.462 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:32.488 INFO:teuthology.orchestra.run.smithi163.stdout:107374182436 2024-03-21T12:59:32.488 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-21T12:59:32.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:32 smithi204 ceph-mon[33153]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-21T12:59:32.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:32 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:59:32.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:32 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:59:32.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:32 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:32.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:32 smithi204 ceph-mon[33153]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T12:59:34.081 INFO:teuthology.orchestra.run.smithi163.stdout:167503724568 2024-03-21T12:59:34.081 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-21T12:59:34.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:34 smithi204 ceph-mon[33153]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 750 B/s rd, 0 op/s 2024-03-21T12:59:34.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:34 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:34.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:34 smithi163 ceph-mon[29621]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 750 B/s rd, 0 op/s 2024-03-21T12:59:34.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:34 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:34.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:34 smithi163 ceph-mon[40675]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 750 B/s rd, 0 op/s 2024-03-21T12:59:34.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:34 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:35.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:12:59:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T12:59:35.810 INFO:teuthology.orchestra.run.smithi163.stdout:77309411369 2024-03-21T12:59:35.810 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-21T12:59:36.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:36 smithi163 ceph-mon[29621]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-03-21T12:59:36.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:36 smithi163 ceph-mon[29621]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T12:59:36.470 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:36 smithi163 ceph-mon[40675]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-03-21T12:59:36.470 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:36 smithi163 ceph-mon[40675]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T12:59:36.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:36 smithi204 ceph-mon[33153]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-03-21T12:59:36.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:36 smithi204 ceph-mon[33153]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T12:59:37.425 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:37 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:59:37.426 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:37 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:59:37.426 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:37 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:59:37.426 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:37 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:59:37.451 INFO:teuthology.orchestra.run.smithi163.stdout:201863462934 2024-03-21T12:59:37.451 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-21T12:59:37.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:37 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:59:37.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:37 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:59:37.655 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:37.656 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:37.657 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:38.160 INFO:teuthology.orchestra.run.smithi163.stdout:34359738420 2024-03-21T12:59:38.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:38 smithi163 ceph-mon[29621]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:38.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:38 smithi163 ceph-mon[29621]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T12:59:38.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:38 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2176930439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T12:59:38.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:38 smithi163 ceph-mon[40675]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:38.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:38 smithi163 ceph-mon[40675]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T12:59:38.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:38 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2176930439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T12:59:38.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:38 smithi204 ceph-mon[33153]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:38.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:38 smithi204 ceph-mon[33153]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T12:59:38.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:38 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2176930439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T12:59:40.369 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:40 smithi163 ceph-mon[29621]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 688 B/s rd, 0 op/s 2024-03-21T12:59:40.369 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:40 smithi163 ceph-mon[40675]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 688 B/s rd, 0 op/s 2024-03-21T12:59:40.405 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738420 for osd.0 2024-03-21T12:59:40.406 DEBUG:teuthology.parallel:result is None 2024-03-21T12:59:40.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:40 smithi204 ceph-mon[33153]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 688 B/s rd, 0 op/s 2024-03-21T12:59:41.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 12:59:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T12:59:41.639 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:42.045 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:42.416 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:42 smithi204 ceph-mon[33153]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:42.417 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:42 smithi204 ceph-mon[33153]: 5.1a scrub starts 2024-03-21T12:59:42.417 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:42 smithi204 ceph-mon[33153]: 5.1a scrub ok 2024-03-21T12:59:42.417 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:42 smithi204 ceph-mon[33153]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T12:59:42.417 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:42 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:42.424 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:42 smithi163 ceph-mon[29621]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:42.424 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:42 smithi163 ceph-mon[29621]: 5.1a scrub starts 2024-03-21T12:59:42.425 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:42 smithi163 ceph-mon[29621]: 5.1a scrub ok 2024-03-21T12:59:42.425 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:42 smithi163 ceph-mon[29621]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T12:59:42.425 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:42 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:42.425 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:42 smithi163 ceph-mon[40675]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:42.425 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:42 smithi163 ceph-mon[40675]: 5.1a scrub starts 2024-03-21T12:59:42.425 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:42 smithi163 ceph-mon[40675]: 5.1a scrub ok 2024-03-21T12:59:42.425 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:42 smithi163 ceph-mon[40675]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T12:59:42.425 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:42 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:43.459 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:43 smithi163 ceph-mon[29621]: 3.8 scrub starts 2024-03-21T12:59:43.460 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:43 smithi163 ceph-mon[29621]: 3.8 scrub ok 2024-03-21T12:59:43.460 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:43 smithi163 ceph-mon[40675]: 3.8 scrub starts 2024-03-21T12:59:43.460 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:43 smithi163 ceph-mon[40675]: 3.8 scrub ok 2024-03-21T12:59:43.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:43 smithi204 ceph-mon[33153]: 3.8 scrub starts 2024-03-21T12:59:43.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:43 smithi204 ceph-mon[33153]: 3.8 scrub ok 2024-03-21T12:59:44.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:44 smithi163 ceph-mon[29621]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-21T12:59:44.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:44 smithi163 ceph-mon[40675]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-21T12:59:44.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:44 smithi204 ceph-mon[33153]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-21T12:59:45.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:45 smithi163 ceph-mon[29621]: 3.7 deep-scrub starts 2024-03-21T12:59:45.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:45 smithi163 ceph-mon[29621]: 3.7 deep-scrub ok 2024-03-21T12:59:45.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:45 smithi163 ceph-mon[29621]: 5.12 scrub starts 2024-03-21T12:59:45.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:45 smithi163 ceph-mon[29621]: 5.12 scrub ok 2024-03-21T12:59:45.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:45 smithi163 ceph-mon[40675]: 3.7 deep-scrub starts 2024-03-21T12:59:45.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:45 smithi163 ceph-mon[40675]: 3.7 deep-scrub ok 2024-03-21T12:59:45.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:45 smithi163 ceph-mon[40675]: 5.12 scrub starts 2024-03-21T12:59:45.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:45 smithi163 ceph-mon[40675]: 5.12 scrub ok 2024-03-21T12:59:45.310 INFO:teuthology.orchestra.run.smithi163.stdout:201863462936 2024-03-21T12:59:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:45 smithi204 ceph-mon[33153]: 3.7 deep-scrub starts 2024-03-21T12:59:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:45 smithi204 ceph-mon[33153]: 3.7 deep-scrub ok 2024-03-21T12:59:45.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:45 smithi204 ceph-mon[33153]: 5.12 scrub starts 2024-03-21T12:59:45.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:45 smithi204 ceph-mon[33153]: 5.12 scrub ok 2024-03-21T12:59:45.855 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:12:59:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T12:59:46.132 INFO:teuthology.orchestra.run.smithi163.stdout:167503724572 2024-03-21T12:59:46.348 INFO:teuthology.orchestra.run.smithi163.stdout:55834574896 2024-03-21T12:59:46.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:46 smithi204 ceph-mon[33153]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:59:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:46 smithi204 ceph-mon[33153]: 3.1d scrub starts 2024-03-21T12:59:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:46 smithi204 ceph-mon[33153]: 3.1d scrub ok 2024-03-21T12:59:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:46 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1088235169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T12:59:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:46 smithi204 ceph-mon[33153]: 4.2 deep-scrub starts 2024-03-21T12:59:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:46 smithi204 ceph-mon[33153]: 4.2 deep-scrub ok 2024-03-21T12:59:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:46 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2638322710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T12:59:46.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[29621]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:59:46.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[29621]: 3.1d scrub starts 2024-03-21T12:59:46.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[29621]: 3.1d scrub ok 2024-03-21T12:59:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1088235169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T12:59:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[29621]: 4.2 deep-scrub starts 2024-03-21T12:59:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[29621]: 4.2 deep-scrub ok 2024-03-21T12:59:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2638322710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T12:59:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[40675]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:59:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[40675]: 3.1d scrub starts 2024-03-21T12:59:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[40675]: 3.1d scrub ok 2024-03-21T12:59:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1088235169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T12:59:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[40675]: 4.2 deep-scrub starts 2024-03-21T12:59:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[40675]: 4.2 deep-scrub ok 2024-03-21T12:59:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:46 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2638322710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T12:59:47.413 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[40675]: 3.14 deep-scrub starts 2024-03-21T12:59:47.413 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[40675]: 3.14 deep-scrub ok 2024-03-21T12:59:47.414 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[40675]: 3.d scrub starts 2024-03-21T12:59:47.414 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[40675]: 3.d scrub ok 2024-03-21T12:59:47.414 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[40675]: 4.17 scrub starts 2024-03-21T12:59:47.414 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[40675]: 4.17 scrub ok 2024-03-21T12:59:47.414 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2376171060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T12:59:47.414 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[40675]: 4.b deep-scrub starts 2024-03-21T12:59:47.414 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[40675]: 4.b deep-scrub ok 2024-03-21T12:59:47.414 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[29621]: 3.14 deep-scrub starts 2024-03-21T12:59:47.414 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[29621]: 3.14 deep-scrub ok 2024-03-21T12:59:47.414 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[29621]: 3.d scrub starts 2024-03-21T12:59:47.415 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[29621]: 3.d scrub ok 2024-03-21T12:59:47.415 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[29621]: 4.17 scrub starts 2024-03-21T12:59:47.415 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[29621]: 4.17 scrub ok 2024-03-21T12:59:47.415 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2376171060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T12:59:47.415 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[29621]: 4.b deep-scrub starts 2024-03-21T12:59:47.415 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:47 smithi163 ceph-mon[29621]: 4.b deep-scrub ok 2024-03-21T12:59:47.439 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724568 got 167503724572 for osd.5 2024-03-21T12:59:47.439 DEBUG:teuthology.parallel:result is None 2024-03-21T12:59:47.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:47 smithi204 ceph-mon[33153]: 3.14 deep-scrub starts 2024-03-21T12:59:47.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:47 smithi204 ceph-mon[33153]: 3.14 deep-scrub ok 2024-03-21T12:59:47.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:47 smithi204 ceph-mon[33153]: 3.d scrub starts 2024-03-21T12:59:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:47 smithi204 ceph-mon[33153]: 3.d scrub ok 2024-03-21T12:59:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:47 smithi204 ceph-mon[33153]: 4.17 scrub starts 2024-03-21T12:59:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:47 smithi204 ceph-mon[33153]: 4.17 scrub ok 2024-03-21T12:59:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:47 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2376171060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T12:59:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:47 smithi204 ceph-mon[33153]: 4.b deep-scrub starts 2024-03-21T12:59:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:47 smithi204 ceph-mon[33153]: 4.b deep-scrub ok 2024-03-21T12:59:47.532 INFO:teuthology.orchestra.run.smithi163.stdout:128849018913 2024-03-21T12:59:48.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:48 smithi163 ceph-mon[29621]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:48.422 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:48 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1222655197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T12:59:48.422 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:48 smithi163 ceph-mon[29621]: 3.11 scrub starts 2024-03-21T12:59:48.422 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:48 smithi163 ceph-mon[29621]: 3.11 scrub ok 2024-03-21T12:59:48.422 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:48 smithi163 ceph-mon[29621]: 4.12 deep-scrub starts 2024-03-21T12:59:48.422 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:48 smithi163 ceph-mon[29621]: 4.12 deep-scrub ok 2024-03-21T12:59:48.422 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:48 smithi163 ceph-mon[40675]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:48.422 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:48 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1222655197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T12:59:48.422 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:48 smithi163 ceph-mon[40675]: 3.11 scrub starts 2024-03-21T12:59:48.422 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:48 smithi163 ceph-mon[40675]: 3.11 scrub ok 2024-03-21T12:59:48.422 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:48 smithi163 ceph-mon[40675]: 4.12 deep-scrub starts 2024-03-21T12:59:48.423 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:48 smithi163 ceph-mon[40675]: 4.12 deep-scrub ok 2024-03-21T12:59:48.448 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018909 got 128849018913 for osd.4 2024-03-21T12:59:48.448 DEBUG:teuthology.parallel:result is None 2024-03-21T12:59:48.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:48 smithi204 ceph-mon[33153]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:48.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:48 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1222655197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T12:59:48.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:48 smithi204 ceph-mon[33153]: 3.11 scrub starts 2024-03-21T12:59:48.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:48 smithi204 ceph-mon[33153]: 3.11 scrub ok 2024-03-21T12:59:48.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:48 smithi204 ceph-mon[33153]: 4.12 deep-scrub starts 2024-03-21T12:59:48.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:48 smithi204 ceph-mon[33153]: 4.12 deep-scrub ok 2024-03-21T12:59:48.881 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:48.882 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:48.966 INFO:teuthology.orchestra.run.smithi163.stdout:107374182439 2024-03-21T12:59:49.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[29621]: 3.c scrub starts 2024-03-21T12:59:49.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[29621]: 3.c scrub ok 2024-03-21T12:59:49.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[29621]: 3.3 scrub starts 2024-03-21T12:59:49.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[29621]: 3.3 scrub ok 2024-03-21T12:59:49.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[29621]: 4.0 scrub starts 2024-03-21T12:59:49.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[29621]: 4.0 scrub ok 2024-03-21T12:59:49.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[29621]: 5.3 scrub starts 2024-03-21T12:59:49.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[29621]: 5.3 scrub ok 2024-03-21T12:59:49.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:49.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:49.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:49.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1470522510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T12:59:49.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[40675]: 3.c scrub starts 2024-03-21T12:59:49.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[40675]: 3.c scrub ok 2024-03-21T12:59:49.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[40675]: 3.3 scrub starts 2024-03-21T12:59:49.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[40675]: 3.3 scrub ok 2024-03-21T12:59:49.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[40675]: 4.0 scrub starts 2024-03-21T12:59:49.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[40675]: 4.0 scrub ok 2024-03-21T12:59:49.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[40675]: 5.3 scrub starts 2024-03-21T12:59:49.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[40675]: 5.3 scrub ok 2024-03-21T12:59:49.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:49.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:49.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:49.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:49 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1470522510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T12:59:49.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:49 smithi204 ceph-mon[33153]: 3.c scrub starts 2024-03-21T12:59:49.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:49 smithi204 ceph-mon[33153]: 3.c scrub ok 2024-03-21T12:59:49.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:49 smithi204 ceph-mon[33153]: 3.3 scrub starts 2024-03-21T12:59:49.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:49 smithi204 ceph-mon[33153]: 3.3 scrub ok 2024-03-21T12:59:49.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:49 smithi204 ceph-mon[33153]: 4.0 scrub starts 2024-03-21T12:59:49.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:49 smithi204 ceph-mon[33153]: 4.0 scrub ok 2024-03-21T12:59:49.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:49 smithi204 ceph-mon[33153]: 5.3 scrub starts 2024-03-21T12:59:49.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:49 smithi204 ceph-mon[33153]: 5.3 scrub ok 2024-03-21T12:59:49.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:49 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:49.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:49 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:49.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:49 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:49.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:49 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1470522510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T12:59:49.518 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574896 for osd.1 2024-03-21T12:59:49.518 DEBUG:teuthology.parallel:result is None 2024-03-21T12:59:49.971 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462934 got 201863462936 for osd.6 2024-03-21T12:59:49.971 DEBUG:teuthology.parallel:result is None 2024-03-21T12:59:50.192 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[40675]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:59:50.192 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[40675]: 3.18 scrub starts 2024-03-21T12:59:50.192 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[40675]: 3.18 scrub ok 2024-03-21T12:59:50.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:50 smithi204 ceph-mon[33153]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:59:50.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:50 smithi204 ceph-mon[33153]: 3.18 scrub starts 2024-03-21T12:59:50.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:50 smithi204 ceph-mon[33153]: 3.18 scrub ok 2024-03-21T12:59:50.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:50 smithi204 ceph-mon[33153]: 3.a scrub starts 2024-03-21T12:59:50.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:50 smithi204 ceph-mon[33153]: 3.a scrub ok 2024-03-21T12:59:50.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:50 smithi204 ceph-mon[33153]: 5.c scrub starts 2024-03-21T12:59:50.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:50 smithi204 ceph-mon[33153]: 5.c scrub ok 2024-03-21T12:59:50.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[29621]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:59:50.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[29621]: 3.18 scrub starts 2024-03-21T12:59:50.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[29621]: 3.18 scrub ok 2024-03-21T12:59:50.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[29621]: 3.a scrub starts 2024-03-21T12:59:50.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[29621]: 3.a scrub ok 2024-03-21T12:59:50.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[29621]: 5.c scrub starts 2024-03-21T12:59:50.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[29621]: 5.c scrub ok 2024-03-21T12:59:50.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[40675]: 3.a scrub starts 2024-03-21T12:59:50.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[40675]: 3.a scrub ok 2024-03-21T12:59:50.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[40675]: 5.c scrub starts 2024-03-21T12:59:50.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:50 smithi163 ceph-mon[40675]: 5.c scrub ok 2024-03-21T12:59:51.006 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182436 got 107374182439 for osd.3 2024-03-21T12:59:51.006 DEBUG:teuthology.parallel:result is None 2024-03-21T12:59:51.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T12:59:51.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: 6.12 scrub starts 2024-03-21T12:59:51.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: 6.12 scrub ok 2024-03-21T12:59:51.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:51.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:51.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:51.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:51.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: Adjusting osd_memory_target on smithi163 to 3760M 2024-03-21T12:59:51.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:51.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:59:51.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: Updating smithi163:/etc/ceph/ceph.conf 2024-03-21T12:59:51.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-21T12:59:51.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: 4.1e scrub starts 2024-03-21T12:59:51.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: 4.1e scrub ok 2024-03-21T12:59:51.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: Updating smithi163:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:59:51.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:59:51.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:59:51.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:59:51.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:59:51.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:59:51.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:59:51.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: Updating smithi163:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:59:51.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:59:51.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:51.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:51 smithi204 ceph-mon[33153]: Deploying daemon alertmanager.a on smithi163 2024-03-21T12:59:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: 6.12 scrub starts 2024-03-21T12:59:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: 6.12 scrub ok 2024-03-21T12:59:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: Adjusting osd_memory_target on smithi163 to 3760M 2024-03-21T12:59:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:59:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: Updating smithi163:/etc/ceph/ceph.conf 2024-03-21T12:59:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-21T12:59:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: 4.1e scrub starts 2024-03-21T12:59:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: 4.1e scrub ok 2024-03-21T12:59:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: Updating smithi163:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:59:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:59:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:59:51.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:59:51.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:59:51.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:59:51.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:59:51.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: Updating smithi163:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:59:51.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:59:51.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:51.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[29621]: Deploying daemon alertmanager.a on smithi163 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: 6.12 scrub starts 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: 6.12 scrub ok 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: Adjusting osd_memory_target on smithi163 to 3760M 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: Updating smithi163:/etc/ceph/ceph.conf 2024-03-21T12:59:51.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: 4.1e scrub starts 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: 4.1e scrub ok 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: Updating smithi163:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: Updating smithi163:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: Updating smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:51.766 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.766 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.766 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T12:59:51.766 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:51 smithi163 ceph-mon[40675]: Deploying daemon alertmanager.a on smithi163 2024-03-21T12:59:52.310 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[29621]: 5.15 scrub starts 2024-03-21T12:59:52.310 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[29621]: 5.15 scrub ok 2024-03-21T12:59:52.310 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[29621]: 5.1c scrub starts 2024-03-21T12:59:52.310 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[29621]: 5.1c scrub ok 2024-03-21T12:59:52.310 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:59:52.311 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:59:52.311 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[29621]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T12:59:52.619 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[40675]: 5.15 scrub starts 2024-03-21T12:59:52.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[40675]: 5.15 scrub ok 2024-03-21T12:59:52.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[40675]: 5.1c scrub starts 2024-03-21T12:59:52.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[40675]: 5.1c scrub ok 2024-03-21T12:59:52.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:59:52.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:59:52.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[40675]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T12:59:52.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[40675]: 4.11 scrub starts 2024-03-21T12:59:52.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[40675]: 4.11 scrub ok 2024-03-21T12:59:52.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[29621]: 4.11 scrub starts 2024-03-21T12:59:52.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:52 smithi163 ceph-mon[29621]: 4.11 scrub ok 2024-03-21T12:59:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:52 smithi204 ceph-mon[33153]: 5.15 scrub starts 2024-03-21T12:59:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:52 smithi204 ceph-mon[33153]: 5.15 scrub ok 2024-03-21T12:59:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:52 smithi204 ceph-mon[33153]: 5.1c scrub starts 2024-03-21T12:59:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:52 smithi204 ceph-mon[33153]: 5.1c scrub ok 2024-03-21T12:59:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:52 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:59:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:52 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:59:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:52 smithi204 ceph-mon[33153]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T12:59:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:52 smithi204 ceph-mon[33153]: 4.11 scrub starts 2024-03-21T12:59:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:52 smithi204 ceph-mon[33153]: 4.11 scrub ok 2024-03-21T12:59:53.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[40675]: 6.14 deep-scrub starts 2024-03-21T12:59:53.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[40675]: 6.14 deep-scrub ok 2024-03-21T12:59:53.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[40675]: 3.10 scrub starts 2024-03-21T12:59:53.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[40675]: 3.10 scrub ok 2024-03-21T12:59:53.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T12:59:53.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T12:59:53.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[40675]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:59:53.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[29621]: 6.14 deep-scrub starts 2024-03-21T12:59:53.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[29621]: 6.14 deep-scrub ok 2024-03-21T12:59:53.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[29621]: 3.10 scrub starts 2024-03-21T12:59:53.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[29621]: 3.10 scrub ok 2024-03-21T12:59:53.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T12:59:53.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T12:59:53.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:53 smithi163 ceph-mon[29621]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:59:53.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:53 smithi204 ceph-mon[33153]: 6.14 deep-scrub starts 2024-03-21T12:59:53.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:53 smithi204 ceph-mon[33153]: 6.14 deep-scrub ok 2024-03-21T12:59:53.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:53 smithi204 ceph-mon[33153]: 3.10 scrub starts 2024-03-21T12:59:53.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:53 smithi204 ceph-mon[33153]: 3.10 scrub ok 2024-03-21T12:59:53.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:53 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T12:59:53.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:53 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T12:59:53.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:53 smithi204 ceph-mon[33153]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:59:54.288 INFO:teuthology.orchestra.run.smithi163.stdout:77309411373 2024-03-21T12:59:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:54 smithi204 ceph-mon[33153]: 6.7 scrub starts 2024-03-21T12:59:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:54 smithi204 ceph-mon[33153]: 6.7 scrub ok 2024-03-21T12:59:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:54 smithi204 ceph-mon[33153]: 4.1 deep-scrub starts 2024-03-21T12:59:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:54 smithi204 ceph-mon[33153]: 4.1 deep-scrub ok 2024-03-21T12:59:54.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:54 smithi204 ceph-mon[33153]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T12:59:54.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:54 smithi204 ceph-mon[33153]: 3.1f scrub starts 2024-03-21T12:59:54.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:54 smithi204 ceph-mon[33153]: 3.1f scrub ok 2024-03-21T12:59:54.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:54 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2603718407' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T12:59:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[29621]: 6.7 scrub starts 2024-03-21T12:59:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[29621]: 6.7 scrub ok 2024-03-21T12:59:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[29621]: 4.1 deep-scrub starts 2024-03-21T12:59:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[29621]: 4.1 deep-scrub ok 2024-03-21T12:59:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[29621]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T12:59:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[29621]: 3.1f scrub starts 2024-03-21T12:59:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[29621]: 3.1f scrub ok 2024-03-21T12:59:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2603718407' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T12:59:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[40675]: 6.7 scrub starts 2024-03-21T12:59:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[40675]: 6.7 scrub ok 2024-03-21T12:59:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[40675]: 4.1 deep-scrub starts 2024-03-21T12:59:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[40675]: 4.1 deep-scrub ok 2024-03-21T12:59:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[40675]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T12:59:54.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[40675]: 3.1f scrub starts 2024-03-21T12:59:54.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[40675]: 3.1f scrub ok 2024-03-21T12:59:54.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:54 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2603718407' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T12:59:55.401 INFO:teuthology.orchestra.run.smithi163.stdout:240518168596 2024-03-21T12:59:55.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[29621]: 4.1b deep-scrub starts 2024-03-21T12:59:55.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[29621]: 4.1b deep-scrub ok 2024-03-21T12:59:55.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[29621]: 6.5 scrub starts 2024-03-21T12:59:55.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[29621]: 6.5 scrub ok 2024-03-21T12:59:55.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[29621]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:55.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[29621]: 5.1d deep-scrub starts 2024-03-21T12:59:55.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[29621]: 5.1d deep-scrub ok 2024-03-21T12:59:55.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[40675]: 4.1b deep-scrub starts 2024-03-21T12:59:55.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[40675]: 4.1b deep-scrub ok 2024-03-21T12:59:55.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[40675]: 6.5 scrub starts 2024-03-21T12:59:55.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[40675]: 6.5 scrub ok 2024-03-21T12:59:55.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[40675]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:55.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[40675]: 5.1d deep-scrub starts 2024-03-21T12:59:55.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-mon[40675]: 5.1d deep-scrub ok 2024-03-21T12:59:55.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:55 smithi204 ceph-mon[33153]: 4.1b deep-scrub starts 2024-03-21T12:59:55.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:55 smithi204 ceph-mon[33153]: 4.1b deep-scrub ok 2024-03-21T12:59:55.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:55 smithi204 ceph-mon[33153]: 6.5 scrub starts 2024-03-21T12:59:55.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:55 smithi204 ceph-mon[33153]: 6.5 scrub ok 2024-03-21T12:59:55.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:55 smithi204 ceph-mon[33153]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:55.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:55 smithi204 ceph-mon[33153]: 5.1d deep-scrub starts 2024-03-21T12:59:55.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:55 smithi204 ceph-mon[33153]: 5.1d deep-scrub ok 2024-03-21T12:59:56.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 12:59:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:12:59:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T12:59:56.455 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411373 for osd.2 2024-03-21T12:59:56.456 DEBUG:teuthology.parallel:result is None 2024-03-21T12:59:56.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:56 smithi204 ceph-mon[33153]: 3.1a deep-scrub starts 2024-03-21T12:59:56.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:56 smithi204 ceph-mon[33153]: 3.1a deep-scrub ok 2024-03-21T12:59:56.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:56 smithi204 ceph-mon[33153]: 5.1f scrub starts 2024-03-21T12:59:56.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:56 smithi204 ceph-mon[33153]: 5.1f scrub ok 2024-03-21T12:59:56.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:56 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1340867574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T12:59:56.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:56 smithi204 ceph-mon[33153]: 2.2 scrub starts 2024-03-21T12:59:56.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:56 smithi204 ceph-mon[33153]: 2.2 scrub ok 2024-03-21T12:59:56.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:56 smithi204 ceph-mon[33153]: 4.1c scrub starts 2024-03-21T12:59:56.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:56 smithi204 ceph-mon[33153]: 4.1c scrub ok 2024-03-21T12:59:56.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[29621]: 3.1a deep-scrub starts 2024-03-21T12:59:56.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[29621]: 3.1a deep-scrub ok 2024-03-21T12:59:56.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[29621]: 5.1f scrub starts 2024-03-21T12:59:56.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[29621]: 5.1f scrub ok 2024-03-21T12:59:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1340867574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T12:59:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[29621]: 2.2 scrub starts 2024-03-21T12:59:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[29621]: 2.2 scrub ok 2024-03-21T12:59:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[29621]: 4.1c scrub starts 2024-03-21T12:59:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[29621]: 4.1c scrub ok 2024-03-21T12:59:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[40675]: 3.1a deep-scrub starts 2024-03-21T12:59:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[40675]: 3.1a deep-scrub ok 2024-03-21T12:59:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[40675]: 5.1f scrub starts 2024-03-21T12:59:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[40675]: 5.1f scrub ok 2024-03-21T12:59:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1340867574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T12:59:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[40675]: 2.2 scrub starts 2024-03-21T12:59:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[40675]: 2.2 scrub ok 2024-03-21T12:59:56.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[40675]: 4.1c scrub starts 2024-03-21T12:59:56.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:56 smithi163 ceph-mon[40675]: 4.1c scrub ok 2024-03-21T12:59:57.130 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168590 got 240518168596 for osd.7 2024-03-21T12:59:57.130 DEBUG:teuthology.parallel:result is None 2024-03-21T12:59:57.130 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-21T12:59:57.130 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T12:59:57.509 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:57 smithi163 ceph-mon[29621]: 6.1e scrub starts 2024-03-21T12:59:57.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:57 smithi163 ceph-mon[29621]: 6.1e scrub ok 2024-03-21T12:59:57.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:57 smithi163 ceph-mon[29621]: 4.5 scrub starts 2024-03-21T12:59:57.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:57 smithi163 ceph-mon[29621]: 4.5 scrub ok 2024-03-21T12:59:57.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:57 smithi163 ceph-mon[29621]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:57.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:57 smithi163 ceph-mon[40675]: 6.1e scrub starts 2024-03-21T12:59:57.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:57 smithi163 ceph-mon[40675]: 6.1e scrub ok 2024-03-21T12:59:57.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:57 smithi163 ceph-mon[40675]: 4.5 scrub starts 2024-03-21T12:59:57.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:57 smithi163 ceph-mon[40675]: 4.5 scrub ok 2024-03-21T12:59:57.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:57 smithi163 ceph-mon[40675]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:57 smithi204 ceph-mon[33153]: 6.1e scrub starts 2024-03-21T12:59:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:57 smithi204 ceph-mon[33153]: 6.1e scrub ok 2024-03-21T12:59:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:57 smithi204 ceph-mon[33153]: 4.5 scrub starts 2024-03-21T12:59:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:57 smithi204 ceph-mon[33153]: 4.5 scrub ok 2024-03-21T12:59:57.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:57 smithi204 ceph-mon[33153]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:58.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:58 smithi163 ceph-mon[29621]: 3.4 scrub starts 2024-03-21T12:59:58.709 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:58 smithi163 ceph-mon[29621]: 3.4 scrub ok 2024-03-21T12:59:58.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:58 smithi163 ceph-mon[40675]: 3.4 scrub starts 2024-03-21T12:59:58.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:58 smithi163 ceph-mon[40675]: 3.4 scrub ok 2024-03-21T12:59:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:58 smithi204 ceph-mon[33153]: 3.4 scrub starts 2024-03-21T12:59:58.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:58 smithi204 ceph-mon[33153]: 3.4 scrub ok 2024-03-21T12:59:58.858 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:59:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:59 smithi204 ceph-mon[33153]: 6.d scrub starts 2024-03-21T12:59:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:59 smithi204 ceph-mon[33153]: 6.d scrub ok 2024-03-21T12:59:59.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:59 smithi204 ceph-mon[33153]: 2.1 scrub starts 2024-03-21T12:59:59.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:59 smithi204 ceph-mon[33153]: 2.1 scrub ok 2024-03-21T12:59:59.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 12:59:59 smithi204 ceph-mon[33153]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:59:59.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:59 smithi163 ceph-mon[29621]: 6.d scrub starts 2024-03-21T12:59:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:59 smithi163 ceph-mon[29621]: 6.d scrub ok 2024-03-21T12:59:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:59 smithi163 ceph-mon[29621]: 2.1 scrub starts 2024-03-21T12:59:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:59 smithi163 ceph-mon[29621]: 2.1 scrub ok 2024-03-21T12:59:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 12:59:59 smithi163 ceph-mon[29621]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:59:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:59 smithi163 ceph-mon[40675]: 6.d scrub starts 2024-03-21T12:59:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:59 smithi163 ceph-mon[40675]: 6.d scrub ok 2024-03-21T12:59:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:59 smithi163 ceph-mon[40675]: 2.1 scrub starts 2024-03-21T12:59:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:59 smithi163 ceph-mon[40675]: 2.1 scrub ok 2024-03-21T12:59:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 12:59:59 smithi163 ceph-mon[40675]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:00:00.380 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[29621]: 5.8 scrub starts 2024-03-21T13:00:00.380 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[29621]: 5.8 scrub ok 2024-03-21T13:00:00.380 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[29621]: 6.1a deep-scrub starts 2024-03-21T13:00:00.380 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[29621]: 6.1a deep-scrub ok 2024-03-21T13:00:00.381 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[29621]: 3.e scrub starts 2024-03-21T13:00:00.381 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[29621]: 3.e scrub ok 2024-03-21T13:00:00.381 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[29621]: 4.a scrub starts 2024-03-21T13:00:00.381 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[29621]: 4.a scrub ok 2024-03-21T13:00:00.381 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[29621]: overall HEALTH_OK 2024-03-21T13:00:00.381 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[40675]: 5.8 scrub starts 2024-03-21T13:00:00.381 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[40675]: 5.8 scrub ok 2024-03-21T13:00:00.381 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[40675]: 6.1a deep-scrub starts 2024-03-21T13:00:00.381 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[40675]: 6.1a deep-scrub ok 2024-03-21T13:00:00.381 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[40675]: 3.e scrub starts 2024-03-21T13:00:00.381 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[40675]: 3.e scrub ok 2024-03-21T13:00:00.381 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[40675]: 4.a scrub starts 2024-03-21T13:00:00.381 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[40675]: 4.a scrub ok 2024-03-21T13:00:00.382 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:00 smithi163 ceph-mon[40675]: overall HEALTH_OK 2024-03-21T13:00:00.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:00 smithi204 ceph-mon[33153]: 5.8 scrub starts 2024-03-21T13:00:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:00 smithi204 ceph-mon[33153]: 5.8 scrub ok 2024-03-21T13:00:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:00 smithi204 ceph-mon[33153]: 6.1a deep-scrub starts 2024-03-21T13:00:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:00 smithi204 ceph-mon[33153]: 6.1a deep-scrub ok 2024-03-21T13:00:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:00 smithi204 ceph-mon[33153]: 3.e scrub starts 2024-03-21T13:00:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:00 smithi204 ceph-mon[33153]: 3.e scrub ok 2024-03-21T13:00:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:00 smithi204 ceph-mon[33153]: 4.a scrub starts 2024-03-21T13:00:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:00 smithi204 ceph-mon[33153]: 4.a scrub ok 2024-03-21T13:00:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:00 smithi204 ceph-mon[33153]: overall HEALTH_OK 2024-03-21T13:00:01.290 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:01 smithi163 systemd[1]: Starting Ceph alertmanager.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:00:01.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:01 smithi204 ceph-mon[33153]: 3.1c scrub starts 2024-03-21T13:00:01.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:01 smithi204 ceph-mon[33153]: 3.1c scrub ok 2024-03-21T13:00:01.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:01 smithi204 ceph-mon[33153]: 6.1c scrub starts 2024-03-21T13:00:01.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:01 smithi204 ceph-mon[33153]: 6.1c scrub ok 2024-03-21T13:00:01.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:01 smithi204 ceph-mon[33153]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:00:01.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:01 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:01.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:00:01 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:00:01.573 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:01 smithi163 ceph-mon[40675]: 3.1c scrub starts 2024-03-21T13:00:01.573 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:01 smithi163 ceph-mon[40675]: 3.1c scrub ok 2024-03-21T13:00:01.573 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:01 smithi163 ceph-mon[40675]: 6.1c scrub starts 2024-03-21T13:00:01.573 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:01 smithi163 ceph-mon[40675]: 6.1c scrub ok 2024-03-21T13:00:01.573 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:01 smithi163 ceph-mon[40675]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:00:01.573 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:01 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:01.574 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:01 smithi163 ceph-mon[29621]: 3.1c scrub starts 2024-03-21T13:00:01.574 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:01 smithi163 ceph-mon[29621]: 3.1c scrub ok 2024-03-21T13:00:01.574 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:01 smithi163 ceph-mon[29621]: 6.1c scrub starts 2024-03-21T13:00:01.574 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:01 smithi163 ceph-mon[29621]: 6.1c scrub ok 2024-03-21T13:00:01.574 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:01 smithi163 ceph-mon[29621]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:00:01.574 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:01 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:02.261 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:01 smithi163 podman[75133]: 2024-03-21 13:00:01.871385278 +0000 UTC m=+0.197669076 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-21T13:00:02.261 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:01 smithi163 podman[75133]: 2024-03-21 13:00:01.99345428 +0000 UTC m=+0.319738071 volume create d7f5bf520d3e515760b26142a3c7e5f250aa7d579e3e6fea79fa6f641f700197 2024-03-21T13:00:02.261 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:02 smithi163 podman[75133]: 2024-03-21 13:00:02.085441513 +0000 UTC m=+0.411725306 container create 846b4d858ab0fecb43a36280744e3bb3e0fb3170315f16e7de3bfa260c63c1fe (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:00:02.513 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[29621]: 6.a scrub starts 2024-03-21T13:00:02.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[29621]: 6.a scrub ok 2024-03-21T13:00:02.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[29621]: 4.1f scrub starts 2024-03-21T13:00:02.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[29621]: 4.1f scrub ok 2024-03-21T13:00:02.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[29621]: 4.6 scrub starts 2024-03-21T13:00:02.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[29621]: 4.6 scrub ok 2024-03-21T13:00:02.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[29621]: 5.a scrub starts 2024-03-21T13:00:02.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[29621]: 5.a scrub ok 2024-03-21T13:00:02.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[29621]: 4.4 scrub starts 2024-03-21T13:00:02.514 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[29621]: 4.4 scrub ok 2024-03-21T13:00:02.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[40675]: 6.a scrub starts 2024-03-21T13:00:02.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[40675]: 6.a scrub ok 2024-03-21T13:00:02.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[40675]: 4.1f scrub starts 2024-03-21T13:00:02.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[40675]: 4.1f scrub ok 2024-03-21T13:00:02.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[40675]: 4.6 scrub starts 2024-03-21T13:00:02.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[40675]: 4.6 scrub ok 2024-03-21T13:00:02.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[40675]: 5.a scrub starts 2024-03-21T13:00:02.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[40675]: 5.a scrub ok 2024-03-21T13:00:02.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[40675]: 4.4 scrub starts 2024-03-21T13:00:02.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-mon[40675]: 4.4 scrub ok 2024-03-21T13:00:02.515 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:02 smithi163 podman[75133]: 2024-03-21 13:00:02.405496645 +0000 UTC m=+0.731780435 container init 846b4d858ab0fecb43a36280744e3bb3e0fb3170315f16e7de3bfa260c63c1fe (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:00:02.515 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:02 smithi163 podman[75133]: 2024-03-21 13:00:02.408231894 +0000 UTC m=+0.734515680 container start 846b4d858ab0fecb43a36280744e3bb3e0fb3170315f16e7de3bfa260c63c1fe (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:00:02.515 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[75154]: ts=2024-03-21T13:00:02.420Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-21T13:00:02.516 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[75154]: ts=2024-03-21T13:00:02.420Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-21T13:00:02.516 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[75154]: ts=2024-03-21T13:00:02.421Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.163 port=9094 2024-03-21T13:00:02.516 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[75154]: ts=2024-03-21T13:00:02.422Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-21T13:00:02.516 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[75154]: ts=2024-03-21T13:00:02.444Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T13:00:02.516 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[75154]: ts=2024-03-21T13:00:02.445Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T13:00:02.516 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[75154]: ts=2024-03-21T13:00:02.447Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-21T13:00:02.516 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[75154]: ts=2024-03-21T13:00:02.447Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-21T13:00:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:02 smithi204 ceph-mon[33153]: 6.a scrub starts 2024-03-21T13:00:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:02 smithi204 ceph-mon[33153]: 6.a scrub ok 2024-03-21T13:00:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:02 smithi204 ceph-mon[33153]: 4.1f scrub starts 2024-03-21T13:00:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:02 smithi204 ceph-mon[33153]: 4.1f scrub ok 2024-03-21T13:00:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:02 smithi204 ceph-mon[33153]: 4.6 scrub starts 2024-03-21T13:00:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:02 smithi204 ceph-mon[33153]: 4.6 scrub ok 2024-03-21T13:00:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:02 smithi204 ceph-mon[33153]: 5.a scrub starts 2024-03-21T13:00:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:02 smithi204 ceph-mon[33153]: 5.a scrub ok 2024-03-21T13:00:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:02 smithi204 ceph-mon[33153]: 4.4 scrub starts 2024-03-21T13:00:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:02 smithi204 ceph-mon[33153]: 4.4 scrub ok 2024-03-21T13:00:02.778 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:02 smithi163 bash[75133]: 846b4d858ab0fecb43a36280744e3bb3e0fb3170315f16e7de3bfa260c63c1fe 2024-03-21T13:00:02.778 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:02 smithi163 systemd[1]: Started Ceph alertmanager.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T13:00:02.778 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:00:02] ENGINE Bus STOPPING 2024-03-21T13:00:02.778 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:00:02] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:00:02.778 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:00:02] ENGINE Bus STOPPED 2024-03-21T13:00:02.779 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:00:02] ENGINE Bus STARTING 2024-03-21T13:00:02.779 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:00:02] ENGINE Serving on http://:::9283 2024-03-21T13:00:02.779 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:00:02] ENGINE Bus STARTED 2024-03-21T13:00:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:00:03.007 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:00:03.847 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.847 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.847 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.847 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.847 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T13:00:03.847 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: 6.18 scrub starts 2024-03-21T13:00:03.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: 6.18 scrub ok 2024-03-21T13:00:03.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:03.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T13:00:03.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T13:00:03.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: Deploying daemon grafana.a on smithi204 2024-03-21T13:00:03.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: 6.19 scrub starts 2024-03-21T13:00:03.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: 6.19 scrub ok 2024-03-21T13:00:03.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[29621]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:00:03.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T13:00:03.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: 6.18 scrub starts 2024-03-21T13:00:03.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: 6.18 scrub ok 2024-03-21T13:00:03.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:03.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T13:00:03.850 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T13:00:03.850 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:03.850 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: Deploying daemon grafana.a on smithi204 2024-03-21T13:00:03.850 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: 6.19 scrub starts 2024-03-21T13:00:03.850 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: 6.19 scrub ok 2024-03-21T13:00:03.850 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:03 smithi163 ceph-mon[40675]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:00:03.874 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-21T13:00:02.623642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1008,"num_read_kb":854,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":624,"ondisk_log_size":624,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249920,"kb_used_data":8828,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749711296,"statfs":{"total":767960285184,"available":767704367104,"internally_reserved":0,"allocated":9039872,"data_stored":4792088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002431"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:58.846732+0000","last_change":"2024-03-21T12:59:58.846777+0000","last_active":"2024-03-21T12:59:58.846732+0000","last_peered":"2024-03-21T12:59:58.846732+0000","last_clean":"2024-03-21T12:59:58.846732+0000","last_became_active":"2024-03-21T12:58:43.182506+0000","last_became_peered":"2024-03-21T12:58:43.182506+0000","last_unstale":"2024-03-21T12:59:58.846732+0000","last_undegraded":"2024-03-21T12:59:58.846732+0000","last_fullsized":"2024-03-21T12:59:58.846732+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:58.846634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:58.846634+0000","last_clean_scrub_stamp":"2024-03-21T12:59:58.846634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.412490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710452+0000","last_change":"2024-03-21T12:58:41.160668+0000","last_active":"2024-03-21T12:59:53.710452+0000","last_peered":"2024-03-21T12:59:53.710452+0000","last_clean":"2024-03-21T12:59:53.710452+0000","last_became_active":"2024-03-21T12:58:41.160342+0000","last_became_peered":"2024-03-21T12:58:41.160342+0000","last_unstale":"2024-03-21T12:59:53.710452+0000","last_undegraded":"2024-03-21T12:59:53.710452+0000","last_fullsized":"2024-03-21T12:59:53.710452+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:07.771079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.478199+0000","last_change":"2024-03-21T12:59:53.478218+0000","last_active":"2024-03-21T12:59:53.478199+0000","last_peered":"2024-03-21T12:59:53.478199+0000","last_clean":"2024-03-21T12:59:53.478199+0000","last_became_active":"2024-03-21T12:58:37.731393+0000","last_became_peered":"2024-03-21T12:58:37.731393+0000","last_unstale":"2024-03-21T12:59:53.478199+0000","last_undegraded":"2024-03-21T12:59:53.478199+0000","last_fullsized":"2024-03-21T12:59:53.478199+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:53.478141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:53.478141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:57.521483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"71'9","reported_seq":86,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840002+0000","last_change":"2024-03-21T12:59:01.166943+0000","last_active":"2024-03-21T12:59:53.840002+0000","last_peered":"2024-03-21T12:59:53.840002+0000","last_clean":"2024-03-21T12:59:53.840002+0000","last_became_active":"2024-03-21T12:59:01.161098+0000","last_became_peered":"2024-03-21T12:59:01.161098+0000","last_unstale":"2024-03-21T12:59:53.840002+0000","last_undegraded":"2024-03-21T12:59:53.840002+0000","last_fullsized":"2024-03-21T12:59:53.840002+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:57.399487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315279+0000","last_change":"2024-03-21T12:58:43.352892+0000","last_active":"2024-03-21T12:59:53.315279+0000","last_peered":"2024-03-21T12:59:53.315279+0000","last_clean":"2024-03-21T12:59:53.315279+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T12:59:53.315279+0000","last_undegraded":"2024-03-21T12:59:53.315279+0000","last_fullsized":"2024-03-21T12:59:53.315279+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840116+0000","last_change":"2024-03-21T12:58:41.168108+0000","last_active":"2024-03-21T12:59:53.840116+0000","last_peered":"2024-03-21T12:59:53.840116+0000","last_clean":"2024-03-21T12:59:53.840116+0000","last_became_active":"2024-03-21T12:58:41.167710+0000","last_became_peered":"2024-03-21T12:58:41.167710+0000","last_unstale":"2024-03-21T12:59:53.840116+0000","last_undegraded":"2024-03-21T12:59:53.840116+0000","last_fullsized":"2024-03-21T12:59:53.840116+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:56.955691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":64,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316055+0000","last_change":"2024-03-21T12:58:37.728682+0000","last_active":"2024-03-21T12:59:53.316055+0000","last_peered":"2024-03-21T12:59:53.316055+0000","last_clean":"2024-03-21T12:59:53.316055+0000","last_became_active":"2024-03-21T12:58:37.728383+0000","last_became_peered":"2024-03-21T12:58:37.728383+0000","last_unstale":"2024-03-21T12:59:53.316055+0000","last_undegraded":"2024-03-21T12:59:53.316055+0000","last_fullsized":"2024-03-21T12:59:53.316055+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:04.926301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"71'15","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315281+0000","last_change":"2024-03-21T12:58:39.751346+0000","last_active":"2024-03-21T12:59:53.315281+0000","last_peered":"2024-03-21T12:59:53.315281+0000","last_clean":"2024-03-21T12:59:53.315281+0000","last_became_active":"2024-03-21T12:58:39.750396+0000","last_became_peered":"2024-03-21T12:58:39.750396+0000","last_unstale":"2024-03-21T12:59:53.315281+0000","last_undegraded":"2024-03-21T12:59:53.315281+0000","last_fullsized":"2024-03-21T12:59:53.315281+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:54.161286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339278+0000","last_change":"2024-03-21T12:58:43.180242+0000","last_active":"2024-03-21T12:59:53.339278+0000","last_peered":"2024-03-21T12:59:53.339278+0000","last_clean":"2024-03-21T12:59:53.339278+0000","last_became_active":"2024-03-21T12:58:43.179920+0000","last_became_peered":"2024-03-21T12:58:43.179920+0000","last_unstale":"2024-03-21T12:59:53.339278+0000","last_undegraded":"2024-03-21T12:59:53.339278+0000","last_fullsized":"2024-03-21T12:59:53.339278+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315533+0000","last_change":"2024-03-21T12:58:41.169432+0000","last_active":"2024-03-21T12:59:53.315533+0000","last_peered":"2024-03-21T12:59:53.315533+0000","last_clean":"2024-03-21T12:59:53.315533+0000","last_became_active":"2024-03-21T12:58:41.168978+0000","last_became_peered":"2024-03-21T12:58:41.168978+0000","last_unstale":"2024-03-21T12:59:53.315533+0000","last_undegraded":"2024-03-21T12:59:53.315533+0000","last_fullsized":"2024-03-21T12:59:53.315533+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:54.624190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315779+0000","last_change":"2024-03-21T12:59:44.661395+0000","last_active":"2024-03-21T12:59:53.315779+0000","last_peered":"2024-03-21T12:59:53.315779+0000","last_clean":"2024-03-21T12:59:53.315779+0000","last_became_active":"2024-03-21T12:58:37.733705+0000","last_became_peered":"2024-03-21T12:58:37.733705+0000","last_unstale":"2024-03-21T12:59:53.315779+0000","last_undegraded":"2024-03-21T12:59:53.315779+0000","last_fullsized":"2024-03-21T12:59:53.315779+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:44.661289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:44.661289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:45.014731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"71'9","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840441+0000","last_change":"2024-03-21T12:58:39.749391+0000","last_active":"2024-03-21T12:59:53.840441+0000","last_peered":"2024-03-21T12:59:53.840441+0000","last_clean":"2024-03-21T12:59:53.840441+0000","last_became_active":"2024-03-21T12:58:39.749010+0000","last_became_peered":"2024-03-21T12:58:39.749010+0000","last_unstale":"2024-03-21T12:59:53.840441+0000","last_undegraded":"2024-03-21T12:59:53.840441+0000","last_fullsized":"2024-03-21T12:59:53.840441+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:04.526693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315376+0000","last_change":"2024-03-21T12:58:43.181803+0000","last_active":"2024-03-21T12:59:53.315376+0000","last_peered":"2024-03-21T12:59:53.315376+0000","last_clean":"2024-03-21T12:59:53.315376+0000","last_became_active":"2024-03-21T12:58:43.181494+0000","last_became_peered":"2024-03-21T12:58:43.181494+0000","last_unstale":"2024-03-21T12:59:53.315376+0000","last_undegraded":"2024-03-21T12:59:53.315376+0000","last_fullsized":"2024-03-21T12:59:53.315376+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840365+0000","last_change":"2024-03-21T12:59:41.158247+0000","last_active":"2024-03-21T12:59:53.840365+0000","last_peered":"2024-03-21T12:59:53.840365+0000","last_clean":"2024-03-21T12:59:53.840365+0000","last_became_active":"2024-03-21T12:58:41.163517+0000","last_became_peered":"2024-03-21T12:58:41.163517+0000","last_unstale":"2024-03-21T12:59:53.840365+0000","last_undegraded":"2024-03-21T12:59:53.840365+0000","last_fullsized":"2024-03-21T12:59:53.840365+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:41.158105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:41.158105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:45.036271+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:59.594002+0000","last_change":"2024-03-21T12:59:59.594026+0000","last_active":"2024-03-21T12:59:59.594002+0000","last_peered":"2024-03-21T12:59:59.594002+0000","last_clean":"2024-03-21T12:59:59.594002+0000","last_became_active":"2024-03-21T12:58:37.731058+0000","last_became_peered":"2024-03-21T12:58:37.731058+0000","last_unstale":"2024-03-21T12:59:59.594002+0000","last_undegraded":"2024-03-21T12:59:59.594002+0000","last_fullsized":"2024-03-21T12:59:59.594002+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:59.593927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.593927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:14.344496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.952486+0000","last_change":"2024-03-21T12:59:53.952540+0000","last_active":"2024-03-21T12:59:53.952486+0000","last_peered":"2024-03-21T12:59:53.952486+0000","last_clean":"2024-03-21T12:59:53.952486+0000","last_became_active":"2024-03-21T12:58:39.749211+0000","last_became_peered":"2024-03-21T12:58:39.749211+0000","last_unstale":"2024-03-21T12:59:53.952486+0000","last_undegraded":"2024-03-21T12:59:53.952486+0000","last_fullsized":"2024-03-21T12:59:53.952486+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T12:59:53.952401+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.263330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:55.899413+0000","last_change":"2024-03-21T12:59:55.899438+0000","last_active":"2024-03-21T12:59:55.899413+0000","last_peered":"2024-03-21T12:59:55.899413+0000","last_clean":"2024-03-21T12:59:55.899413+0000","last_became_active":"2024-03-21T12:58:43.351768+0000","last_became_peered":"2024-03-21T12:58:43.351768+0000","last_unstale":"2024-03-21T12:59:55.899413+0000","last_undegraded":"2024-03-21T12:59:55.899413+0000","last_fullsized":"2024-03-21T12:59:55.899413+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:55.899344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.899344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:59.849673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:55.014533+0000","last_change":"2024-03-21T12:59:55.014569+0000","last_active":"2024-03-21T12:59:55.014533+0000","last_peered":"2024-03-21T12:59:55.014533+0000","last_clean":"2024-03-21T12:59:55.014533+0000","last_became_active":"2024-03-21T12:58:41.165783+0000","last_became_peered":"2024-03-21T12:58:41.165783+0000","last_unstale":"2024-03-21T12:59:55.014533+0000","last_undegraded":"2024-03-21T12:59:55.014533+0000","last_fullsized":"2024-03-21T12:59:55.014533+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.014453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:10.470418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339649+0000","last_change":"2024-03-21T12:58:37.733421+0000","last_active":"2024-03-21T12:59:53.339649+0000","last_peered":"2024-03-21T12:59:53.339649+0000","last_clean":"2024-03-21T12:59:53.339649+0000","last_became_active":"2024-03-21T12:58:37.733064+0000","last_became_peered":"2024-03-21T12:58:37.733064+0000","last_unstale":"2024-03-21T12:59:53.339649+0000","last_undegraded":"2024-03-21T12:59:53.339649+0000","last_fullsized":"2024-03-21T12:59:53.339649+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:54.238132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":88,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:55.664722+0000","last_change":"2024-03-21T12:59:55.664752+0000","last_active":"2024-03-21T12:59:55.664722+0000","last_peered":"2024-03-21T12:59:55.664722+0000","last_clean":"2024-03-21T12:59:55.664722+0000","last_became_active":"2024-03-21T12:58:39.749130+0000","last_became_peered":"2024-03-21T12:58:39.749130+0000","last_unstale":"2024-03-21T12:59:55.664722+0000","last_undegraded":"2024-03-21T12:59:55.664722+0000","last_fullsized":"2024-03-21T12:59:55.664722+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T12:59:55.664649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.664649+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:11.693713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316207+0000","last_change":"2024-03-21T12:58:43.352883+0000","last_active":"2024-03-21T12:59:53.316207+0000","last_peered":"2024-03-21T12:59:53.316207+0000","last_clean":"2024-03-21T12:59:53.316207+0000","last_became_active":"2024-03-21T12:58:43.352510+0000","last_became_peered":"2024-03-21T12:58:43.352510+0000","last_unstale":"2024-03-21T12:59:53.316207+0000","last_undegraded":"2024-03-21T12:59:53.316207+0000","last_fullsized":"2024-03-21T12:59:53.316207+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:56.333194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840998+0000","last_change":"2024-03-21T12:59:50.904504+0000","last_active":"2024-03-21T12:59:53.840998+0000","last_peered":"2024-03-21T12:59:53.840998+0000","last_clean":"2024-03-21T12:59:53.840998+0000","last_became_active":"2024-03-21T12:58:41.167961+0000","last_became_peered":"2024-03-21T12:58:41.167961+0000","last_unstale":"2024-03-21T12:59:53.840998+0000","last_undegraded":"2024-03-21T12:59:53.840998+0000","last_fullsized":"2024-03-21T12:59:53.840998+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.904419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.904419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.905923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:54.948164+0000","last_change":"2024-03-21T12:59:54.948197+0000","last_active":"2024-03-21T12:59:54.948164+0000","last_peered":"2024-03-21T12:59:54.948164+0000","last_clean":"2024-03-21T12:59:54.948164+0000","last_became_active":"2024-03-21T12:58:37.732355+0000","last_became_peered":"2024-03-21T12:58:37.732355+0000","last_unstale":"2024-03-21T12:59:54.948164+0000","last_undegraded":"2024-03-21T12:59:54.948164+0000","last_fullsized":"2024-03-21T12:59:54.948164+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_clean_scrub_stamp":"2024-03-21T12:59:54.948081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:03.565967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'12","reported_seq":95,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315512+0000","last_change":"2024-03-21T12:59:01.168089+0000","last_active":"2024-03-21T12:59:53.315512+0000","last_peered":"2024-03-21T12:59:53.315512+0000","last_clean":"2024-03-21T12:59:53.315512+0000","last_became_active":"2024-03-21T12:59:01.161520+0000","last_became_peered":"2024-03-21T12:59:01.161520+0000","last_unstale":"2024-03-21T12:59:53.315512+0000","last_undegraded":"2024-03-21T12:59:53.315512+0000","last_fullsized":"2024-03-21T12:59:53.315512+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:08.420065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840537+0000","last_change":"2024-03-21T12:58:43.184768+0000","last_active":"2024-03-21T12:59:53.840537+0000","last_peered":"2024-03-21T12:59:53.840537+0000","last_clean":"2024-03-21T12:59:53.840537+0000","last_became_active":"2024-03-21T12:58:43.184473+0000","last_became_peered":"2024-03-21T12:58:43.184473+0000","last_unstale":"2024-03-21T12:59:53.840537+0000","last_undegraded":"2024-03-21T12:59:53.840537+0000","last_fullsized":"2024-03-21T12:59:53.840537+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":86,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:55.277279+0000","last_change":"2024-03-21T12:59:55.277279+0000","last_active":"2024-03-21T12:59:55.277279+0000","last_peered":"2024-03-21T12:59:55.277279+0000","last_clean":"2024-03-21T12:59:55.277279+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T12:59:55.277279+0000","last_undegraded":"2024-03-21T12:59:55.277279+0000","last_fullsized":"2024-03-21T12:59:55.277279+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:55.277165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.277165+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:19.101757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710555+0000","last_change":"2024-03-21T12:58:37.736753+0000","last_active":"2024-03-21T12:59:53.710555+0000","last_peered":"2024-03-21T12:59:53.710555+0000","last_clean":"2024-03-21T12:59:53.710555+0000","last_became_active":"2024-03-21T12:58:37.736455+0000","last_became_peered":"2024-03-21T12:58:37.736455+0000","last_unstale":"2024-03-21T12:59:53.710555+0000","last_undegraded":"2024-03-21T12:59:53.710555+0000","last_fullsized":"2024-03-21T12:59:53.710555+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:56.602984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"71'10","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339627+0000","last_change":"2024-03-21T12:59:50.414098+0000","last_active":"2024-03-21T12:59:53.339627+0000","last_peered":"2024-03-21T12:59:53.339627+0000","last_clean":"2024-03-21T12:59:53.339627+0000","last_became_active":"2024-03-21T12:58:39.754548+0000","last_became_peered":"2024-03-21T12:58:39.754548+0000","last_unstale":"2024-03-21T12:59:53.339627+0000","last_undegraded":"2024-03-21T12:59:53.339627+0000","last_fullsized":"2024-03-21T12:59:53.339627+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T12:59:50.413983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.413983+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.575091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710921+0000","last_change":"2024-03-21T12:58:43.186597+0000","last_active":"2024-03-21T12:59:53.710921+0000","last_peered":"2024-03-21T12:59:53.710921+0000","last_clean":"2024-03-21T12:59:53.710921+0000","last_became_active":"2024-03-21T12:58:43.186289+0000","last_became_peered":"2024-03-21T12:58:43.186289+0000","last_unstale":"2024-03-21T12:59:53.710921+0000","last_undegraded":"2024-03-21T12:59:53.710921+0000","last_fullsized":"2024-03-21T12:59:53.710921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:00.777226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339724+0000","last_change":"2024-03-21T12:58:41.167760+0000","last_active":"2024-03-21T12:59:53.339724+0000","last_peered":"2024-03-21T12:59:53.339724+0000","last_clean":"2024-03-21T12:59:53.339724+0000","last_became_active":"2024-03-21T12:58:41.167443+0000","last_became_peered":"2024-03-21T12:58:41.167443+0000","last_unstale":"2024-03-21T12:59:53.339724+0000","last_undegraded":"2024-03-21T12:59:53.339724+0000","last_fullsized":"2024-03-21T12:59:53.339724+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:03.895693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.314370+0000","last_change":"2024-03-21T12:59:49.178866+0000","last_active":"2024-03-21T12:59:53.314370+0000","last_peered":"2024-03-21T12:59:53.314370+0000","last_clean":"2024-03-21T12:59:53.314370+0000","last_became_active":"2024-03-21T12:58:37.728914+0000","last_became_peered":"2024-03-21T12:58:37.728914+0000","last_unstale":"2024-03-21T12:59:53.314370+0000","last_undegraded":"2024-03-21T12:59:53.314370+0000","last_fullsized":"2024-03-21T12:59:53.314370+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:49.178785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:49.178785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.644002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'11","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.755043+0000","last_change":"2024-03-21T12:58:39.755749+0000","last_active":"2024-03-21T12:59:53.755043+0000","last_peered":"2024-03-21T12:59:53.755043+0000","last_clean":"2024-03-21T12:59:53.755043+0000","last_became_active":"2024-03-21T12:58:39.755294+0000","last_became_peered":"2024-03-21T12:58:39.755294+0000","last_unstale":"2024-03-21T12:59:53.755043+0000","last_undegraded":"2024-03-21T12:59:53.755043+0000","last_fullsized":"2024-03-21T12:59:53.755043+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:57.124138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840256+0000","last_change":"2024-03-21T12:59:50.209251+0000","last_active":"2024-03-21T12:59:53.840256+0000","last_peered":"2024-03-21T12:59:53.840256+0000","last_clean":"2024-03-21T12:59:53.840256+0000","last_became_active":"2024-03-21T12:58:43.183475+0000","last_became_peered":"2024-03-21T12:58:43.183475+0000","last_unstale":"2024-03-21T12:59:53.840256+0000","last_undegraded":"2024-03-21T12:59:53.840256+0000","last_fullsized":"2024-03-21T12:59:53.840256+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.209152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.209152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:18.566644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.754433+0000","last_change":"2024-03-21T12:58:41.167214+0000","last_active":"2024-03-21T12:59:53.754433+0000","last_peered":"2024-03-21T12:59:53.754433+0000","last_clean":"2024-03-21T12:59:53.754433+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T12:59:53.754433+0000","last_undegraded":"2024-03-21T12:59:53.754433+0000","last_fullsized":"2024-03-21T12:59:53.754433+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:58.390621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339693+0000","last_change":"2024-03-21T12:58:37.732416+0000","last_active":"2024-03-21T12:59:53.339693+0000","last_peered":"2024-03-21T12:59:53.339693+0000","last_clean":"2024-03-21T12:59:53.339693+0000","last_became_active":"2024-03-21T12:58:37.732144+0000","last_became_peered":"2024-03-21T12:58:37.732144+0000","last_unstale":"2024-03-21T12:59:53.339693+0000","last_undegraded":"2024-03-21T12:59:53.339693+0000","last_fullsized":"2024-03-21T12:59:53.339693+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:05.140648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":60,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.314430+0000","last_change":"2024-03-21T12:58:39.755658+0000","last_active":"2024-03-21T12:59:53.314430+0000","last_peered":"2024-03-21T12:59:53.314430+0000","last_clean":"2024-03-21T12:59:53.314430+0000","last_became_active":"2024-03-21T12:58:39.755336+0000","last_became_peered":"2024-03-21T12:58:39.755336+0000","last_unstale":"2024-03-21T12:59:53.314430+0000","last_undegraded":"2024-03-21T12:59:53.314430+0000","last_fullsized":"2024-03-21T12:59:53.314430+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:03.263219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315097+0000","last_change":"2024-03-21T12:58:43.352917+0000","last_active":"2024-03-21T12:59:53.315097+0000","last_peered":"2024-03-21T12:59:53.315097+0000","last_clean":"2024-03-21T12:59:53.315097+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T12:59:53.315097+0000","last_undegraded":"2024-03-21T12:59:53.315097+0000","last_fullsized":"2024-03-21T12:59:53.315097+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840086+0000","last_change":"2024-03-21T12:58:41.165123+0000","last_active":"2024-03-21T12:59:53.840086+0000","last_peered":"2024-03-21T12:59:53.840086+0000","last_clean":"2024-03-21T12:59:53.840086+0000","last_became_active":"2024-03-21T12:58:41.164784+0000","last_became_peered":"2024-03-21T12:58:41.164784+0000","last_unstale":"2024-03-21T12:59:53.840086+0000","last_undegraded":"2024-03-21T12:59:53.840086+0000","last_fullsized":"2024-03-21T12:59:53.840086+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:04.486718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316024+0000","last_change":"2024-03-21T12:58:37.736422+0000","last_active":"2024-03-21T12:59:53.316024+0000","last_peered":"2024-03-21T12:59:53.316024+0000","last_clean":"2024-03-21T12:59:53.316024+0000","last_became_active":"2024-03-21T12:58:37.736127+0000","last_became_peered":"2024-03-21T12:58:37.736127+0000","last_unstale":"2024-03-21T12:59:53.316024+0000","last_undegraded":"2024-03-21T12:59:53.316024+0000","last_fullsized":"2024-03-21T12:59:53.316024+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:56.711736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"71'11","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315179+0000","last_change":"2024-03-21T12:59:52.181828+0000","last_active":"2024-03-21T12:59:53.315179+0000","last_peered":"2024-03-21T12:59:53.315179+0000","last_clean":"2024-03-21T12:59:53.315179+0000","last_became_active":"2024-03-21T12:58:39.753014+0000","last_became_peered":"2024-03-21T12:58:39.753014+0000","last_unstale":"2024-03-21T12:59:53.315179+0000","last_undegraded":"2024-03-21T12:59:53.315179+0000","last_fullsized":"2024-03-21T12:59:53.315179+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T12:59:52.181697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.181697+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.248453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339098+0000","last_change":"2024-03-21T12:58:43.178915+0000","last_active":"2024-03-21T12:59:53.339098+0000","last_peered":"2024-03-21T12:59:53.339098+0000","last_clean":"2024-03-21T12:59:53.339098+0000","last_became_active":"2024-03-21T12:58:43.178589+0000","last_became_peered":"2024-03-21T12:58:43.178589+0000","last_unstale":"2024-03-21T12:59:53.339098+0000","last_undegraded":"2024-03-21T12:59:53.339098+0000","last_fullsized":"2024-03-21T12:59:53.339098+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:07.484568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315361+0000","last_change":"2024-03-21T12:58:41.167969+0000","last_active":"2024-03-21T12:59:53.315361+0000","last_peered":"2024-03-21T12:59:53.315361+0000","last_clean":"2024-03-21T12:59:53.315361+0000","last_became_active":"2024-03-21T12:58:41.167371+0000","last_became_peered":"2024-03-21T12:58:41.167371+0000","last_unstale":"2024-03-21T12:59:53.315361+0000","last_undegraded":"2024-03-21T12:59:53.315361+0000","last_fullsized":"2024-03-21T12:59:53.315361+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:59.050477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840721+0000","last_change":"2024-03-21T12:58:37.733949+0000","last_active":"2024-03-21T12:59:53.840721+0000","last_peered":"2024-03-21T12:59:53.840721+0000","last_clean":"2024-03-21T12:59:53.840721+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T12:59:53.840721+0000","last_undegraded":"2024-03-21T12:59:53.840721+0000","last_fullsized":"2024-03-21T12:59:53.840721+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:05.144477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"86'13","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710766+0000","last_change":"2024-03-21T12:59:48.085870+0000","last_active":"2024-03-21T12:59:53.710766+0000","last_peered":"2024-03-21T12:59:53.710766+0000","last_clean":"2024-03-21T12:59:53.710766+0000","last_became_active":"2024-03-21T12:58:39.753224+0000","last_became_peered":"2024-03-21T12:58:39.753224+0000","last_unstale":"2024-03-21T12:59:53.710766+0000","last_undegraded":"2024-03-21T12:59:53.710766+0000","last_fullsized":"2024-03-21T12:59:53.710766+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.085774+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:53.042862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.314797+0000","last_change":"2024-03-21T12:58:43.184148+0000","last_active":"2024-03-21T12:59:53.314797+0000","last_peered":"2024-03-21T12:59:53.314797+0000","last_clean":"2024-03-21T12:59:53.314797+0000","last_became_active":"2024-03-21T12:58:43.183712+0000","last_became_peered":"2024-03-21T12:58:43.183712+0000","last_unstale":"2024-03-21T12:59:53.314797+0000","last_undegraded":"2024-03-21T12:59:53.314797+0000","last_fullsized":"2024-03-21T12:59:53.314797+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:12.054567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710756+0000","last_change":"2024-03-21T12:59:45.088950+0000","last_active":"2024-03-21T12:59:53.710756+0000","last_peered":"2024-03-21T12:59:53.710756+0000","last_clean":"2024-03-21T12:59:53.710756+0000","last_became_active":"2024-03-21T12:58:41.164463+0000","last_became_peered":"2024-03-21T12:58:41.164463+0000","last_unstale":"2024-03-21T12:59:53.710756+0000","last_undegraded":"2024-03-21T12:59:53.710756+0000","last_fullsized":"2024-03-21T12:59:53.710756+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:45.088861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.088861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:11.118145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841177+0000","last_change":"2024-03-21T12:59:45.833435+0000","last_active":"2024-03-21T12:59:53.841177+0000","last_peered":"2024-03-21T12:59:53.841177+0000","last_clean":"2024-03-21T12:59:53.841177+0000","last_became_active":"2024-03-21T12:58:37.731659+0000","last_became_peered":"2024-03-21T12:58:37.731659+0000","last_unstale":"2024-03-21T12:59:53.841177+0000","last_undegraded":"2024-03-21T12:59:53.841177+0000","last_fullsized":"2024-03-21T12:59:53.841177+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.833304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:49.946967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841051+0000","last_change":"2024-03-21T12:58:39.746514+0000","last_active":"2024-03-21T12:59:53.841051+0000","last_peered":"2024-03-21T12:59:53.841051+0000","last_clean":"2024-03-21T12:59:53.841051+0000","last_became_active":"2024-03-21T12:58:39.746182+0000","last_became_peered":"2024-03-21T12:58:39.746182+0000","last_unstale":"2024-03-21T12:59:53.841051+0000","last_undegraded":"2024-03-21T12:59:53.841051+0000","last_fullsized":"2024-03-21T12:59:53.841051+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:08.200205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339384+0000","last_change":"2024-03-21T12:58:43.181024+0000","last_active":"2024-03-21T12:59:53.339384+0000","last_peered":"2024-03-21T12:59:53.339384+0000","last_clean":"2024-03-21T12:59:53.339384+0000","last_became_active":"2024-03-21T12:58:43.180773+0000","last_became_peered":"2024-03-21T12:58:43.180773+0000","last_unstale":"2024-03-21T12:59:53.339384+0000","last_undegraded":"2024-03-21T12:59:53.339384+0000","last_fullsized":"2024-03-21T12:59:53.339384+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:07.203930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":86,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315904+0000","last_change":"2024-03-21T12:59:50.724334+0000","last_active":"2024-03-21T12:59:53.315904+0000","last_peered":"2024-03-21T12:59:53.315904+0000","last_clean":"2024-03-21T12:59:53.315904+0000","last_became_active":"2024-03-21T12:58:41.167956+0000","last_became_peered":"2024-03-21T12:58:41.167956+0000","last_unstale":"2024-03-21T12:59:53.315904+0000","last_undegraded":"2024-03-21T12:59:53.315904+0000","last_fullsized":"2024-03-21T12:59:53.315904+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:50.724242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.724242+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:53.691391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840661+0000","last_change":"2024-03-21T12:58:37.732282+0000","last_active":"2024-03-21T12:59:53.840661+0000","last_peered":"2024-03-21T12:59:53.840661+0000","last_clean":"2024-03-21T12:59:53.840661+0000","last_became_active":"2024-03-21T12:58:37.731988+0000","last_became_peered":"2024-03-21T12:58:37.731988+0000","last_unstale":"2024-03-21T12:59:53.840661+0000","last_undegraded":"2024-03-21T12:59:53.840661+0000","last_fullsized":"2024-03-21T12:59:53.840661+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:00.577765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"71'10","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315898+0000","last_change":"2024-03-21T12:58:39.758132+0000","last_active":"2024-03-21T12:59:53.315898+0000","last_peered":"2024-03-21T12:59:53.315898+0000","last_clean":"2024-03-21T12:59:53.315898+0000","last_became_active":"2024-03-21T12:58:39.757806+0000","last_became_peered":"2024-03-21T12:58:39.757806+0000","last_unstale":"2024-03-21T12:59:53.315898+0000","last_undegraded":"2024-03-21T12:59:53.315898+0000","last_fullsized":"2024-03-21T12:59:53.315898+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841610+0000","last_change":"2024-03-21T12:58:43.186611+0000","last_active":"2024-03-21T12:59:53.841610+0000","last_peered":"2024-03-21T12:59:53.841610+0000","last_clean":"2024-03-21T12:59:53.841610+0000","last_became_active":"2024-03-21T12:58:43.186155+0000","last_became_peered":"2024-03-21T12:58:43.186155+0000","last_unstale":"2024-03-21T12:59:53.841610+0000","last_undegraded":"2024-03-21T12:59:53.841610+0000","last_fullsized":"2024-03-21T12:59:53.841610+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:01.571810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":75,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316046+0000","last_change":"2024-03-21T12:58:41.168116+0000","last_active":"2024-03-21T12:59:53.316046+0000","last_peered":"2024-03-21T12:59:53.316046+0000","last_clean":"2024-03-21T12:59:53.316046+0000","last_became_active":"2024-03-21T12:58:41.167826+0000","last_became_peered":"2024-03-21T12:58:41.167826+0000","last_unstale":"2024-03-21T12:59:53.316046+0000","last_undegraded":"2024-03-21T12:59:53.316046+0000","last_fullsized":"2024-03-21T12:59:53.316046+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:59.174914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339299+0000","last_change":"2024-03-21T12:58:37.732471+0000","last_active":"2024-03-21T12:59:53.339299+0000","last_peered":"2024-03-21T12:59:53.339299+0000","last_clean":"2024-03-21T12:59:53.339299+0000","last_became_active":"2024-03-21T12:58:37.732166+0000","last_became_peered":"2024-03-21T12:58:37.732166+0000","last_unstale":"2024-03-21T12:59:53.339299+0000","last_undegraded":"2024-03-21T12:59:53.339299+0000","last_fullsized":"2024-03-21T12:59:53.339299+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:06.434461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'9","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316076+0000","last_change":"2024-03-21T12:58:39.751200+0000","last_active":"2024-03-21T12:59:53.316076+0000","last_peered":"2024-03-21T12:59:53.316076+0000","last_clean":"2024-03-21T12:59:53.316076+0000","last_became_active":"2024-03-21T12:58:39.750820+0000","last_became_peered":"2024-03-21T12:58:39.750820+0000","last_unstale":"2024-03-21T12:59:53.316076+0000","last_undegraded":"2024-03-21T12:59:53.316076+0000","last_fullsized":"2024-03-21T12:59:53.316076+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:00.043713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.317216+0000","last_change":"2024-03-21T12:59:51.730551+0000","last_active":"2024-03-21T12:59:53.317216+0000","last_peered":"2024-03-21T12:59:53.317216+0000","last_clean":"2024-03-21T12:59:53.317216+0000","last_became_active":"2024-03-21T12:58:43.182126+0000","last_became_peered":"2024-03-21T12:58:43.182126+0000","last_unstale":"2024-03-21T12:59:53.317216+0000","last_undegraded":"2024-03-21T12:59:53.317216+0000","last_fullsized":"2024-03-21T12:59:53.317216+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T12:59:51.730413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.791726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316600+0000","last_change":"2024-03-21T12:58:41.173262+0000","last_active":"2024-03-21T12:59:53.316600+0000","last_peered":"2024-03-21T12:59:53.316600+0000","last_clean":"2024-03-21T12:59:53.316600+0000","last_became_active":"2024-03-21T12:58:41.172904+0000","last_became_peered":"2024-03-21T12:58:41.172904+0000","last_unstale":"2024-03-21T12:59:53.316600+0000","last_undegraded":"2024-03-21T12:59:53.316600+0000","last_fullsized":"2024-03-21T12:59:53.316600+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.839994+0000","last_change":"2024-03-21T12:59:47.203592+0000","last_active":"2024-03-21T12:59:53.839994+0000","last_peered":"2024-03-21T12:59:53.839994+0000","last_clean":"2024-03-21T12:59:53.839994+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T12:59:53.839994+0000","last_undegraded":"2024-03-21T12:59:53.839994+0000","last_fullsized":"2024-03-21T12:59:53.839994+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T12:59:47.203453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.203453+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.098187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'9","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339622+0000","last_change":"2024-03-21T12:58:39.753073+0000","last_active":"2024-03-21T12:59:53.339622+0000","last_peered":"2024-03-21T12:59:53.339622+0000","last_clean":"2024-03-21T12:59:53.339622+0000","last_became_active":"2024-03-21T12:58:39.752722+0000","last_became_peered":"2024-03-21T12:58:39.752722+0000","last_unstale":"2024-03-21T12:59:53.339622+0000","last_undegraded":"2024-03-21T12:59:53.339622+0000","last_fullsized":"2024-03-21T12:59:53.339622+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:53.379100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840407+0000","last_change":"2024-03-21T12:58:43.353751+0000","last_active":"2024-03-21T12:59:53.840407+0000","last_peered":"2024-03-21T12:59:53.840407+0000","last_clean":"2024-03-21T12:59:53.840407+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T12:59:53.840407+0000","last_undegraded":"2024-03-21T12:59:53.840407+0000","last_fullsized":"2024-03-21T12:59:53.840407+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:57.020516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315924+0000","last_change":"2024-03-21T12:58:41.162672+0000","last_active":"2024-03-21T12:59:53.315924+0000","last_peered":"2024-03-21T12:59:53.315924+0000","last_clean":"2024-03-21T12:59:53.315924+0000","last_became_active":"2024-03-21T12:58:41.162343+0000","last_became_peered":"2024-03-21T12:58:41.162343+0000","last_unstale":"2024-03-21T12:59:53.315924+0000","last_undegraded":"2024-03-21T12:59:53.315924+0000","last_fullsized":"2024-03-21T12:59:53.315924+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:06.284594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.754353+0000","last_change":"2024-03-21T12:59:52.319029+0000","last_active":"2024-03-21T12:59:53.754353+0000","last_peered":"2024-03-21T12:59:53.754353+0000","last_clean":"2024-03-21T12:59:53.754353+0000","last_became_active":"2024-03-21T12:58:37.731932+0000","last_became_peered":"2024-03-21T12:58:37.731932+0000","last_unstale":"2024-03-21T12:59:53.754353+0000","last_undegraded":"2024-03-21T12:59:53.754353+0000","last_fullsized":"2024-03-21T12:59:53.754353+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:52.318927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.318927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:20.356137+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'6","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316455+0000","last_change":"2024-03-21T12:59:46.176940+0000","last_active":"2024-03-21T12:59:53.316455+0000","last_peered":"2024-03-21T12:59:53.316455+0000","last_clean":"2024-03-21T12:59:53.316455+0000","last_became_active":"2024-03-21T12:58:39.755419+0000","last_became_peered":"2024-03-21T12:58:39.755419+0000","last_unstale":"2024-03-21T12:59:53.316455+0000","last_undegraded":"2024-03-21T12:59:53.316455+0000","last_fullsized":"2024-03-21T12:59:53.316455+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T12:59:46.176819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.176819+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:06.815464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315710+0000","last_change":"2024-03-21T12:58:43.351431+0000","last_active":"2024-03-21T12:59:53.315710+0000","last_peered":"2024-03-21T12:59:53.315710+0000","last_clean":"2024-03-21T12:59:53.315710+0000","last_became_active":"2024-03-21T12:58:43.351103+0000","last_became_peered":"2024-03-21T12:58:43.351103+0000","last_unstale":"2024-03-21T12:59:53.315710+0000","last_undegraded":"2024-03-21T12:59:53.315710+0000","last_fullsized":"2024-03-21T12:59:53.315710+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841274+0000","last_change":"2024-03-21T12:58:37.735326+0000","last_active":"2024-03-21T12:59:53.841274+0000","last_peered":"2024-03-21T12:59:53.841274+0000","last_clean":"2024-03-21T12:59:53.841274+0000","last_became_active":"2024-03-21T12:58:37.735037+0000","last_became_peered":"2024-03-21T12:58:37.735037+0000","last_unstale":"2024-03-21T12:59:53.841274+0000","last_undegraded":"2024-03-21T12:59:53.841274+0000","last_fullsized":"2024-03-21T12:59:53.841274+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:54.657655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'15","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315400+0000","last_change":"2024-03-21T12:58:39.750902+0000","last_active":"2024-03-21T12:59:53.315400+0000","last_peered":"2024-03-21T12:59:53.315400+0000","last_clean":"2024-03-21T12:59:53.315400+0000","last_became_active":"2024-03-21T12:58:39.750546+0000","last_became_peered":"2024-03-21T12:58:39.750546+0000","last_unstale":"2024-03-21T12:59:53.315400+0000","last_undegraded":"2024-03-21T12:59:53.315400+0000","last_fullsized":"2024-03-21T12:59:53.315400+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:02.801886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"71'8","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840421+0000","last_change":"2024-03-21T12:58:41.166431+0000","last_active":"2024-03-21T12:59:53.840421+0000","last_peered":"2024-03-21T12:59:53.840421+0000","last_clean":"2024-03-21T12:59:53.840421+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T12:59:53.840421+0000","last_undegraded":"2024-03-21T12:59:53.840421+0000","last_fullsized":"2024-03-21T12:59:53.840421+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:08.044850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315189+0000","last_change":"2024-03-21T12:58:43.181756+0000","last_active":"2024-03-21T12:59:53.315189+0000","last_peered":"2024-03-21T12:59:53.315189+0000","last_clean":"2024-03-21T12:59:53.315189+0000","last_became_active":"2024-03-21T12:58:43.181036+0000","last_became_peered":"2024-03-21T12:58:43.181036+0000","last_unstale":"2024-03-21T12:59:53.315189+0000","last_undegraded":"2024-03-21T12:59:53.315189+0000","last_fullsized":"2024-03-21T12:59:53.315189+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:59.136913+0000","last_change":"2024-03-21T12:59:59.136937+0000","last_active":"2024-03-21T12:59:59.136913+0000","last_peered":"2024-03-21T12:59:59.136913+0000","last_clean":"2024-03-21T12:59:59.136913+0000","last_became_active":"2024-03-21T12:58:37.737012+0000","last_became_peered":"2024-03-21T12:58:37.737012+0000","last_unstale":"2024-03-21T12:59:59.136913+0000","last_undegraded":"2024-03-21T12:59:59.136913+0000","last_fullsized":"2024-03-21T12:59:59.136913+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:59.136829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.136829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:03.237208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"71'12","reported_seq":77,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840417+0000","last_change":"2024-03-21T12:58:39.751648+0000","last_active":"2024-03-21T12:59:53.840417+0000","last_peered":"2024-03-21T12:59:53.840417+0000","last_clean":"2024-03-21T12:59:53.840417+0000","last_became_active":"2024-03-21T12:58:39.751236+0000","last_became_peered":"2024-03-21T12:58:39.751236+0000","last_unstale":"2024-03-21T12:59:53.840417+0000","last_undegraded":"2024-03-21T12:59:53.840417+0000","last_fullsized":"2024-03-21T12:59:53.840417+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:04.196572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:58.733332+0000","last_change":"2024-03-21T12:59:58.733374+0000","last_active":"2024-03-21T12:59:58.733332+0000","last_peered":"2024-03-21T12:59:58.733332+0000","last_clean":"2024-03-21T12:59:58.733332+0000","last_became_active":"2024-03-21T12:58:41.171443+0000","last_became_peered":"2024-03-21T12:58:41.171443+0000","last_unstale":"2024-03-21T12:59:58.733332+0000","last_undegraded":"2024-03-21T12:59:58.733332+0000","last_fullsized":"2024-03-21T12:59:58.733332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:58.733235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:58.733235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.428021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.839798+0000","last_change":"2024-03-21T12:58:43.179289+0000","last_active":"2024-03-21T12:59:53.839798+0000","last_peered":"2024-03-21T12:59:53.839798+0000","last_clean":"2024-03-21T12:59:53.839798+0000","last_became_active":"2024-03-21T12:58:43.179007+0000","last_became_peered":"2024-03-21T12:58:43.179007+0000","last_unstale":"2024-03-21T12:59:53.839798+0000","last_undegraded":"2024-03-21T12:59:53.839798+0000","last_fullsized":"2024-03-21T12:59:53.839798+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316145+0000","last_change":"2024-03-21T12:58:41.166265+0000","last_active":"2024-03-21T12:59:53.316145+0000","last_peered":"2024-03-21T12:59:53.316145+0000","last_clean":"2024-03-21T12:59:53.316145+0000","last_became_active":"2024-03-21T12:58:41.165203+0000","last_became_peered":"2024-03-21T12:58:41.165203+0000","last_unstale":"2024-03-21T12:59:53.316145+0000","last_undegraded":"2024-03-21T12:59:53.316145+0000","last_fullsized":"2024-03-21T12:59:53.316145+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:08.244050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840816+0000","last_change":"2024-03-21T12:59:46.169468+0000","last_active":"2024-03-21T12:59:53.840816+0000","last_peered":"2024-03-21T12:59:53.840816+0000","last_clean":"2024-03-21T12:59:53.840816+0000","last_became_active":"2024-03-21T12:58:37.731448+0000","last_became_peered":"2024-03-21T12:58:37.731448+0000","last_unstale":"2024-03-21T12:59:53.840816+0000","last_undegraded":"2024-03-21T12:59:53.840816+0000","last_fullsized":"2024-03-21T12:59:53.840816+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:46.169315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.169315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:51.199764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"71'19","reported_seq":93,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:59.392934+0000","last_change":"2024-03-21T12:59:59.392987+0000","last_active":"2024-03-21T12:59:59.392934+0000","last_peered":"2024-03-21T12:59:59.392934+0000","last_clean":"2024-03-21T12:59:59.392934+0000","last_became_active":"2024-03-21T12:58:39.754761+0000","last_became_peered":"2024-03-21T12:58:39.754761+0000","last_unstale":"2024-03-21T12:59:59.392934+0000","last_undegraded":"2024-03-21T12:59:59.392934+0000","last_fullsized":"2024-03-21T12:59:59.392934+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T12:59:59.392834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.392834+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.114737+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339138+0000","last_change":"2024-03-21T12:58:43.180724+0000","last_active":"2024-03-21T12:59:53.339138+0000","last_peered":"2024-03-21T12:59:53.339138+0000","last_clean":"2024-03-21T12:59:53.339138+0000","last_became_active":"2024-03-21T12:58:43.180393+0000","last_became_peered":"2024-03-21T12:58:43.180393+0000","last_unstale":"2024-03-21T12:59:53.339138+0000","last_undegraded":"2024-03-21T12:59:53.339138+0000","last_fullsized":"2024-03-21T12:59:53.339138+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:02.348923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.317040+0000","last_change":"2024-03-21T12:58:41.164684+0000","last_active":"2024-03-21T12:59:53.317040+0000","last_peered":"2024-03-21T12:59:53.317040+0000","last_clean":"2024-03-21T12:59:53.317040+0000","last_became_active":"2024-03-21T12:58:41.163843+0000","last_became_peered":"2024-03-21T12:58:41.163843+0000","last_unstale":"2024-03-21T12:59:53.317040+0000","last_undegraded":"2024-03-21T12:59:53.317040+0000","last_fullsized":"2024-03-21T12:59:53.317040+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:53.709873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315887+0000","last_change":"2024-03-21T12:59:47.676292+0000","last_active":"2024-03-21T12:59:53.315887+0000","last_peered":"2024-03-21T12:59:53.315887+0000","last_clean":"2024-03-21T12:59:53.315887+0000","last_became_active":"2024-03-21T12:58:37.730518+0000","last_became_peered":"2024-03-21T12:58:37.730518+0000","last_unstale":"2024-03-21T12:59:53.315887+0000","last_undegraded":"2024-03-21T12:59:53.315887+0000","last_fullsized":"2024-03-21T12:59:53.315887+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:47.676198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.676198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.156465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"86'13","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339817+0000","last_change":"2024-03-21T12:59:46.421228+0000","last_active":"2024-03-21T12:59:53.339817+0000","last_peered":"2024-03-21T12:59:53.339817+0000","last_clean":"2024-03-21T12:59:53.339817+0000","last_became_active":"2024-03-21T12:58:39.748144+0000","last_became_peered":"2024-03-21T12:58:39.748144+0000","last_unstale":"2024-03-21T12:59:53.339817+0000","last_undegraded":"2024-03-21T12:59:53.339817+0000","last_fullsized":"2024-03-21T12:59:53.339817+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.421144+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:06.815237+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315865+0000","last_change":"2024-03-21T12:58:43.180710+0000","last_active":"2024-03-21T12:59:53.315865+0000","last_peered":"2024-03-21T12:59:53.315865+0000","last_clean":"2024-03-21T12:59:53.315865+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T12:59:53.315865+0000","last_undegraded":"2024-03-21T12:59:53.315865+0000","last_fullsized":"2024-03-21T12:59:53.315865+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:10.959635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"67'2","reported_seq":77,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:55.541514+0000","last_change":"2024-03-21T12:59:55.541552+0000","last_active":"2024-03-21T12:59:55.541514+0000","last_peered":"2024-03-21T12:59:55.541514+0000","last_clean":"2024-03-21T12:59:55.541514+0000","last_became_active":"2024-03-21T12:58:36.724010+0000","last_became_peered":"2024-03-21T12:58:36.724010+0000","last_unstale":"2024-03-21T12:59:55.541514+0000","last_undegraded":"2024-03-21T12:59:55.541514+0000","last_fullsized":"2024-03-21T12:59:55.541514+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-21T12:59:55.541410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.541410+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:17.828687+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840854+0000","last_change":"2024-03-21T12:59:47.840935+0000","last_active":"2024-03-21T12:59:53.840854+0000","last_peered":"2024-03-21T12:59:53.840854+0000","last_clean":"2024-03-21T12:59:53.840854+0000","last_became_active":"2024-03-21T12:58:37.729454+0000","last_became_peered":"2024-03-21T12:58:37.729454+0000","last_unstale":"2024-03-21T12:59:53.840854+0000","last_undegraded":"2024-03-21T12:59:53.840854+0000","last_fullsized":"2024-03-21T12:59:53.840854+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:47.840831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.840831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.579290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"71'28","reported_seq":119,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710985+0000","last_change":"2024-03-21T12:58:39.756346+0000","last_active":"2024-03-21T12:59:53.710985+0000","last_peered":"2024-03-21T12:59:53.710985+0000","last_clean":"2024-03-21T12:59:53.710985+0000","last_became_active":"2024-03-21T12:58:39.756052+0000","last_became_peered":"2024-03-21T12:58:39.756052+0000","last_unstale":"2024-03-21T12:59:53.710985+0000","last_undegraded":"2024-03-21T12:59:53.710985+0000","last_fullsized":"2024-03-21T12:59:53.710985+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339496+0000","last_change":"2024-03-21T12:58:41.170763+0000","last_active":"2024-03-21T12:59:53.339496+0000","last_peered":"2024-03-21T12:59:53.339496+0000","last_clean":"2024-03-21T12:59:53.339496+0000","last_became_active":"2024-03-21T12:58:41.170460+0000","last_became_peered":"2024-03-21T12:58:41.170460+0000","last_unstale":"2024-03-21T12:59:53.339496+0000","last_undegraded":"2024-03-21T12:59:53.339496+0000","last_fullsized":"2024-03-21T12:59:53.339496+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:06.578225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:54.191221+0000","last_change":"2024-03-21T12:59:54.191245+0000","last_active":"2024-03-21T12:59:54.191221+0000","last_peered":"2024-03-21T12:59:54.191221+0000","last_clean":"2024-03-21T12:59:54.191221+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T12:59:54.191221+0000","last_undegraded":"2024-03-21T12:59:54.191221+0000","last_fullsized":"2024-03-21T12:59:54.191221+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:54.191150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:54.191150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:15.445044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710593+0000","last_change":"2024-03-21T12:58:37.733226+0000","last_active":"2024-03-21T12:59:53.710593+0000","last_peered":"2024-03-21T12:59:53.710593+0000","last_clean":"2024-03-21T12:59:53.710593+0000","last_became_active":"2024-03-21T12:58:37.731386+0000","last_became_peered":"2024-03-21T12:58:37.731386+0000","last_unstale":"2024-03-21T12:59:53.710593+0000","last_undegraded":"2024-03-21T12:59:53.710593+0000","last_fullsized":"2024-03-21T12:59:53.710593+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:03.233816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.734474+0000","last_change":"2024-03-21T12:59:57.734511+0000","last_active":"2024-03-21T12:59:57.734474+0000","last_peered":"2024-03-21T12:59:57.734474+0000","last_clean":"2024-03-21T12:59:57.734474+0000","last_became_active":"2024-03-21T12:58:36.724067+0000","last_became_peered":"2024-03-21T12:58:36.724067+0000","last_unstale":"2024-03-21T12:59:57.734474+0000","last_undegraded":"2024-03-21T12:59:57.734474+0000","last_fullsized":"2024-03-21T12:59:57.734474+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.734379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.734379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.789374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000743678,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'13","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.711251+0000","last_change":"2024-03-21T12:58:39.750379+0000","last_active":"2024-03-21T12:59:53.711251+0000","last_peered":"2024-03-21T12:59:53.711251+0000","last_clean":"2024-03-21T12:59:53.711251+0000","last_became_active":"2024-03-21T12:58:39.750068+0000","last_became_peered":"2024-03-21T12:58:39.750068+0000","last_unstale":"2024-03-21T12:59:53.711251+0000","last_undegraded":"2024-03-21T12:59:53.711251+0000","last_fullsized":"2024-03-21T12:59:53.711251+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:58.393230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316373+0000","last_change":"2024-03-21T12:58:41.165072+0000","last_active":"2024-03-21T12:59:53.316373+0000","last_peered":"2024-03-21T12:59:53.316373+0000","last_clean":"2024-03-21T12:59:53.316373+0000","last_became_active":"2024-03-21T12:58:41.164725+0000","last_became_peered":"2024-03-21T12:58:41.164725+0000","last_unstale":"2024-03-21T12:59:53.316373+0000","last_undegraded":"2024-03-21T12:59:53.316373+0000","last_fullsized":"2024-03-21T12:59:53.316373+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:59.205553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316074+0000","last_change":"2024-03-21T12:59:52.658473+0000","last_active":"2024-03-21T12:59:53.316074+0000","last_peered":"2024-03-21T12:59:53.316074+0000","last_clean":"2024-03-21T12:59:53.316074+0000","last_became_active":"2024-03-21T12:58:43.183113+0000","last_became_peered":"2024-03-21T12:58:43.183113+0000","last_unstale":"2024-03-21T12:59:53.316074+0000","last_undegraded":"2024-03-21T12:59:53.316074+0000","last_fullsized":"2024-03-21T12:59:53.316074+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:52.658377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.658377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.565790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":314,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.839938+0000","last_change":"2024-03-21T12:59:02.091964+0000","last_active":"2024-03-21T12:59:53.839938+0000","last_peered":"2024-03-21T12:59:53.839938+0000","last_clean":"2024-03-21T12:59:53.839938+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T12:59:53.839938+0000","last_undegraded":"2024-03-21T12:59:53.839938+0000","last_fullsized":"2024-03-21T12:59:53.839938+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'119","last_scrub_stamp":"2024-03-21T12:59:02.091835+0000","last_deep_scrub":"76'119","last_deep_scrub_stamp":"2024-03-21T12:59:02.091835+0000","last_clean_scrub_stamp":"2024-03-21T12:59:02.091835+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:02.880876+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.314920+0000","last_change":"2024-03-21T12:58:37.729254+0000","last_active":"2024-03-21T12:59:53.314920+0000","last_peered":"2024-03-21T12:59:53.314920+0000","last_clean":"2024-03-21T12:59:53.314920+0000","last_became_active":"2024-03-21T12:58:37.728911+0000","last_became_peered":"2024-03-21T12:58:37.728911+0000","last_unstale":"2024-03-21T12:59:53.314920+0000","last_undegraded":"2024-03-21T12:59:53.314920+0000","last_fullsized":"2024-03-21T12:59:53.314920+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:57.212471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":101,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:56.298894+0000","last_change":"2024-03-21T12:59:56.298894+0000","last_active":"2024-03-21T12:59:56.298894+0000","last_peered":"2024-03-21T12:59:56.298894+0000","last_clean":"2024-03-21T12:59:56.298894+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T12:59:56.298894+0000","last_undegraded":"2024-03-21T12:59:56.298894+0000","last_fullsized":"2024-03-21T12:59:56.298894+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-21T12:59:56.298768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:56.298768+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.767366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840609+0000","last_change":"2024-03-21T12:58:41.168317+0000","last_active":"2024-03-21T12:59:53.840609+0000","last_peered":"2024-03-21T12:59:53.840609+0000","last_clean":"2024-03-21T12:59:53.840609+0000","last_became_active":"2024-03-21T12:58:41.168003+0000","last_became_peered":"2024-03-21T12:58:41.168003+0000","last_unstale":"2024-03-21T12:59:53.840609+0000","last_undegraded":"2024-03-21T12:59:53.840609+0000","last_fullsized":"2024-03-21T12:59:53.840609+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:58.477533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710736+0000","last_change":"2024-03-21T12:58:43.184189+0000","last_active":"2024-03-21T12:59:53.710736+0000","last_peered":"2024-03-21T12:59:53.710736+0000","last_clean":"2024-03-21T12:59:53.710736+0000","last_became_active":"2024-03-21T12:58:43.183860+0000","last_became_peered":"2024-03-21T12:58:43.183860+0000","last_unstale":"2024-03-21T12:59:53.710736+0000","last_undegraded":"2024-03-21T12:59:53.710736+0000","last_fullsized":"2024-03-21T12:59:53.710736+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:01.953966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339323+0000","last_change":"2024-03-21T12:58:37.734459+0000","last_active":"2024-03-21T12:59:53.339323+0000","last_peered":"2024-03-21T12:59:53.339323+0000","last_clean":"2024-03-21T12:59:53.339323+0000","last_became_active":"2024-03-21T12:58:37.734176+0000","last_became_peered":"2024-03-21T12:58:37.734176+0000","last_unstale":"2024-03-21T12:59:53.339323+0000","last_undegraded":"2024-03-21T12:59:53.339323+0000","last_fullsized":"2024-03-21T12:59:53.339323+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:56.304166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":143,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:58.223366+0000","last_change":"2024-03-21T12:58:40.754485+0000","last_active":"2024-03-21T12:59:58.223366+0000","last_peered":"2024-03-21T12:59:58.223366+0000","last_clean":"2024-03-21T12:59:58.223366+0000","last_became_active":"2024-03-21T12:58:36.725735+0000","last_became_peered":"2024-03-21T12:58:36.725735+0000","last_unstale":"2024-03-21T12:59:58.223366+0000","last_undegraded":"2024-03-21T12:59:58.223366+0000","last_fullsized":"2024-03-21T12:59:58.223366+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:58:35.713454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:02.864092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339478+0000","last_change":"2024-03-21T12:58:39.747623+0000","last_active":"2024-03-21T12:59:53.339478+0000","last_peered":"2024-03-21T12:59:53.339478+0000","last_clean":"2024-03-21T12:59:53.339478+0000","last_became_active":"2024-03-21T12:58:39.747315+0000","last_became_peered":"2024-03-21T12:58:39.747315+0000","last_unstale":"2024-03-21T12:59:53.339478+0000","last_undegraded":"2024-03-21T12:59:53.339478+0000","last_fullsized":"2024-03-21T12:59:53.339478+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316244+0000","last_change":"2024-03-21T12:58:41.167475+0000","last_active":"2024-03-21T12:59:53.316244+0000","last_peered":"2024-03-21T12:59:53.316244+0000","last_clean":"2024-03-21T12:59:53.316244+0000","last_became_active":"2024-03-21T12:58:41.167131+0000","last_became_peered":"2024-03-21T12:58:41.167131+0000","last_unstale":"2024-03-21T12:59:53.316244+0000","last_undegraded":"2024-03-21T12:59:53.316244+0000","last_fullsized":"2024-03-21T12:59:53.316244+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:58.954135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710418+0000","last_change":"2024-03-21T12:58:43.351489+0000","last_active":"2024-03-21T12:59:53.710418+0000","last_peered":"2024-03-21T12:59:53.710418+0000","last_clean":"2024-03-21T12:59:53.710418+0000","last_became_active":"2024-03-21T12:58:43.351138+0000","last_became_peered":"2024-03-21T12:58:43.351138+0000","last_unstale":"2024-03-21T12:59:53.710418+0000","last_undegraded":"2024-03-21T12:59:53.710418+0000","last_fullsized":"2024-03-21T12:59:53.710418+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:56.857483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.925271+0000","last_change":"2024-03-21T12:59:57.925311+0000","last_active":"2024-03-21T12:59:57.925271+0000","last_peered":"2024-03-21T12:59:57.925271+0000","last_clean":"2024-03-21T12:59:57.925271+0000","last_became_active":"2024-03-21T12:58:37.729818+0000","last_became_peered":"2024-03-21T12:58:37.729818+0000","last_unstale":"2024-03-21T12:59:57.925271+0000","last_undegraded":"2024-03-21T12:59:57.925271+0000","last_fullsized":"2024-03-21T12:59:57.925271+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.925179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.925179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.845352+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"71'19","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339719+0000","last_change":"2024-03-21T12:58:39.751068+0000","last_active":"2024-03-21T12:59:53.339719+0000","last_peered":"2024-03-21T12:59:53.339719+0000","last_clean":"2024-03-21T12:59:53.339719+0000","last_became_active":"2024-03-21T12:58:39.750216+0000","last_became_peered":"2024-03-21T12:58:39.750216+0000","last_unstale":"2024-03-21T12:59:53.339719+0000","last_undegraded":"2024-03-21T12:59:53.339719+0000","last_fullsized":"2024-03-21T12:59:53.339719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:57.408510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.754642+0000","last_change":"2024-03-21T12:58:41.166808+0000","last_active":"2024-03-21T12:59:53.754642+0000","last_peered":"2024-03-21T12:59:53.754642+0000","last_clean":"2024-03-21T12:59:53.754642+0000","last_became_active":"2024-03-21T12:58:41.166495+0000","last_became_peered":"2024-03-21T12:58:41.166495+0000","last_unstale":"2024-03-21T12:59:53.754642+0000","last_undegraded":"2024-03-21T12:59:53.754642+0000","last_fullsized":"2024-03-21T12:59:53.754642+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:59.378636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339098+0000","last_change":"2024-03-21T12:58:43.177740+0000","last_active":"2024-03-21T12:59:53.339098+0000","last_peered":"2024-03-21T12:59:53.339098+0000","last_clean":"2024-03-21T12:59:53.339098+0000","last_became_active":"2024-03-21T12:58:43.177409+0000","last_became_peered":"2024-03-21T12:58:43.177409+0000","last_unstale":"2024-03-21T12:59:53.339098+0000","last_undegraded":"2024-03-21T12:59:53.339098+0000","last_fullsized":"2024-03-21T12:59:53.339098+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315902+0000","last_change":"2024-03-21T12:58:37.730839+0000","last_active":"2024-03-21T12:59:53.315902+0000","last_peered":"2024-03-21T12:59:53.315902+0000","last_clean":"2024-03-21T12:59:53.315902+0000","last_became_active":"2024-03-21T12:58:37.730547+0000","last_became_peered":"2024-03-21T12:58:37.730547+0000","last_unstale":"2024-03-21T12:59:53.315902+0000","last_undegraded":"2024-03-21T12:59:53.315902+0000","last_fullsized":"2024-03-21T12:59:53.315902+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:01.180130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"86'15","reported_seq":60,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316813+0000","last_change":"2024-03-21T12:59:45.653329+0000","last_active":"2024-03-21T12:59:53.316813+0000","last_peered":"2024-03-21T12:59:53.316813+0000","last_clean":"2024-03-21T12:59:53.316813+0000","last_became_active":"2024-03-21T12:59:01.160456+0000","last_became_peered":"2024-03-21T12:59:01.160456+0000","last_unstale":"2024-03-21T12:59:53.316813+0000","last_undegraded":"2024-03-21T12:59:53.316813+0000","last_fullsized":"2024-03-21T12:59:53.316813+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.653197+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.205091+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339579+0000","last_change":"2024-03-21T12:59:48.409996+0000","last_active":"2024-03-21T12:59:53.339579+0000","last_peered":"2024-03-21T12:59:53.339579+0000","last_clean":"2024-03-21T12:59:53.339579+0000","last_became_active":"2024-03-21T12:58:41.171770+0000","last_became_peered":"2024-03-21T12:58:41.171770+0000","last_unstale":"2024-03-21T12:59:53.339579+0000","last_undegraded":"2024-03-21T12:59:53.339579+0000","last_fullsized":"2024-03-21T12:59:53.339579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:48.409905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.409905+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.077210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841012+0000","last_change":"2024-03-21T12:58:43.353379+0000","last_active":"2024-03-21T12:59:53.841012+0000","last_peered":"2024-03-21T12:59:53.841012+0000","last_clean":"2024-03-21T12:59:53.841012+0000","last_became_active":"2024-03-21T12:58:43.353049+0000","last_became_peered":"2024-03-21T12:58:43.353049+0000","last_unstale":"2024-03-21T12:59:53.841012+0000","last_undegraded":"2024-03-21T12:59:53.841012+0000","last_fullsized":"2024-03-21T12:59:53.841012+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:03.153074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339519+0000","last_change":"2024-03-21T12:58:37.735715+0000","last_active":"2024-03-21T12:59:53.339519+0000","last_peered":"2024-03-21T12:59:53.339519+0000","last_clean":"2024-03-21T12:59:53.339519+0000","last_became_active":"2024-03-21T12:58:37.735421+0000","last_became_peered":"2024-03-21T12:58:37.735421+0000","last_unstale":"2024-03-21T12:59:53.339519+0000","last_undegraded":"2024-03-21T12:59:53.339519+0000","last_fullsized":"2024-03-21T12:59:53.339519+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:04.750303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"87'21","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840274+0000","last_change":"2024-03-21T12:59:52.916371+0000","last_active":"2024-03-21T12:59:53.840274+0000","last_peered":"2024-03-21T12:59:53.840274+0000","last_clean":"2024-03-21T12:59:53.840274+0000","last_became_active":"2024-03-21T12:58:39.743592+0000","last_became_peered":"2024-03-21T12:58:39.743592+0000","last_unstale":"2024-03-21T12:59:53.840274+0000","last_undegraded":"2024-03-21T12:59:53.840274+0000","last_fullsized":"2024-03-21T12:59:53.840274+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'21","last_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.916260+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.636952+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"71'8","reported_seq":75,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.314631+0000","last_change":"2024-03-21T12:58:41.174471+0000","last_active":"2024-03-21T12:59:53.314631+0000","last_peered":"2024-03-21T12:59:53.314631+0000","last_clean":"2024-03-21T12:59:53.314631+0000","last_became_active":"2024-03-21T12:58:41.174137+0000","last_became_peered":"2024-03-21T12:58:41.174137+0000","last_unstale":"2024-03-21T12:59:53.314631+0000","last_undegraded":"2024-03-21T12:59:53.314631+0000","last_fullsized":"2024-03-21T12:59:53.314631+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:59.446102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840340+0000","last_change":"2024-03-21T12:58:43.181411+0000","last_active":"2024-03-21T12:59:53.840340+0000","last_peered":"2024-03-21T12:59:53.840340+0000","last_clean":"2024-03-21T12:59:53.840340+0000","last_became_active":"2024-03-21T12:58:43.180939+0000","last_became_peered":"2024-03-21T12:58:43.180939+0000","last_unstale":"2024-03-21T12:59:53.840340+0000","last_undegraded":"2024-03-21T12:59:53.840340+0000","last_fullsized":"2024-03-21T12:59:53.840340+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316308+0000","last_change":"2024-03-21T12:59:43.240826+0000","last_active":"2024-03-21T12:59:53.316308+0000","last_peered":"2024-03-21T12:59:53.316308+0000","last_clean":"2024-03-21T12:59:53.316308+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T12:59:53.316308+0000","last_undegraded":"2024-03-21T12:59:53.316308+0000","last_fullsized":"2024-03-21T12:59:53.316308+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T12:59:43.240674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.075915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"71'18","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.314541+0000","last_change":"2024-03-21T12:59:48.179286+0000","last_active":"2024-03-21T12:59:53.314541+0000","last_peered":"2024-03-21T12:59:53.314541+0000","last_clean":"2024-03-21T12:59:53.314541+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T12:59:53.314541+0000","last_undegraded":"2024-03-21T12:59:53.314541+0000","last_fullsized":"2024-03-21T12:59:53.314541+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T12:59:48.179188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.179188+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:15.921161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840082+0000","last_change":"2024-03-21T12:58:41.165071+0000","last_active":"2024-03-21T12:59:53.840082+0000","last_peered":"2024-03-21T12:59:53.840082+0000","last_clean":"2024-03-21T12:59:53.840082+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T12:59:53.840082+0000","last_undegraded":"2024-03-21T12:59:53.840082+0000","last_fullsized":"2024-03-21T12:59:53.840082+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:06.598245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.599170+0000","last_change":"2024-03-21T12:59:57.599220+0000","last_active":"2024-03-21T12:59:57.599170+0000","last_peered":"2024-03-21T12:59:57.599170+0000","last_clean":"2024-03-21T12:59:57.599170+0000","last_became_active":"2024-03-21T12:58:43.183614+0000","last_became_peered":"2024-03-21T12:58:43.183614+0000","last_unstale":"2024-03-21T12:59:57.599170+0000","last_undegraded":"2024-03-21T12:59:57.599170+0000","last_fullsized":"2024-03-21T12:59:57.599170+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.599076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.599076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:00.058326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":75,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841723+0000","last_change":"2024-03-21T12:58:41.172355+0000","last_active":"2024-03-21T12:59:53.841723+0000","last_peered":"2024-03-21T12:59:53.841723+0000","last_clean":"2024-03-21T12:59:53.841723+0000","last_became_active":"2024-03-21T12:58:41.172019+0000","last_became_peered":"2024-03-21T12:58:41.172019+0000","last_unstale":"2024-03-21T12:59:53.841723+0000","last_undegraded":"2024-03-21T12:59:53.841723+0000","last_fullsized":"2024-03-21T12:59:53.841723+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:01.420292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.314481+0000","last_change":"2024-03-21T12:59:41.231430+0000","last_active":"2024-03-21T12:59:53.314481+0000","last_peered":"2024-03-21T12:59:53.314481+0000","last_clean":"2024-03-21T12:59:53.314481+0000","last_became_active":"2024-03-21T12:58:37.731327+0000","last_became_peered":"2024-03-21T12:58:37.731327+0000","last_unstale":"2024-03-21T12:59:53.314481+0000","last_undegraded":"2024-03-21T12:59:53.314481+0000","last_fullsized":"2024-03-21T12:59:53.314481+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:41.231340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:41.231340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.247538+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.711208+0000","last_change":"2024-03-21T12:58:39.755619+0000","last_active":"2024-03-21T12:59:53.711208+0000","last_peered":"2024-03-21T12:59:53.711208+0000","last_clean":"2024-03-21T12:59:53.711208+0000","last_became_active":"2024-03-21T12:58:39.755326+0000","last_became_peered":"2024-03-21T12:58:39.755326+0000","last_unstale":"2024-03-21T12:59:53.711208+0000","last_undegraded":"2024-03-21T12:59:53.711208+0000","last_fullsized":"2024-03-21T12:59:53.711208+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:03.535907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315577+0000","last_change":"2024-03-21T12:58:43.351417+0000","last_active":"2024-03-21T12:59:53.315577+0000","last_peered":"2024-03-21T12:59:53.315577+0000","last_clean":"2024-03-21T12:59:53.315577+0000","last_became_active":"2024-03-21T12:58:43.350921+0000","last_became_peered":"2024-03-21T12:58:43.350921+0000","last_unstale":"2024-03-21T12:59:53.315577+0000","last_undegraded":"2024-03-21T12:59:53.315577+0000","last_fullsized":"2024-03-21T12:59:53.315577+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:53.390578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316213+0000","last_change":"2024-03-21T12:58:41.169093+0000","last_active":"2024-03-21T12:59:53.316213+0000","last_peered":"2024-03-21T12:59:53.316213+0000","last_clean":"2024-03-21T12:59:53.316213+0000","last_became_active":"2024-03-21T12:58:41.168784+0000","last_became_peered":"2024-03-21T12:58:41.168784+0000","last_unstale":"2024-03-21T12:59:53.316213+0000","last_undegraded":"2024-03-21T12:59:53.316213+0000","last_fullsized":"2024-03-21T12:59:53.316213+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:55.289928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840178+0000","last_change":"2024-03-21T12:58:37.731156+0000","last_active":"2024-03-21T12:59:53.840178+0000","last_peered":"2024-03-21T12:59:53.840178+0000","last_clean":"2024-03-21T12:59:53.840178+0000","last_became_active":"2024-03-21T12:58:37.730858+0000","last_became_peered":"2024-03-21T12:58:37.730858+0000","last_unstale":"2024-03-21T12:59:53.840178+0000","last_undegraded":"2024-03-21T12:59:53.840178+0000","last_fullsized":"2024-03-21T12:59:53.840178+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:53.876749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'11","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841547+0000","last_change":"2024-03-21T12:58:39.754932+0000","last_active":"2024-03-21T12:59:53.841547+0000","last_peered":"2024-03-21T12:59:53.841547+0000","last_clean":"2024-03-21T12:59:53.841547+0000","last_became_active":"2024-03-21T12:58:39.754337+0000","last_became_peered":"2024-03-21T12:58:39.754337+0000","last_unstale":"2024-03-21T12:59:53.841547+0000","last_undegraded":"2024-03-21T12:59:53.841547+0000","last_fullsized":"2024-03-21T12:59:53.841547+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:04.723925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316645+0000","last_change":"2024-03-21T12:58:43.183848+0000","last_active":"2024-03-21T12:59:53.316645+0000","last_peered":"2024-03-21T12:59:53.316645+0000","last_clean":"2024-03-21T12:59:53.316645+0000","last_became_active":"2024-03-21T12:58:43.183380+0000","last_became_peered":"2024-03-21T12:58:43.183380+0000","last_unstale":"2024-03-21T12:59:53.316645+0000","last_undegraded":"2024-03-21T12:59:53.316645+0000","last_fullsized":"2024-03-21T12:59:53.316645+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:11.881657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.711049+0000","last_change":"2024-03-21T12:59:50.039168+0000","last_active":"2024-03-21T12:59:53.711049+0000","last_peered":"2024-03-21T12:59:53.711049+0000","last_clean":"2024-03-21T12:59:53.711049+0000","last_became_active":"2024-03-21T12:58:41.163145+0000","last_became_peered":"2024-03-21T12:58:41.163145+0000","last_unstale":"2024-03-21T12:59:53.711049+0000","last_undegraded":"2024-03-21T12:59:53.711049+0000","last_fullsized":"2024-03-21T12:59:53.711049+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.039091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.039091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.361130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.754942+0000","last_change":"2024-03-21T12:59:49.204557+0000","last_active":"2024-03-21T12:59:53.754942+0000","last_peered":"2024-03-21T12:59:53.754942+0000","last_clean":"2024-03-21T12:59:53.754942+0000","last_became_active":"2024-03-21T12:58:37.729792+0000","last_became_peered":"2024-03-21T12:58:37.729792+0000","last_unstale":"2024-03-21T12:59:53.754942+0000","last_undegraded":"2024-03-21T12:59:53.754942+0000","last_fullsized":"2024-03-21T12:59:53.754942+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:49.204419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:49.204419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.549632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841068+0000","last_change":"2024-03-21T12:58:39.746134+0000","last_active":"2024-03-21T12:59:53.841068+0000","last_peered":"2024-03-21T12:59:53.841068+0000","last_clean":"2024-03-21T12:59:53.841068+0000","last_became_active":"2024-03-21T12:58:39.745807+0000","last_became_peered":"2024-03-21T12:58:39.745807+0000","last_unstale":"2024-03-21T12:59:53.841068+0000","last_undegraded":"2024-03-21T12:59:53.841068+0000","last_fullsized":"2024-03-21T12:59:53.841068+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:06.303038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841680+0000","last_change":"2024-03-21T12:58:43.185167+0000","last_active":"2024-03-21T12:59:53.841680+0000","last_peered":"2024-03-21T12:59:53.841680+0000","last_clean":"2024-03-21T12:59:53.841680+0000","last_became_active":"2024-03-21T12:58:43.184576+0000","last_became_peered":"2024-03-21T12:58:43.184576+0000","last_unstale":"2024-03-21T12:59:53.841680+0000","last_undegraded":"2024-03-21T12:59:53.841680+0000","last_fullsized":"2024-03-21T12:59:53.841680+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:00.146215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"71'8","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316355+0000","last_change":"2024-03-21T12:58:41.165903+0000","last_active":"2024-03-21T12:59:53.316355+0000","last_peered":"2024-03-21T12:59:53.316355+0000","last_clean":"2024-03-21T12:59:53.316355+0000","last_became_active":"2024-03-21T12:58:41.165560+0000","last_became_peered":"2024-03-21T12:58:41.165560+0000","last_unstale":"2024-03-21T12:59:53.316355+0000","last_undegraded":"2024-03-21T12:59:53.316355+0000","last_fullsized":"2024-03-21T12:59:53.316355+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:09.702381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315408+0000","last_change":"2024-03-21T12:58:37.732264+0000","last_active":"2024-03-21T12:59:53.315408+0000","last_peered":"2024-03-21T12:59:53.315408+0000","last_clean":"2024-03-21T12:59:53.315408+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T12:59:53.315408+0000","last_undegraded":"2024-03-21T12:59:53.315408+0000","last_fullsized":"2024-03-21T12:59:53.315408+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841020+0000","last_change":"2024-03-21T12:58:39.750295+0000","last_active":"2024-03-21T12:59:53.841020+0000","last_peered":"2024-03-21T12:59:53.841020+0000","last_clean":"2024-03-21T12:59:53.841020+0000","last_became_active":"2024-03-21T12:58:39.749956+0000","last_became_peered":"2024-03-21T12:58:39.749956+0000","last_unstale":"2024-03-21T12:59:53.841020+0000","last_undegraded":"2024-03-21T12:59:53.841020+0000","last_fullsized":"2024-03-21T12:59:53.841020+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":415,"ondisk_log_size":415,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168597,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1499136,"data_stored":968748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462939,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32628,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712524,"statfs":{"total":95995035648,"available":95961624576,"internally_reserved":0,"allocated":1486848,"data_stored":968145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":377872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1499136,"data_stored":967579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":892928,"data_stored":377298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":377757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":913408,"data_stored":377933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724575,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":376756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:00:03.880 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:00:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T13:00:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: 6.18 scrub starts 2024-03-21T13:00:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: 6.18 scrub ok 2024-03-21T13:00:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:04.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T13:00:04.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T13:00:04.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:04.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: Deploying daemon grafana.a on smithi204 2024-03-21T13:00:04.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: 6.19 scrub starts 2024-03-21T13:00:04.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: 6.19 scrub ok 2024-03-21T13:00:04.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:03 smithi204 ceph-mon[33153]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:00:04.163 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:04.525 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:04 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[75154]: ts=2024-03-21T13:00:04.423Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000887332s 2024-03-21T13:00:04.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:04 smithi163 ceph-mon[29621]: 5.11 scrub starts 2024-03-21T13:00:04.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:04 smithi163 ceph-mon[29621]: 5.11 scrub ok 2024-03-21T13:00:04.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:04 smithi163 ceph-mon[29621]: 5.1b deep-scrub starts 2024-03-21T13:00:04.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:04 smithi163 ceph-mon[29621]: 5.1b deep-scrub ok 2024-03-21T13:00:04.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:04 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:04.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:04 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:04.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:04 smithi163 ceph-mon[40675]: 5.11 scrub starts 2024-03-21T13:00:04.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:04 smithi163 ceph-mon[40675]: 5.11 scrub ok 2024-03-21T13:00:04.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:04 smithi163 ceph-mon[40675]: 5.1b deep-scrub starts 2024-03-21T13:00:04.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:04 smithi163 ceph-mon[40675]: 5.1b deep-scrub ok 2024-03-21T13:00:04.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:04 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:04.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:04 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:04 smithi204 ceph-mon[33153]: 5.11 scrub starts 2024-03-21T13:00:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:04 smithi204 ceph-mon[33153]: 5.11 scrub ok 2024-03-21T13:00:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:04 smithi204 ceph-mon[33153]: 5.1b deep-scrub starts 2024-03-21T13:00:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:04 smithi204 ceph-mon[33153]: 5.1b deep-scrub ok 2024-03-21T13:00:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:04 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:04 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:05.666 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:00:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:00:05.666 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[29621]: 5.6 scrub starts 2024-03-21T13:00:05.666 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[29621]: 5.6 scrub ok 2024-03-21T13:00:05.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[29621]: 6.8 scrub starts 2024-03-21T13:00:05.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[29621]: 6.8 scrub ok 2024-03-21T13:00:05.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[29621]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:00:05.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[29621]: 6.1b deep-scrub starts 2024-03-21T13:00:05.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[29621]: 6.1b deep-scrub ok 2024-03-21T13:00:05.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[40675]: 5.6 scrub starts 2024-03-21T13:00:05.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[40675]: 5.6 scrub ok 2024-03-21T13:00:05.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[40675]: 6.8 scrub starts 2024-03-21T13:00:05.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[40675]: 6.8 scrub ok 2024-03-21T13:00:05.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[40675]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:00:05.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[40675]: 6.1b deep-scrub starts 2024-03-21T13:00:05.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:05 smithi163 ceph-mon[40675]: 6.1b deep-scrub ok 2024-03-21T13:00:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:05 smithi204 ceph-mon[33153]: 5.6 scrub starts 2024-03-21T13:00:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:05 smithi204 ceph-mon[33153]: 5.6 scrub ok 2024-03-21T13:00:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:05 smithi204 ceph-mon[33153]: 6.8 scrub starts 2024-03-21T13:00:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:05 smithi204 ceph-mon[33153]: 6.8 scrub ok 2024-03-21T13:00:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:05 smithi204 ceph-mon[33153]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:00:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:05 smithi204 ceph-mon[33153]: 6.1b deep-scrub starts 2024-03-21T13:00:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:05 smithi204 ceph-mon[33153]: 6.1b deep-scrub ok 2024-03-21T13:00:06.121 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:00:06.123 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:00:06.907 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:06 smithi163 ceph-mon[29621]: 6.0 scrub starts 2024-03-21T13:00:06.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:06 smithi163 ceph-mon[29621]: 6.0 scrub ok 2024-03-21T13:00:06.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:06 smithi163 ceph-mon[29621]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:00:06.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:06 smithi163 ceph-mon[29621]: 4.14 scrub starts 2024-03-21T13:00:06.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:06 smithi163 ceph-mon[29621]: 4.14 scrub ok 2024-03-21T13:00:06.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:06 smithi163 ceph-mon[40675]: 6.0 scrub starts 2024-03-21T13:00:06.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:06 smithi163 ceph-mon[40675]: 6.0 scrub ok 2024-03-21T13:00:06.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:06 smithi163 ceph-mon[40675]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:00:06.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:06 smithi163 ceph-mon[40675]: 4.14 scrub starts 2024-03-21T13:00:06.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:06 smithi163 ceph-mon[40675]: 4.14 scrub ok 2024-03-21T13:00:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:06 smithi204 ceph-mon[33153]: 6.0 scrub starts 2024-03-21T13:00:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:06 smithi204 ceph-mon[33153]: 6.0 scrub ok 2024-03-21T13:00:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:06 smithi204 ceph-mon[33153]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:00:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:06 smithi204 ceph-mon[33153]: 4.14 scrub starts 2024-03-21T13:00:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:06 smithi204 ceph-mon[33153]: 4.14 scrub ok 2024-03-21T13:00:07.216 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-03-21T13:00:04.623958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1013,"num_read_kb":859,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":624,"ondisk_log_size":624,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249916,"kb_used_data":8828,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749711300,"statfs":{"total":767960285184,"available":767704371200,"internally_reserved":0,"allocated":9039872,"data_stored":4792088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.310641"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:58.846732+0000","last_change":"2024-03-21T12:59:58.846777+0000","last_active":"2024-03-21T12:59:58.846732+0000","last_peered":"2024-03-21T12:59:58.846732+0000","last_clean":"2024-03-21T12:59:58.846732+0000","last_became_active":"2024-03-21T12:58:43.182506+0000","last_became_peered":"2024-03-21T12:58:43.182506+0000","last_unstale":"2024-03-21T12:59:58.846732+0000","last_undegraded":"2024-03-21T12:59:58.846732+0000","last_fullsized":"2024-03-21T12:59:58.846732+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:58.846634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:58.846634+0000","last_clean_scrub_stamp":"2024-03-21T12:59:58.846634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.412490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710452+0000","last_change":"2024-03-21T12:58:41.160668+0000","last_active":"2024-03-21T12:59:53.710452+0000","last_peered":"2024-03-21T12:59:53.710452+0000","last_clean":"2024-03-21T12:59:53.710452+0000","last_became_active":"2024-03-21T12:58:41.160342+0000","last_became_peered":"2024-03-21T12:58:41.160342+0000","last_unstale":"2024-03-21T12:59:53.710452+0000","last_undegraded":"2024-03-21T12:59:53.710452+0000","last_fullsized":"2024-03-21T12:59:53.710452+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:07.771079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.478199+0000","last_change":"2024-03-21T12:59:53.478218+0000","last_active":"2024-03-21T12:59:53.478199+0000","last_peered":"2024-03-21T12:59:53.478199+0000","last_clean":"2024-03-21T12:59:53.478199+0000","last_became_active":"2024-03-21T12:58:37.731393+0000","last_became_peered":"2024-03-21T12:58:37.731393+0000","last_unstale":"2024-03-21T12:59:53.478199+0000","last_undegraded":"2024-03-21T12:59:53.478199+0000","last_fullsized":"2024-03-21T12:59:53.478199+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:53.478141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:53.478141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:57.521483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"71'9","reported_seq":86,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840002+0000","last_change":"2024-03-21T12:59:01.166943+0000","last_active":"2024-03-21T12:59:53.840002+0000","last_peered":"2024-03-21T12:59:53.840002+0000","last_clean":"2024-03-21T12:59:53.840002+0000","last_became_active":"2024-03-21T12:59:01.161098+0000","last_became_peered":"2024-03-21T12:59:01.161098+0000","last_unstale":"2024-03-21T12:59:53.840002+0000","last_undegraded":"2024-03-21T12:59:53.840002+0000","last_fullsized":"2024-03-21T12:59:53.840002+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:57.399487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315279+0000","last_change":"2024-03-21T12:58:43.352892+0000","last_active":"2024-03-21T12:59:53.315279+0000","last_peered":"2024-03-21T12:59:53.315279+0000","last_clean":"2024-03-21T12:59:53.315279+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T12:59:53.315279+0000","last_undegraded":"2024-03-21T12:59:53.315279+0000","last_fullsized":"2024-03-21T12:59:53.315279+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840116+0000","last_change":"2024-03-21T12:58:41.168108+0000","last_active":"2024-03-21T12:59:53.840116+0000","last_peered":"2024-03-21T12:59:53.840116+0000","last_clean":"2024-03-21T12:59:53.840116+0000","last_became_active":"2024-03-21T12:58:41.167710+0000","last_became_peered":"2024-03-21T12:58:41.167710+0000","last_unstale":"2024-03-21T12:59:53.840116+0000","last_undegraded":"2024-03-21T12:59:53.840116+0000","last_fullsized":"2024-03-21T12:59:53.840116+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:56.955691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":64,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316055+0000","last_change":"2024-03-21T12:58:37.728682+0000","last_active":"2024-03-21T12:59:53.316055+0000","last_peered":"2024-03-21T12:59:53.316055+0000","last_clean":"2024-03-21T12:59:53.316055+0000","last_became_active":"2024-03-21T12:58:37.728383+0000","last_became_peered":"2024-03-21T12:58:37.728383+0000","last_unstale":"2024-03-21T12:59:53.316055+0000","last_undegraded":"2024-03-21T12:59:53.316055+0000","last_fullsized":"2024-03-21T12:59:53.316055+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:04.926301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"71'15","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315281+0000","last_change":"2024-03-21T12:58:39.751346+0000","last_active":"2024-03-21T12:59:53.315281+0000","last_peered":"2024-03-21T12:59:53.315281+0000","last_clean":"2024-03-21T12:59:53.315281+0000","last_became_active":"2024-03-21T12:58:39.750396+0000","last_became_peered":"2024-03-21T12:58:39.750396+0000","last_unstale":"2024-03-21T12:59:53.315281+0000","last_undegraded":"2024-03-21T12:59:53.315281+0000","last_fullsized":"2024-03-21T12:59:53.315281+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:54.161286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339278+0000","last_change":"2024-03-21T12:58:43.180242+0000","last_active":"2024-03-21T12:59:53.339278+0000","last_peered":"2024-03-21T12:59:53.339278+0000","last_clean":"2024-03-21T12:59:53.339278+0000","last_became_active":"2024-03-21T12:58:43.179920+0000","last_became_peered":"2024-03-21T12:58:43.179920+0000","last_unstale":"2024-03-21T12:59:53.339278+0000","last_undegraded":"2024-03-21T12:59:53.339278+0000","last_fullsized":"2024-03-21T12:59:53.339278+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:03.644278+0000","last_change":"2024-03-21T13:00:03.644301+0000","last_active":"2024-03-21T13:00:03.644278+0000","last_peered":"2024-03-21T13:00:03.644278+0000","last_clean":"2024-03-21T13:00:03.644278+0000","last_became_active":"2024-03-21T12:58:41.168978+0000","last_became_peered":"2024-03-21T12:58:41.168978+0000","last_unstale":"2024-03-21T13:00:03.644278+0000","last_undegraded":"2024-03-21T13:00:03.644278+0000","last_fullsized":"2024-03-21T13:00:03.644278+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:03.644209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:03.644209+0000","last_clean_scrub_stamp":"2024-03-21T13:00:03.644209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:18.843767+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315779+0000","last_change":"2024-03-21T12:59:44.661395+0000","last_active":"2024-03-21T12:59:53.315779+0000","last_peered":"2024-03-21T12:59:53.315779+0000","last_clean":"2024-03-21T12:59:53.315779+0000","last_became_active":"2024-03-21T12:58:37.733705+0000","last_became_peered":"2024-03-21T12:58:37.733705+0000","last_unstale":"2024-03-21T12:59:53.315779+0000","last_undegraded":"2024-03-21T12:59:53.315779+0000","last_fullsized":"2024-03-21T12:59:53.315779+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:44.661289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:44.661289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:45.014731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"71'9","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840441+0000","last_change":"2024-03-21T12:58:39.749391+0000","last_active":"2024-03-21T12:59:53.840441+0000","last_peered":"2024-03-21T12:59:53.840441+0000","last_clean":"2024-03-21T12:59:53.840441+0000","last_became_active":"2024-03-21T12:58:39.749010+0000","last_became_peered":"2024-03-21T12:58:39.749010+0000","last_unstale":"2024-03-21T12:59:53.840441+0000","last_undegraded":"2024-03-21T12:59:53.840441+0000","last_fullsized":"2024-03-21T12:59:53.840441+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:04.526693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:02.683922+0000","last_change":"2024-03-21T13:00:02.683946+0000","last_active":"2024-03-21T13:00:02.683922+0000","last_peered":"2024-03-21T13:00:02.683922+0000","last_clean":"2024-03-21T13:00:02.683922+0000","last_became_active":"2024-03-21T12:58:43.181494+0000","last_became_peered":"2024-03-21T12:58:43.181494+0000","last_unstale":"2024-03-21T13:00:02.683922+0000","last_undegraded":"2024-03-21T13:00:02.683922+0000","last_fullsized":"2024-03-21T13:00:02.683922+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:02.683841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:02.683841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:10.243871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840365+0000","last_change":"2024-03-21T12:59:41.158247+0000","last_active":"2024-03-21T12:59:53.840365+0000","last_peered":"2024-03-21T12:59:53.840365+0000","last_clean":"2024-03-21T12:59:53.840365+0000","last_became_active":"2024-03-21T12:58:41.163517+0000","last_became_peered":"2024-03-21T12:58:41.163517+0000","last_unstale":"2024-03-21T12:59:53.840365+0000","last_undegraded":"2024-03-21T12:59:53.840365+0000","last_fullsized":"2024-03-21T12:59:53.840365+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:41.158105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:41.158105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:45.036271+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:59.594002+0000","last_change":"2024-03-21T12:59:59.594026+0000","last_active":"2024-03-21T12:59:59.594002+0000","last_peered":"2024-03-21T12:59:59.594002+0000","last_clean":"2024-03-21T12:59:59.594002+0000","last_became_active":"2024-03-21T12:58:37.731058+0000","last_became_peered":"2024-03-21T12:58:37.731058+0000","last_unstale":"2024-03-21T12:59:59.594002+0000","last_undegraded":"2024-03-21T12:59:59.594002+0000","last_fullsized":"2024-03-21T12:59:59.594002+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:59.593927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.593927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:14.344496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.952486+0000","last_change":"2024-03-21T12:59:53.952540+0000","last_active":"2024-03-21T12:59:53.952486+0000","last_peered":"2024-03-21T12:59:53.952486+0000","last_clean":"2024-03-21T12:59:53.952486+0000","last_became_active":"2024-03-21T12:58:39.749211+0000","last_became_peered":"2024-03-21T12:58:39.749211+0000","last_unstale":"2024-03-21T12:59:53.952486+0000","last_undegraded":"2024-03-21T12:59:53.952486+0000","last_fullsized":"2024-03-21T12:59:53.952486+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T12:59:53.952401+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.263330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:55.899413+0000","last_change":"2024-03-21T12:59:55.899438+0000","last_active":"2024-03-21T12:59:55.899413+0000","last_peered":"2024-03-21T12:59:55.899413+0000","last_clean":"2024-03-21T12:59:55.899413+0000","last_became_active":"2024-03-21T12:58:43.351768+0000","last_became_peered":"2024-03-21T12:58:43.351768+0000","last_unstale":"2024-03-21T12:59:55.899413+0000","last_undegraded":"2024-03-21T12:59:55.899413+0000","last_fullsized":"2024-03-21T12:59:55.899413+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:55.899344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.899344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:59.849673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:55.014533+0000","last_change":"2024-03-21T12:59:55.014569+0000","last_active":"2024-03-21T12:59:55.014533+0000","last_peered":"2024-03-21T12:59:55.014533+0000","last_clean":"2024-03-21T12:59:55.014533+0000","last_became_active":"2024-03-21T12:58:41.165783+0000","last_became_peered":"2024-03-21T12:58:41.165783+0000","last_unstale":"2024-03-21T12:59:55.014533+0000","last_undegraded":"2024-03-21T12:59:55.014533+0000","last_fullsized":"2024-03-21T12:59:55.014533+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.014453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:10.470418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339649+0000","last_change":"2024-03-21T12:58:37.733421+0000","last_active":"2024-03-21T12:59:53.339649+0000","last_peered":"2024-03-21T12:59:53.339649+0000","last_clean":"2024-03-21T12:59:53.339649+0000","last_became_active":"2024-03-21T12:58:37.733064+0000","last_became_peered":"2024-03-21T12:58:37.733064+0000","last_unstale":"2024-03-21T12:59:53.339649+0000","last_undegraded":"2024-03-21T12:59:53.339649+0000","last_fullsized":"2024-03-21T12:59:53.339649+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:54.238132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":88,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:55.664722+0000","last_change":"2024-03-21T12:59:55.664752+0000","last_active":"2024-03-21T12:59:55.664722+0000","last_peered":"2024-03-21T12:59:55.664722+0000","last_clean":"2024-03-21T12:59:55.664722+0000","last_became_active":"2024-03-21T12:58:39.749130+0000","last_became_peered":"2024-03-21T12:58:39.749130+0000","last_unstale":"2024-03-21T12:59:55.664722+0000","last_undegraded":"2024-03-21T12:59:55.664722+0000","last_fullsized":"2024-03-21T12:59:55.664722+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T12:59:55.664649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.664649+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:11.693713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316207+0000","last_change":"2024-03-21T12:58:43.352883+0000","last_active":"2024-03-21T12:59:53.316207+0000","last_peered":"2024-03-21T12:59:53.316207+0000","last_clean":"2024-03-21T12:59:53.316207+0000","last_became_active":"2024-03-21T12:58:43.352510+0000","last_became_peered":"2024-03-21T12:58:43.352510+0000","last_unstale":"2024-03-21T12:59:53.316207+0000","last_undegraded":"2024-03-21T12:59:53.316207+0000","last_fullsized":"2024-03-21T12:59:53.316207+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:56.333194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840998+0000","last_change":"2024-03-21T12:59:50.904504+0000","last_active":"2024-03-21T12:59:53.840998+0000","last_peered":"2024-03-21T12:59:53.840998+0000","last_clean":"2024-03-21T12:59:53.840998+0000","last_became_active":"2024-03-21T12:58:41.167961+0000","last_became_peered":"2024-03-21T12:58:41.167961+0000","last_unstale":"2024-03-21T12:59:53.840998+0000","last_undegraded":"2024-03-21T12:59:53.840998+0000","last_fullsized":"2024-03-21T12:59:53.840998+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.904419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.904419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.905923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:54.948164+0000","last_change":"2024-03-21T12:59:54.948197+0000","last_active":"2024-03-21T12:59:54.948164+0000","last_peered":"2024-03-21T12:59:54.948164+0000","last_clean":"2024-03-21T12:59:54.948164+0000","last_became_active":"2024-03-21T12:58:37.732355+0000","last_became_peered":"2024-03-21T12:58:37.732355+0000","last_unstale":"2024-03-21T12:59:54.948164+0000","last_undegraded":"2024-03-21T12:59:54.948164+0000","last_fullsized":"2024-03-21T12:59:54.948164+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_clean_scrub_stamp":"2024-03-21T12:59:54.948081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:03.565967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'12","reported_seq":95,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315512+0000","last_change":"2024-03-21T12:59:01.168089+0000","last_active":"2024-03-21T12:59:53.315512+0000","last_peered":"2024-03-21T12:59:53.315512+0000","last_clean":"2024-03-21T12:59:53.315512+0000","last_became_active":"2024-03-21T12:59:01.161520+0000","last_became_peered":"2024-03-21T12:59:01.161520+0000","last_unstale":"2024-03-21T12:59:53.315512+0000","last_undegraded":"2024-03-21T12:59:53.315512+0000","last_fullsized":"2024-03-21T12:59:53.315512+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:08.420065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:00.168588+0000","last_change":"2024-03-21T13:00:00.168621+0000","last_active":"2024-03-21T13:00:00.168588+0000","last_peered":"2024-03-21T13:00:00.168588+0000","last_clean":"2024-03-21T13:00:00.168588+0000","last_became_active":"2024-03-21T12:58:43.184473+0000","last_became_peered":"2024-03-21T12:58:43.184473+0000","last_unstale":"2024-03-21T13:00:00.168588+0000","last_undegraded":"2024-03-21T13:00:00.168588+0000","last_fullsized":"2024-03-21T13:00:00.168588+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:00:00.168509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:00.168509+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.057928+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":87,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.787781+0000","last_change":"2024-03-21T12:59:55.277279+0000","last_active":"2024-03-21T12:59:57.787781+0000","last_peered":"2024-03-21T12:59:57.787781+0000","last_clean":"2024-03-21T12:59:57.787781+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T12:59:57.787781+0000","last_undegraded":"2024-03-21T12:59:57.787781+0000","last_fullsized":"2024-03-21T12:59:57.787781+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:55.277165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.277165+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:19.101757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710555+0000","last_change":"2024-03-21T12:58:37.736753+0000","last_active":"2024-03-21T12:59:53.710555+0000","last_peered":"2024-03-21T12:59:53.710555+0000","last_clean":"2024-03-21T12:59:53.710555+0000","last_became_active":"2024-03-21T12:58:37.736455+0000","last_became_peered":"2024-03-21T12:58:37.736455+0000","last_unstale":"2024-03-21T12:59:53.710555+0000","last_undegraded":"2024-03-21T12:59:53.710555+0000","last_fullsized":"2024-03-21T12:59:53.710555+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:56.602984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"71'10","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339627+0000","last_change":"2024-03-21T12:59:50.414098+0000","last_active":"2024-03-21T12:59:53.339627+0000","last_peered":"2024-03-21T12:59:53.339627+0000","last_clean":"2024-03-21T12:59:53.339627+0000","last_became_active":"2024-03-21T12:58:39.754548+0000","last_became_peered":"2024-03-21T12:58:39.754548+0000","last_unstale":"2024-03-21T12:59:53.339627+0000","last_undegraded":"2024-03-21T12:59:53.339627+0000","last_fullsized":"2024-03-21T12:59:53.339627+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T12:59:50.413983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.413983+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.575091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710921+0000","last_change":"2024-03-21T12:58:43.186597+0000","last_active":"2024-03-21T12:59:53.710921+0000","last_peered":"2024-03-21T12:59:53.710921+0000","last_clean":"2024-03-21T12:59:53.710921+0000","last_became_active":"2024-03-21T12:58:43.186289+0000","last_became_peered":"2024-03-21T12:58:43.186289+0000","last_unstale":"2024-03-21T12:59:53.710921+0000","last_undegraded":"2024-03-21T12:59:53.710921+0000","last_fullsized":"2024-03-21T12:59:53.710921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:00.777226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339724+0000","last_change":"2024-03-21T12:58:41.167760+0000","last_active":"2024-03-21T12:59:53.339724+0000","last_peered":"2024-03-21T12:59:53.339724+0000","last_clean":"2024-03-21T12:59:53.339724+0000","last_became_active":"2024-03-21T12:58:41.167443+0000","last_became_peered":"2024-03-21T12:58:41.167443+0000","last_unstale":"2024-03-21T12:59:53.339724+0000","last_undegraded":"2024-03-21T12:59:53.339724+0000","last_fullsized":"2024-03-21T12:59:53.339724+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:03.895693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.314370+0000","last_change":"2024-03-21T12:59:49.178866+0000","last_active":"2024-03-21T12:59:53.314370+0000","last_peered":"2024-03-21T12:59:53.314370+0000","last_clean":"2024-03-21T12:59:53.314370+0000","last_became_active":"2024-03-21T12:58:37.728914+0000","last_became_peered":"2024-03-21T12:58:37.728914+0000","last_unstale":"2024-03-21T12:59:53.314370+0000","last_undegraded":"2024-03-21T12:59:53.314370+0000","last_fullsized":"2024-03-21T12:59:53.314370+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:49.178785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:49.178785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.644002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'11","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:01.396831+0000","last_change":"2024-03-21T13:00:01.396865+0000","last_active":"2024-03-21T13:00:01.396831+0000","last_peered":"2024-03-21T13:00:01.396831+0000","last_clean":"2024-03-21T13:00:01.396831+0000","last_became_active":"2024-03-21T12:58:39.755294+0000","last_became_peered":"2024-03-21T12:58:39.755294+0000","last_unstale":"2024-03-21T13:00:01.396831+0000","last_undegraded":"2024-03-21T13:00:01.396831+0000","last_fullsized":"2024-03-21T13:00:01.396831+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:00:01.396749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.396749+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.505311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840256+0000","last_change":"2024-03-21T12:59:50.209251+0000","last_active":"2024-03-21T12:59:53.840256+0000","last_peered":"2024-03-21T12:59:53.840256+0000","last_clean":"2024-03-21T12:59:53.840256+0000","last_became_active":"2024-03-21T12:58:43.183475+0000","last_became_peered":"2024-03-21T12:58:43.183475+0000","last_unstale":"2024-03-21T12:59:53.840256+0000","last_undegraded":"2024-03-21T12:59:53.840256+0000","last_fullsized":"2024-03-21T12:59:53.840256+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.209152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.209152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:18.566644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:03.421362+0000","last_change":"2024-03-21T13:00:03.421382+0000","last_active":"2024-03-21T13:00:03.421362+0000","last_peered":"2024-03-21T13:00:03.421362+0000","last_clean":"2024-03-21T13:00:03.421362+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:00:03.421362+0000","last_undegraded":"2024-03-21T13:00:03.421362+0000","last_fullsized":"2024-03-21T13:00:03.421362+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:03.421305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:03.421305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.521044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339693+0000","last_change":"2024-03-21T12:58:37.732416+0000","last_active":"2024-03-21T12:59:53.339693+0000","last_peered":"2024-03-21T12:59:53.339693+0000","last_clean":"2024-03-21T12:59:53.339693+0000","last_became_active":"2024-03-21T12:58:37.732144+0000","last_became_peered":"2024-03-21T12:58:37.732144+0000","last_unstale":"2024-03-21T12:59:53.339693+0000","last_undegraded":"2024-03-21T12:59:53.339693+0000","last_fullsized":"2024-03-21T12:59:53.339693+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:05.140648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":60,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.314430+0000","last_change":"2024-03-21T12:58:39.755658+0000","last_active":"2024-03-21T12:59:53.314430+0000","last_peered":"2024-03-21T12:59:53.314430+0000","last_clean":"2024-03-21T12:59:53.314430+0000","last_became_active":"2024-03-21T12:58:39.755336+0000","last_became_peered":"2024-03-21T12:58:39.755336+0000","last_unstale":"2024-03-21T12:59:53.314430+0000","last_undegraded":"2024-03-21T12:59:53.314430+0000","last_fullsized":"2024-03-21T12:59:53.314430+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:03.263219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315097+0000","last_change":"2024-03-21T12:58:43.352917+0000","last_active":"2024-03-21T12:59:53.315097+0000","last_peered":"2024-03-21T12:59:53.315097+0000","last_clean":"2024-03-21T12:59:53.315097+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T12:59:53.315097+0000","last_undegraded":"2024-03-21T12:59:53.315097+0000","last_fullsized":"2024-03-21T12:59:53.315097+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840086+0000","last_change":"2024-03-21T12:58:41.165123+0000","last_active":"2024-03-21T12:59:53.840086+0000","last_peered":"2024-03-21T12:59:53.840086+0000","last_clean":"2024-03-21T12:59:53.840086+0000","last_became_active":"2024-03-21T12:58:41.164784+0000","last_became_peered":"2024-03-21T12:58:41.164784+0000","last_unstale":"2024-03-21T12:59:53.840086+0000","last_undegraded":"2024-03-21T12:59:53.840086+0000","last_fullsized":"2024-03-21T12:59:53.840086+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:04.486718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316024+0000","last_change":"2024-03-21T12:58:37.736422+0000","last_active":"2024-03-21T12:59:53.316024+0000","last_peered":"2024-03-21T12:59:53.316024+0000","last_clean":"2024-03-21T12:59:53.316024+0000","last_became_active":"2024-03-21T12:58:37.736127+0000","last_became_peered":"2024-03-21T12:58:37.736127+0000","last_unstale":"2024-03-21T12:59:53.316024+0000","last_undegraded":"2024-03-21T12:59:53.316024+0000","last_fullsized":"2024-03-21T12:59:53.316024+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:56.711736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"71'11","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315179+0000","last_change":"2024-03-21T12:59:52.181828+0000","last_active":"2024-03-21T12:59:53.315179+0000","last_peered":"2024-03-21T12:59:53.315179+0000","last_clean":"2024-03-21T12:59:53.315179+0000","last_became_active":"2024-03-21T12:58:39.753014+0000","last_became_peered":"2024-03-21T12:58:39.753014+0000","last_unstale":"2024-03-21T12:59:53.315179+0000","last_undegraded":"2024-03-21T12:59:53.315179+0000","last_fullsized":"2024-03-21T12:59:53.315179+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T12:59:52.181697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.181697+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.248453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339098+0000","last_change":"2024-03-21T12:58:43.178915+0000","last_active":"2024-03-21T12:59:53.339098+0000","last_peered":"2024-03-21T12:59:53.339098+0000","last_clean":"2024-03-21T12:59:53.339098+0000","last_became_active":"2024-03-21T12:58:43.178589+0000","last_became_peered":"2024-03-21T12:58:43.178589+0000","last_unstale":"2024-03-21T12:59:53.339098+0000","last_undegraded":"2024-03-21T12:59:53.339098+0000","last_fullsized":"2024-03-21T12:59:53.339098+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:07.484568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315361+0000","last_change":"2024-03-21T12:58:41.167969+0000","last_active":"2024-03-21T12:59:53.315361+0000","last_peered":"2024-03-21T12:59:53.315361+0000","last_clean":"2024-03-21T12:59:53.315361+0000","last_became_active":"2024-03-21T12:58:41.167371+0000","last_became_peered":"2024-03-21T12:58:41.167371+0000","last_unstale":"2024-03-21T12:59:53.315361+0000","last_undegraded":"2024-03-21T12:59:53.315361+0000","last_fullsized":"2024-03-21T12:59:53.315361+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:59.050477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840721+0000","last_change":"2024-03-21T12:58:37.733949+0000","last_active":"2024-03-21T12:59:53.840721+0000","last_peered":"2024-03-21T12:59:53.840721+0000","last_clean":"2024-03-21T12:59:53.840721+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T12:59:53.840721+0000","last_undegraded":"2024-03-21T12:59:53.840721+0000","last_fullsized":"2024-03-21T12:59:53.840721+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:05.144477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"86'13","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710766+0000","last_change":"2024-03-21T12:59:48.085870+0000","last_active":"2024-03-21T12:59:53.710766+0000","last_peered":"2024-03-21T12:59:53.710766+0000","last_clean":"2024-03-21T12:59:53.710766+0000","last_became_active":"2024-03-21T12:58:39.753224+0000","last_became_peered":"2024-03-21T12:58:39.753224+0000","last_unstale":"2024-03-21T12:59:53.710766+0000","last_undegraded":"2024-03-21T12:59:53.710766+0000","last_fullsized":"2024-03-21T12:59:53.710766+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.085774+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:53.042862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.314797+0000","last_change":"2024-03-21T12:58:43.184148+0000","last_active":"2024-03-21T12:59:53.314797+0000","last_peered":"2024-03-21T12:59:53.314797+0000","last_clean":"2024-03-21T12:59:53.314797+0000","last_became_active":"2024-03-21T12:58:43.183712+0000","last_became_peered":"2024-03-21T12:58:43.183712+0000","last_unstale":"2024-03-21T12:59:53.314797+0000","last_undegraded":"2024-03-21T12:59:53.314797+0000","last_fullsized":"2024-03-21T12:59:53.314797+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:12.054567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710756+0000","last_change":"2024-03-21T12:59:45.088950+0000","last_active":"2024-03-21T12:59:53.710756+0000","last_peered":"2024-03-21T12:59:53.710756+0000","last_clean":"2024-03-21T12:59:53.710756+0000","last_became_active":"2024-03-21T12:58:41.164463+0000","last_became_peered":"2024-03-21T12:58:41.164463+0000","last_unstale":"2024-03-21T12:59:53.710756+0000","last_undegraded":"2024-03-21T12:59:53.710756+0000","last_fullsized":"2024-03-21T12:59:53.710756+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:45.088861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.088861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:11.118145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841177+0000","last_change":"2024-03-21T12:59:45.833435+0000","last_active":"2024-03-21T12:59:53.841177+0000","last_peered":"2024-03-21T12:59:53.841177+0000","last_clean":"2024-03-21T12:59:53.841177+0000","last_became_active":"2024-03-21T12:58:37.731659+0000","last_became_peered":"2024-03-21T12:58:37.731659+0000","last_unstale":"2024-03-21T12:59:53.841177+0000","last_undegraded":"2024-03-21T12:59:53.841177+0000","last_fullsized":"2024-03-21T12:59:53.841177+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.833304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:49.946967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841051+0000","last_change":"2024-03-21T12:58:39.746514+0000","last_active":"2024-03-21T12:59:53.841051+0000","last_peered":"2024-03-21T12:59:53.841051+0000","last_clean":"2024-03-21T12:59:53.841051+0000","last_became_active":"2024-03-21T12:58:39.746182+0000","last_became_peered":"2024-03-21T12:58:39.746182+0000","last_unstale":"2024-03-21T12:59:53.841051+0000","last_undegraded":"2024-03-21T12:59:53.841051+0000","last_fullsized":"2024-03-21T12:59:53.841051+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:08.200205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339384+0000","last_change":"2024-03-21T12:58:43.181024+0000","last_active":"2024-03-21T12:59:53.339384+0000","last_peered":"2024-03-21T12:59:53.339384+0000","last_clean":"2024-03-21T12:59:53.339384+0000","last_became_active":"2024-03-21T12:58:43.180773+0000","last_became_peered":"2024-03-21T12:58:43.180773+0000","last_unstale":"2024-03-21T12:59:53.339384+0000","last_undegraded":"2024-03-21T12:59:53.339384+0000","last_fullsized":"2024-03-21T12:59:53.339384+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:07.203930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":87,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.787745+0000","last_change":"2024-03-21T12:59:50.724334+0000","last_active":"2024-03-21T12:59:57.787745+0000","last_peered":"2024-03-21T12:59:57.787745+0000","last_clean":"2024-03-21T12:59:57.787745+0000","last_became_active":"2024-03-21T12:58:41.167956+0000","last_became_peered":"2024-03-21T12:58:41.167956+0000","last_unstale":"2024-03-21T12:59:57.787745+0000","last_undegraded":"2024-03-21T12:59:57.787745+0000","last_fullsized":"2024-03-21T12:59:57.787745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:50.724242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.724242+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:53.691391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840661+0000","last_change":"2024-03-21T12:58:37.732282+0000","last_active":"2024-03-21T12:59:53.840661+0000","last_peered":"2024-03-21T12:59:53.840661+0000","last_clean":"2024-03-21T12:59:53.840661+0000","last_became_active":"2024-03-21T12:58:37.731988+0000","last_became_peered":"2024-03-21T12:58:37.731988+0000","last_unstale":"2024-03-21T12:59:53.840661+0000","last_undegraded":"2024-03-21T12:59:53.840661+0000","last_fullsized":"2024-03-21T12:59:53.840661+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:00.577765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"71'10","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315898+0000","last_change":"2024-03-21T12:58:39.758132+0000","last_active":"2024-03-21T12:59:53.315898+0000","last_peered":"2024-03-21T12:59:53.315898+0000","last_clean":"2024-03-21T12:59:53.315898+0000","last_became_active":"2024-03-21T12:58:39.757806+0000","last_became_peered":"2024-03-21T12:58:39.757806+0000","last_unstale":"2024-03-21T12:59:53.315898+0000","last_undegraded":"2024-03-21T12:59:53.315898+0000","last_fullsized":"2024-03-21T12:59:53.315898+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841610+0000","last_change":"2024-03-21T12:58:43.186611+0000","last_active":"2024-03-21T12:59:53.841610+0000","last_peered":"2024-03-21T12:59:53.841610+0000","last_clean":"2024-03-21T12:59:53.841610+0000","last_became_active":"2024-03-21T12:58:43.186155+0000","last_became_peered":"2024-03-21T12:58:43.186155+0000","last_unstale":"2024-03-21T12:59:53.841610+0000","last_undegraded":"2024-03-21T12:59:53.841610+0000","last_fullsized":"2024-03-21T12:59:53.841610+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:01.571810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.787668+0000","last_change":"2024-03-21T12:58:41.168116+0000","last_active":"2024-03-21T12:59:57.787668+0000","last_peered":"2024-03-21T12:59:57.787668+0000","last_clean":"2024-03-21T12:59:57.787668+0000","last_became_active":"2024-03-21T12:58:41.167826+0000","last_became_peered":"2024-03-21T12:58:41.167826+0000","last_unstale":"2024-03-21T12:59:57.787668+0000","last_undegraded":"2024-03-21T12:59:57.787668+0000","last_fullsized":"2024-03-21T12:59:57.787668+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339299+0000","last_change":"2024-03-21T12:58:37.732471+0000","last_active":"2024-03-21T12:59:53.339299+0000","last_peered":"2024-03-21T12:59:53.339299+0000","last_clean":"2024-03-21T12:59:53.339299+0000","last_became_active":"2024-03-21T12:58:37.732166+0000","last_became_peered":"2024-03-21T12:58:37.732166+0000","last_unstale":"2024-03-21T12:59:53.339299+0000","last_undegraded":"2024-03-21T12:59:53.339299+0000","last_fullsized":"2024-03-21T12:59:53.339299+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:06.434461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'9","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316076+0000","last_change":"2024-03-21T12:58:39.751200+0000","last_active":"2024-03-21T12:59:53.316076+0000","last_peered":"2024-03-21T12:59:53.316076+0000","last_clean":"2024-03-21T12:59:53.316076+0000","last_became_active":"2024-03-21T12:58:39.750820+0000","last_became_peered":"2024-03-21T12:58:39.750820+0000","last_unstale":"2024-03-21T12:59:53.316076+0000","last_undegraded":"2024-03-21T12:59:53.316076+0000","last_fullsized":"2024-03-21T12:59:53.316076+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:00.043713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.317216+0000","last_change":"2024-03-21T12:59:51.730551+0000","last_active":"2024-03-21T12:59:53.317216+0000","last_peered":"2024-03-21T12:59:53.317216+0000","last_clean":"2024-03-21T12:59:53.317216+0000","last_became_active":"2024-03-21T12:58:43.182126+0000","last_became_peered":"2024-03-21T12:58:43.182126+0000","last_unstale":"2024-03-21T12:59:53.317216+0000","last_undegraded":"2024-03-21T12:59:53.317216+0000","last_fullsized":"2024-03-21T12:59:53.317216+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T12:59:51.730413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.791726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316600+0000","last_change":"2024-03-21T12:58:41.173262+0000","last_active":"2024-03-21T12:59:53.316600+0000","last_peered":"2024-03-21T12:59:53.316600+0000","last_clean":"2024-03-21T12:59:53.316600+0000","last_became_active":"2024-03-21T12:58:41.172904+0000","last_became_peered":"2024-03-21T12:58:41.172904+0000","last_unstale":"2024-03-21T12:59:53.316600+0000","last_undegraded":"2024-03-21T12:59:53.316600+0000","last_fullsized":"2024-03-21T12:59:53.316600+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.839994+0000","last_change":"2024-03-21T12:59:47.203592+0000","last_active":"2024-03-21T12:59:53.839994+0000","last_peered":"2024-03-21T12:59:53.839994+0000","last_clean":"2024-03-21T12:59:53.839994+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T12:59:53.839994+0000","last_undegraded":"2024-03-21T12:59:53.839994+0000","last_fullsized":"2024-03-21T12:59:53.839994+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T12:59:47.203453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.203453+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.098187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'9","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339622+0000","last_change":"2024-03-21T12:58:39.753073+0000","last_active":"2024-03-21T12:59:53.339622+0000","last_peered":"2024-03-21T12:59:53.339622+0000","last_clean":"2024-03-21T12:59:53.339622+0000","last_became_active":"2024-03-21T12:58:39.752722+0000","last_became_peered":"2024-03-21T12:58:39.752722+0000","last_unstale":"2024-03-21T12:59:53.339622+0000","last_undegraded":"2024-03-21T12:59:53.339622+0000","last_fullsized":"2024-03-21T12:59:53.339622+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:53.379100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840407+0000","last_change":"2024-03-21T12:58:43.353751+0000","last_active":"2024-03-21T12:59:53.840407+0000","last_peered":"2024-03-21T12:59:53.840407+0000","last_clean":"2024-03-21T12:59:53.840407+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T12:59:53.840407+0000","last_undegraded":"2024-03-21T12:59:53.840407+0000","last_fullsized":"2024-03-21T12:59:53.840407+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:57.020516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315924+0000","last_change":"2024-03-21T12:58:41.162672+0000","last_active":"2024-03-21T12:59:53.315924+0000","last_peered":"2024-03-21T12:59:53.315924+0000","last_clean":"2024-03-21T12:59:53.315924+0000","last_became_active":"2024-03-21T12:58:41.162343+0000","last_became_peered":"2024-03-21T12:58:41.162343+0000","last_unstale":"2024-03-21T12:59:53.315924+0000","last_undegraded":"2024-03-21T12:59:53.315924+0000","last_fullsized":"2024-03-21T12:59:53.315924+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:06.284594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.754353+0000","last_change":"2024-03-21T12:59:52.319029+0000","last_active":"2024-03-21T12:59:53.754353+0000","last_peered":"2024-03-21T12:59:53.754353+0000","last_clean":"2024-03-21T12:59:53.754353+0000","last_became_active":"2024-03-21T12:58:37.731932+0000","last_became_peered":"2024-03-21T12:58:37.731932+0000","last_unstale":"2024-03-21T12:59:53.754353+0000","last_undegraded":"2024-03-21T12:59:53.754353+0000","last_fullsized":"2024-03-21T12:59:53.754353+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:52.318927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.318927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:20.356137+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'6","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316455+0000","last_change":"2024-03-21T12:59:46.176940+0000","last_active":"2024-03-21T12:59:53.316455+0000","last_peered":"2024-03-21T12:59:53.316455+0000","last_clean":"2024-03-21T12:59:53.316455+0000","last_became_active":"2024-03-21T12:58:39.755419+0000","last_became_peered":"2024-03-21T12:58:39.755419+0000","last_unstale":"2024-03-21T12:59:53.316455+0000","last_undegraded":"2024-03-21T12:59:53.316455+0000","last_fullsized":"2024-03-21T12:59:53.316455+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T12:59:46.176819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.176819+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:06.815464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:00.622170+0000","last_change":"2024-03-21T13:00:00.622196+0000","last_active":"2024-03-21T13:00:00.622170+0000","last_peered":"2024-03-21T13:00:00.622170+0000","last_clean":"2024-03-21T13:00:00.622170+0000","last_became_active":"2024-03-21T12:58:43.351103+0000","last_became_peered":"2024-03-21T12:58:43.351103+0000","last_unstale":"2024-03-21T13:00:00.622170+0000","last_undegraded":"2024-03-21T13:00:00.622170+0000","last_fullsized":"2024-03-21T13:00:00.622170+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:00.622094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:00.622094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:21.931294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841274+0000","last_change":"2024-03-21T12:58:37.735326+0000","last_active":"2024-03-21T12:59:53.841274+0000","last_peered":"2024-03-21T12:59:53.841274+0000","last_clean":"2024-03-21T12:59:53.841274+0000","last_became_active":"2024-03-21T12:58:37.735037+0000","last_became_peered":"2024-03-21T12:58:37.735037+0000","last_unstale":"2024-03-21T12:59:53.841274+0000","last_undegraded":"2024-03-21T12:59:53.841274+0000","last_fullsized":"2024-03-21T12:59:53.841274+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:54.657655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'15","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315400+0000","last_change":"2024-03-21T12:58:39.750902+0000","last_active":"2024-03-21T12:59:53.315400+0000","last_peered":"2024-03-21T12:59:53.315400+0000","last_clean":"2024-03-21T12:59:53.315400+0000","last_became_active":"2024-03-21T12:58:39.750546+0000","last_became_peered":"2024-03-21T12:58:39.750546+0000","last_unstale":"2024-03-21T12:59:53.315400+0000","last_undegraded":"2024-03-21T12:59:53.315400+0000","last_fullsized":"2024-03-21T12:59:53.315400+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:02.801886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"71'8","reported_seq":75,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.787779+0000","last_change":"2024-03-21T12:58:41.166431+0000","last_active":"2024-03-21T12:59:57.787779+0000","last_peered":"2024-03-21T12:59:57.787779+0000","last_clean":"2024-03-21T12:59:57.787779+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T12:59:57.787779+0000","last_undegraded":"2024-03-21T12:59:57.787779+0000","last_fullsized":"2024-03-21T12:59:57.787779+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:08.044850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315189+0000","last_change":"2024-03-21T12:58:43.181756+0000","last_active":"2024-03-21T12:59:53.315189+0000","last_peered":"2024-03-21T12:59:53.315189+0000","last_clean":"2024-03-21T12:59:53.315189+0000","last_became_active":"2024-03-21T12:58:43.181036+0000","last_became_peered":"2024-03-21T12:58:43.181036+0000","last_unstale":"2024-03-21T12:59:53.315189+0000","last_undegraded":"2024-03-21T12:59:53.315189+0000","last_fullsized":"2024-03-21T12:59:53.315189+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:59.136913+0000","last_change":"2024-03-21T12:59:59.136937+0000","last_active":"2024-03-21T12:59:59.136913+0000","last_peered":"2024-03-21T12:59:59.136913+0000","last_clean":"2024-03-21T12:59:59.136913+0000","last_became_active":"2024-03-21T12:58:37.737012+0000","last_became_peered":"2024-03-21T12:58:37.737012+0000","last_unstale":"2024-03-21T12:59:59.136913+0000","last_undegraded":"2024-03-21T12:59:59.136913+0000","last_fullsized":"2024-03-21T12:59:59.136913+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:59.136829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.136829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:03.237208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"71'12","reported_seq":77,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840417+0000","last_change":"2024-03-21T12:58:39.751648+0000","last_active":"2024-03-21T12:59:53.840417+0000","last_peered":"2024-03-21T12:59:53.840417+0000","last_clean":"2024-03-21T12:59:53.840417+0000","last_became_active":"2024-03-21T12:58:39.751236+0000","last_became_peered":"2024-03-21T12:58:39.751236+0000","last_unstale":"2024-03-21T12:59:53.840417+0000","last_undegraded":"2024-03-21T12:59:53.840417+0000","last_fullsized":"2024-03-21T12:59:53.840417+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:04.196572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:58.733332+0000","last_change":"2024-03-21T12:59:58.733374+0000","last_active":"2024-03-21T12:59:58.733332+0000","last_peered":"2024-03-21T12:59:58.733332+0000","last_clean":"2024-03-21T12:59:58.733332+0000","last_became_active":"2024-03-21T12:58:41.171443+0000","last_became_peered":"2024-03-21T12:58:41.171443+0000","last_unstale":"2024-03-21T12:59:58.733332+0000","last_undegraded":"2024-03-21T12:59:58.733332+0000","last_fullsized":"2024-03-21T12:59:58.733332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:58.733235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:58.733235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.428021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:04.200931+0000","last_change":"2024-03-21T13:00:04.200955+0000","last_active":"2024-03-21T13:00:04.200931+0000","last_peered":"2024-03-21T13:00:04.200931+0000","last_clean":"2024-03-21T13:00:04.200931+0000","last_became_active":"2024-03-21T12:58:43.179007+0000","last_became_peered":"2024-03-21T12:58:43.179007+0000","last_unstale":"2024-03-21T13:00:04.200931+0000","last_undegraded":"2024-03-21T13:00:04.200931+0000","last_fullsized":"2024-03-21T13:00:04.200931+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:04.200849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:04.200849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.029947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316145+0000","last_change":"2024-03-21T12:58:41.166265+0000","last_active":"2024-03-21T12:59:53.316145+0000","last_peered":"2024-03-21T12:59:53.316145+0000","last_clean":"2024-03-21T12:59:53.316145+0000","last_became_active":"2024-03-21T12:58:41.165203+0000","last_became_peered":"2024-03-21T12:58:41.165203+0000","last_unstale":"2024-03-21T12:59:53.316145+0000","last_undegraded":"2024-03-21T12:59:53.316145+0000","last_fullsized":"2024-03-21T12:59:53.316145+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:08.244050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840816+0000","last_change":"2024-03-21T12:59:46.169468+0000","last_active":"2024-03-21T12:59:53.840816+0000","last_peered":"2024-03-21T12:59:53.840816+0000","last_clean":"2024-03-21T12:59:53.840816+0000","last_became_active":"2024-03-21T12:58:37.731448+0000","last_became_peered":"2024-03-21T12:58:37.731448+0000","last_unstale":"2024-03-21T12:59:53.840816+0000","last_undegraded":"2024-03-21T12:59:53.840816+0000","last_fullsized":"2024-03-21T12:59:53.840816+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:46.169315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.169315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:51.199764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"71'19","reported_seq":93,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:59.392934+0000","last_change":"2024-03-21T12:59:59.392987+0000","last_active":"2024-03-21T12:59:59.392934+0000","last_peered":"2024-03-21T12:59:59.392934+0000","last_clean":"2024-03-21T12:59:59.392934+0000","last_became_active":"2024-03-21T12:58:39.754761+0000","last_became_peered":"2024-03-21T12:58:39.754761+0000","last_unstale":"2024-03-21T12:59:59.392934+0000","last_undegraded":"2024-03-21T12:59:59.392934+0000","last_fullsized":"2024-03-21T12:59:59.392934+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T12:59:59.392834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.392834+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.114737+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339138+0000","last_change":"2024-03-21T12:58:43.180724+0000","last_active":"2024-03-21T12:59:53.339138+0000","last_peered":"2024-03-21T12:59:53.339138+0000","last_clean":"2024-03-21T12:59:53.339138+0000","last_became_active":"2024-03-21T12:58:43.180393+0000","last_became_peered":"2024-03-21T12:58:43.180393+0000","last_unstale":"2024-03-21T12:59:53.339138+0000","last_undegraded":"2024-03-21T12:59:53.339138+0000","last_fullsized":"2024-03-21T12:59:53.339138+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:02.348923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.317040+0000","last_change":"2024-03-21T12:58:41.164684+0000","last_active":"2024-03-21T12:59:53.317040+0000","last_peered":"2024-03-21T12:59:53.317040+0000","last_clean":"2024-03-21T12:59:53.317040+0000","last_became_active":"2024-03-21T12:58:41.163843+0000","last_became_peered":"2024-03-21T12:58:41.163843+0000","last_unstale":"2024-03-21T12:59:53.317040+0000","last_undegraded":"2024-03-21T12:59:53.317040+0000","last_fullsized":"2024-03-21T12:59:53.317040+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:53.709873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315887+0000","last_change":"2024-03-21T12:59:47.676292+0000","last_active":"2024-03-21T12:59:53.315887+0000","last_peered":"2024-03-21T12:59:53.315887+0000","last_clean":"2024-03-21T12:59:53.315887+0000","last_became_active":"2024-03-21T12:58:37.730518+0000","last_became_peered":"2024-03-21T12:58:37.730518+0000","last_unstale":"2024-03-21T12:59:53.315887+0000","last_undegraded":"2024-03-21T12:59:53.315887+0000","last_fullsized":"2024-03-21T12:59:53.315887+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:47.676198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.676198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.156465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"86'13","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339817+0000","last_change":"2024-03-21T12:59:46.421228+0000","last_active":"2024-03-21T12:59:53.339817+0000","last_peered":"2024-03-21T12:59:53.339817+0000","last_clean":"2024-03-21T12:59:53.339817+0000","last_became_active":"2024-03-21T12:58:39.748144+0000","last_became_peered":"2024-03-21T12:58:39.748144+0000","last_unstale":"2024-03-21T12:59:53.339817+0000","last_undegraded":"2024-03-21T12:59:53.339817+0000","last_fullsized":"2024-03-21T12:59:53.339817+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.421144+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:06.815237+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315865+0000","last_change":"2024-03-21T12:58:43.180710+0000","last_active":"2024-03-21T12:59:53.315865+0000","last_peered":"2024-03-21T12:59:53.315865+0000","last_clean":"2024-03-21T12:59:53.315865+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T12:59:53.315865+0000","last_undegraded":"2024-03-21T12:59:53.315865+0000","last_fullsized":"2024-03-21T12:59:53.315865+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:10.959635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"67'2","reported_seq":77,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:55.541514+0000","last_change":"2024-03-21T12:59:55.541552+0000","last_active":"2024-03-21T12:59:55.541514+0000","last_peered":"2024-03-21T12:59:55.541514+0000","last_clean":"2024-03-21T12:59:55.541514+0000","last_became_active":"2024-03-21T12:58:36.724010+0000","last_became_peered":"2024-03-21T12:58:36.724010+0000","last_unstale":"2024-03-21T12:59:55.541514+0000","last_undegraded":"2024-03-21T12:59:55.541514+0000","last_fullsized":"2024-03-21T12:59:55.541514+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-21T12:59:55.541410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.541410+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:17.828687+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840854+0000","last_change":"2024-03-21T12:59:47.840935+0000","last_active":"2024-03-21T12:59:53.840854+0000","last_peered":"2024-03-21T12:59:53.840854+0000","last_clean":"2024-03-21T12:59:53.840854+0000","last_became_active":"2024-03-21T12:58:37.729454+0000","last_became_peered":"2024-03-21T12:58:37.729454+0000","last_unstale":"2024-03-21T12:59:53.840854+0000","last_undegraded":"2024-03-21T12:59:53.840854+0000","last_fullsized":"2024-03-21T12:59:53.840854+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:47.840831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.840831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.579290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"71'28","reported_seq":128,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:01.917692+0000","last_change":"2024-03-21T13:00:01.917692+0000","last_active":"2024-03-21T13:00:01.917692+0000","last_peered":"2024-03-21T13:00:01.917692+0000","last_clean":"2024-03-21T13:00:01.917692+0000","last_became_active":"2024-03-21T12:58:39.756052+0000","last_became_peered":"2024-03-21T12:58:39.756052+0000","last_unstale":"2024-03-21T13:00:01.917692+0000","last_undegraded":"2024-03-21T13:00:01.917692+0000","last_fullsized":"2024-03-21T13:00:01.917692+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'28","last_scrub_stamp":"2024-03-21T13:00:01.917593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.917593+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:26.189859+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339496+0000","last_change":"2024-03-21T12:58:41.170763+0000","last_active":"2024-03-21T12:59:53.339496+0000","last_peered":"2024-03-21T12:59:53.339496+0000","last_clean":"2024-03-21T12:59:53.339496+0000","last_became_active":"2024-03-21T12:58:41.170460+0000","last_became_peered":"2024-03-21T12:58:41.170460+0000","last_unstale":"2024-03-21T12:59:53.339496+0000","last_undegraded":"2024-03-21T12:59:53.339496+0000","last_fullsized":"2024-03-21T12:59:53.339496+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:06.578225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:54.191221+0000","last_change":"2024-03-21T12:59:54.191245+0000","last_active":"2024-03-21T12:59:54.191221+0000","last_peered":"2024-03-21T12:59:54.191221+0000","last_clean":"2024-03-21T12:59:54.191221+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T12:59:54.191221+0000","last_undegraded":"2024-03-21T12:59:54.191221+0000","last_fullsized":"2024-03-21T12:59:54.191221+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:54.191150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:54.191150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:15.445044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710593+0000","last_change":"2024-03-21T12:58:37.733226+0000","last_active":"2024-03-21T12:59:53.710593+0000","last_peered":"2024-03-21T12:59:53.710593+0000","last_clean":"2024-03-21T12:59:53.710593+0000","last_became_active":"2024-03-21T12:58:37.731386+0000","last_became_peered":"2024-03-21T12:58:37.731386+0000","last_unstale":"2024-03-21T12:59:53.710593+0000","last_undegraded":"2024-03-21T12:59:53.710593+0000","last_fullsized":"2024-03-21T12:59:53.710593+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:03.233816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.734474+0000","last_change":"2024-03-21T12:59:57.734511+0000","last_active":"2024-03-21T12:59:57.734474+0000","last_peered":"2024-03-21T12:59:57.734474+0000","last_clean":"2024-03-21T12:59:57.734474+0000","last_became_active":"2024-03-21T12:58:36.724067+0000","last_became_peered":"2024-03-21T12:58:36.724067+0000","last_unstale":"2024-03-21T12:59:57.734474+0000","last_undegraded":"2024-03-21T12:59:57.734474+0000","last_fullsized":"2024-03-21T12:59:57.734474+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.734379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.734379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.789374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000743678,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'13","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.711251+0000","last_change":"2024-03-21T12:58:39.750379+0000","last_active":"2024-03-21T12:59:53.711251+0000","last_peered":"2024-03-21T12:59:53.711251+0000","last_clean":"2024-03-21T12:59:53.711251+0000","last_became_active":"2024-03-21T12:58:39.750068+0000","last_became_peered":"2024-03-21T12:58:39.750068+0000","last_unstale":"2024-03-21T12:59:53.711251+0000","last_undegraded":"2024-03-21T12:59:53.711251+0000","last_fullsized":"2024-03-21T12:59:53.711251+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:58.393230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316373+0000","last_change":"2024-03-21T12:58:41.165072+0000","last_active":"2024-03-21T12:59:53.316373+0000","last_peered":"2024-03-21T12:59:53.316373+0000","last_clean":"2024-03-21T12:59:53.316373+0000","last_became_active":"2024-03-21T12:58:41.164725+0000","last_became_peered":"2024-03-21T12:58:41.164725+0000","last_unstale":"2024-03-21T12:59:53.316373+0000","last_undegraded":"2024-03-21T12:59:53.316373+0000","last_fullsized":"2024-03-21T12:59:53.316373+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:59.205553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316074+0000","last_change":"2024-03-21T12:59:52.658473+0000","last_active":"2024-03-21T12:59:53.316074+0000","last_peered":"2024-03-21T12:59:53.316074+0000","last_clean":"2024-03-21T12:59:53.316074+0000","last_became_active":"2024-03-21T12:58:43.183113+0000","last_became_peered":"2024-03-21T12:58:43.183113+0000","last_unstale":"2024-03-21T12:59:53.316074+0000","last_undegraded":"2024-03-21T12:59:53.316074+0000","last_fullsized":"2024-03-21T12:59:53.316074+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:52.658377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.658377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.565790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":314,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.839938+0000","last_change":"2024-03-21T12:59:02.091964+0000","last_active":"2024-03-21T12:59:53.839938+0000","last_peered":"2024-03-21T12:59:53.839938+0000","last_clean":"2024-03-21T12:59:53.839938+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T12:59:53.839938+0000","last_undegraded":"2024-03-21T12:59:53.839938+0000","last_fullsized":"2024-03-21T12:59:53.839938+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'119","last_scrub_stamp":"2024-03-21T12:59:02.091835+0000","last_deep_scrub":"76'119","last_deep_scrub_stamp":"2024-03-21T12:59:02.091835+0000","last_clean_scrub_stamp":"2024-03-21T12:59:02.091835+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:02.880876+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.314920+0000","last_change":"2024-03-21T12:58:37.729254+0000","last_active":"2024-03-21T12:59:53.314920+0000","last_peered":"2024-03-21T12:59:53.314920+0000","last_clean":"2024-03-21T12:59:53.314920+0000","last_became_active":"2024-03-21T12:58:37.728911+0000","last_became_peered":"2024-03-21T12:58:37.728911+0000","last_unstale":"2024-03-21T12:59:53.314920+0000","last_undegraded":"2024-03-21T12:59:53.314920+0000","last_fullsized":"2024-03-21T12:59:53.314920+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:57.212471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":102,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.787905+0000","last_change":"2024-03-21T12:59:56.298894+0000","last_active":"2024-03-21T12:59:57.787905+0000","last_peered":"2024-03-21T12:59:57.787905+0000","last_clean":"2024-03-21T12:59:57.787905+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T12:59:57.787905+0000","last_undegraded":"2024-03-21T12:59:57.787905+0000","last_fullsized":"2024-03-21T12:59:57.787905+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-21T12:59:56.298768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:56.298768+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.767366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840609+0000","last_change":"2024-03-21T12:58:41.168317+0000","last_active":"2024-03-21T12:59:53.840609+0000","last_peered":"2024-03-21T12:59:53.840609+0000","last_clean":"2024-03-21T12:59:53.840609+0000","last_became_active":"2024-03-21T12:58:41.168003+0000","last_became_peered":"2024-03-21T12:58:41.168003+0000","last_unstale":"2024-03-21T12:59:53.840609+0000","last_undegraded":"2024-03-21T12:59:53.840609+0000","last_fullsized":"2024-03-21T12:59:53.840609+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:58.477533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710736+0000","last_change":"2024-03-21T12:58:43.184189+0000","last_active":"2024-03-21T12:59:53.710736+0000","last_peered":"2024-03-21T12:59:53.710736+0000","last_clean":"2024-03-21T12:59:53.710736+0000","last_became_active":"2024-03-21T12:58:43.183860+0000","last_became_peered":"2024-03-21T12:58:43.183860+0000","last_unstale":"2024-03-21T12:59:53.710736+0000","last_undegraded":"2024-03-21T12:59:53.710736+0000","last_fullsized":"2024-03-21T12:59:53.710736+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:01.953966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339323+0000","last_change":"2024-03-21T12:58:37.734459+0000","last_active":"2024-03-21T12:59:53.339323+0000","last_peered":"2024-03-21T12:59:53.339323+0000","last_clean":"2024-03-21T12:59:53.339323+0000","last_became_active":"2024-03-21T12:58:37.734176+0000","last_became_peered":"2024-03-21T12:58:37.734176+0000","last_unstale":"2024-03-21T12:59:53.339323+0000","last_undegraded":"2024-03-21T12:59:53.339323+0000","last_fullsized":"2024-03-21T12:59:53.339323+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:56.304166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":148,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:03.231289+0000","last_change":"2024-03-21T12:58:40.754485+0000","last_active":"2024-03-21T13:00:03.231289+0000","last_peered":"2024-03-21T13:00:03.231289+0000","last_clean":"2024-03-21T13:00:03.231289+0000","last_became_active":"2024-03-21T12:58:36.725735+0000","last_became_peered":"2024-03-21T12:58:36.725735+0000","last_unstale":"2024-03-21T13:00:03.231289+0000","last_undegraded":"2024-03-21T13:00:03.231289+0000","last_fullsized":"2024-03-21T13:00:03.231289+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:58:35.713454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339478+0000","last_change":"2024-03-21T12:58:39.747623+0000","last_active":"2024-03-21T12:59:53.339478+0000","last_peered":"2024-03-21T12:59:53.339478+0000","last_clean":"2024-03-21T12:59:53.339478+0000","last_became_active":"2024-03-21T12:58:39.747315+0000","last_became_peered":"2024-03-21T12:58:39.747315+0000","last_unstale":"2024-03-21T12:59:53.339478+0000","last_undegraded":"2024-03-21T12:59:53.339478+0000","last_fullsized":"2024-03-21T12:59:53.339478+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316244+0000","last_change":"2024-03-21T12:58:41.167475+0000","last_active":"2024-03-21T12:59:53.316244+0000","last_peered":"2024-03-21T12:59:53.316244+0000","last_clean":"2024-03-21T12:59:53.316244+0000","last_became_active":"2024-03-21T12:58:41.167131+0000","last_became_peered":"2024-03-21T12:58:41.167131+0000","last_unstale":"2024-03-21T12:59:53.316244+0000","last_undegraded":"2024-03-21T12:59:53.316244+0000","last_fullsized":"2024-03-21T12:59:53.316244+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:58.954135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.710418+0000","last_change":"2024-03-21T12:58:43.351489+0000","last_active":"2024-03-21T12:59:53.710418+0000","last_peered":"2024-03-21T12:59:53.710418+0000","last_clean":"2024-03-21T12:59:53.710418+0000","last_became_active":"2024-03-21T12:58:43.351138+0000","last_became_peered":"2024-03-21T12:58:43.351138+0000","last_unstale":"2024-03-21T12:59:53.710418+0000","last_undegraded":"2024-03-21T12:59:53.710418+0000","last_fullsized":"2024-03-21T12:59:53.710418+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:56.857483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.925271+0000","last_change":"2024-03-21T12:59:57.925311+0000","last_active":"2024-03-21T12:59:57.925271+0000","last_peered":"2024-03-21T12:59:57.925271+0000","last_clean":"2024-03-21T12:59:57.925271+0000","last_became_active":"2024-03-21T12:58:37.729818+0000","last_became_peered":"2024-03-21T12:58:37.729818+0000","last_unstale":"2024-03-21T12:59:57.925271+0000","last_undegraded":"2024-03-21T12:59:57.925271+0000","last_fullsized":"2024-03-21T12:59:57.925271+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.925179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.925179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.845352+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"71'19","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339719+0000","last_change":"2024-03-21T12:58:39.751068+0000","last_active":"2024-03-21T12:59:53.339719+0000","last_peered":"2024-03-21T12:59:53.339719+0000","last_clean":"2024-03-21T12:59:53.339719+0000","last_became_active":"2024-03-21T12:58:39.750216+0000","last_became_peered":"2024-03-21T12:58:39.750216+0000","last_unstale":"2024-03-21T12:59:53.339719+0000","last_undegraded":"2024-03-21T12:59:53.339719+0000","last_fullsized":"2024-03-21T12:59:53.339719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:57.408510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.754642+0000","last_change":"2024-03-21T12:58:41.166808+0000","last_active":"2024-03-21T12:59:53.754642+0000","last_peered":"2024-03-21T12:59:53.754642+0000","last_clean":"2024-03-21T12:59:53.754642+0000","last_became_active":"2024-03-21T12:58:41.166495+0000","last_became_peered":"2024-03-21T12:58:41.166495+0000","last_unstale":"2024-03-21T12:59:53.754642+0000","last_undegraded":"2024-03-21T12:59:53.754642+0000","last_fullsized":"2024-03-21T12:59:53.754642+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:59.378636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339098+0000","last_change":"2024-03-21T12:58:43.177740+0000","last_active":"2024-03-21T12:59:53.339098+0000","last_peered":"2024-03-21T12:59:53.339098+0000","last_clean":"2024-03-21T12:59:53.339098+0000","last_became_active":"2024-03-21T12:58:43.177409+0000","last_became_peered":"2024-03-21T12:58:43.177409+0000","last_unstale":"2024-03-21T12:59:53.339098+0000","last_undegraded":"2024-03-21T12:59:53.339098+0000","last_fullsized":"2024-03-21T12:59:53.339098+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315902+0000","last_change":"2024-03-21T12:58:37.730839+0000","last_active":"2024-03-21T12:59:53.315902+0000","last_peered":"2024-03-21T12:59:53.315902+0000","last_clean":"2024-03-21T12:59:53.315902+0000","last_became_active":"2024-03-21T12:58:37.730547+0000","last_became_peered":"2024-03-21T12:58:37.730547+0000","last_unstale":"2024-03-21T12:59:53.315902+0000","last_undegraded":"2024-03-21T12:59:53.315902+0000","last_fullsized":"2024-03-21T12:59:53.315902+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:01.180130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"86'15","reported_seq":60,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316813+0000","last_change":"2024-03-21T12:59:45.653329+0000","last_active":"2024-03-21T12:59:53.316813+0000","last_peered":"2024-03-21T12:59:53.316813+0000","last_clean":"2024-03-21T12:59:53.316813+0000","last_became_active":"2024-03-21T12:59:01.160456+0000","last_became_peered":"2024-03-21T12:59:01.160456+0000","last_unstale":"2024-03-21T12:59:53.316813+0000","last_undegraded":"2024-03-21T12:59:53.316813+0000","last_fullsized":"2024-03-21T12:59:53.316813+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.653197+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.205091+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339579+0000","last_change":"2024-03-21T12:59:48.409996+0000","last_active":"2024-03-21T12:59:53.339579+0000","last_peered":"2024-03-21T12:59:53.339579+0000","last_clean":"2024-03-21T12:59:53.339579+0000","last_became_active":"2024-03-21T12:58:41.171770+0000","last_became_peered":"2024-03-21T12:58:41.171770+0000","last_unstale":"2024-03-21T12:59:53.339579+0000","last_undegraded":"2024-03-21T12:59:53.339579+0000","last_fullsized":"2024-03-21T12:59:53.339579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:48.409905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.409905+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.077210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841012+0000","last_change":"2024-03-21T12:58:43.353379+0000","last_active":"2024-03-21T12:59:53.841012+0000","last_peered":"2024-03-21T12:59:53.841012+0000","last_clean":"2024-03-21T12:59:53.841012+0000","last_became_active":"2024-03-21T12:58:43.353049+0000","last_became_peered":"2024-03-21T12:58:43.353049+0000","last_unstale":"2024-03-21T12:59:53.841012+0000","last_undegraded":"2024-03-21T12:59:53.841012+0000","last_fullsized":"2024-03-21T12:59:53.841012+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:03.153074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339519+0000","last_change":"2024-03-21T12:58:37.735715+0000","last_active":"2024-03-21T12:59:53.339519+0000","last_peered":"2024-03-21T12:59:53.339519+0000","last_clean":"2024-03-21T12:59:53.339519+0000","last_became_active":"2024-03-21T12:58:37.735421+0000","last_became_peered":"2024-03-21T12:58:37.735421+0000","last_unstale":"2024-03-21T12:59:53.339519+0000","last_undegraded":"2024-03-21T12:59:53.339519+0000","last_fullsized":"2024-03-21T12:59:53.339519+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:04.750303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"87'21","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840274+0000","last_change":"2024-03-21T12:59:52.916371+0000","last_active":"2024-03-21T12:59:53.840274+0000","last_peered":"2024-03-21T12:59:53.840274+0000","last_clean":"2024-03-21T12:59:53.840274+0000","last_became_active":"2024-03-21T12:58:39.743592+0000","last_became_peered":"2024-03-21T12:58:39.743592+0000","last_unstale":"2024-03-21T12:59:53.840274+0000","last_undegraded":"2024-03-21T12:59:53.840274+0000","last_fullsized":"2024-03-21T12:59:53.840274+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'21","last_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.916260+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.636952+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"71'8","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.787439+0000","last_change":"2024-03-21T12:58:41.174471+0000","last_active":"2024-03-21T12:59:57.787439+0000","last_peered":"2024-03-21T12:59:57.787439+0000","last_clean":"2024-03-21T12:59:57.787439+0000","last_became_active":"2024-03-21T12:58:41.174137+0000","last_became_peered":"2024-03-21T12:58:41.174137+0000","last_unstale":"2024-03-21T12:59:57.787439+0000","last_undegraded":"2024-03-21T12:59:57.787439+0000","last_fullsized":"2024-03-21T12:59:57.787439+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840340+0000","last_change":"2024-03-21T12:58:43.181411+0000","last_active":"2024-03-21T12:59:53.840340+0000","last_peered":"2024-03-21T12:59:53.840340+0000","last_clean":"2024-03-21T12:59:53.840340+0000","last_became_active":"2024-03-21T12:58:43.180939+0000","last_became_peered":"2024-03-21T12:58:43.180939+0000","last_unstale":"2024-03-21T12:59:53.840340+0000","last_undegraded":"2024-03-21T12:59:53.840340+0000","last_fullsized":"2024-03-21T12:59:53.840340+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316308+0000","last_change":"2024-03-21T12:59:43.240826+0000","last_active":"2024-03-21T12:59:53.316308+0000","last_peered":"2024-03-21T12:59:53.316308+0000","last_clean":"2024-03-21T12:59:53.316308+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T12:59:53.316308+0000","last_undegraded":"2024-03-21T12:59:53.316308+0000","last_fullsized":"2024-03-21T12:59:53.316308+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T12:59:43.240674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.075915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"71'18","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.314541+0000","last_change":"2024-03-21T12:59:48.179286+0000","last_active":"2024-03-21T12:59:53.314541+0000","last_peered":"2024-03-21T12:59:53.314541+0000","last_clean":"2024-03-21T12:59:53.314541+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T12:59:53.314541+0000","last_undegraded":"2024-03-21T12:59:53.314541+0000","last_fullsized":"2024-03-21T12:59:53.314541+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T12:59:48.179188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.179188+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:15.921161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840082+0000","last_change":"2024-03-21T12:58:41.165071+0000","last_active":"2024-03-21T12:59:53.840082+0000","last_peered":"2024-03-21T12:59:53.840082+0000","last_clean":"2024-03-21T12:59:53.840082+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T12:59:53.840082+0000","last_undegraded":"2024-03-21T12:59:53.840082+0000","last_fullsized":"2024-03-21T12:59:53.840082+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:06.598245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.599170+0000","last_change":"2024-03-21T12:59:57.599220+0000","last_active":"2024-03-21T12:59:57.599170+0000","last_peered":"2024-03-21T12:59:57.599170+0000","last_clean":"2024-03-21T12:59:57.599170+0000","last_became_active":"2024-03-21T12:58:43.183614+0000","last_became_peered":"2024-03-21T12:58:43.183614+0000","last_unstale":"2024-03-21T12:59:57.599170+0000","last_undegraded":"2024-03-21T12:59:57.599170+0000","last_fullsized":"2024-03-21T12:59:57.599170+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.599076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.599076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:00.058326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.787745+0000","last_change":"2024-03-21T12:58:41.172355+0000","last_active":"2024-03-21T12:59:57.787745+0000","last_peered":"2024-03-21T12:59:57.787745+0000","last_clean":"2024-03-21T12:59:57.787745+0000","last_became_active":"2024-03-21T12:58:41.172019+0000","last_became_peered":"2024-03-21T12:58:41.172019+0000","last_unstale":"2024-03-21T12:59:57.787745+0000","last_undegraded":"2024-03-21T12:59:57.787745+0000","last_fullsized":"2024-03-21T12:59:57.787745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.314481+0000","last_change":"2024-03-21T12:59:41.231430+0000","last_active":"2024-03-21T12:59:53.314481+0000","last_peered":"2024-03-21T12:59:53.314481+0000","last_clean":"2024-03-21T12:59:53.314481+0000","last_became_active":"2024-03-21T12:58:37.731327+0000","last_became_peered":"2024-03-21T12:58:37.731327+0000","last_unstale":"2024-03-21T12:59:53.314481+0000","last_undegraded":"2024-03-21T12:59:53.314481+0000","last_fullsized":"2024-03-21T12:59:53.314481+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:41.231340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:41.231340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.247538+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.711208+0000","last_change":"2024-03-21T12:58:39.755619+0000","last_active":"2024-03-21T12:59:53.711208+0000","last_peered":"2024-03-21T12:59:53.711208+0000","last_clean":"2024-03-21T12:59:53.711208+0000","last_became_active":"2024-03-21T12:58:39.755326+0000","last_became_peered":"2024-03-21T12:58:39.755326+0000","last_unstale":"2024-03-21T12:59:53.711208+0000","last_undegraded":"2024-03-21T12:59:53.711208+0000","last_fullsized":"2024-03-21T12:59:53.711208+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:03.535907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315577+0000","last_change":"2024-03-21T12:58:43.351417+0000","last_active":"2024-03-21T12:59:53.315577+0000","last_peered":"2024-03-21T12:59:53.315577+0000","last_clean":"2024-03-21T12:59:53.315577+0000","last_became_active":"2024-03-21T12:58:43.350921+0000","last_became_peered":"2024-03-21T12:58:43.350921+0000","last_unstale":"2024-03-21T12:59:53.315577+0000","last_undegraded":"2024-03-21T12:59:53.315577+0000","last_fullsized":"2024-03-21T12:59:53.315577+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:53.390578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316213+0000","last_change":"2024-03-21T12:58:41.169093+0000","last_active":"2024-03-21T12:59:53.316213+0000","last_peered":"2024-03-21T12:59:53.316213+0000","last_clean":"2024-03-21T12:59:53.316213+0000","last_became_active":"2024-03-21T12:58:41.168784+0000","last_became_peered":"2024-03-21T12:58:41.168784+0000","last_unstale":"2024-03-21T12:59:53.316213+0000","last_undegraded":"2024-03-21T12:59:53.316213+0000","last_fullsized":"2024-03-21T12:59:53.316213+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:55.289928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.840178+0000","last_change":"2024-03-21T12:58:37.731156+0000","last_active":"2024-03-21T12:59:53.840178+0000","last_peered":"2024-03-21T12:59:53.840178+0000","last_clean":"2024-03-21T12:59:53.840178+0000","last_became_active":"2024-03-21T12:58:37.730858+0000","last_became_peered":"2024-03-21T12:58:37.730858+0000","last_unstale":"2024-03-21T12:59:53.840178+0000","last_undegraded":"2024-03-21T12:59:53.840178+0000","last_fullsized":"2024-03-21T12:59:53.840178+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:53.876749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'11","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841547+0000","last_change":"2024-03-21T12:58:39.754932+0000","last_active":"2024-03-21T12:59:53.841547+0000","last_peered":"2024-03-21T12:59:53.841547+0000","last_clean":"2024-03-21T12:59:53.841547+0000","last_became_active":"2024-03-21T12:58:39.754337+0000","last_became_peered":"2024-03-21T12:58:39.754337+0000","last_unstale":"2024-03-21T12:59:53.841547+0000","last_undegraded":"2024-03-21T12:59:53.841547+0000","last_fullsized":"2024-03-21T12:59:53.841547+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:04.723925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316645+0000","last_change":"2024-03-21T12:58:43.183848+0000","last_active":"2024-03-21T12:59:53.316645+0000","last_peered":"2024-03-21T12:59:53.316645+0000","last_clean":"2024-03-21T12:59:53.316645+0000","last_became_active":"2024-03-21T12:58:43.183380+0000","last_became_peered":"2024-03-21T12:58:43.183380+0000","last_unstale":"2024-03-21T12:59:53.316645+0000","last_undegraded":"2024-03-21T12:59:53.316645+0000","last_fullsized":"2024-03-21T12:59:53.316645+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:11.881657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.711049+0000","last_change":"2024-03-21T12:59:50.039168+0000","last_active":"2024-03-21T12:59:53.711049+0000","last_peered":"2024-03-21T12:59:53.711049+0000","last_clean":"2024-03-21T12:59:53.711049+0000","last_became_active":"2024-03-21T12:58:41.163145+0000","last_became_peered":"2024-03-21T12:58:41.163145+0000","last_unstale":"2024-03-21T12:59:53.711049+0000","last_undegraded":"2024-03-21T12:59:53.711049+0000","last_fullsized":"2024-03-21T12:59:53.711049+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.039091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.039091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.361130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.754942+0000","last_change":"2024-03-21T12:59:49.204557+0000","last_active":"2024-03-21T12:59:53.754942+0000","last_peered":"2024-03-21T12:59:53.754942+0000","last_clean":"2024-03-21T12:59:53.754942+0000","last_became_active":"2024-03-21T12:58:37.729792+0000","last_became_peered":"2024-03-21T12:58:37.729792+0000","last_unstale":"2024-03-21T12:59:53.754942+0000","last_undegraded":"2024-03-21T12:59:53.754942+0000","last_fullsized":"2024-03-21T12:59:53.754942+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:49.204419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:49.204419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.549632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841068+0000","last_change":"2024-03-21T12:58:39.746134+0000","last_active":"2024-03-21T12:59:53.841068+0000","last_peered":"2024-03-21T12:59:53.841068+0000","last_clean":"2024-03-21T12:59:53.841068+0000","last_became_active":"2024-03-21T12:58:39.745807+0000","last_became_peered":"2024-03-21T12:58:39.745807+0000","last_unstale":"2024-03-21T12:59:53.841068+0000","last_undegraded":"2024-03-21T12:59:53.841068+0000","last_fullsized":"2024-03-21T12:59:53.841068+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:06.303038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841680+0000","last_change":"2024-03-21T12:58:43.185167+0000","last_active":"2024-03-21T12:59:53.841680+0000","last_peered":"2024-03-21T12:59:53.841680+0000","last_clean":"2024-03-21T12:59:53.841680+0000","last_became_active":"2024-03-21T12:58:43.184576+0000","last_became_peered":"2024-03-21T12:58:43.184576+0000","last_unstale":"2024-03-21T12:59:53.841680+0000","last_undegraded":"2024-03-21T12:59:53.841680+0000","last_fullsized":"2024-03-21T12:59:53.841680+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:00.146215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"71'8","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316355+0000","last_change":"2024-03-21T12:58:41.165903+0000","last_active":"2024-03-21T12:59:53.316355+0000","last_peered":"2024-03-21T12:59:53.316355+0000","last_clean":"2024-03-21T12:59:53.316355+0000","last_became_active":"2024-03-21T12:58:41.165560+0000","last_became_peered":"2024-03-21T12:58:41.165560+0000","last_unstale":"2024-03-21T12:59:53.316355+0000","last_undegraded":"2024-03-21T12:59:53.316355+0000","last_fullsized":"2024-03-21T12:59:53.316355+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:09.702381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.315408+0000","last_change":"2024-03-21T12:58:37.732264+0000","last_active":"2024-03-21T12:59:53.315408+0000","last_peered":"2024-03-21T12:59:53.315408+0000","last_clean":"2024-03-21T12:59:53.315408+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T12:59:53.315408+0000","last_undegraded":"2024-03-21T12:59:53.315408+0000","last_fullsized":"2024-03-21T12:59:53.315408+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.841020+0000","last_change":"2024-03-21T12:58:39.750295+0000","last_active":"2024-03-21T12:59:53.841020+0000","last_peered":"2024-03-21T12:59:53.841020+0000","last_clean":"2024-03-21T12:59:53.841020+0000","last_became_active":"2024-03-21T12:58:39.749956+0000","last_became_peered":"2024-03-21T12:58:39.749956+0000","last_unstale":"2024-03-21T12:59:53.841020+0000","last_undegraded":"2024-03-21T12:59:53.841020+0000","last_fullsized":"2024-03-21T12:59:53.841020+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":415,"ondisk_log_size":415,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168598,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1499136,"data_stored":968748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462940,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32628,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712524,"statfs":{"total":95995035648,"available":95961624576,"internally_reserved":0,"allocated":1486848,"data_stored":968145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":377872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1499136,"data_stored":967579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":892928,"data_stored":377298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182443,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":377757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":913408,"data_stored":377933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724576,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":376756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:00:07.221 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-21T13:00:07.221 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-21T13:00:07.222 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-21T13:00:07.222 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-21T13:00:07.470 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:07.868 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[40675]: 3.f scrub starts 2024-03-21T13:00:07.868 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[40675]: 3.f scrub ok 2024-03-21T13:00:07.869 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[40675]: 5.2 scrub starts 2024-03-21T13:00:07.869 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[40675]: 5.2 scrub ok 2024-03-21T13:00:07.869 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[40675]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:07.869 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[40675]: 4.16 scrub starts 2024-03-21T13:00:07.869 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[40675]: 4.16 scrub ok 2024-03-21T13:00:07.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[29621]: 3.f scrub starts 2024-03-21T13:00:07.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[29621]: 3.f scrub ok 2024-03-21T13:00:07.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[29621]: 5.2 scrub starts 2024-03-21T13:00:07.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[29621]: 5.2 scrub ok 2024-03-21T13:00:07.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[29621]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:07.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[29621]: 4.16 scrub starts 2024-03-21T13:00:07.870 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:07 smithi163 ceph-mon[29621]: 4.16 scrub ok 2024-03-21T13:00:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:07 smithi204 ceph-mon[33153]: 3.f scrub starts 2024-03-21T13:00:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:07 smithi204 ceph-mon[33153]: 3.f scrub ok 2024-03-21T13:00:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:07 smithi204 ceph-mon[33153]: 5.2 scrub starts 2024-03-21T13:00:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:07 smithi204 ceph-mon[33153]: 5.2 scrub ok 2024-03-21T13:00:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:07 smithi204 ceph-mon[33153]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:07 smithi204 ceph-mon[33153]: 4.16 scrub starts 2024-03-21T13:00:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:07 smithi204 ceph-mon[33153]: 4.16 scrub ok 2024-03-21T13:00:08.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:08 smithi163 ceph-mon[29621]: 5.b scrub starts 2024-03-21T13:00:08.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:08 smithi163 ceph-mon[29621]: 5.b scrub ok 2024-03-21T13:00:08.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:08 smithi163 ceph-mon[40675]: 5.b scrub starts 2024-03-21T13:00:08.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:08 smithi163 ceph-mon[40675]: 5.b scrub ok 2024-03-21T13:00:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:08 smithi204 ceph-mon[33153]: 5.b scrub starts 2024-03-21T13:00:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:08 smithi204 ceph-mon[33153]: 5.b scrub ok 2024-03-21T13:00:09.452 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:00:09.452 INFO:teuthology.orchestra.run.smithi163.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-21T13:00:10.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:09 smithi204 ceph-mon[33153]: 6.15 scrub starts 2024-03-21T13:00:10.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:09 smithi204 ceph-mon[33153]: 6.15 scrub ok 2024-03-21T13:00:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:09 smithi204 ceph-mon[33153]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:09 smithi204 ceph-mon[33153]: 5.f scrub starts 2024-03-21T13:00:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:09 smithi204 ceph-mon[33153]: 5.f scrub ok 2024-03-21T13:00:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:09 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/623099927' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T13:00:10.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:09 smithi163 ceph-mon[29621]: 6.15 scrub starts 2024-03-21T13:00:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:09 smithi163 ceph-mon[29621]: 6.15 scrub ok 2024-03-21T13:00:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:09 smithi163 ceph-mon[29621]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:09 smithi163 ceph-mon[29621]: 5.f scrub starts 2024-03-21T13:00:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:09 smithi163 ceph-mon[29621]: 5.f scrub ok 2024-03-21T13:00:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:09 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/623099927' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T13:00:10.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:09 smithi163 ceph-mon[40675]: 6.15 scrub starts 2024-03-21T13:00:10.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:09 smithi163 ceph-mon[40675]: 6.15 scrub ok 2024-03-21T13:00:10.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:09 smithi163 ceph-mon[40675]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:10.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:09 smithi163 ceph-mon[40675]: 5.f scrub starts 2024-03-21T13:00:10.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:09 smithi163 ceph-mon[40675]: 5.f scrub ok 2024-03-21T13:00:10.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:09 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/623099927' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T13:00:10.167 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-21T13:00:10.167 INFO:tasks.cephadm:Setup complete, yielding 2024-03-21T13:00:10.168 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-21T13:00:10.183 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-21T13:00:10.183 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-testing-2024-03-21-0818'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '960fbbe4645c2becc8084669be194c95183c7753'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '960fbbe4645c2becc8084669be194c95183c7753'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-guits-testing-2024-03-21-0818', 'sha1': '960fbbe4645c2becc8084669be194c95183c7753'}} 2024-03-21T13:00:10.183 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-21T13:00:10.183 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-21T13:00:10.423 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:10.851 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[29621]: 6.2 deep-scrub starts 2024-03-21T13:00:10.851 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[29621]: 6.2 deep-scrub ok 2024-03-21T13:00:10.851 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[29621]: 3.16 scrub starts 2024-03-21T13:00:10.851 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[29621]: 3.16 scrub ok 2024-03-21T13:00:10.851 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:00:10.851 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:00:10.852 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[29621]: 6.13 scrub starts 2024-03-21T13:00:10.852 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[29621]: 6.13 scrub ok 2024-03-21T13:00:10.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[40675]: 6.2 deep-scrub starts 2024-03-21T13:00:10.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[40675]: 6.2 deep-scrub ok 2024-03-21T13:00:10.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[40675]: 3.16 scrub starts 2024-03-21T13:00:10.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[40675]: 3.16 scrub ok 2024-03-21T13:00:10.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:00:10.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:00:10.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[40675]: 6.13 scrub starts 2024-03-21T13:00:10.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:10 smithi163 ceph-mon[40675]: 6.13 scrub ok 2024-03-21T13:00:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:10 smithi204 ceph-mon[33153]: 6.2 deep-scrub starts 2024-03-21T13:00:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:10 smithi204 ceph-mon[33153]: 6.2 deep-scrub ok 2024-03-21T13:00:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:10 smithi204 ceph-mon[33153]: 3.16 scrub starts 2024-03-21T13:00:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:10 smithi204 ceph-mon[33153]: 3.16 scrub ok 2024-03-21T13:00:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:10 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:00:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:10 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:00:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:10 smithi204 ceph-mon[33153]: 6.13 scrub starts 2024-03-21T13:00:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:10 smithi204 ceph-mon[33153]: 6.13 scrub ok 2024-03-21T13:00:11.419 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:00:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:00:11.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[29621]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:11.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[29621]: 5.7 scrub starts 2024-03-21T13:00:11.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[29621]: 5.7 scrub ok 2024-03-21T13:00:11.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[29621]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T13:00:11.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[29621]: 4.c scrub starts 2024-03-21T13:00:11.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[29621]: 4.c scrub ok 2024-03-21T13:00:11.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[29621]: 3.19 scrub starts 2024-03-21T13:00:11.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[29621]: 3.19 scrub ok 2024-03-21T13:00:11.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:11.823 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[29621]: 5.d scrub starts 2024-03-21T13:00:11.823 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[29621]: 5.d scrub ok 2024-03-21T13:00:11.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[40675]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:11.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[40675]: 5.7 scrub starts 2024-03-21T13:00:11.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[40675]: 5.7 scrub ok 2024-03-21T13:00:11.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[40675]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T13:00:11.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[40675]: 4.c scrub starts 2024-03-21T13:00:11.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[40675]: 4.c scrub ok 2024-03-21T13:00:11.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[40675]: 3.19 scrub starts 2024-03-21T13:00:11.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[40675]: 3.19 scrub ok 2024-03-21T13:00:11.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:11.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[40675]: 5.d scrub starts 2024-03-21T13:00:11.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:11 smithi163 ceph-mon[40675]: 5.d scrub ok 2024-03-21T13:00:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:11 smithi204 ceph-mon[33153]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:11 smithi204 ceph-mon[33153]: 5.7 scrub starts 2024-03-21T13:00:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:11 smithi204 ceph-mon[33153]: 5.7 scrub ok 2024-03-21T13:00:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:11 smithi204 ceph-mon[33153]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T13:00:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:11 smithi204 ceph-mon[33153]: 4.c scrub starts 2024-03-21T13:00:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:11 smithi204 ceph-mon[33153]: 4.c scrub ok 2024-03-21T13:00:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:11 smithi204 ceph-mon[33153]: 3.19 scrub starts 2024-03-21T13:00:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:11 smithi204 ceph-mon[33153]: 3.19 scrub ok 2024-03-21T13:00:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:11 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:11 smithi204 ceph-mon[33153]: 5.d scrub starts 2024-03-21T13:00:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:11 smithi204 ceph-mon[33153]: 5.d scrub ok 2024-03-21T13:00:12.511 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[75154]: ts=2024-03-21T13:00:12.425Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003218412s 2024-03-21T13:00:12.511 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:00:12] ENGINE Bus STOPPING 2024-03-21T13:00:12.511 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:00:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:00:12.511 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:00:12] ENGINE Bus STOPPED 2024-03-21T13:00:12.511 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:00:12] ENGINE Bus STARTING 2024-03-21T13:00:12.511 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:00:12] ENGINE Serving on http://:::9283 2024-03-21T13:00:12.511 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:00:12] ENGINE Bus STARTED 2024-03-21T13:00:12.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:12 smithi204 ceph-mon[33153]: 5.4 deep-scrub starts 2024-03-21T13:00:12.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:12 smithi204 ceph-mon[33153]: 5.4 deep-scrub ok 2024-03-21T13:00:12.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:12 smithi204 ceph-mon[33153]: 6.1 scrub starts 2024-03-21T13:00:12.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:12 smithi204 ceph-mon[33153]: 6.1 scrub ok 2024-03-21T13:00:12.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:12 smithi204 ceph-mon[33153]: 5.17 scrub starts 2024-03-21T13:00:12.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:12 smithi204 ceph-mon[33153]: 5.17 scrub ok 2024-03-21T13:00:12.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:12 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' 2024-03-21T13:00:13.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[40675]: 5.4 deep-scrub starts 2024-03-21T13:00:13.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[40675]: 5.4 deep-scrub ok 2024-03-21T13:00:13.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[40675]: 6.1 scrub starts 2024-03-21T13:00:13.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[40675]: 6.1 scrub ok 2024-03-21T13:00:13.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[40675]: 5.17 scrub starts 2024-03-21T13:00:13.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[40675]: 5.17 scrub ok 2024-03-21T13:00:13.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' 2024-03-21T13:00:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[29621]: 5.4 deep-scrub starts 2024-03-21T13:00:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[29621]: 5.4 deep-scrub ok 2024-03-21T13:00:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[29621]: 6.1 scrub starts 2024-03-21T13:00:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[29621]: 6.1 scrub ok 2024-03-21T13:00:13.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[29621]: 5.17 scrub starts 2024-03-21T13:00:13.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[29621]: 5.17 scrub ok 2024-03-21T13:00:13.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:12 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' 2024-03-21T13:00:13.046 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-21T13:00:13.046 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-21T13:00:13.047 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:00:13.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 systemd[1]: Starting Ceph grafana.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:00:13.294 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:13.427 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 podman[64011]: 2024-03-21 13:00:13.191675406 +0000 UTC m=+0.018955593 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-03-21T13:00:13.427 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 podman[64011]: 2024-03-21 13:00:13.320958548 +0000 UTC m=+0.148238723 container create 146318eeea1aeb9cf3a13aa4bcc912d2a5d79144e080ccb0f6d0930ad768e87b (image=quay.io/ceph/grafana:9.4.12, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:00:13.686 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 podman[64011]: 2024-03-21 13:00:13.531480784 +0000 UTC m=+0.358760971 container init 146318eeea1aeb9cf3a13aa4bcc912d2a5d79144e080ccb0f6d0930ad768e87b (image=quay.io/ceph/grafana:9.4.12, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:00:13.686 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 podman[64011]: 2024-03-21 13:00:13.534373306 +0000 UTC m=+0.361653488 container start 146318eeea1aeb9cf3a13aa4bcc912d2a5d79144e080ccb0f6d0930ad768e87b (image=quay.io/ceph/grafana:9.4.12, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:00:13.686 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 bash[64011]: 146318eeea1aeb9cf3a13aa4bcc912d2a5d79144e080ccb0f6d0930ad768e87b 2024-03-21T13:00:13.686 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 systemd[1]: Started Ceph grafana.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T13:00:13.939 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686390622Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-21T13:00:13.939 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686585429Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-21T13:00:13.939 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686635551Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-21T13:00:13.940 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.6866453Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-21T13:00:13.940 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686650431Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-21T13:00:13.940 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.68665598Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-21T13:00:13.940 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686660571Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-21T13:00:13.940 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686690457Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-21T13:00:13.940 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686695176Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-21T13:00:13.940 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686699951Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-21T13:00:13.940 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686704505Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-21T13:00:13.940 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686708811Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-21T13:00:13.940 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686713289Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-21T13:00:13.940 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686717991Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-21T13:00:13.940 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686722709Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-21T13:00:13.941 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.68672695Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-21T13:00:13.941 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686731159Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-21T13:00:13.941 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686735735Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-21T13:00:13.941 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=settings t=2024-03-21T13:00:13.686740496Z level=info msg="App mode production" 2024-03-21T13:00:13.941 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=sqlstore t=2024-03-21T13:00:13.686797945Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-21T13:00:13.941 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=sqlstore t=2024-03-21T13:00:13.686813798Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-21T13:00:13.941 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.687682054Z level=info msg="Starting DB migrations" 2024-03-21T13:00:13.941 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.687757318Z level=info msg="Executing migration" id="create migration_log table" 2024-03-21T13:00:13.941 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.688301065Z level=info msg="Executing migration" id="create user table" 2024-03-21T13:00:13.941 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.688741629Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-21T13:00:13.941 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.689227353Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-21T13:00:13.941 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.68970534Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-21T13:00:13.941 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.69021703Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-21T13:00:13.942 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.690622436Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-21T13:00:13.942 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.691385579Z level=info msg="Executing migration" id="create user table v2" 2024-03-21T13:00:13.942 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.691780288Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-21T13:00:13.942 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.69218209Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-21T13:00:13.942 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.692590598Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-21T13:00:13.942 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.692892581Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-21T13:00:13.942 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.693266145Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-21T13:00:13.942 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.693672534Z level=info msg="Executing migration" id="Update user table charset" 2024-03-21T13:00:13.942 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.693934337Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-21T13:00:13.942 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.694341259Z level=info msg="Executing migration" id="Add missing user data" 2024-03-21T13:00:13.944 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.694654111Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-21T13:00:13.944 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.69506811Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-21T13:00:13.944 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.695454016Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-21T13:00:13.944 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.695835776Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-21T13:00:13.944 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.697180461Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-21T13:00:13.944 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.697599262Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-21T13:00:13.944 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.698038741Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-21T13:00:13.944 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.698419077Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-21T13:00:13.944 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.698789038Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-21T13:00:13.944 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.699166212Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-21T13:00:13.945 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.699443308Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-21T13:00:13.945 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.69980079Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-21T13:00:13.945 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.700192812Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-21T13:00:13.945 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.700552234Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-21T13:00:13.945 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.700892913Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-21T13:00:13.945 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.70163135Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-21T13:00:13.945 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.702036937Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-21T13:00:13.945 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.702771822Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-21T13:00:13.945 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.70315269Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-21T13:00:13.945 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.70352397Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-21T13:00:13.945 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.703903771Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-21T13:00:13.945 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.704229429Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-21T13:00:13.945 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.704535718Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.704833985Z level=info msg="Executing migration" id="create star table" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.705194894Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.705577408Z level=info msg="Executing migration" id="create org table v1" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.705943648Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.706326205Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.706666116Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.707090313Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.707533169Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.707923783Z level=info msg="Executing migration" id="Update org table charset" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.708290459Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.709052898Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.709378494Z level=info msg="Executing migration" id="create dashboard table" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.709905594Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-21T13:00:13.946 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.71034529Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.71115977Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.711964871Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.712806017Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.713241123Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.714351807Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.717066449Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.719765803Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.720230858Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.722794037Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.72542859Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.728102341Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.730888341Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.733697021Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-21T13:00:13.947 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.736254864Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.738930238Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.741701598Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.744399064Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.744855947Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.747531954Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.750238496Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.752937942Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.753489349Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.756317835Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.758959201Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.761568238Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.762039814Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.764693783Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-21T13:00:13.948 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.767369125Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.769925844Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.770338084Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.773015979Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.775530545Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.778198996Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.778657464Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.781150993Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.781566323Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.783842907Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.786517347Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.786846077Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.789492695Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.790271697Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-21T13:00:13.949 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.791070934Z level=info msg="Executing migration" id="create data_source table" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.791532236Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.792332195Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.793138879Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.793947078Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.794387529Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.795635199Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.796435258Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.797251555Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.797696831Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.798504954Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.799283026Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.800102349Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.800420619Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-21T13:00:13.950 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.80124728Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.801836207Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.802188237Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.802973496Z level=info msg="Executing migration" id="Add uid column" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.80355487Z level=info msg="Executing migration" id="Update uid value" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.803861704Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.804308289Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.804735246Z level=info msg="Executing migration" id="create api_key table" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.805566753Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.806413606Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.806849239Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.807695317Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.808527354Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.809398869Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-21T13:00:13.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.810277552Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.811583544Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.81239861Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.813219394Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.814077958Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.814987771Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.817732878Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.820478204Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.823133373Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.825853314Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.828638126Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.831287326Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.834048954Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.836822425Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-21T13:00:13.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.839429811Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.842188981Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.844932638Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.847645543Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.850331651Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.852915231Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.855642578Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.85835946Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.861185013Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.863946613Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.866597762Z level=info msg="Executing migration" id="create quota table v1" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.869326811Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.871864118Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.874409057Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-21T13:00:13.953 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.877064509Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.879743473Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.882389539Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.884977957Z level=info msg="Executing migration" id="create session table" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.887561155Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.890278912Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.892974161Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.895662779Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.898347372Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.900926711Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.903593116Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.906268658Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.908975132Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.909783776Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-21T13:00:13.954 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.910556654Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-21T13:00:13.955 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.911318854Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-21T13:00:13.955 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.912137064Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-21T13:00:13.955 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.912962665Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-21T13:00:13.955 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.913740206Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-21T13:00:13.955 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.914557189Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-21T13:00:13.955 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.915385491Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-21T13:00:13.955 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.916187767Z level=info msg="Executing migration" id="create alert table v1" 2024-03-21T13:00:13.955 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.916998979Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-21T13:00:13.955 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.917787662Z level=info msg="Executing migration" id="add index alert state" 2024-03-21T13:00:13.955 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.918588739Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-21T13:00:13.955 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.91937721Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-21T13:00:13.955 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.92016397Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-21T13:00:13.955 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.920985567Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.921761487Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.923437686Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.924318456Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.925101718Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.925421671Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.925764787Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.926140213Z level=info msg="Executing migration" id="Add column is_default" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.926848868Z level=info msg="Executing migration" id="Add column frequency" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.927537959Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.928231226Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.928928022Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.929335599Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.929582878Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-21T13:00:13.956 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.929829755Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-21T13:00:13.957 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.930197577Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-21T13:00:13.957 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.930589165Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-21T13:00:13.957 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.931021528Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-21T13:00:13.957 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.933687289Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-21T13:00:13.957 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.936488798Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-21T13:00:13.957 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.939211039Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-21T13:00:13.957 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.941925053Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-21T13:00:14.189 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.944668752Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-21T13:00:14.190 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.947246812Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-21T13:00:14.190 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.94988091Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-21T13:00:14.190 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.952511157Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-21T13:00:14.190 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.95510707Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-21T13:00:14.190 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.957881073Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-21T13:00:14.190 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.960563878Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-21T13:00:14.190 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.963321352Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-21T13:00:14.190 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.96624748Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-21T13:00:14.190 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.96889016Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-21T13:00:14.190 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.971472916Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-21T13:00:14.190 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.974089993Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-21T13:00:14.190 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.976610559Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-21T13:00:14.190 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.979000043Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.981729327Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.984330338Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.987025779Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.989656231Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.992297349Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.994727272Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:13 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.997342914Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:13.999987953Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.002541033Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.005083928Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.007773317Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.010183005Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.010972586Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-21T13:00:14.191 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.011801861Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-21T13:00:14.192 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.012624052Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-21T13:00:14.192 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.013420814Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-21T13:00:14.192 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.014279478Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-21T13:00:14.192 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.015067242Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-21T13:00:14.192 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.015920048Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-21T13:00:14.192 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.016702631Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-21T13:00:14.192 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.01748329Z level=info msg="Executing migration" id="Move region to single row" 2024-03-21T13:00:14.192 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.01831478Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-21T13:00:14.192 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.019080998Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-21T13:00:14.193 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.01989197Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-21T13:00:14.193 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.02068185Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-21T13:00:14.193 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.021463106Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-21T13:00:14.194 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.022273149Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-21T13:00:14.194 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.023013709Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-21T13:00:14.194 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.023836462Z level=info msg="Executing migration" id="create test_data table" 2024-03-21T13:00:14.194 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.024556548Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-21T13:00:14.194 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.024964401Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-21T13:00:14.194 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.025778159Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-21T13:00:14.194 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.026599609Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-21T13:00:14.194 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.027402627Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-21T13:00:14.194 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.028239091Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-21T13:00:14.194 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.029030644Z level=info msg="Executing migration" id="create team table" 2024-03-21T13:00:14.194 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.029846879Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-21T13:00:14.194 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.030709939Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-21T13:00:14.194 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.031534071Z level=info msg="Executing migration" id="create team member table" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.032349595Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.033150716Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.033979351Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.034785512Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.03567042Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.03654564Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.037436898Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.038209827Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.038994347Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.04158455Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.042070498Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.044885922Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.045347343Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.045809676Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-21T13:00:14.195 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.046274549Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.046719118Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.047045013Z level=info msg="Executing migration" id="create tag table" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.047464266Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.047938774Z level=info msg="Executing migration" id="create login attempt table" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.048351686Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.048788743Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.049239075Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.051611299Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.052026097Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.05246524Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.052773981Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.053158394Z level=info msg="Executing migration" id="create user auth table" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.053578897Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-21T13:00:14.196 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.054032531Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.054359197Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.055227408Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.056096126Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.056971674Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.057906083Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.05837779Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.059326012Z level=info msg="Executing migration" id="create server_lock table" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.059753516Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.060242372Z level=info msg="Executing migration" id="create user auth token table" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.060705241Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.063338842Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.063830611Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.064308365Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.065202906Z level=info msg="Executing migration" id="create cache_data table" 2024-03-21T13:00:14.197 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.065647192Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.066122181Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.066583995Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.069249249Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.069602241Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.069907183Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.070386617Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.070854671Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.071315767Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.071573412Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.071980125Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.07238206Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.072774775Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.073200143Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-21T13:00:14.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.074065641Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.074482031Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.074746018Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.07727801Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.080159942Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.080579414Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.080828719Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.083248291Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.085866745Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.08857295Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.088983843Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.089860644Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.090676175Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.091562517Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-21T13:00:14.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.095992063Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.100405912Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.101216674Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.102091252Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.10307993Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.103908128Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.104691108Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.105504287Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.10598181Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.106762781Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.10771361Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.108640401Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.109602395Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.110461096Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-21T13:00:14.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.111336962Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-21T13:00:14.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.11399139Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-21T13:00:14.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.114920947Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-21T13:00:14.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.117692697Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-21T13:00:14.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.120284385Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-21T13:00:14.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.122998137Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-21T13:00:14.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.123348039Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-21T13:00:14.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.126219269Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-21T13:00:14.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.128797989Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-21T13:00:14.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.131589581Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-21T13:00:14.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.134294136Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-21T13:00:14.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.13706319Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-21T13:00:14.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.139776054Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-21T13:00:14.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.142491449Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.143474613Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.146403778Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.148997683Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.151778462Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.152757615Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.155535158Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.158110254Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.16070792Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.161717334Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.164372568Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.167100474Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.169636063Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.170076289Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-21T13:00:14.202 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.172681533Z level=info msg="Executing migration" id="create alert_image table" 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.175317267Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.177954473Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.178278345Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.181009126Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.183510621Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.183581286Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.186221304Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.18658258Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.189186923Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.189328777Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.189440539Z level=info msg="alerts found to migrate" alerts=0 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.18989511Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.190356248Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-21T13:00:14.203 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.191160803Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-21T13:00:14.204 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.191579033Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-21T13:00:14.204 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.192380886Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-21T13:00:14.204 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.193200407Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-21T13:00:14.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-mon[33153]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:14.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:14.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:14.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:14.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:14.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-mon[33153]: 4.7 scrub starts 2024-03-21T13:00:14.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-mon[33153]: 4.7 scrub ok 2024-03-21T13:00:14.504 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.193972586Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-21T13:00:14.504 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.194363651Z level=info msg="Executing migration" id="create data_keys table" 2024-03-21T13:00:14.504 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.195170736Z level=info msg="Executing migration" id="create secrets table" 2024-03-21T13:00:14.504 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.19595888Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-21T13:00:14.504 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.201545377Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-21T13:00:14.504 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.202631573Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-21T13:00:14.504 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.203441397Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-21T13:00:14.504 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.208999918Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-21T13:00:14.505 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.214643141Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-21T13:00:14.505 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.215059085Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-21T13:00:14.505 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.216028975Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-21T13:00:14.505 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.216712167Z level=info msg="Executing migration" id="create permission table" 2024-03-21T13:00:14.505 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.217592813Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-21T13:00:14.505 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.218021911Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-21T13:00:14.505 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.218986337Z level=info msg="Executing migration" id="create role table" 2024-03-21T13:00:14.505 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.219628581Z level=info msg="Executing migration" id="add column display_name" 2024-03-21T13:00:14.505 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.220710996Z level=info msg="Executing migration" id="add column group_name" 2024-03-21T13:00:14.505 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.221759091Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-21T13:00:14.505 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.222611771Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-21T13:00:14.505 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.223444671Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-21T13:00:14.505 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.224296637Z level=info msg="Executing migration" id="create team role table" 2024-03-21T13:00:14.506 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.225064716Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-21T13:00:14.506 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.226007711Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-21T13:00:14.506 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.226824931Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-21T13:00:14.506 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.227762239Z level=info msg="Executing migration" id="create user role table" 2024-03-21T13:00:14.506 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.228269213Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-21T13:00:14.506 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.229114525Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-21T13:00:14.506 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.229984189Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-21T13:00:14.506 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.23087706Z level=info msg="Executing migration" id="create builtin role table" 2024-03-21T13:00:14.506 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.231323074Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-21T13:00:14.506 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.234229851Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-21T13:00:14.506 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.236736742Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-21T13:00:14.506 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.239332599Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-21T13:00:14.506 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.239808514Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.242571087Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.245066787Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.247923629Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.250426312Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.250893011Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.253650053Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.256622742Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.259225647Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.261855429Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.262347578Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.265046293Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.26543304Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.267621003Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-21T13:00:14.507 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.268005644Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.270317604Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.270786483Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.273465861Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.276083421Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.278879073Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.279341338Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.281970845Z level=info msg="Executing migration" id="add correlation config column" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.284593373Z level=info msg="Executing migration" id="create entity_events table" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.287275941Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.287712328Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.287792279Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.290280099Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.290377797Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T13:00:14.508 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.293068409Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.295520952Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.295972263Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.298654653Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.301297459Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.301964274Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.302403897Z level=info msg="Executing migration" id="Drop public config table" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.303165898Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.303592152Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.304030727Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.304811545Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.305263571Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.309034648Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.310210964Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-21T13:00:14.509 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.311462976Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.311776262Z level=info msg="Executing migration" id="add share column" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.312938486Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.313837981Z level=info msg="Executing migration" id="create file table" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.314477387Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.314930331Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.315659204Z level=info msg="Executing migration" id="create file_meta table" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.316110335Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.316840608Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.317159662Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.317974053Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.318303198Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.318725582Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.319908742Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-21T13:00:14.510 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.320230092Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-21T13:00:14.511 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.320687209Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-21T13:00:14.511 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.321557717Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-21T13:00:14.511 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.322465675Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-21T13:00:14.511 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.323126323Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-21T13:00:14.511 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.324306238Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-21T13:00:14.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[29621]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:14.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:14.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:14.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:14.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:14.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[29621]: 4.7 scrub starts 2024-03-21T13:00:14.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[29621]: 4.7 scrub ok 2024-03-21T13:00:14.512 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.325525356Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-21T13:00:14.512 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.326331453Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-21T13:00:14.512 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.338879844Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-21T13:00:14.512 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.339349225Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-21T13:00:14.512 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.340153888Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-21T13:00:14.513 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.344510327Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-21T13:00:14.513 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.345386503Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-21T13:00:14.513 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.345898982Z level=info msg="Executing migration" id="create folder table" 2024-03-21T13:00:14.513 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.348810248Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-21T13:00:14.513 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.351472086Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-21T13:00:14.513 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.354154411Z level=info msg="Executing migration" id="Update folder title length" 2024-03-21T13:00:14.513 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.354568062Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-21T13:00:14.513 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=migrator t=2024-03-21T13:00:14.357605763Z level=info msg="migrations completed" performed=485 skipped=0 duration=669.865779ms 2024-03-21T13:00:14.513 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=sqlstore t=2024-03-21T13:00:14.35804133Z level=info msg="Created default organization" 2024-03-21T13:00:14.513 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=plugin.finder t=2024-03-21T13:00:14.392132875Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-21T13:00:14.513 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=secrets t=2024-03-21T13:00:14.392216816Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-21T13:00:14.513 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=query_data t=2024-03-21T13:00:14.394435714Z level=info msg="Query Service initialization" 2024-03-21T13:00:14.513 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=live.push_http t=2024-03-21T13:00:14.398659771Z level=info msg="Live Push Gateway initialization" 2024-03-21T13:00:14.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[40675]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:14.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:14.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:14.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:14.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:14.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[40675]: 4.7 scrub starts 2024-03-21T13:00:14.514 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:14 smithi163 ceph-mon[40675]: 4.7 scrub ok 2024-03-21T13:00:15.410 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:00:15.412 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:00:15.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[29621]: Deploying daemon node-exporter.a on smithi163 2024-03-21T13:00:15.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[29621]: 3.13 scrub starts 2024-03-21T13:00:15.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[29621]: 3.13 scrub ok 2024-03-21T13:00:15.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[29621]: 3.1b scrub starts 2024-03-21T13:00:15.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[29621]: 3.1b scrub ok 2024-03-21T13:00:15.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[29621]: 6.1d scrub starts 2024-03-21T13:00:15.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[29621]: 6.1d scrub ok 2024-03-21T13:00:15.421 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[40675]: Deploying daemon node-exporter.a on smithi163 2024-03-21T13:00:15.421 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[40675]: 3.13 scrub starts 2024-03-21T13:00:15.421 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[40675]: 3.13 scrub ok 2024-03-21T13:00:15.422 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[40675]: 3.1b scrub starts 2024-03-21T13:00:15.422 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[40675]: 3.1b scrub ok 2024-03-21T13:00:15.422 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[40675]: 6.1d scrub starts 2024-03-21T13:00:15.422 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-mon[40675]: 6.1d scrub ok 2024-03-21T13:00:15.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-mon[33153]: Deploying daemon node-exporter.a on smithi163 2024-03-21T13:00:15.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-mon[33153]: 3.13 scrub starts 2024-03-21T13:00:15.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-mon[33153]: 3.13 scrub ok 2024-03-21T13:00:15.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-mon[33153]: 3.1b scrub starts 2024-03-21T13:00:15.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-mon[33153]: 3.1b scrub ok 2024-03-21T13:00:15.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-mon[33153]: 6.1d scrub starts 2024-03-21T13:00:15.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-mon[33153]: 6.1d scrub ok 2024-03-21T13:00:15.503 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=infra.usagestats.collector t=2024-03-21T13:00:15.005020113Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-21T13:00:15.503 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=provisioning.datasources t=2024-03-21T13:00:15.006008821Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-21T13:00:15.503 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=provisioning.datasources t=2024-03-21T13:00:15.016963197Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-21T13:00:15.503 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=provisioning.alerting t=2024-03-21T13:00:15.027473028Z level=info msg="starting to provision alerting" 2024-03-21T13:00:15.503 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=provisioning.alerting t=2024-03-21T13:00:15.027483281Z level=info msg="finished to provision alerting" 2024-03-21T13:00:15.503 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=grafanaStorageLogger t=2024-03-21T13:00:15.027640562Z level=info msg="storage starting" 2024-03-21T13:00:15.503 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=ngalert.state.manager t=2024-03-21T13:00:15.027725013Z level=info msg="Warming state cache for startup" 2024-03-21T13:00:15.503 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=http.server t=2024-03-21T13:00:15.029454718Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-21T13:00:15.503 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=ngalert.state.manager t=2024-03-21T13:00:15.033813837Z level=info msg="State cache has been initialized" states=0 duration=6.088254ms 2024-03-21T13:00:15.504 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=ngalert.multiorg.alertmanager t=2024-03-21T13:00:15.033854713Z level=info msg="starting MultiOrg Alertmanager" 2024-03-21T13:00:15.504 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=ticker t=2024-03-21T13:00:15.033881422Z level=info msg=starting first_tick=2024-03-21T13:00:20Z 2024-03-21T13:00:15.504 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=sqlstore.transactions t=2024-03-21T13:00:15.038100611Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-21T13:00:15.504 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=sqlstore.transactions t=2024-03-21T13:00:15.048289135Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-21T13:00:15.504 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:00:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=sqlstore.transactions t=2024-03-21T13:00:15.059031038Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-21T13:00:15.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:00:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:00:16.339 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-03-21T13:00:14.626674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1023,"num_read_kb":869,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":624,"ondisk_log_size":624,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249764,"kb_used_data":8180,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749711452,"statfs":{"total":767960285184,"available":767704526848,"internally_reserved":0,"allocated":8376320,"data_stored":4659284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002715"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728486+0000","last_change":"2024-03-21T12:59:58.846777+0000","last_active":"2024-03-21T13:00:10.728486+0000","last_peered":"2024-03-21T13:00:10.728486+0000","last_clean":"2024-03-21T13:00:10.728486+0000","last_became_active":"2024-03-21T12:58:43.182506+0000","last_became_peered":"2024-03-21T12:58:43.182506+0000","last_unstale":"2024-03-21T13:00:10.728486+0000","last_undegraded":"2024-03-21T13:00:10.728486+0000","last_fullsized":"2024-03-21T13:00:10.728486+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:58.846634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:58.846634+0000","last_clean_scrub_stamp":"2024-03-21T12:59:58.846634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.412490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836484+0000","last_change":"2024-03-21T12:58:41.160668+0000","last_active":"2024-03-21T13:00:10.836484+0000","last_peered":"2024-03-21T13:00:10.836484+0000","last_clean":"2024-03-21T13:00:10.836484+0000","last_became_active":"2024-03-21T12:58:41.160342+0000","last_became_peered":"2024-03-21T12:58:41.160342+0000","last_unstale":"2024-03-21T13:00:10.836484+0000","last_undegraded":"2024-03-21T13:00:10.836484+0000","last_fullsized":"2024-03-21T13:00:10.836484+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.478199+0000","last_change":"2024-03-21T12:59:53.478218+0000","last_active":"2024-03-21T12:59:53.478199+0000","last_peered":"2024-03-21T12:59:53.478199+0000","last_clean":"2024-03-21T12:59:53.478199+0000","last_became_active":"2024-03-21T12:58:37.731393+0000","last_became_peered":"2024-03-21T12:58:37.731393+0000","last_unstale":"2024-03-21T12:59:53.478199+0000","last_undegraded":"2024-03-21T12:59:53.478199+0000","last_fullsized":"2024-03-21T12:59:53.478199+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:53.478141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:53.478141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:57.521483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"71'9","reported_seq":88,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728359+0000","last_change":"2024-03-21T12:59:01.166943+0000","last_active":"2024-03-21T13:00:10.728359+0000","last_peered":"2024-03-21T13:00:10.728359+0000","last_clean":"2024-03-21T13:00:10.728359+0000","last_became_active":"2024-03-21T12:59:01.161098+0000","last_became_peered":"2024-03-21T12:59:01.161098+0000","last_unstale":"2024-03-21T13:00:10.728359+0000","last_undegraded":"2024-03-21T13:00:10.728359+0000","last_fullsized":"2024-03-21T13:00:10.728359+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781719+0000","last_change":"2024-03-21T13:00:05.173374+0000","last_active":"2024-03-21T13:00:10.781719+0000","last_peered":"2024-03-21T13:00:10.781719+0000","last_clean":"2024-03-21T13:00:10.781719+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:00:10.781719+0000","last_undegraded":"2024-03-21T13:00:10.781719+0000","last_fullsized":"2024-03-21T13:00:10.781719+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:00:05.173276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:14.939464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728367+0000","last_change":"2024-03-21T12:58:41.168108+0000","last_active":"2024-03-21T13:00:10.728367+0000","last_peered":"2024-03-21T13:00:10.728367+0000","last_clean":"2024-03-21T13:00:10.728367+0000","last_became_active":"2024-03-21T12:58:41.167710+0000","last_became_peered":"2024-03-21T12:58:41.167710+0000","last_unstale":"2024-03-21T13:00:10.728367+0000","last_undegraded":"2024-03-21T13:00:10.728367+0000","last_fullsized":"2024-03-21T13:00:10.728367+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781984+0000","last_change":"2024-03-21T12:58:37.728682+0000","last_active":"2024-03-21T13:00:10.781984+0000","last_peered":"2024-03-21T13:00:10.781984+0000","last_clean":"2024-03-21T13:00:10.781984+0000","last_became_active":"2024-03-21T12:58:37.728383+0000","last_became_peered":"2024-03-21T12:58:37.728383+0000","last_unstale":"2024-03-21T13:00:10.781984+0000","last_undegraded":"2024-03-21T13:00:10.781984+0000","last_fullsized":"2024-03-21T13:00:10.781984+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"71'15","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781698+0000","last_change":"2024-03-21T12:58:39.751346+0000","last_active":"2024-03-21T13:00:10.781698+0000","last_peered":"2024-03-21T13:00:10.781698+0000","last_clean":"2024-03-21T13:00:10.781698+0000","last_became_active":"2024-03-21T12:58:39.750396+0000","last_became_peered":"2024-03-21T12:58:39.750396+0000","last_unstale":"2024-03-21T13:00:10.781698+0000","last_undegraded":"2024-03-21T13:00:10.781698+0000","last_fullsized":"2024-03-21T13:00:10.781698+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:02.599761+0000","last_change":"2024-03-21T13:00:02.599805+0000","last_active":"2024-03-21T13:00:02.599761+0000","last_peered":"2024-03-21T13:00:02.599761+0000","last_clean":"2024-03-21T13:00:02.599761+0000","last_became_active":"2024-03-21T12:58:43.179920+0000","last_became_peered":"2024-03-21T12:58:43.179920+0000","last_unstale":"2024-03-21T13:00:02.599761+0000","last_undegraded":"2024-03-21T13:00:02.599761+0000","last_fullsized":"2024-03-21T13:00:02.599761+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:02.599679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:02.599679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:10.324225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014482+0000","last_change":"2024-03-21T13:00:03.644301+0000","last_active":"2024-03-21T13:00:11.014482+0000","last_peered":"2024-03-21T13:00:11.014482+0000","last_clean":"2024-03-21T13:00:11.014482+0000","last_became_active":"2024-03-21T12:58:41.168978+0000","last_became_peered":"2024-03-21T12:58:41.168978+0000","last_unstale":"2024-03-21T13:00:11.014482+0000","last_undegraded":"2024-03-21T13:00:11.014482+0000","last_fullsized":"2024-03-21T13:00:11.014482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:03.644209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:03.644209+0000","last_clean_scrub_stamp":"2024-03-21T13:00:03.644209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:18.843767+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015243+0000","last_change":"2024-03-21T12:59:44.661395+0000","last_active":"2024-03-21T13:00:11.015243+0000","last_peered":"2024-03-21T13:00:11.015243+0000","last_clean":"2024-03-21T13:00:11.015243+0000","last_became_active":"2024-03-21T12:58:37.733705+0000","last_became_peered":"2024-03-21T12:58:37.733705+0000","last_unstale":"2024-03-21T13:00:11.015243+0000","last_undegraded":"2024-03-21T13:00:11.015243+0000","last_fullsized":"2024-03-21T13:00:11.015243+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:44.661289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:44.661289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:45.014731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"71'9","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728617+0000","last_change":"2024-03-21T12:58:39.749391+0000","last_active":"2024-03-21T13:00:10.728617+0000","last_peered":"2024-03-21T13:00:10.728617+0000","last_clean":"2024-03-21T13:00:10.728617+0000","last_became_active":"2024-03-21T12:58:39.749010+0000","last_became_peered":"2024-03-21T12:58:39.749010+0000","last_unstale":"2024-03-21T13:00:10.728617+0000","last_undegraded":"2024-03-21T13:00:10.728617+0000","last_fullsized":"2024-03-21T13:00:10.728617+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014998+0000","last_change":"2024-03-21T13:00:02.683946+0000","last_active":"2024-03-21T13:00:11.014998+0000","last_peered":"2024-03-21T13:00:11.014998+0000","last_clean":"2024-03-21T13:00:11.014998+0000","last_became_active":"2024-03-21T12:58:43.181494+0000","last_became_peered":"2024-03-21T12:58:43.181494+0000","last_unstale":"2024-03-21T13:00:11.014998+0000","last_undegraded":"2024-03-21T13:00:11.014998+0000","last_fullsized":"2024-03-21T13:00:11.014998+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:02.683841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:02.683841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:10.243871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":59,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014096+0000","last_change":"2024-03-21T12:59:41.158247+0000","last_active":"2024-03-21T13:00:11.014096+0000","last_peered":"2024-03-21T13:00:11.014096+0000","last_clean":"2024-03-21T13:00:11.014096+0000","last_became_active":"2024-03-21T12:58:41.163517+0000","last_became_peered":"2024-03-21T12:58:41.163517+0000","last_unstale":"2024-03-21T13:00:11.014096+0000","last_undegraded":"2024-03-21T13:00:11.014096+0000","last_fullsized":"2024-03-21T13:00:11.014096+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:41.158105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:41.158105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:45.036271+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014761+0000","last_change":"2024-03-21T12:59:59.594026+0000","last_active":"2024-03-21T13:00:11.014761+0000","last_peered":"2024-03-21T13:00:11.014761+0000","last_clean":"2024-03-21T13:00:11.014761+0000","last_became_active":"2024-03-21T12:58:37.731058+0000","last_became_peered":"2024-03-21T12:58:37.731058+0000","last_unstale":"2024-03-21T13:00:11.014761+0000","last_undegraded":"2024-03-21T13:00:11.014761+0000","last_fullsized":"2024-03-21T13:00:11.014761+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:59.593927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.593927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:14.344496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":76,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729004+0000","last_change":"2024-03-21T12:59:53.952540+0000","last_active":"2024-03-21T13:00:10.729004+0000","last_peered":"2024-03-21T13:00:10.729004+0000","last_clean":"2024-03-21T13:00:10.729004+0000","last_became_active":"2024-03-21T12:58:39.749211+0000","last_became_peered":"2024-03-21T12:58:39.749211+0000","last_unstale":"2024-03-21T13:00:10.729004+0000","last_undegraded":"2024-03-21T13:00:10.729004+0000","last_fullsized":"2024-03-21T13:00:10.729004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T12:59:53.952401+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.263330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729179+0000","last_change":"2024-03-21T12:59:55.899438+0000","last_active":"2024-03-21T13:00:10.729179+0000","last_peered":"2024-03-21T13:00:10.729179+0000","last_clean":"2024-03-21T13:00:10.729179+0000","last_became_active":"2024-03-21T12:58:43.351768+0000","last_became_peered":"2024-03-21T12:58:43.351768+0000","last_unstale":"2024-03-21T13:00:10.729179+0000","last_undegraded":"2024-03-21T13:00:10.729179+0000","last_fullsized":"2024-03-21T13:00:10.729179+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:55.899344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.899344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:59.849673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.837061+0000","last_change":"2024-03-21T12:59:55.014569+0000","last_active":"2024-03-21T13:00:10.837061+0000","last_peered":"2024-03-21T13:00:10.837061+0000","last_clean":"2024-03-21T13:00:10.837061+0000","last_became_active":"2024-03-21T12:58:41.165783+0000","last_became_peered":"2024-03-21T12:58:41.165783+0000","last_unstale":"2024-03-21T13:00:10.837061+0000","last_undegraded":"2024-03-21T13:00:10.837061+0000","last_fullsized":"2024-03-21T13:00:10.837061+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.014453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:10.470418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339649+0000","last_change":"2024-03-21T12:58:37.733421+0000","last_active":"2024-03-21T12:59:53.339649+0000","last_peered":"2024-03-21T12:59:53.339649+0000","last_clean":"2024-03-21T12:59:53.339649+0000","last_became_active":"2024-03-21T12:58:37.733064+0000","last_became_peered":"2024-03-21T12:58:37.733064+0000","last_unstale":"2024-03-21T12:59:53.339649+0000","last_undegraded":"2024-03-21T12:59:53.339649+0000","last_fullsized":"2024-03-21T12:59:53.339649+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:54.238132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":88,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:55.664722+0000","last_change":"2024-03-21T12:59:55.664752+0000","last_active":"2024-03-21T12:59:55.664722+0000","last_peered":"2024-03-21T12:59:55.664722+0000","last_clean":"2024-03-21T12:59:55.664722+0000","last_became_active":"2024-03-21T12:58:39.749130+0000","last_became_peered":"2024-03-21T12:58:39.749130+0000","last_unstale":"2024-03-21T12:59:55.664722+0000","last_undegraded":"2024-03-21T12:59:55.664722+0000","last_fullsized":"2024-03-21T12:59:55.664722+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T12:59:55.664649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.664649+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:11.693713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.782004+0000","last_change":"2024-03-21T12:58:43.352883+0000","last_active":"2024-03-21T13:00:10.782004+0000","last_peered":"2024-03-21T13:00:10.782004+0000","last_clean":"2024-03-21T13:00:10.782004+0000","last_became_active":"2024-03-21T12:58:43.352510+0000","last_became_peered":"2024-03-21T12:58:43.352510+0000","last_unstale":"2024-03-21T13:00:10.782004+0000","last_undegraded":"2024-03-21T13:00:10.782004+0000","last_fullsized":"2024-03-21T13:00:10.782004+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728820+0000","last_change":"2024-03-21T12:59:50.904504+0000","last_active":"2024-03-21T13:00:10.728820+0000","last_peered":"2024-03-21T13:00:10.728820+0000","last_clean":"2024-03-21T13:00:10.728820+0000","last_became_active":"2024-03-21T12:58:41.167961+0000","last_became_peered":"2024-03-21T12:58:41.167961+0000","last_unstale":"2024-03-21T13:00:10.728820+0000","last_undegraded":"2024-03-21T13:00:10.728820+0000","last_fullsized":"2024-03-21T13:00:10.728820+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.904419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.904419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.905923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728638+0000","last_change":"2024-03-21T12:59:54.948197+0000","last_active":"2024-03-21T13:00:10.728638+0000","last_peered":"2024-03-21T13:00:10.728638+0000","last_clean":"2024-03-21T13:00:10.728638+0000","last_became_active":"2024-03-21T12:58:37.732355+0000","last_became_peered":"2024-03-21T12:58:37.732355+0000","last_unstale":"2024-03-21T13:00:10.728638+0000","last_undegraded":"2024-03-21T13:00:10.728638+0000","last_fullsized":"2024-03-21T13:00:10.728638+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_clean_scrub_stamp":"2024-03-21T12:59:54.948081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:03.565967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'12","reported_seq":97,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781778+0000","last_change":"2024-03-21T12:59:01.168089+0000","last_active":"2024-03-21T13:00:10.781778+0000","last_peered":"2024-03-21T13:00:10.781778+0000","last_clean":"2024-03-21T13:00:10.781778+0000","last_became_active":"2024-03-21T12:59:01.161520+0000","last_became_peered":"2024-03-21T12:59:01.161520+0000","last_unstale":"2024-03-21T13:00:10.781778+0000","last_undegraded":"2024-03-21T13:00:10.781778+0000","last_fullsized":"2024-03-21T13:00:10.781778+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014256+0000","last_change":"2024-03-21T13:00:00.168621+0000","last_active":"2024-03-21T13:00:11.014256+0000","last_peered":"2024-03-21T13:00:11.014256+0000","last_clean":"2024-03-21T13:00:11.014256+0000","last_became_active":"2024-03-21T12:58:43.184473+0000","last_became_peered":"2024-03-21T12:58:43.184473+0000","last_unstale":"2024-03-21T13:00:11.014256+0000","last_undegraded":"2024-03-21T13:00:11.014256+0000","last_fullsized":"2024-03-21T13:00:11.014256+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:00:00.168509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:00.168509+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.057928+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.784305+0000","last_change":"2024-03-21T12:59:55.277279+0000","last_active":"2024-03-21T13:00:10.784305+0000","last_peered":"2024-03-21T13:00:10.784305+0000","last_clean":"2024-03-21T13:00:10.784305+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:00:10.784305+0000","last_undegraded":"2024-03-21T13:00:10.784305+0000","last_fullsized":"2024-03-21T13:00:10.784305+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:55.277165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.277165+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:19.101757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":69,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.838018+0000","last_change":"2024-03-21T13:00:10.838057+0000","last_active":"2024-03-21T13:00:10.838018+0000","last_peered":"2024-03-21T13:00:10.838018+0000","last_clean":"2024-03-21T13:00:10.838018+0000","last_became_active":"2024-03-21T12:58:37.736455+0000","last_became_peered":"2024-03-21T12:58:37.736455+0000","last_unstale":"2024-03-21T13:00:10.838018+0000","last_undegraded":"2024-03-21T13:00:10.838018+0000","last_fullsized":"2024-03-21T13:00:10.838018+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:00:10.837949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.837949+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:29.483399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"71'10","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339627+0000","last_change":"2024-03-21T12:59:50.414098+0000","last_active":"2024-03-21T12:59:53.339627+0000","last_peered":"2024-03-21T12:59:53.339627+0000","last_clean":"2024-03-21T12:59:53.339627+0000","last_became_active":"2024-03-21T12:58:39.754548+0000","last_became_peered":"2024-03-21T12:58:39.754548+0000","last_unstale":"2024-03-21T12:59:53.339627+0000","last_undegraded":"2024-03-21T12:59:53.339627+0000","last_fullsized":"2024-03-21T12:59:53.339627+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T12:59:50.413983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.413983+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.575091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836853+0000","last_change":"2024-03-21T12:58:43.186597+0000","last_active":"2024-03-21T13:00:10.836853+0000","last_peered":"2024-03-21T13:00:10.836853+0000","last_clean":"2024-03-21T13:00:10.836853+0000","last_became_active":"2024-03-21T12:58:43.186289+0000","last_became_peered":"2024-03-21T12:58:43.186289+0000","last_unstale":"2024-03-21T13:00:10.836853+0000","last_undegraded":"2024-03-21T13:00:10.836853+0000","last_fullsized":"2024-03-21T13:00:10.836853+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339724+0000","last_change":"2024-03-21T12:58:41.167760+0000","last_active":"2024-03-21T12:59:53.339724+0000","last_peered":"2024-03-21T12:59:53.339724+0000","last_clean":"2024-03-21T12:59:53.339724+0000","last_became_active":"2024-03-21T12:58:41.167443+0000","last_became_peered":"2024-03-21T12:58:41.167443+0000","last_unstale":"2024-03-21T12:59:53.339724+0000","last_undegraded":"2024-03-21T12:59:53.339724+0000","last_fullsized":"2024-03-21T12:59:53.339724+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:03.895693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781277+0000","last_change":"2024-03-21T12:59:49.178866+0000","last_active":"2024-03-21T13:00:10.781277+0000","last_peered":"2024-03-21T13:00:10.781277+0000","last_clean":"2024-03-21T13:00:10.781277+0000","last_became_active":"2024-03-21T12:58:37.728914+0000","last_became_peered":"2024-03-21T12:58:37.728914+0000","last_unstale":"2024-03-21T13:00:10.781277+0000","last_undegraded":"2024-03-21T13:00:10.781277+0000","last_fullsized":"2024-03-21T13:00:10.781277+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:49.178785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:49.178785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.644002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'11","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.784280+0000","last_change":"2024-03-21T13:00:01.396865+0000","last_active":"2024-03-21T13:00:10.784280+0000","last_peered":"2024-03-21T13:00:10.784280+0000","last_clean":"2024-03-21T13:00:10.784280+0000","last_became_active":"2024-03-21T12:58:39.755294+0000","last_became_peered":"2024-03-21T12:58:39.755294+0000","last_unstale":"2024-03-21T13:00:10.784280+0000","last_undegraded":"2024-03-21T13:00:10.784280+0000","last_fullsized":"2024-03-21T13:00:10.784280+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:00:01.396749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.396749+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.505311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.013984+0000","last_change":"2024-03-21T12:59:50.209251+0000","last_active":"2024-03-21T13:00:11.013984+0000","last_peered":"2024-03-21T13:00:11.013984+0000","last_clean":"2024-03-21T13:00:11.013984+0000","last_became_active":"2024-03-21T12:58:43.183475+0000","last_became_peered":"2024-03-21T12:58:43.183475+0000","last_unstale":"2024-03-21T13:00:11.013984+0000","last_undegraded":"2024-03-21T13:00:11.013984+0000","last_fullsized":"2024-03-21T13:00:11.013984+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.209152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.209152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:18.566644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.783594+0000","last_change":"2024-03-21T13:00:03.421382+0000","last_active":"2024-03-21T13:00:10.783594+0000","last_peered":"2024-03-21T13:00:10.783594+0000","last_clean":"2024-03-21T13:00:10.783594+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:00:10.783594+0000","last_undegraded":"2024-03-21T13:00:10.783594+0000","last_fullsized":"2024-03-21T13:00:10.783594+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:03.421305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:03.421305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.521044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339693+0000","last_change":"2024-03-21T12:58:37.732416+0000","last_active":"2024-03-21T12:59:53.339693+0000","last_peered":"2024-03-21T12:59:53.339693+0000","last_clean":"2024-03-21T12:59:53.339693+0000","last_became_active":"2024-03-21T12:58:37.732144+0000","last_became_peered":"2024-03-21T12:58:37.732144+0000","last_unstale":"2024-03-21T12:59:53.339693+0000","last_undegraded":"2024-03-21T12:59:53.339693+0000","last_fullsized":"2024-03-21T12:59:53.339693+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:05.140648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781352+0000","last_change":"2024-03-21T12:58:39.755658+0000","last_active":"2024-03-21T13:00:10.781352+0000","last_peered":"2024-03-21T13:00:10.781352+0000","last_clean":"2024-03-21T13:00:10.781352+0000","last_became_active":"2024-03-21T12:58:39.755336+0000","last_became_peered":"2024-03-21T12:58:39.755336+0000","last_unstale":"2024-03-21T13:00:10.781352+0000","last_undegraded":"2024-03-21T13:00:10.781352+0000","last_fullsized":"2024-03-21T13:00:10.781352+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781472+0000","last_change":"2024-03-21T13:00:10.110413+0000","last_active":"2024-03-21T13:00:10.781472+0000","last_peered":"2024-03-21T13:00:10.781472+0000","last_clean":"2024-03-21T13:00:10.781472+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:00:10.781472+0000","last_undegraded":"2024-03-21T13:00:10.781472+0000","last_fullsized":"2024-03-21T13:00:10.781472+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:10.110335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.110335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:30.400061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014143+0000","last_change":"2024-03-21T12:58:41.165123+0000","last_active":"2024-03-21T13:00:11.014143+0000","last_peered":"2024-03-21T13:00:11.014143+0000","last_clean":"2024-03-21T13:00:11.014143+0000","last_became_active":"2024-03-21T12:58:41.164784+0000","last_became_peered":"2024-03-21T12:58:41.164784+0000","last_unstale":"2024-03-21T13:00:11.014143+0000","last_undegraded":"2024-03-21T13:00:11.014143+0000","last_fullsized":"2024-03-21T13:00:11.014143+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014691+0000","last_change":"2024-03-21T13:00:09.658052+0000","last_active":"2024-03-21T13:00:11.014691+0000","last_peered":"2024-03-21T13:00:11.014691+0000","last_clean":"2024-03-21T13:00:11.014691+0000","last_became_active":"2024-03-21T12:58:37.736127+0000","last_became_peered":"2024-03-21T12:58:37.736127+0000","last_unstale":"2024-03-21T13:00:11.014691+0000","last_undegraded":"2024-03-21T13:00:11.014691+0000","last_fullsized":"2024-03-21T13:00:11.014691+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:09.657971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:09.657971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:30.873328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"71'11","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781642+0000","last_change":"2024-03-21T12:59:52.181828+0000","last_active":"2024-03-21T13:00:10.781642+0000","last_peered":"2024-03-21T13:00:10.781642+0000","last_clean":"2024-03-21T13:00:10.781642+0000","last_became_active":"2024-03-21T12:58:39.753014+0000","last_became_peered":"2024-03-21T12:58:39.753014+0000","last_unstale":"2024-03-21T13:00:10.781642+0000","last_undegraded":"2024-03-21T13:00:10.781642+0000","last_fullsized":"2024-03-21T13:00:10.781642+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T12:59:52.181697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.181697+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.248453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339098+0000","last_change":"2024-03-21T12:58:43.178915+0000","last_active":"2024-03-21T12:59:53.339098+0000","last_peered":"2024-03-21T12:59:53.339098+0000","last_clean":"2024-03-21T12:59:53.339098+0000","last_became_active":"2024-03-21T12:58:43.178589+0000","last_became_peered":"2024-03-21T12:58:43.178589+0000","last_unstale":"2024-03-21T12:59:53.339098+0000","last_undegraded":"2024-03-21T12:59:53.339098+0000","last_fullsized":"2024-03-21T12:59:53.339098+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:07.484568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781725+0000","last_change":"2024-03-21T12:58:41.167969+0000","last_active":"2024-03-21T13:00:10.781725+0000","last_peered":"2024-03-21T13:00:10.781725+0000","last_clean":"2024-03-21T13:00:10.781725+0000","last_became_active":"2024-03-21T12:58:41.167371+0000","last_became_peered":"2024-03-21T12:58:41.167371+0000","last_unstale":"2024-03-21T13:00:10.781725+0000","last_undegraded":"2024-03-21T13:00:10.781725+0000","last_fullsized":"2024-03-21T13:00:10.781725+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":59,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014850+0000","last_change":"2024-03-21T12:58:37.733949+0000","last_active":"2024-03-21T13:00:11.014850+0000","last_peered":"2024-03-21T13:00:11.014850+0000","last_clean":"2024-03-21T13:00:11.014850+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:00:11.014850+0000","last_undegraded":"2024-03-21T13:00:11.014850+0000","last_fullsized":"2024-03-21T13:00:11.014850+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"86'13","reported_seq":84,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836777+0000","last_change":"2024-03-21T12:59:48.085870+0000","last_active":"2024-03-21T13:00:10.836777+0000","last_peered":"2024-03-21T13:00:10.836777+0000","last_clean":"2024-03-21T13:00:10.836777+0000","last_became_active":"2024-03-21T12:58:39.753224+0000","last_became_peered":"2024-03-21T12:58:39.753224+0000","last_unstale":"2024-03-21T13:00:10.836777+0000","last_undegraded":"2024-03-21T13:00:10.836777+0000","last_fullsized":"2024-03-21T13:00:10.836777+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.085774+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:53.042862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781350+0000","last_change":"2024-03-21T12:58:43.184148+0000","last_active":"2024-03-21T13:00:10.781350+0000","last_peered":"2024-03-21T13:00:10.781350+0000","last_clean":"2024-03-21T13:00:10.781350+0000","last_became_active":"2024-03-21T12:58:43.183712+0000","last_became_peered":"2024-03-21T12:58:43.183712+0000","last_unstale":"2024-03-21T13:00:10.781350+0000","last_undegraded":"2024-03-21T13:00:10.781350+0000","last_fullsized":"2024-03-21T13:00:10.781350+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:12.054567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836713+0000","last_change":"2024-03-21T12:59:45.088950+0000","last_active":"2024-03-21T13:00:10.836713+0000","last_peered":"2024-03-21T13:00:10.836713+0000","last_clean":"2024-03-21T13:00:10.836713+0000","last_became_active":"2024-03-21T12:58:41.164463+0000","last_became_peered":"2024-03-21T12:58:41.164463+0000","last_unstale":"2024-03-21T13:00:10.836713+0000","last_undegraded":"2024-03-21T13:00:10.836713+0000","last_fullsized":"2024-03-21T13:00:10.836713+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:45.088861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.088861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:11.118145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728925+0000","last_change":"2024-03-21T12:59:45.833435+0000","last_active":"2024-03-21T13:00:10.728925+0000","last_peered":"2024-03-21T13:00:10.728925+0000","last_clean":"2024-03-21T13:00:10.728925+0000","last_became_active":"2024-03-21T12:58:37.731659+0000","last_became_peered":"2024-03-21T12:58:37.731659+0000","last_unstale":"2024-03-21T13:00:10.728925+0000","last_undegraded":"2024-03-21T13:00:10.728925+0000","last_fullsized":"2024-03-21T13:00:10.728925+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.833304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:49.946967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":75,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728879+0000","last_change":"2024-03-21T12:58:39.746514+0000","last_active":"2024-03-21T13:00:10.728879+0000","last_peered":"2024-03-21T13:00:10.728879+0000","last_clean":"2024-03-21T13:00:10.728879+0000","last_became_active":"2024-03-21T12:58:39.746182+0000","last_became_peered":"2024-03-21T12:58:39.746182+0000","last_unstale":"2024-03-21T13:00:10.728879+0000","last_undegraded":"2024-03-21T13:00:10.728879+0000","last_fullsized":"2024-03-21T13:00:10.728879+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339384+0000","last_change":"2024-03-21T12:58:43.181024+0000","last_active":"2024-03-21T12:59:53.339384+0000","last_peered":"2024-03-21T12:59:53.339384+0000","last_clean":"2024-03-21T12:59:53.339384+0000","last_became_active":"2024-03-21T12:58:43.180773+0000","last_became_peered":"2024-03-21T12:58:43.180773+0000","last_unstale":"2024-03-21T12:59:53.339384+0000","last_undegraded":"2024-03-21T12:59:53.339384+0000","last_fullsized":"2024-03-21T12:59:53.339384+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:07.203930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015405+0000","last_change":"2024-03-21T12:59:50.724334+0000","last_active":"2024-03-21T13:00:11.015405+0000","last_peered":"2024-03-21T13:00:11.015405+0000","last_clean":"2024-03-21T13:00:11.015405+0000","last_became_active":"2024-03-21T12:58:41.167956+0000","last_became_peered":"2024-03-21T12:58:41.167956+0000","last_unstale":"2024-03-21T13:00:11.015405+0000","last_undegraded":"2024-03-21T13:00:11.015405+0000","last_fullsized":"2024-03-21T13:00:11.015405+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:50.724242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.724242+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:53.691391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:14.161211+0000","last_change":"2024-03-21T13:00:14.161251+0000","last_active":"2024-03-21T13:00:14.161211+0000","last_peered":"2024-03-21T13:00:14.161211+0000","last_clean":"2024-03-21T13:00:14.161211+0000","last_became_active":"2024-03-21T12:58:37.731988+0000","last_became_peered":"2024-03-21T12:58:37.731988+0000","last_unstale":"2024-03-21T13:00:14.161211+0000","last_undegraded":"2024-03-21T13:00:14.161211+0000","last_fullsized":"2024-03-21T13:00:14.161211+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:14.161136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:14.161136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:26.647081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"71'10","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781902+0000","last_change":"2024-03-21T13:00:06.198017+0000","last_active":"2024-03-21T13:00:10.781902+0000","last_peered":"2024-03-21T13:00:10.781902+0000","last_clean":"2024-03-21T13:00:10.781902+0000","last_became_active":"2024-03-21T12:58:39.757806+0000","last_became_peered":"2024-03-21T12:58:39.757806+0000","last_unstale":"2024-03-21T13:00:10.781902+0000","last_undegraded":"2024-03-21T13:00:10.781902+0000","last_fullsized":"2024-03-21T13:00:10.781902+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:00:06.197903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.197903+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:24.909636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729358+0000","last_change":"2024-03-21T12:58:43.186611+0000","last_active":"2024-03-21T13:00:10.729358+0000","last_peered":"2024-03-21T13:00:10.729358+0000","last_clean":"2024-03-21T13:00:10.729358+0000","last_became_active":"2024-03-21T12:58:43.186155+0000","last_became_peered":"2024-03-21T12:58:43.186155+0000","last_unstale":"2024-03-21T13:00:10.729358+0000","last_undegraded":"2024-03-21T13:00:10.729358+0000","last_fullsized":"2024-03-21T13:00:10.729358+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.782088+0000","last_change":"2024-03-21T12:58:41.168116+0000","last_active":"2024-03-21T13:00:10.782088+0000","last_peered":"2024-03-21T13:00:10.782088+0000","last_clean":"2024-03-21T13:00:10.782088+0000","last_became_active":"2024-03-21T12:58:41.167826+0000","last_became_peered":"2024-03-21T12:58:41.167826+0000","last_unstale":"2024-03-21T13:00:10.782088+0000","last_undegraded":"2024-03-21T13:00:10.782088+0000","last_fullsized":"2024-03-21T13:00:10.782088+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339299+0000","last_change":"2024-03-21T12:58:37.732471+0000","last_active":"2024-03-21T12:59:53.339299+0000","last_peered":"2024-03-21T12:59:53.339299+0000","last_clean":"2024-03-21T12:59:53.339299+0000","last_became_active":"2024-03-21T12:58:37.732166+0000","last_became_peered":"2024-03-21T12:58:37.732166+0000","last_unstale":"2024-03-21T12:59:53.339299+0000","last_undegraded":"2024-03-21T12:59:53.339299+0000","last_fullsized":"2024-03-21T12:59:53.339299+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:06.434461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'9","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015482+0000","last_change":"2024-03-21T12:58:39.751200+0000","last_active":"2024-03-21T13:00:11.015482+0000","last_peered":"2024-03-21T13:00:11.015482+0000","last_clean":"2024-03-21T13:00:11.015482+0000","last_became_active":"2024-03-21T12:58:39.750820+0000","last_became_peered":"2024-03-21T12:58:39.750820+0000","last_unstale":"2024-03-21T13:00:11.015482+0000","last_undegraded":"2024-03-21T13:00:11.015482+0000","last_fullsized":"2024-03-21T13:00:11.015482+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.317216+0000","last_change":"2024-03-21T12:59:51.730551+0000","last_active":"2024-03-21T12:59:53.317216+0000","last_peered":"2024-03-21T12:59:53.317216+0000","last_clean":"2024-03-21T12:59:53.317216+0000","last_became_active":"2024-03-21T12:58:43.182126+0000","last_became_peered":"2024-03-21T12:58:43.182126+0000","last_unstale":"2024-03-21T12:59:53.317216+0000","last_undegraded":"2024-03-21T12:59:53.317216+0000","last_fullsized":"2024-03-21T12:59:53.317216+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T12:59:51.730413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.791726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:12.122039+0000","last_change":"2024-03-21T13:00:12.122078+0000","last_active":"2024-03-21T13:00:12.122039+0000","last_peered":"2024-03-21T13:00:12.122039+0000","last_clean":"2024-03-21T13:00:12.122039+0000","last_became_active":"2024-03-21T12:58:41.172904+0000","last_became_peered":"2024-03-21T12:58:41.172904+0000","last_unstale":"2024-03-21T13:00:12.122039+0000","last_undegraded":"2024-03-21T13:00:12.122039+0000","last_fullsized":"2024-03-21T13:00:12.122039+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:12.121970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:12.121970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:31.573284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014672+0000","last_change":"2024-03-21T12:59:47.203592+0000","last_active":"2024-03-21T13:00:11.014672+0000","last_peered":"2024-03-21T13:00:11.014672+0000","last_clean":"2024-03-21T13:00:11.014672+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:00:11.014672+0000","last_undegraded":"2024-03-21T13:00:11.014672+0000","last_fullsized":"2024-03-21T13:00:11.014672+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T12:59:47.203453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.203453+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.098187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'9","reported_seq":77,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:06.685145+0000","last_change":"2024-03-21T13:00:06.685183+0000","last_active":"2024-03-21T13:00:06.685145+0000","last_peered":"2024-03-21T13:00:06.685145+0000","last_clean":"2024-03-21T13:00:06.685145+0000","last_became_active":"2024-03-21T12:58:39.752722+0000","last_became_peered":"2024-03-21T12:58:39.752722+0000","last_unstale":"2024-03-21T13:00:06.685145+0000","last_undegraded":"2024-03-21T13:00:06.685145+0000","last_fullsized":"2024-03-21T13:00:06.685145+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:00:06.685071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.685071+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:35.157380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014762+0000","last_change":"2024-03-21T13:00:08.197105+0000","last_active":"2024-03-21T13:00:11.014762+0000","last_peered":"2024-03-21T13:00:11.014762+0000","last_clean":"2024-03-21T13:00:11.014762+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:00:11.014762+0000","last_undegraded":"2024-03-21T13:00:11.014762+0000","last_fullsized":"2024-03-21T13:00:11.014762+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.196979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.196979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:33.771136+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014654+0000","last_change":"2024-03-21T12:58:41.162672+0000","last_active":"2024-03-21T13:00:11.014654+0000","last_peered":"2024-03-21T13:00:11.014654+0000","last_clean":"2024-03-21T13:00:11.014654+0000","last_became_active":"2024-03-21T12:58:41.162343+0000","last_became_peered":"2024-03-21T12:58:41.162343+0000","last_unstale":"2024-03-21T13:00:11.014654+0000","last_undegraded":"2024-03-21T13:00:11.014654+0000","last_fullsized":"2024-03-21T13:00:11.014654+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.783537+0000","last_change":"2024-03-21T12:59:52.319029+0000","last_active":"2024-03-21T13:00:10.783537+0000","last_peered":"2024-03-21T13:00:10.783537+0000","last_clean":"2024-03-21T13:00:10.783537+0000","last_became_active":"2024-03-21T12:58:37.731932+0000","last_became_peered":"2024-03-21T12:58:37.731932+0000","last_unstale":"2024-03-21T13:00:10.783537+0000","last_undegraded":"2024-03-21T13:00:10.783537+0000","last_fullsized":"2024-03-21T13:00:10.783537+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:52.318927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.318927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:20.356137+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'6","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.782202+0000","last_change":"2024-03-21T12:59:46.176940+0000","last_active":"2024-03-21T13:00:10.782202+0000","last_peered":"2024-03-21T13:00:10.782202+0000","last_clean":"2024-03-21T13:00:10.782202+0000","last_became_active":"2024-03-21T12:58:39.755419+0000","last_became_peered":"2024-03-21T12:58:39.755419+0000","last_unstale":"2024-03-21T13:00:10.782202+0000","last_undegraded":"2024-03-21T13:00:10.782202+0000","last_fullsized":"2024-03-21T13:00:10.782202+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T12:59:46.176819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.176819+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:06.815464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014477+0000","last_change":"2024-03-21T13:00:00.622196+0000","last_active":"2024-03-21T13:00:11.014477+0000","last_peered":"2024-03-21T13:00:11.014477+0000","last_clean":"2024-03-21T13:00:11.014477+0000","last_became_active":"2024-03-21T12:58:43.351103+0000","last_became_peered":"2024-03-21T12:58:43.351103+0000","last_unstale":"2024-03-21T13:00:11.014477+0000","last_undegraded":"2024-03-21T13:00:11.014477+0000","last_fullsized":"2024-03-21T13:00:11.014477+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:00.622094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:00.622094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:21.931294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015609+0000","last_change":"2024-03-21T13:00:06.188217+0000","last_active":"2024-03-21T13:00:11.015609+0000","last_peered":"2024-03-21T13:00:11.015609+0000","last_clean":"2024-03-21T13:00:11.015609+0000","last_became_active":"2024-03-21T12:58:37.735037+0000","last_became_peered":"2024-03-21T12:58:37.735037+0000","last_unstale":"2024-03-21T13:00:11.015609+0000","last_undegraded":"2024-03-21T13:00:11.015609+0000","last_fullsized":"2024-03-21T13:00:11.015609+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:00:06.188118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.188118+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.485685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'15","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014066+0000","last_change":"2024-03-21T12:58:39.750902+0000","last_active":"2024-03-21T13:00:11.014066+0000","last_peered":"2024-03-21T13:00:11.014066+0000","last_clean":"2024-03-21T13:00:11.014066+0000","last_became_active":"2024-03-21T12:58:39.750546+0000","last_became_peered":"2024-03-21T12:58:39.750546+0000","last_unstale":"2024-03-21T13:00:11.014066+0000","last_undegraded":"2024-03-21T13:00:11.014066+0000","last_fullsized":"2024-03-21T13:00:11.014066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"71'8","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014976+0000","last_change":"2024-03-21T12:58:41.166431+0000","last_active":"2024-03-21T13:00:11.014976+0000","last_peered":"2024-03-21T13:00:11.014976+0000","last_clean":"2024-03-21T13:00:11.014976+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:00:11.014976+0000","last_undegraded":"2024-03-21T13:00:11.014976+0000","last_fullsized":"2024-03-21T13:00:11.014976+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781606+0000","last_change":"2024-03-21T12:58:43.181756+0000","last_active":"2024-03-21T13:00:10.781606+0000","last_peered":"2024-03-21T13:00:10.781606+0000","last_clean":"2024-03-21T13:00:10.781606+0000","last_became_active":"2024-03-21T12:58:43.181036+0000","last_became_peered":"2024-03-21T12:58:43.181036+0000","last_unstale":"2024-03-21T13:00:10.781606+0000","last_undegraded":"2024-03-21T13:00:10.781606+0000","last_fullsized":"2024-03-21T13:00:10.781606+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015424+0000","last_change":"2024-03-21T12:59:59.136937+0000","last_active":"2024-03-21T13:00:11.015424+0000","last_peered":"2024-03-21T13:00:11.015424+0000","last_clean":"2024-03-21T13:00:11.015424+0000","last_became_active":"2024-03-21T12:58:37.737012+0000","last_became_peered":"2024-03-21T12:58:37.737012+0000","last_unstale":"2024-03-21T13:00:11.015424+0000","last_undegraded":"2024-03-21T13:00:11.015424+0000","last_fullsized":"2024-03-21T13:00:11.015424+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:59.136829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.136829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:03.237208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"71'12","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728637+0000","last_change":"2024-03-21T12:58:39.751648+0000","last_active":"2024-03-21T13:00:10.728637+0000","last_peered":"2024-03-21T13:00:10.728637+0000","last_clean":"2024-03-21T13:00:10.728637+0000","last_became_active":"2024-03-21T12:58:39.751236+0000","last_became_peered":"2024-03-21T12:58:39.751236+0000","last_unstale":"2024-03-21T13:00:10.728637+0000","last_undegraded":"2024-03-21T13:00:10.728637+0000","last_fullsized":"2024-03-21T13:00:10.728637+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:58.733332+0000","last_change":"2024-03-21T12:59:58.733374+0000","last_active":"2024-03-21T12:59:58.733332+0000","last_peered":"2024-03-21T12:59:58.733332+0000","last_clean":"2024-03-21T12:59:58.733332+0000","last_became_active":"2024-03-21T12:58:41.171443+0000","last_became_peered":"2024-03-21T12:58:41.171443+0000","last_unstale":"2024-03-21T12:59:58.733332+0000","last_undegraded":"2024-03-21T12:59:58.733332+0000","last_fullsized":"2024-03-21T12:59:58.733332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:58.733235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:58.733235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.428021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.013827+0000","last_change":"2024-03-21T13:00:04.200955+0000","last_active":"2024-03-21T13:00:11.013827+0000","last_peered":"2024-03-21T13:00:11.013827+0000","last_clean":"2024-03-21T13:00:11.013827+0000","last_became_active":"2024-03-21T12:58:43.179007+0000","last_became_peered":"2024-03-21T12:58:43.179007+0000","last_unstale":"2024-03-21T13:00:11.013827+0000","last_undegraded":"2024-03-21T13:00:11.013827+0000","last_fullsized":"2024-03-21T13:00:11.013827+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:04.200849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:04.200849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.029947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:08.625350+0000","last_change":"2024-03-21T13:00:08.625392+0000","last_active":"2024-03-21T13:00:08.625350+0000","last_peered":"2024-03-21T13:00:08.625350+0000","last_clean":"2024-03-21T13:00:08.625350+0000","last_became_active":"2024-03-21T12:58:41.165203+0000","last_became_peered":"2024-03-21T12:58:41.165203+0000","last_unstale":"2024-03-21T13:00:08.625350+0000","last_undegraded":"2024-03-21T13:00:08.625350+0000","last_fullsized":"2024-03-21T13:00:08.625350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.625230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.625230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:13.968472+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015067+0000","last_change":"2024-03-21T12:59:46.169468+0000","last_active":"2024-03-21T13:00:11.015067+0000","last_peered":"2024-03-21T13:00:11.015067+0000","last_clean":"2024-03-21T13:00:11.015067+0000","last_became_active":"2024-03-21T12:58:37.731448+0000","last_became_peered":"2024-03-21T12:58:37.731448+0000","last_unstale":"2024-03-21T13:00:11.015067+0000","last_undegraded":"2024-03-21T13:00:11.015067+0000","last_fullsized":"2024-03-21T13:00:11.015067+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:46.169315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.169315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:51.199764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"71'19","reported_seq":95,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.783798+0000","last_change":"2024-03-21T12:59:59.392987+0000","last_active":"2024-03-21T13:00:10.783798+0000","last_peered":"2024-03-21T13:00:10.783798+0000","last_clean":"2024-03-21T13:00:10.783798+0000","last_became_active":"2024-03-21T12:58:39.754761+0000","last_became_peered":"2024-03-21T12:58:39.754761+0000","last_unstale":"2024-03-21T13:00:10.783798+0000","last_undegraded":"2024-03-21T13:00:10.783798+0000","last_fullsized":"2024-03-21T13:00:10.783798+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T12:59:59.392834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.392834+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.114737+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339138+0000","last_change":"2024-03-21T12:58:43.180724+0000","last_active":"2024-03-21T12:59:53.339138+0000","last_peered":"2024-03-21T12:59:53.339138+0000","last_clean":"2024-03-21T12:59:53.339138+0000","last_became_active":"2024-03-21T12:58:43.180393+0000","last_became_peered":"2024-03-21T12:58:43.180393+0000","last_unstale":"2024-03-21T12:59:53.339138+0000","last_undegraded":"2024-03-21T12:59:53.339138+0000","last_fullsized":"2024-03-21T12:59:53.339138+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:02.348923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:01.667344+0000","last_change":"2024-03-21T13:00:01.667365+0000","last_active":"2024-03-21T13:00:01.667344+0000","last_peered":"2024-03-21T13:00:01.667344+0000","last_clean":"2024-03-21T13:00:01.667344+0000","last_became_active":"2024-03-21T12:58:41.163843+0000","last_became_peered":"2024-03-21T12:58:41.163843+0000","last_unstale":"2024-03-21T13:00:01.667344+0000","last_undegraded":"2024-03-21T13:00:01.667344+0000","last_fullsized":"2024-03-21T13:00:01.667344+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:01.667285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.667285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:31.061685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014826+0000","last_change":"2024-03-21T12:59:47.676292+0000","last_active":"2024-03-21T13:00:11.014826+0000","last_peered":"2024-03-21T13:00:11.014826+0000","last_clean":"2024-03-21T13:00:11.014826+0000","last_became_active":"2024-03-21T12:58:37.730518+0000","last_became_peered":"2024-03-21T12:58:37.730518+0000","last_unstale":"2024-03-21T13:00:11.014826+0000","last_undegraded":"2024-03-21T13:00:11.014826+0000","last_fullsized":"2024-03-21T13:00:11.014826+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:47.676198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.676198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.156465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"86'13","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339817+0000","last_change":"2024-03-21T12:59:46.421228+0000","last_active":"2024-03-21T12:59:53.339817+0000","last_peered":"2024-03-21T12:59:53.339817+0000","last_clean":"2024-03-21T12:59:53.339817+0000","last_became_active":"2024-03-21T12:58:39.748144+0000","last_became_peered":"2024-03-21T12:58:39.748144+0000","last_unstale":"2024-03-21T12:59:53.339817+0000","last_undegraded":"2024-03-21T12:59:53.339817+0000","last_fullsized":"2024-03-21T12:59:53.339817+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.421144+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:06.815237+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781924+0000","last_change":"2024-03-21T12:58:43.180710+0000","last_active":"2024-03-21T13:00:10.781924+0000","last_peered":"2024-03-21T13:00:10.781924+0000","last_clean":"2024-03-21T13:00:10.781924+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:00:10.781924+0000","last_undegraded":"2024-03-21T13:00:10.781924+0000","last_fullsized":"2024-03-21T13:00:10.781924+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:10.959635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"67'2","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014329+0000","last_change":"2024-03-21T12:59:55.541552+0000","last_active":"2024-03-21T13:00:11.014329+0000","last_peered":"2024-03-21T13:00:11.014329+0000","last_clean":"2024-03-21T13:00:11.014329+0000","last_became_active":"2024-03-21T12:58:36.724010+0000","last_became_peered":"2024-03-21T12:58:36.724010+0000","last_unstale":"2024-03-21T13:00:11.014329+0000","last_undegraded":"2024-03-21T13:00:11.014329+0000","last_fullsized":"2024-03-21T13:00:11.014329+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-21T12:59:55.541410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.541410+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:17.828687+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728683+0000","last_change":"2024-03-21T12:59:47.840935+0000","last_active":"2024-03-21T13:00:10.728683+0000","last_peered":"2024-03-21T13:00:10.728683+0000","last_clean":"2024-03-21T13:00:10.728683+0000","last_became_active":"2024-03-21T12:58:37.729454+0000","last_became_peered":"2024-03-21T12:58:37.729454+0000","last_unstale":"2024-03-21T13:00:10.728683+0000","last_undegraded":"2024-03-21T13:00:10.728683+0000","last_fullsized":"2024-03-21T13:00:10.728683+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:47.840831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.840831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.579290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"71'28","reported_seq":132,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836992+0000","last_change":"2024-03-21T13:00:01.917692+0000","last_active":"2024-03-21T13:00:10.836992+0000","last_peered":"2024-03-21T13:00:10.836992+0000","last_clean":"2024-03-21T13:00:10.836992+0000","last_became_active":"2024-03-21T12:58:39.756052+0000","last_became_peered":"2024-03-21T12:58:39.756052+0000","last_unstale":"2024-03-21T13:00:10.836992+0000","last_undegraded":"2024-03-21T13:00:10.836992+0000","last_fullsized":"2024-03-21T13:00:10.836992+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'28","last_scrub_stamp":"2024-03-21T13:00:01.917593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.917593+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:26.189859+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339496+0000","last_change":"2024-03-21T12:58:41.170763+0000","last_active":"2024-03-21T12:59:53.339496+0000","last_peered":"2024-03-21T12:59:53.339496+0000","last_clean":"2024-03-21T12:59:53.339496+0000","last_became_active":"2024-03-21T12:58:41.170460+0000","last_became_peered":"2024-03-21T12:58:41.170460+0000","last_unstale":"2024-03-21T12:59:53.339496+0000","last_undegraded":"2024-03-21T12:59:53.339496+0000","last_fullsized":"2024-03-21T12:59:53.339496+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:06.578225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015221+0000","last_change":"2024-03-21T12:59:54.191245+0000","last_active":"2024-03-21T13:00:11.015221+0000","last_peered":"2024-03-21T13:00:11.015221+0000","last_clean":"2024-03-21T13:00:11.015221+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:00:11.015221+0000","last_undegraded":"2024-03-21T13:00:11.015221+0000","last_fullsized":"2024-03-21T13:00:11.015221+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:54.191150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:54.191150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:15.445044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836570+0000","last_change":"2024-03-21T12:58:37.733226+0000","last_active":"2024-03-21T13:00:10.836570+0000","last_peered":"2024-03-21T13:00:10.836570+0000","last_clean":"2024-03-21T13:00:10.836570+0000","last_became_active":"2024-03-21T12:58:37.731386+0000","last_became_peered":"2024-03-21T12:58:37.731386+0000","last_unstale":"2024-03-21T13:00:10.836570+0000","last_undegraded":"2024-03-21T13:00:10.836570+0000","last_fullsized":"2024-03-21T13:00:10.836570+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:57.734474+0000","last_change":"2024-03-21T12:59:57.734511+0000","last_active":"2024-03-21T12:59:57.734474+0000","last_peered":"2024-03-21T12:59:57.734474+0000","last_clean":"2024-03-21T12:59:57.734474+0000","last_became_active":"2024-03-21T12:58:36.724067+0000","last_became_peered":"2024-03-21T12:58:36.724067+0000","last_unstale":"2024-03-21T12:59:57.734474+0000","last_undegraded":"2024-03-21T12:59:57.734474+0000","last_fullsized":"2024-03-21T12:59:57.734474+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.734379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.734379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.789374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000743678,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'13","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:13.796147+0000","last_change":"2024-03-21T13:00:13.796181+0000","last_active":"2024-03-21T13:00:13.796147+0000","last_peered":"2024-03-21T13:00:13.796147+0000","last_clean":"2024-03-21T13:00:13.796147+0000","last_became_active":"2024-03-21T12:58:39.750068+0000","last_became_peered":"2024-03-21T12:58:39.750068+0000","last_unstale":"2024-03-21T13:00:13.796147+0000","last_undegraded":"2024-03-21T13:00:13.796147+0000","last_fullsized":"2024-03-21T13:00:13.796147+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'13","last_scrub_stamp":"2024-03-21T13:00:13.796084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:13.796084+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.166815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:03.688376+0000","last_change":"2024-03-21T13:00:03.688398+0000","last_active":"2024-03-21T13:00:03.688376+0000","last_peered":"2024-03-21T13:00:03.688376+0000","last_clean":"2024-03-21T13:00:03.688376+0000","last_became_active":"2024-03-21T12:58:41.164725+0000","last_became_peered":"2024-03-21T12:58:41.164725+0000","last_unstale":"2024-03-21T13:00:03.688376+0000","last_undegraded":"2024-03-21T13:00:03.688376+0000","last_fullsized":"2024-03-21T13:00:03.688376+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:03.688311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:03.688311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:03.827045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014656+0000","last_change":"2024-03-21T12:59:52.658473+0000","last_active":"2024-03-21T13:00:11.014656+0000","last_peered":"2024-03-21T13:00:11.014656+0000","last_clean":"2024-03-21T13:00:11.014656+0000","last_became_active":"2024-03-21T12:58:43.183113+0000","last_became_peered":"2024-03-21T12:58:43.183113+0000","last_unstale":"2024-03-21T13:00:11.014656+0000","last_undegraded":"2024-03-21T13:00:11.014656+0000","last_fullsized":"2024-03-21T13:00:11.014656+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:52.658377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.658377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.565790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":316,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.013961+0000","last_change":"2024-03-21T12:59:02.091964+0000","last_active":"2024-03-21T13:00:11.013961+0000","last_peered":"2024-03-21T13:00:11.013961+0000","last_clean":"2024-03-21T13:00:11.013961+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:00:11.013961+0000","last_undegraded":"2024-03-21T13:00:11.013961+0000","last_fullsized":"2024-03-21T13:00:11.013961+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'119","last_scrub_stamp":"2024-03-21T12:59:02.091835+0000","last_deep_scrub":"76'119","last_deep_scrub_stamp":"2024-03-21T12:59:02.091835+0000","last_clean_scrub_stamp":"2024-03-21T12:59:02.091835+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781685+0000","last_change":"2024-03-21T12:58:37.729254+0000","last_active":"2024-03-21T13:00:10.781685+0000","last_peered":"2024-03-21T13:00:10.781685+0000","last_clean":"2024-03-21T13:00:10.781685+0000","last_became_active":"2024-03-21T12:58:37.728911+0000","last_became_peered":"2024-03-21T12:58:37.728911+0000","last_unstale":"2024-03-21T13:00:10.781685+0000","last_undegraded":"2024-03-21T13:00:10.781685+0000","last_fullsized":"2024-03-21T13:00:10.781685+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":106,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.784096+0000","last_change":"2024-03-21T12:59:56.298894+0000","last_active":"2024-03-21T13:00:10.784096+0000","last_peered":"2024-03-21T13:00:10.784096+0000","last_clean":"2024-03-21T13:00:10.784096+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:00:10.784096+0000","last_undegraded":"2024-03-21T13:00:10.784096+0000","last_fullsized":"2024-03-21T13:00:10.784096+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-21T12:59:56.298768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:56.298768+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.767366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":59,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.121578+0000","last_change":"2024-03-21T13:00:11.121619+0000","last_active":"2024-03-21T13:00:11.121578+0000","last_peered":"2024-03-21T13:00:11.121578+0000","last_clean":"2024-03-21T13:00:11.121578+0000","last_became_active":"2024-03-21T12:58:41.168003+0000","last_became_peered":"2024-03-21T12:58:41.168003+0000","last_unstale":"2024-03-21T13:00:11.121578+0000","last_undegraded":"2024-03-21T13:00:11.121578+0000","last_fullsized":"2024-03-21T13:00:11.121578+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:11.121495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:11.121495+0000","last_clean_scrub_stamp":"2024-03-21T13:00:11.121495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.671209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836582+0000","last_change":"2024-03-21T12:58:43.184189+0000","last_active":"2024-03-21T13:00:10.836582+0000","last_peered":"2024-03-21T13:00:10.836582+0000","last_clean":"2024-03-21T13:00:10.836582+0000","last_became_active":"2024-03-21T12:58:43.183860+0000","last_became_peered":"2024-03-21T12:58:43.183860+0000","last_unstale":"2024-03-21T13:00:10.836582+0000","last_undegraded":"2024-03-21T13:00:10.836582+0000","last_fullsized":"2024-03-21T13:00:10.836582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339323+0000","last_change":"2024-03-21T12:58:37.734459+0000","last_active":"2024-03-21T12:59:53.339323+0000","last_peered":"2024-03-21T12:59:53.339323+0000","last_clean":"2024-03-21T12:59:53.339323+0000","last_became_active":"2024-03-21T12:58:37.734176+0000","last_became_peered":"2024-03-21T12:58:37.734176+0000","last_unstale":"2024-03-21T12:59:53.339323+0000","last_undegraded":"2024-03-21T12:59:53.339323+0000","last_fullsized":"2024-03-21T12:59:53.339323+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:56.304166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":160,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:13.243336+0000","last_change":"2024-03-21T12:58:40.754485+0000","last_active":"2024-03-21T13:00:13.243336+0000","last_peered":"2024-03-21T13:00:13.243336+0000","last_clean":"2024-03-21T13:00:13.243336+0000","last_became_active":"2024-03-21T12:58:36.725735+0000","last_became_peered":"2024-03-21T12:58:36.725735+0000","last_unstale":"2024-03-21T13:00:13.243336+0000","last_undegraded":"2024-03-21T13:00:13.243336+0000","last_fullsized":"2024-03-21T13:00:13.243336+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:58:35.713454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:01.563906+0000","last_change":"2024-03-21T13:00:01.563940+0000","last_active":"2024-03-21T13:00:01.563906+0000","last_peered":"2024-03-21T13:00:01.563906+0000","last_clean":"2024-03-21T13:00:01.563906+0000","last_became_active":"2024-03-21T12:58:39.747315+0000","last_became_peered":"2024-03-21T12:58:39.747315+0000","last_unstale":"2024-03-21T13:00:01.563906+0000","last_undegraded":"2024-03-21T13:00:01.563906+0000","last_fullsized":"2024-03-21T13:00:01.563906+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:00:01.563844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.563844+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:12.072972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014942+0000","last_change":"2024-03-21T13:00:10.646160+0000","last_active":"2024-03-21T13:00:11.014942+0000","last_peered":"2024-03-21T13:00:11.014942+0000","last_clean":"2024-03-21T13:00:11.014942+0000","last_became_active":"2024-03-21T12:58:41.167131+0000","last_became_peered":"2024-03-21T12:58:41.167131+0000","last_unstale":"2024-03-21T13:00:11.014942+0000","last_undegraded":"2024-03-21T13:00:11.014942+0000","last_fullsized":"2024-03-21T13:00:11.014942+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:10.646083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.646083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.616835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.888250+0000","last_change":"2024-03-21T13:00:11.888272+0000","last_active":"2024-03-21T13:00:11.888250+0000","last_peered":"2024-03-21T13:00:11.888250+0000","last_clean":"2024-03-21T13:00:11.888250+0000","last_became_active":"2024-03-21T12:58:43.351138+0000","last_became_peered":"2024-03-21T12:58:43.351138+0000","last_unstale":"2024-03-21T13:00:11.888250+0000","last_undegraded":"2024-03-21T13:00:11.888250+0000","last_fullsized":"2024-03-21T13:00:11.888250+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:11.888191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:11.888191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.059106+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836809+0000","last_change":"2024-03-21T12:59:57.925311+0000","last_active":"2024-03-21T13:00:10.836809+0000","last_peered":"2024-03-21T13:00:10.836809+0000","last_clean":"2024-03-21T13:00:10.836809+0000","last_became_active":"2024-03-21T12:58:37.729818+0000","last_became_peered":"2024-03-21T12:58:37.729818+0000","last_unstale":"2024-03-21T13:00:10.836809+0000","last_undegraded":"2024-03-21T13:00:10.836809+0000","last_fullsized":"2024-03-21T13:00:10.836809+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.925179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.925179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.845352+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"71'19","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339719+0000","last_change":"2024-03-21T12:58:39.751068+0000","last_active":"2024-03-21T12:59:53.339719+0000","last_peered":"2024-03-21T12:59:53.339719+0000","last_clean":"2024-03-21T12:59:53.339719+0000","last_became_active":"2024-03-21T12:58:39.750216+0000","last_became_peered":"2024-03-21T12:58:39.750216+0000","last_unstale":"2024-03-21T12:59:53.339719+0000","last_undegraded":"2024-03-21T12:59:53.339719+0000","last_fullsized":"2024-03-21T12:59:53.339719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:57.408510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.783878+0000","last_change":"2024-03-21T13:00:06.485807+0000","last_active":"2024-03-21T13:00:10.783878+0000","last_peered":"2024-03-21T13:00:10.783878+0000","last_clean":"2024-03-21T13:00:10.783878+0000","last_became_active":"2024-03-21T12:58:41.166495+0000","last_became_peered":"2024-03-21T12:58:41.166495+0000","last_unstale":"2024-03-21T13:00:10.783878+0000","last_undegraded":"2024-03-21T13:00:10.783878+0000","last_fullsized":"2024-03-21T13:00:10.783878+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:06.485726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.485726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.711578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:05.703925+0000","last_change":"2024-03-21T13:00:05.703953+0000","last_active":"2024-03-21T13:00:05.703925+0000","last_peered":"2024-03-21T13:00:05.703925+0000","last_clean":"2024-03-21T13:00:05.703925+0000","last_became_active":"2024-03-21T12:58:43.177409+0000","last_became_peered":"2024-03-21T12:58:43.177409+0000","last_unstale":"2024-03-21T13:00:05.703925+0000","last_undegraded":"2024-03-21T13:00:05.703925+0000","last_fullsized":"2024-03-21T13:00:05.703925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:05.703839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:05.703839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:29.696086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015352+0000","last_change":"2024-03-21T12:58:37.730839+0000","last_active":"2024-03-21T13:00:11.015352+0000","last_peered":"2024-03-21T13:00:11.015352+0000","last_clean":"2024-03-21T13:00:11.015352+0000","last_became_active":"2024-03-21T12:58:37.730547+0000","last_became_peered":"2024-03-21T12:58:37.730547+0000","last_unstale":"2024-03-21T13:00:11.015352+0000","last_undegraded":"2024-03-21T13:00:11.015352+0000","last_fullsized":"2024-03-21T13:00:11.015352+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"86'15","reported_seq":60,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316813+0000","last_change":"2024-03-21T12:59:45.653329+0000","last_active":"2024-03-21T12:59:53.316813+0000","last_peered":"2024-03-21T12:59:53.316813+0000","last_clean":"2024-03-21T12:59:53.316813+0000","last_became_active":"2024-03-21T12:59:01.160456+0000","last_became_peered":"2024-03-21T12:59:01.160456+0000","last_unstale":"2024-03-21T12:59:53.316813+0000","last_undegraded":"2024-03-21T12:59:53.316813+0000","last_fullsized":"2024-03-21T12:59:53.316813+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.653197+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.205091+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":84,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:02.788621+0000","last_change":"2024-03-21T12:59:48.409996+0000","last_active":"2024-03-21T13:00:02.788621+0000","last_peered":"2024-03-21T13:00:02.788621+0000","last_clean":"2024-03-21T13:00:02.788621+0000","last_became_active":"2024-03-21T12:58:41.171770+0000","last_became_peered":"2024-03-21T12:58:41.171770+0000","last_unstale":"2024-03-21T13:00:02.788621+0000","last_undegraded":"2024-03-21T13:00:02.788621+0000","last_fullsized":"2024-03-21T13:00:02.788621+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:48.409905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.409905+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.077210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014648+0000","last_change":"2024-03-21T12:58:43.353379+0000","last_active":"2024-03-21T13:00:11.014648+0000","last_peered":"2024-03-21T13:00:11.014648+0000","last_clean":"2024-03-21T13:00:11.014648+0000","last_became_active":"2024-03-21T12:58:43.353049+0000","last_became_peered":"2024-03-21T12:58:43.353049+0000","last_unstale":"2024-03-21T13:00:11.014648+0000","last_undegraded":"2024-03-21T13:00:11.014648+0000","last_fullsized":"2024-03-21T13:00:11.014648+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.339519+0000","last_change":"2024-03-21T12:58:37.735715+0000","last_active":"2024-03-21T12:59:53.339519+0000","last_peered":"2024-03-21T12:59:53.339519+0000","last_clean":"2024-03-21T12:59:53.339519+0000","last_became_active":"2024-03-21T12:58:37.735421+0000","last_became_peered":"2024-03-21T12:58:37.735421+0000","last_unstale":"2024-03-21T12:59:53.339519+0000","last_undegraded":"2024-03-21T12:59:53.339519+0000","last_fullsized":"2024-03-21T12:59:53.339519+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:04.750303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"87'21","reported_seq":92,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728506+0000","last_change":"2024-03-21T12:59:52.916371+0000","last_active":"2024-03-21T13:00:10.728506+0000","last_peered":"2024-03-21T13:00:10.728506+0000","last_clean":"2024-03-21T13:00:10.728506+0000","last_became_active":"2024-03-21T12:58:39.743592+0000","last_became_peered":"2024-03-21T12:58:39.743592+0000","last_unstale":"2024-03-21T13:00:10.728506+0000","last_undegraded":"2024-03-21T13:00:10.728506+0000","last_fullsized":"2024-03-21T13:00:10.728506+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'21","last_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.916260+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.636952+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"71'8","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781529+0000","last_change":"2024-03-21T12:58:41.174471+0000","last_active":"2024-03-21T13:00:10.781529+0000","last_peered":"2024-03-21T13:00:10.781529+0000","last_clean":"2024-03-21T13:00:10.781529+0000","last_became_active":"2024-03-21T12:58:41.174137+0000","last_became_peered":"2024-03-21T12:58:41.174137+0000","last_unstale":"2024-03-21T13:00:10.781529+0000","last_undegraded":"2024-03-21T13:00:10.781529+0000","last_fullsized":"2024-03-21T13:00:10.781529+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728509+0000","last_change":"2024-03-21T13:00:08.823941+0000","last_active":"2024-03-21T13:00:10.728509+0000","last_peered":"2024-03-21T13:00:10.728509+0000","last_clean":"2024-03-21T13:00:10.728509+0000","last_became_active":"2024-03-21T12:58:43.180939+0000","last_became_peered":"2024-03-21T12:58:43.180939+0000","last_unstale":"2024-03-21T13:00:10.728509+0000","last_undegraded":"2024-03-21T13:00:10.728509+0000","last_fullsized":"2024-03-21T13:00:10.728509+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.823841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:28.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.782151+0000","last_change":"2024-03-21T12:59:43.240826+0000","last_active":"2024-03-21T13:00:10.782151+0000","last_peered":"2024-03-21T13:00:10.782151+0000","last_clean":"2024-03-21T13:00:10.782151+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:00:10.782151+0000","last_undegraded":"2024-03-21T13:00:10.782151+0000","last_fullsized":"2024-03-21T13:00:10.782151+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T12:59:43.240674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.075915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"71'18","reported_seq":96,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781462+0000","last_change":"2024-03-21T12:59:48.179286+0000","last_active":"2024-03-21T13:00:10.781462+0000","last_peered":"2024-03-21T13:00:10.781462+0000","last_clean":"2024-03-21T13:00:10.781462+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:00:10.781462+0000","last_undegraded":"2024-03-21T13:00:10.781462+0000","last_fullsized":"2024-03-21T13:00:10.781462+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T12:59:48.179188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.179188+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:15.921161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728351+0000","last_change":"2024-03-21T12:58:41.165071+0000","last_active":"2024-03-21T13:00:10.728351+0000","last_peered":"2024-03-21T13:00:10.728351+0000","last_clean":"2024-03-21T13:00:10.728351+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:00:10.728351+0000","last_undegraded":"2024-03-21T13:00:10.728351+0000","last_fullsized":"2024-03-21T13:00:10.728351+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015144+0000","last_change":"2024-03-21T12:59:57.599220+0000","last_active":"2024-03-21T13:00:11.015144+0000","last_peered":"2024-03-21T13:00:11.015144+0000","last_clean":"2024-03-21T13:00:11.015144+0000","last_became_active":"2024-03-21T12:58:43.183614+0000","last_became_peered":"2024-03-21T12:58:43.183614+0000","last_unstale":"2024-03-21T13:00:11.015144+0000","last_undegraded":"2024-03-21T13:00:11.015144+0000","last_fullsized":"2024-03-21T13:00:11.015144+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.599076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.599076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:00.058326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729352+0000","last_change":"2024-03-21T12:58:41.172355+0000","last_active":"2024-03-21T13:00:10.729352+0000","last_peered":"2024-03-21T13:00:10.729352+0000","last_clean":"2024-03-21T13:00:10.729352+0000","last_became_active":"2024-03-21T12:58:41.172019+0000","last_became_peered":"2024-03-21T12:58:41.172019+0000","last_unstale":"2024-03-21T13:00:10.729352+0000","last_undegraded":"2024-03-21T13:00:10.729352+0000","last_fullsized":"2024-03-21T13:00:10.729352+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781381+0000","last_change":"2024-03-21T12:59:41.231430+0000","last_active":"2024-03-21T13:00:10.781381+0000","last_peered":"2024-03-21T13:00:10.781381+0000","last_clean":"2024-03-21T13:00:10.781381+0000","last_became_active":"2024-03-21T12:58:37.731327+0000","last_became_peered":"2024-03-21T12:58:37.731327+0000","last_unstale":"2024-03-21T13:00:10.781381+0000","last_undegraded":"2024-03-21T13:00:10.781381+0000","last_fullsized":"2024-03-21T13:00:10.781381+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:41.231340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:41.231340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.247538+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.837206+0000","last_change":"2024-03-21T12:58:39.755619+0000","last_active":"2024-03-21T13:00:10.837206+0000","last_peered":"2024-03-21T13:00:10.837206+0000","last_clean":"2024-03-21T13:00:10.837206+0000","last_became_active":"2024-03-21T12:58:39.755326+0000","last_became_peered":"2024-03-21T12:58:39.755326+0000","last_unstale":"2024-03-21T13:00:10.837206+0000","last_undegraded":"2024-03-21T13:00:10.837206+0000","last_fullsized":"2024-03-21T13:00:10.837206+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781606+0000","last_change":"2024-03-21T12:58:43.351417+0000","last_active":"2024-03-21T13:00:10.781606+0000","last_peered":"2024-03-21T13:00:10.781606+0000","last_clean":"2024-03-21T13:00:10.781606+0000","last_became_active":"2024-03-21T12:58:43.350921+0000","last_became_peered":"2024-03-21T12:58:43.350921+0000","last_unstale":"2024-03-21T13:00:10.781606+0000","last_undegraded":"2024-03-21T13:00:10.781606+0000","last_fullsized":"2024-03-21T13:00:10.781606+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014889+0000","last_change":"2024-03-21T13:00:08.639131+0000","last_active":"2024-03-21T13:00:11.014889+0000","last_peered":"2024-03-21T13:00:11.014889+0000","last_clean":"2024-03-21T13:00:11.014889+0000","last_became_active":"2024-03-21T12:58:41.168784+0000","last_became_peered":"2024-03-21T12:58:41.168784+0000","last_unstale":"2024-03-21T13:00:11.014889+0000","last_undegraded":"2024-03-21T13:00:11.014889+0000","last_fullsized":"2024-03-21T13:00:11.014889+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.639041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.639041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:17.524978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728385+0000","last_change":"2024-03-21T12:58:37.731156+0000","last_active":"2024-03-21T13:00:10.728385+0000","last_peered":"2024-03-21T13:00:10.728385+0000","last_clean":"2024-03-21T13:00:10.728385+0000","last_became_active":"2024-03-21T12:58:37.730858+0000","last_became_peered":"2024-03-21T12:58:37.730858+0000","last_unstale":"2024-03-21T13:00:10.728385+0000","last_undegraded":"2024-03-21T13:00:10.728385+0000","last_fullsized":"2024-03-21T13:00:10.728385+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'11","reported_seq":75,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729188+0000","last_change":"2024-03-21T12:58:39.754932+0000","last_active":"2024-03-21T13:00:10.729188+0000","last_peered":"2024-03-21T13:00:10.729188+0000","last_clean":"2024-03-21T13:00:10.729188+0000","last_became_active":"2024-03-21T12:58:39.754337+0000","last_became_peered":"2024-03-21T12:58:39.754337+0000","last_unstale":"2024-03-21T13:00:10.729188+0000","last_undegraded":"2024-03-21T13:00:10.729188+0000","last_fullsized":"2024-03-21T13:00:10.729188+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T12:59:53.316645+0000","last_change":"2024-03-21T12:58:43.183848+0000","last_active":"2024-03-21T12:59:53.316645+0000","last_peered":"2024-03-21T12:59:53.316645+0000","last_clean":"2024-03-21T12:59:53.316645+0000","last_became_active":"2024-03-21T12:58:43.183380+0000","last_became_peered":"2024-03-21T12:58:43.183380+0000","last_unstale":"2024-03-21T12:59:53.316645+0000","last_undegraded":"2024-03-21T12:59:53.316645+0000","last_fullsized":"2024-03-21T12:59:53.316645+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:11.881657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.837068+0000","last_change":"2024-03-21T12:59:50.039168+0000","last_active":"2024-03-21T13:00:10.837068+0000","last_peered":"2024-03-21T13:00:10.837068+0000","last_clean":"2024-03-21T13:00:10.837068+0000","last_became_active":"2024-03-21T12:58:41.163145+0000","last_became_peered":"2024-03-21T12:58:41.163145+0000","last_unstale":"2024-03-21T13:00:10.837068+0000","last_undegraded":"2024-03-21T13:00:10.837068+0000","last_fullsized":"2024-03-21T13:00:10.837068+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.039091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.039091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.361130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.783929+0000","last_change":"2024-03-21T12:59:49.204557+0000","last_active":"2024-03-21T13:00:10.783929+0000","last_peered":"2024-03-21T13:00:10.783929+0000","last_clean":"2024-03-21T13:00:10.783929+0000","last_became_active":"2024-03-21T12:58:37.729792+0000","last_became_peered":"2024-03-21T12:58:37.729792+0000","last_unstale":"2024-03-21T13:00:10.783929+0000","last_undegraded":"2024-03-21T13:00:10.783929+0000","last_fullsized":"2024-03-21T13:00:10.783929+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:49.204419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:49.204419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.549632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":84,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729058+0000","last_change":"2024-03-21T12:58:39.746134+0000","last_active":"2024-03-21T13:00:10.729058+0000","last_peered":"2024-03-21T13:00:10.729058+0000","last_clean":"2024-03-21T13:00:10.729058+0000","last_became_active":"2024-03-21T12:58:39.745807+0000","last_became_peered":"2024-03-21T12:58:39.745807+0000","last_unstale":"2024-03-21T13:00:10.729058+0000","last_undegraded":"2024-03-21T13:00:10.729058+0000","last_fullsized":"2024-03-21T13:00:10.729058+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729305+0000","last_change":"2024-03-21T12:58:43.185167+0000","last_active":"2024-03-21T13:00:10.729305+0000","last_peered":"2024-03-21T13:00:10.729305+0000","last_clean":"2024-03-21T13:00:10.729305+0000","last_became_active":"2024-03-21T12:58:43.184576+0000","last_became_peered":"2024-03-21T12:58:43.184576+0000","last_unstale":"2024-03-21T13:00:10.729305+0000","last_undegraded":"2024-03-21T13:00:10.729305+0000","last_fullsized":"2024-03-21T13:00:10.729305+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"71'8","reported_seq":78,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-21T13:00:02.788608+0000","last_change":"2024-03-21T12:58:41.165903+0000","last_active":"2024-03-21T13:00:02.788608+0000","last_peered":"2024-03-21T13:00:02.788608+0000","last_clean":"2024-03-21T13:00:02.788608+0000","last_became_active":"2024-03-21T12:58:41.165560+0000","last_became_peered":"2024-03-21T12:58:41.165560+0000","last_unstale":"2024-03-21T13:00:02.788608+0000","last_undegraded":"2024-03-21T13:00:02.788608+0000","last_fullsized":"2024-03-21T13:00:02.788608+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:09.702381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781836+0000","last_change":"2024-03-21T12:58:37.732264+0000","last_active":"2024-03-21T13:00:10.781836+0000","last_peered":"2024-03-21T13:00:10.781836+0000","last_clean":"2024-03-21T13:00:10.781836+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:00:10.781836+0000","last_undegraded":"2024-03-21T13:00:10.781836+0000","last_fullsized":"2024-03-21T13:00:10.781836+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.785829+0000","last_change":"2024-03-21T13:00:10.785895+0000","last_active":"2024-03-21T13:00:10.785829+0000","last_peered":"2024-03-21T13:00:10.785829+0000","last_clean":"2024-03-21T13:00:10.785829+0000","last_became_active":"2024-03-21T12:58:39.749956+0000","last_became_peered":"2024-03-21T12:58:39.749956+0000","last_unstale":"2024-03-21T13:00:10.785829+0000","last_undegraded":"2024-03-21T13:00:10.785829+0000","last_fullsized":"2024-03-21T13:00:10.785829+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:00:10.785752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.785752+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:30.125302+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":415,"ondisk_log_size":415,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168600,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1388544,"data_stored":946614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462942,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32644,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712508,"statfs":{"total":95995035648,"available":95961608192,"internally_reserved":0,"allocated":1376256,"data_stored":946011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31980,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713172,"statfs":{"total":95995035648,"available":95962288128,"internally_reserved":0,"allocated":823296,"data_stored":355738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1499136,"data_stored":967579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":892928,"data_stored":377298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":815104,"data_stored":355623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018919,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":802816,"data_stored":355799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724578,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":778240,"data_stored":354622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:00:16.345 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:00:16.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:16 smithi204 ceph-mon[33153]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:16.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:16 smithi204 ceph-mon[33153]: 2.0 scrub starts 2024-03-21T13:00:16.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:16 smithi204 ceph-mon[33153]: 2.0 scrub ok 2024-03-21T13:00:16.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:16 smithi204 ceph-mon[33153]: 3.9 scrub starts 2024-03-21T13:00:16.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:16 smithi204 ceph-mon[33153]: 3.9 scrub ok 2024-03-21T13:00:16.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:16 smithi163 ceph-mon[29621]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:16.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:16 smithi163 ceph-mon[29621]: 2.0 scrub starts 2024-03-21T13:00:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:16 smithi163 ceph-mon[29621]: 2.0 scrub ok 2024-03-21T13:00:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:16 smithi163 ceph-mon[29621]: 3.9 scrub starts 2024-03-21T13:00:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:16 smithi163 ceph-mon[29621]: 3.9 scrub ok 2024-03-21T13:00:16.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:16 smithi163 ceph-mon[40675]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:16.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:16 smithi163 ceph-mon[40675]: 2.0 scrub starts 2024-03-21T13:00:16.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:16 smithi163 ceph-mon[40675]: 2.0 scrub ok 2024-03-21T13:00:16.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:16 smithi163 ceph-mon[40675]: 3.9 scrub starts 2024-03-21T13:00:16.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:16 smithi163 ceph-mon[40675]: 3.9 scrub ok 2024-03-21T13:00:16.627 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:17 smithi163 ceph-mon[29621]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:00:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:17 smithi163 ceph-mon[29621]: 1.0 deep-scrub starts 2024-03-21T13:00:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:17 smithi163 ceph-mon[29621]: 1.0 deep-scrub ok 2024-03-21T13:00:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:17 smithi163 ceph-mon[29621]: 5.18 deep-scrub starts 2024-03-21T13:00:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:17 smithi163 ceph-mon[29621]: 5.18 deep-scrub ok 2024-03-21T13:00:17.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:17 smithi163 ceph-mon[40675]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:00:17.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:17 smithi163 ceph-mon[40675]: 1.0 deep-scrub starts 2024-03-21T13:00:17.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:17 smithi163 ceph-mon[40675]: 1.0 deep-scrub ok 2024-03-21T13:00:17.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:17 smithi163 ceph-mon[40675]: 5.18 deep-scrub starts 2024-03-21T13:00:17.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:17 smithi163 ceph-mon[40675]: 5.18 deep-scrub ok 2024-03-21T13:00:17.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:17 smithi204 ceph-mon[33153]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:00:17.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:17 smithi204 ceph-mon[33153]: 1.0 deep-scrub starts 2024-03-21T13:00:17.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:17 smithi204 ceph-mon[33153]: 1.0 deep-scrub ok 2024-03-21T13:00:17.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:17 smithi204 ceph-mon[33153]: 5.18 deep-scrub starts 2024-03-21T13:00:17.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:17 smithi204 ceph-mon[33153]: 5.18 deep-scrub ok 2024-03-21T13:00:18.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:18 smithi163 ceph-mon[29621]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:18.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:18 smithi163 ceph-mon[29621]: 6.3 scrub starts 2024-03-21T13:00:18.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:18 smithi163 ceph-mon[29621]: 6.3 scrub ok 2024-03-21T13:00:18.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:18 smithi163 ceph-mon[29621]: 3.1 scrub starts 2024-03-21T13:00:18.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:18 smithi163 ceph-mon[29621]: 3.1 scrub ok 2024-03-21T13:00:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:18 smithi163 ceph-mon[40675]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:18 smithi163 ceph-mon[40675]: 6.3 scrub starts 2024-03-21T13:00:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:18 smithi163 ceph-mon[40675]: 6.3 scrub ok 2024-03-21T13:00:18.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:18 smithi163 ceph-mon[40675]: 3.1 scrub starts 2024-03-21T13:00:18.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:18 smithi163 ceph-mon[40675]: 3.1 scrub ok 2024-03-21T13:00:18.732 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:00:18.734 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:00:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:18 smithi204 ceph-mon[33153]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:18 smithi204 ceph-mon[33153]: 6.3 scrub starts 2024-03-21T13:00:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:18 smithi204 ceph-mon[33153]: 6.3 scrub ok 2024-03-21T13:00:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:18 smithi204 ceph-mon[33153]: 3.1 scrub starts 2024-03-21T13:00:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:18 smithi204 ceph-mon[33153]: 3.1 scrub ok 2024-03-21T13:00:20.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:19 smithi204 ceph-mon[33153]: 5.10 scrub starts 2024-03-21T13:00:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:19 smithi204 ceph-mon[33153]: 5.10 scrub ok 2024-03-21T13:00:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:19 smithi204 ceph-mon[33153]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:19 smithi204 ceph-mon[33153]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:00:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:19 smithi163 ceph-mon[29621]: 5.10 scrub starts 2024-03-21T13:00:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:19 smithi163 ceph-mon[29621]: 5.10 scrub ok 2024-03-21T13:00:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:19 smithi163 ceph-mon[29621]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:19 smithi163 ceph-mon[29621]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:00:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:20.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:19 smithi163 ceph-mon[40675]: 5.10 scrub starts 2024-03-21T13:00:20.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:19 smithi163 ceph-mon[40675]: 5.10 scrub ok 2024-03-21T13:00:20.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:19 smithi163 ceph-mon[40675]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:20.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:20.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:19 smithi163 ceph-mon[40675]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:00:20.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:20.012 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:19 smithi163 bash[76112]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-21T13:00:20.510 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:20 smithi163 bash[76112]: Writing manifest to image destination 2024-03-21T13:00:20.664 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-03-21T13:00:18.627840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1023,"num_read_kb":869,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":624,"ondisk_log_size":624,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249672,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749711544,"statfs":{"total":767960285184,"available":767704621056,"internally_reserved":0,"allocated":8155136,"data_stored":4615016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002697"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728486+0000","last_change":"2024-03-21T12:59:58.846777+0000","last_active":"2024-03-21T13:00:10.728486+0000","last_peered":"2024-03-21T13:00:10.728486+0000","last_clean":"2024-03-21T13:00:10.728486+0000","last_became_active":"2024-03-21T12:58:43.182506+0000","last_became_peered":"2024-03-21T12:58:43.182506+0000","last_unstale":"2024-03-21T13:00:10.728486+0000","last_undegraded":"2024-03-21T13:00:10.728486+0000","last_fullsized":"2024-03-21T13:00:10.728486+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:58.846634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:58.846634+0000","last_clean_scrub_stamp":"2024-03-21T12:59:58.846634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.412490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836484+0000","last_change":"2024-03-21T12:58:41.160668+0000","last_active":"2024-03-21T13:00:10.836484+0000","last_peered":"2024-03-21T13:00:10.836484+0000","last_clean":"2024-03-21T13:00:10.836484+0000","last_became_active":"2024-03-21T12:58:41.160342+0000","last_became_peered":"2024-03-21T12:58:41.160342+0000","last_unstale":"2024-03-21T13:00:10.836484+0000","last_undegraded":"2024-03-21T13:00:10.836484+0000","last_fullsized":"2024-03-21T13:00:10.836484+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.731290+0000","last_change":"2024-03-21T12:59:53.478218+0000","last_active":"2024-03-21T13:00:10.731290+0000","last_peered":"2024-03-21T13:00:10.731290+0000","last_clean":"2024-03-21T13:00:10.731290+0000","last_became_active":"2024-03-21T12:58:37.731393+0000","last_became_peered":"2024-03-21T12:58:37.731393+0000","last_unstale":"2024-03-21T13:00:10.731290+0000","last_undegraded":"2024-03-21T13:00:10.731290+0000","last_fullsized":"2024-03-21T13:00:10.731290+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:53.478141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:53.478141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:57.521483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"71'9","reported_seq":88,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728359+0000","last_change":"2024-03-21T12:59:01.166943+0000","last_active":"2024-03-21T13:00:10.728359+0000","last_peered":"2024-03-21T13:00:10.728359+0000","last_clean":"2024-03-21T13:00:10.728359+0000","last_became_active":"2024-03-21T12:59:01.161098+0000","last_became_peered":"2024-03-21T12:59:01.161098+0000","last_unstale":"2024-03-21T13:00:10.728359+0000","last_undegraded":"2024-03-21T13:00:10.728359+0000","last_fullsized":"2024-03-21T13:00:10.728359+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781719+0000","last_change":"2024-03-21T13:00:05.173374+0000","last_active":"2024-03-21T13:00:10.781719+0000","last_peered":"2024-03-21T13:00:10.781719+0000","last_clean":"2024-03-21T13:00:10.781719+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:00:10.781719+0000","last_undegraded":"2024-03-21T13:00:10.781719+0000","last_fullsized":"2024-03-21T13:00:10.781719+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:00:05.173276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:14.939464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728367+0000","last_change":"2024-03-21T12:58:41.168108+0000","last_active":"2024-03-21T13:00:10.728367+0000","last_peered":"2024-03-21T13:00:10.728367+0000","last_clean":"2024-03-21T13:00:10.728367+0000","last_became_active":"2024-03-21T12:58:41.167710+0000","last_became_peered":"2024-03-21T12:58:41.167710+0000","last_unstale":"2024-03-21T13:00:10.728367+0000","last_undegraded":"2024-03-21T13:00:10.728367+0000","last_fullsized":"2024-03-21T13:00:10.728367+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781984+0000","last_change":"2024-03-21T12:58:37.728682+0000","last_active":"2024-03-21T13:00:10.781984+0000","last_peered":"2024-03-21T13:00:10.781984+0000","last_clean":"2024-03-21T13:00:10.781984+0000","last_became_active":"2024-03-21T12:58:37.728383+0000","last_became_peered":"2024-03-21T12:58:37.728383+0000","last_unstale":"2024-03-21T13:00:10.781984+0000","last_undegraded":"2024-03-21T13:00:10.781984+0000","last_fullsized":"2024-03-21T13:00:10.781984+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"71'15","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781698+0000","last_change":"2024-03-21T12:58:39.751346+0000","last_active":"2024-03-21T13:00:10.781698+0000","last_peered":"2024-03-21T13:00:10.781698+0000","last_clean":"2024-03-21T13:00:10.781698+0000","last_became_active":"2024-03-21T12:58:39.750396+0000","last_became_peered":"2024-03-21T12:58:39.750396+0000","last_unstale":"2024-03-21T13:00:10.781698+0000","last_undegraded":"2024-03-21T13:00:10.781698+0000","last_fullsized":"2024-03-21T13:00:10.781698+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729595+0000","last_change":"2024-03-21T13:00:02.599805+0000","last_active":"2024-03-21T13:00:10.729595+0000","last_peered":"2024-03-21T13:00:10.729595+0000","last_clean":"2024-03-21T13:00:10.729595+0000","last_became_active":"2024-03-21T12:58:43.179920+0000","last_became_peered":"2024-03-21T12:58:43.179920+0000","last_unstale":"2024-03-21T13:00:10.729595+0000","last_undegraded":"2024-03-21T13:00:10.729595+0000","last_fullsized":"2024-03-21T13:00:10.729595+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:02.599679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:02.599679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:10.324225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014482+0000","last_change":"2024-03-21T13:00:03.644301+0000","last_active":"2024-03-21T13:00:11.014482+0000","last_peered":"2024-03-21T13:00:11.014482+0000","last_clean":"2024-03-21T13:00:11.014482+0000","last_became_active":"2024-03-21T12:58:41.168978+0000","last_became_peered":"2024-03-21T12:58:41.168978+0000","last_unstale":"2024-03-21T13:00:11.014482+0000","last_undegraded":"2024-03-21T13:00:11.014482+0000","last_fullsized":"2024-03-21T13:00:11.014482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:03.644209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:03.644209+0000","last_clean_scrub_stamp":"2024-03-21T13:00:03.644209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:18.843767+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015243+0000","last_change":"2024-03-21T12:59:44.661395+0000","last_active":"2024-03-21T13:00:11.015243+0000","last_peered":"2024-03-21T13:00:11.015243+0000","last_clean":"2024-03-21T13:00:11.015243+0000","last_became_active":"2024-03-21T12:58:37.733705+0000","last_became_peered":"2024-03-21T12:58:37.733705+0000","last_unstale":"2024-03-21T13:00:11.015243+0000","last_undegraded":"2024-03-21T13:00:11.015243+0000","last_fullsized":"2024-03-21T13:00:11.015243+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:44.661289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:44.661289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:45.014731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"71'9","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728617+0000","last_change":"2024-03-21T12:58:39.749391+0000","last_active":"2024-03-21T13:00:10.728617+0000","last_peered":"2024-03-21T13:00:10.728617+0000","last_clean":"2024-03-21T13:00:10.728617+0000","last_became_active":"2024-03-21T12:58:39.749010+0000","last_became_peered":"2024-03-21T12:58:39.749010+0000","last_unstale":"2024-03-21T13:00:10.728617+0000","last_undegraded":"2024-03-21T13:00:10.728617+0000","last_fullsized":"2024-03-21T13:00:10.728617+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014998+0000","last_change":"2024-03-21T13:00:02.683946+0000","last_active":"2024-03-21T13:00:11.014998+0000","last_peered":"2024-03-21T13:00:11.014998+0000","last_clean":"2024-03-21T13:00:11.014998+0000","last_became_active":"2024-03-21T12:58:43.181494+0000","last_became_peered":"2024-03-21T12:58:43.181494+0000","last_unstale":"2024-03-21T13:00:11.014998+0000","last_undegraded":"2024-03-21T13:00:11.014998+0000","last_fullsized":"2024-03-21T13:00:11.014998+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:02.683841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:02.683841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:10.243871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":59,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014096+0000","last_change":"2024-03-21T12:59:41.158247+0000","last_active":"2024-03-21T13:00:11.014096+0000","last_peered":"2024-03-21T13:00:11.014096+0000","last_clean":"2024-03-21T13:00:11.014096+0000","last_became_active":"2024-03-21T12:58:41.163517+0000","last_became_peered":"2024-03-21T12:58:41.163517+0000","last_unstale":"2024-03-21T13:00:11.014096+0000","last_undegraded":"2024-03-21T13:00:11.014096+0000","last_fullsized":"2024-03-21T13:00:11.014096+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:41.158105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:41.158105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:45.036271+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014761+0000","last_change":"2024-03-21T12:59:59.594026+0000","last_active":"2024-03-21T13:00:11.014761+0000","last_peered":"2024-03-21T13:00:11.014761+0000","last_clean":"2024-03-21T13:00:11.014761+0000","last_became_active":"2024-03-21T12:58:37.731058+0000","last_became_peered":"2024-03-21T12:58:37.731058+0000","last_unstale":"2024-03-21T13:00:11.014761+0000","last_undegraded":"2024-03-21T13:00:11.014761+0000","last_fullsized":"2024-03-21T13:00:11.014761+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:59.593927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.593927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:14.344496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":76,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729004+0000","last_change":"2024-03-21T12:59:53.952540+0000","last_active":"2024-03-21T13:00:10.729004+0000","last_peered":"2024-03-21T13:00:10.729004+0000","last_clean":"2024-03-21T13:00:10.729004+0000","last_became_active":"2024-03-21T12:58:39.749211+0000","last_became_peered":"2024-03-21T12:58:39.749211+0000","last_unstale":"2024-03-21T13:00:10.729004+0000","last_undegraded":"2024-03-21T13:00:10.729004+0000","last_fullsized":"2024-03-21T13:00:10.729004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T12:59:53.952401+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.263330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729179+0000","last_change":"2024-03-21T12:59:55.899438+0000","last_active":"2024-03-21T13:00:10.729179+0000","last_peered":"2024-03-21T13:00:10.729179+0000","last_clean":"2024-03-21T13:00:10.729179+0000","last_became_active":"2024-03-21T12:58:43.351768+0000","last_became_peered":"2024-03-21T12:58:43.351768+0000","last_unstale":"2024-03-21T13:00:10.729179+0000","last_undegraded":"2024-03-21T13:00:10.729179+0000","last_fullsized":"2024-03-21T13:00:10.729179+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:55.899344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.899344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:59.849673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.837061+0000","last_change":"2024-03-21T12:59:55.014569+0000","last_active":"2024-03-21T13:00:10.837061+0000","last_peered":"2024-03-21T13:00:10.837061+0000","last_clean":"2024-03-21T13:00:10.837061+0000","last_became_active":"2024-03-21T12:58:41.165783+0000","last_became_peered":"2024-03-21T12:58:41.165783+0000","last_unstale":"2024-03-21T13:00:10.837061+0000","last_undegraded":"2024-03-21T13:00:10.837061+0000","last_fullsized":"2024-03-21T13:00:10.837061+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.014453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:10.470418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":59,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.730144+0000","last_change":"2024-03-21T12:58:37.733421+0000","last_active":"2024-03-21T13:00:10.730144+0000","last_peered":"2024-03-21T13:00:10.730144+0000","last_clean":"2024-03-21T13:00:10.730144+0000","last_became_active":"2024-03-21T12:58:37.733064+0000","last_became_peered":"2024-03-21T12:58:37.733064+0000","last_unstale":"2024-03-21T13:00:10.730144+0000","last_undegraded":"2024-03-21T13:00:10.730144+0000","last_fullsized":"2024-03-21T13:00:10.730144+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":90,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.730339+0000","last_change":"2024-03-21T12:59:55.664752+0000","last_active":"2024-03-21T13:00:10.730339+0000","last_peered":"2024-03-21T13:00:10.730339+0000","last_clean":"2024-03-21T13:00:10.730339+0000","last_became_active":"2024-03-21T12:58:39.749130+0000","last_became_peered":"2024-03-21T12:58:39.749130+0000","last_unstale":"2024-03-21T13:00:10.730339+0000","last_undegraded":"2024-03-21T13:00:10.730339+0000","last_fullsized":"2024-03-21T13:00:10.730339+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T12:59:55.664649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.664649+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:11.693713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.782004+0000","last_change":"2024-03-21T12:58:43.352883+0000","last_active":"2024-03-21T13:00:10.782004+0000","last_peered":"2024-03-21T13:00:10.782004+0000","last_clean":"2024-03-21T13:00:10.782004+0000","last_became_active":"2024-03-21T12:58:43.352510+0000","last_became_peered":"2024-03-21T12:58:43.352510+0000","last_unstale":"2024-03-21T13:00:10.782004+0000","last_undegraded":"2024-03-21T13:00:10.782004+0000","last_fullsized":"2024-03-21T13:00:10.782004+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728820+0000","last_change":"2024-03-21T12:59:50.904504+0000","last_active":"2024-03-21T13:00:10.728820+0000","last_peered":"2024-03-21T13:00:10.728820+0000","last_clean":"2024-03-21T13:00:10.728820+0000","last_became_active":"2024-03-21T12:58:41.167961+0000","last_became_peered":"2024-03-21T12:58:41.167961+0000","last_unstale":"2024-03-21T13:00:10.728820+0000","last_undegraded":"2024-03-21T13:00:10.728820+0000","last_fullsized":"2024-03-21T13:00:10.728820+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.904419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.904419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.905923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728638+0000","last_change":"2024-03-21T12:59:54.948197+0000","last_active":"2024-03-21T13:00:10.728638+0000","last_peered":"2024-03-21T13:00:10.728638+0000","last_clean":"2024-03-21T13:00:10.728638+0000","last_became_active":"2024-03-21T12:58:37.732355+0000","last_became_peered":"2024-03-21T12:58:37.732355+0000","last_unstale":"2024-03-21T13:00:10.728638+0000","last_undegraded":"2024-03-21T13:00:10.728638+0000","last_fullsized":"2024-03-21T13:00:10.728638+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_clean_scrub_stamp":"2024-03-21T12:59:54.948081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:03.565967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'12","reported_seq":97,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781778+0000","last_change":"2024-03-21T12:59:01.168089+0000","last_active":"2024-03-21T13:00:10.781778+0000","last_peered":"2024-03-21T13:00:10.781778+0000","last_clean":"2024-03-21T13:00:10.781778+0000","last_became_active":"2024-03-21T12:59:01.161520+0000","last_became_peered":"2024-03-21T12:59:01.161520+0000","last_unstale":"2024-03-21T13:00:10.781778+0000","last_undegraded":"2024-03-21T13:00:10.781778+0000","last_fullsized":"2024-03-21T13:00:10.781778+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014256+0000","last_change":"2024-03-21T13:00:00.168621+0000","last_active":"2024-03-21T13:00:11.014256+0000","last_peered":"2024-03-21T13:00:11.014256+0000","last_clean":"2024-03-21T13:00:11.014256+0000","last_became_active":"2024-03-21T12:58:43.184473+0000","last_became_peered":"2024-03-21T12:58:43.184473+0000","last_unstale":"2024-03-21T13:00:11.014256+0000","last_undegraded":"2024-03-21T13:00:11.014256+0000","last_fullsized":"2024-03-21T13:00:11.014256+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:00:00.168509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:00.168509+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.057928+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.784305+0000","last_change":"2024-03-21T12:59:55.277279+0000","last_active":"2024-03-21T13:00:10.784305+0000","last_peered":"2024-03-21T13:00:10.784305+0000","last_clean":"2024-03-21T13:00:10.784305+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:00:10.784305+0000","last_undegraded":"2024-03-21T13:00:10.784305+0000","last_fullsized":"2024-03-21T13:00:10.784305+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:55.277165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.277165+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:19.101757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":69,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.838018+0000","last_change":"2024-03-21T13:00:10.838057+0000","last_active":"2024-03-21T13:00:10.838018+0000","last_peered":"2024-03-21T13:00:10.838018+0000","last_clean":"2024-03-21T13:00:10.838018+0000","last_became_active":"2024-03-21T12:58:37.736455+0000","last_became_peered":"2024-03-21T12:58:37.736455+0000","last_unstale":"2024-03-21T13:00:10.838018+0000","last_undegraded":"2024-03-21T13:00:10.838018+0000","last_fullsized":"2024-03-21T13:00:10.838018+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:00:10.837949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.837949+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:29.483399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"71'10","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.731077+0000","last_change":"2024-03-21T12:59:50.414098+0000","last_active":"2024-03-21T13:00:10.731077+0000","last_peered":"2024-03-21T13:00:10.731077+0000","last_clean":"2024-03-21T13:00:10.731077+0000","last_became_active":"2024-03-21T12:58:39.754548+0000","last_became_peered":"2024-03-21T12:58:39.754548+0000","last_unstale":"2024-03-21T13:00:10.731077+0000","last_undegraded":"2024-03-21T13:00:10.731077+0000","last_fullsized":"2024-03-21T13:00:10.731077+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T12:59:50.413983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.413983+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.575091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836853+0000","last_change":"2024-03-21T12:58:43.186597+0000","last_active":"2024-03-21T13:00:10.836853+0000","last_peered":"2024-03-21T13:00:10.836853+0000","last_clean":"2024-03-21T13:00:10.836853+0000","last_became_active":"2024-03-21T12:58:43.186289+0000","last_became_peered":"2024-03-21T12:58:43.186289+0000","last_unstale":"2024-03-21T13:00:10.836853+0000","last_undegraded":"2024-03-21T13:00:10.836853+0000","last_fullsized":"2024-03-21T13:00:10.836853+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.731251+0000","last_change":"2024-03-21T12:58:41.167760+0000","last_active":"2024-03-21T13:00:10.731251+0000","last_peered":"2024-03-21T13:00:10.731251+0000","last_clean":"2024-03-21T13:00:10.731251+0000","last_became_active":"2024-03-21T12:58:41.167443+0000","last_became_peered":"2024-03-21T12:58:41.167443+0000","last_unstale":"2024-03-21T13:00:10.731251+0000","last_undegraded":"2024-03-21T13:00:10.731251+0000","last_fullsized":"2024-03-21T13:00:10.731251+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781277+0000","last_change":"2024-03-21T12:59:49.178866+0000","last_active":"2024-03-21T13:00:10.781277+0000","last_peered":"2024-03-21T13:00:10.781277+0000","last_clean":"2024-03-21T13:00:10.781277+0000","last_became_active":"2024-03-21T12:58:37.728914+0000","last_became_peered":"2024-03-21T12:58:37.728914+0000","last_unstale":"2024-03-21T13:00:10.781277+0000","last_undegraded":"2024-03-21T13:00:10.781277+0000","last_fullsized":"2024-03-21T13:00:10.781277+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:49.178785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:49.178785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.644002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'11","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.784280+0000","last_change":"2024-03-21T13:00:01.396865+0000","last_active":"2024-03-21T13:00:10.784280+0000","last_peered":"2024-03-21T13:00:10.784280+0000","last_clean":"2024-03-21T13:00:10.784280+0000","last_became_active":"2024-03-21T12:58:39.755294+0000","last_became_peered":"2024-03-21T12:58:39.755294+0000","last_unstale":"2024-03-21T13:00:10.784280+0000","last_undegraded":"2024-03-21T13:00:10.784280+0000","last_fullsized":"2024-03-21T13:00:10.784280+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:00:01.396749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.396749+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.505311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.013984+0000","last_change":"2024-03-21T12:59:50.209251+0000","last_active":"2024-03-21T13:00:11.013984+0000","last_peered":"2024-03-21T13:00:11.013984+0000","last_clean":"2024-03-21T13:00:11.013984+0000","last_became_active":"2024-03-21T12:58:43.183475+0000","last_became_peered":"2024-03-21T12:58:43.183475+0000","last_unstale":"2024-03-21T13:00:11.013984+0000","last_undegraded":"2024-03-21T13:00:11.013984+0000","last_fullsized":"2024-03-21T13:00:11.013984+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.209152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.209152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:18.566644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.783594+0000","last_change":"2024-03-21T13:00:03.421382+0000","last_active":"2024-03-21T13:00:10.783594+0000","last_peered":"2024-03-21T13:00:10.783594+0000","last_clean":"2024-03-21T13:00:10.783594+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:00:10.783594+0000","last_undegraded":"2024-03-21T13:00:10.783594+0000","last_fullsized":"2024-03-21T13:00:10.783594+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:03.421305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:03.421305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.521044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":59,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.731282+0000","last_change":"2024-03-21T12:58:37.732416+0000","last_active":"2024-03-21T13:00:10.731282+0000","last_peered":"2024-03-21T13:00:10.731282+0000","last_clean":"2024-03-21T13:00:10.731282+0000","last_became_active":"2024-03-21T12:58:37.732144+0000","last_became_peered":"2024-03-21T12:58:37.732144+0000","last_unstale":"2024-03-21T13:00:10.731282+0000","last_undegraded":"2024-03-21T13:00:10.731282+0000","last_fullsized":"2024-03-21T13:00:10.731282+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781352+0000","last_change":"2024-03-21T12:58:39.755658+0000","last_active":"2024-03-21T13:00:10.781352+0000","last_peered":"2024-03-21T13:00:10.781352+0000","last_clean":"2024-03-21T13:00:10.781352+0000","last_became_active":"2024-03-21T12:58:39.755336+0000","last_became_peered":"2024-03-21T12:58:39.755336+0000","last_unstale":"2024-03-21T13:00:10.781352+0000","last_undegraded":"2024-03-21T13:00:10.781352+0000","last_fullsized":"2024-03-21T13:00:10.781352+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781472+0000","last_change":"2024-03-21T13:00:10.110413+0000","last_active":"2024-03-21T13:00:10.781472+0000","last_peered":"2024-03-21T13:00:10.781472+0000","last_clean":"2024-03-21T13:00:10.781472+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:00:10.781472+0000","last_undegraded":"2024-03-21T13:00:10.781472+0000","last_fullsized":"2024-03-21T13:00:10.781472+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:10.110335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.110335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:30.400061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014143+0000","last_change":"2024-03-21T12:58:41.165123+0000","last_active":"2024-03-21T13:00:11.014143+0000","last_peered":"2024-03-21T13:00:11.014143+0000","last_clean":"2024-03-21T13:00:11.014143+0000","last_became_active":"2024-03-21T12:58:41.164784+0000","last_became_peered":"2024-03-21T12:58:41.164784+0000","last_unstale":"2024-03-21T13:00:11.014143+0000","last_undegraded":"2024-03-21T13:00:11.014143+0000","last_fullsized":"2024-03-21T13:00:11.014143+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014691+0000","last_change":"2024-03-21T13:00:09.658052+0000","last_active":"2024-03-21T13:00:11.014691+0000","last_peered":"2024-03-21T13:00:11.014691+0000","last_clean":"2024-03-21T13:00:11.014691+0000","last_became_active":"2024-03-21T12:58:37.736127+0000","last_became_peered":"2024-03-21T12:58:37.736127+0000","last_unstale":"2024-03-21T13:00:11.014691+0000","last_undegraded":"2024-03-21T13:00:11.014691+0000","last_fullsized":"2024-03-21T13:00:11.014691+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:09.657971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:09.657971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:30.873328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"71'11","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781642+0000","last_change":"2024-03-21T12:59:52.181828+0000","last_active":"2024-03-21T13:00:10.781642+0000","last_peered":"2024-03-21T13:00:10.781642+0000","last_clean":"2024-03-21T13:00:10.781642+0000","last_became_active":"2024-03-21T12:58:39.753014+0000","last_became_peered":"2024-03-21T12:58:39.753014+0000","last_unstale":"2024-03-21T13:00:10.781642+0000","last_undegraded":"2024-03-21T13:00:10.781642+0000","last_fullsized":"2024-03-21T13:00:10.781642+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T12:59:52.181697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.181697+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.248453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729383+0000","last_change":"2024-03-21T12:58:43.178915+0000","last_active":"2024-03-21T13:00:10.729383+0000","last_peered":"2024-03-21T13:00:10.729383+0000","last_clean":"2024-03-21T13:00:10.729383+0000","last_became_active":"2024-03-21T12:58:43.178589+0000","last_became_peered":"2024-03-21T12:58:43.178589+0000","last_unstale":"2024-03-21T13:00:10.729383+0000","last_undegraded":"2024-03-21T13:00:10.729383+0000","last_fullsized":"2024-03-21T13:00:10.729383+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781725+0000","last_change":"2024-03-21T12:58:41.167969+0000","last_active":"2024-03-21T13:00:10.781725+0000","last_peered":"2024-03-21T13:00:10.781725+0000","last_clean":"2024-03-21T13:00:10.781725+0000","last_became_active":"2024-03-21T12:58:41.167371+0000","last_became_peered":"2024-03-21T12:58:41.167371+0000","last_unstale":"2024-03-21T13:00:10.781725+0000","last_undegraded":"2024-03-21T13:00:10.781725+0000","last_fullsized":"2024-03-21T13:00:10.781725+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":59,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014850+0000","last_change":"2024-03-21T12:58:37.733949+0000","last_active":"2024-03-21T13:00:11.014850+0000","last_peered":"2024-03-21T13:00:11.014850+0000","last_clean":"2024-03-21T13:00:11.014850+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:00:11.014850+0000","last_undegraded":"2024-03-21T13:00:11.014850+0000","last_fullsized":"2024-03-21T13:00:11.014850+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"86'13","reported_seq":84,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836777+0000","last_change":"2024-03-21T12:59:48.085870+0000","last_active":"2024-03-21T13:00:10.836777+0000","last_peered":"2024-03-21T13:00:10.836777+0000","last_clean":"2024-03-21T13:00:10.836777+0000","last_became_active":"2024-03-21T12:58:39.753224+0000","last_became_peered":"2024-03-21T12:58:39.753224+0000","last_unstale":"2024-03-21T13:00:10.836777+0000","last_undegraded":"2024-03-21T13:00:10.836777+0000","last_fullsized":"2024-03-21T13:00:10.836777+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.085774+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:53.042862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781350+0000","last_change":"2024-03-21T12:58:43.184148+0000","last_active":"2024-03-21T13:00:10.781350+0000","last_peered":"2024-03-21T13:00:10.781350+0000","last_clean":"2024-03-21T13:00:10.781350+0000","last_became_active":"2024-03-21T12:58:43.183712+0000","last_became_peered":"2024-03-21T12:58:43.183712+0000","last_unstale":"2024-03-21T13:00:10.781350+0000","last_undegraded":"2024-03-21T13:00:10.781350+0000","last_fullsized":"2024-03-21T13:00:10.781350+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:12.054567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836713+0000","last_change":"2024-03-21T12:59:45.088950+0000","last_active":"2024-03-21T13:00:10.836713+0000","last_peered":"2024-03-21T13:00:10.836713+0000","last_clean":"2024-03-21T13:00:10.836713+0000","last_became_active":"2024-03-21T12:58:41.164463+0000","last_became_peered":"2024-03-21T12:58:41.164463+0000","last_unstale":"2024-03-21T13:00:10.836713+0000","last_undegraded":"2024-03-21T13:00:10.836713+0000","last_fullsized":"2024-03-21T13:00:10.836713+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:45.088861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.088861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:11.118145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728925+0000","last_change":"2024-03-21T12:59:45.833435+0000","last_active":"2024-03-21T13:00:10.728925+0000","last_peered":"2024-03-21T13:00:10.728925+0000","last_clean":"2024-03-21T13:00:10.728925+0000","last_became_active":"2024-03-21T12:58:37.731659+0000","last_became_peered":"2024-03-21T12:58:37.731659+0000","last_unstale":"2024-03-21T13:00:10.728925+0000","last_undegraded":"2024-03-21T13:00:10.728925+0000","last_fullsized":"2024-03-21T13:00:10.728925+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.833304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:49.946967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":75,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728879+0000","last_change":"2024-03-21T12:58:39.746514+0000","last_active":"2024-03-21T13:00:10.728879+0000","last_peered":"2024-03-21T13:00:10.728879+0000","last_clean":"2024-03-21T13:00:10.728879+0000","last_became_active":"2024-03-21T12:58:39.746182+0000","last_became_peered":"2024-03-21T12:58:39.746182+0000","last_unstale":"2024-03-21T13:00:10.728879+0000","last_undegraded":"2024-03-21T13:00:10.728879+0000","last_fullsized":"2024-03-21T13:00:10.728879+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.730874+0000","last_change":"2024-03-21T12:58:43.181024+0000","last_active":"2024-03-21T13:00:10.730874+0000","last_peered":"2024-03-21T13:00:10.730874+0000","last_clean":"2024-03-21T13:00:10.730874+0000","last_became_active":"2024-03-21T12:58:43.180773+0000","last_became_peered":"2024-03-21T12:58:43.180773+0000","last_unstale":"2024-03-21T13:00:10.730874+0000","last_undegraded":"2024-03-21T13:00:10.730874+0000","last_fullsized":"2024-03-21T13:00:10.730874+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015405+0000","last_change":"2024-03-21T12:59:50.724334+0000","last_active":"2024-03-21T13:00:11.015405+0000","last_peered":"2024-03-21T13:00:11.015405+0000","last_clean":"2024-03-21T13:00:11.015405+0000","last_became_active":"2024-03-21T12:58:41.167956+0000","last_became_peered":"2024-03-21T12:58:41.167956+0000","last_unstale":"2024-03-21T13:00:11.015405+0000","last_undegraded":"2024-03-21T13:00:11.015405+0000","last_fullsized":"2024-03-21T13:00:11.015405+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:50.724242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.724242+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:53.691391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:14.161211+0000","last_change":"2024-03-21T13:00:14.161251+0000","last_active":"2024-03-21T13:00:14.161211+0000","last_peered":"2024-03-21T13:00:14.161211+0000","last_clean":"2024-03-21T13:00:14.161211+0000","last_became_active":"2024-03-21T12:58:37.731988+0000","last_became_peered":"2024-03-21T12:58:37.731988+0000","last_unstale":"2024-03-21T13:00:14.161211+0000","last_undegraded":"2024-03-21T13:00:14.161211+0000","last_fullsized":"2024-03-21T13:00:14.161211+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:14.161136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:14.161136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:26.647081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"71'10","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781902+0000","last_change":"2024-03-21T13:00:06.198017+0000","last_active":"2024-03-21T13:00:10.781902+0000","last_peered":"2024-03-21T13:00:10.781902+0000","last_clean":"2024-03-21T13:00:10.781902+0000","last_became_active":"2024-03-21T12:58:39.757806+0000","last_became_peered":"2024-03-21T12:58:39.757806+0000","last_unstale":"2024-03-21T13:00:10.781902+0000","last_undegraded":"2024-03-21T13:00:10.781902+0000","last_fullsized":"2024-03-21T13:00:10.781902+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:00:06.197903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.197903+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:24.909636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729358+0000","last_change":"2024-03-21T12:58:43.186611+0000","last_active":"2024-03-21T13:00:10.729358+0000","last_peered":"2024-03-21T13:00:10.729358+0000","last_clean":"2024-03-21T13:00:10.729358+0000","last_became_active":"2024-03-21T12:58:43.186155+0000","last_became_peered":"2024-03-21T12:58:43.186155+0000","last_unstale":"2024-03-21T13:00:10.729358+0000","last_undegraded":"2024-03-21T13:00:10.729358+0000","last_fullsized":"2024-03-21T13:00:10.729358+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.782088+0000","last_change":"2024-03-21T12:58:41.168116+0000","last_active":"2024-03-21T13:00:10.782088+0000","last_peered":"2024-03-21T13:00:10.782088+0000","last_clean":"2024-03-21T13:00:10.782088+0000","last_became_active":"2024-03-21T12:58:41.167826+0000","last_became_peered":"2024-03-21T12:58:41.167826+0000","last_unstale":"2024-03-21T13:00:10.782088+0000","last_undegraded":"2024-03-21T13:00:10.782088+0000","last_fullsized":"2024-03-21T13:00:10.782088+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":59,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.730469+0000","last_change":"2024-03-21T12:58:37.732471+0000","last_active":"2024-03-21T13:00:10.730469+0000","last_peered":"2024-03-21T13:00:10.730469+0000","last_clean":"2024-03-21T13:00:10.730469+0000","last_became_active":"2024-03-21T12:58:37.732166+0000","last_became_peered":"2024-03-21T12:58:37.732166+0000","last_unstale":"2024-03-21T13:00:10.730469+0000","last_undegraded":"2024-03-21T13:00:10.730469+0000","last_fullsized":"2024-03-21T13:00:10.730469+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'9","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015482+0000","last_change":"2024-03-21T12:58:39.751200+0000","last_active":"2024-03-21T13:00:11.015482+0000","last_peered":"2024-03-21T13:00:11.015482+0000","last_clean":"2024-03-21T13:00:11.015482+0000","last_became_active":"2024-03-21T12:58:39.750820+0000","last_became_peered":"2024-03-21T12:58:39.750820+0000","last_unstale":"2024-03-21T13:00:11.015482+0000","last_undegraded":"2024-03-21T13:00:11.015482+0000","last_fullsized":"2024-03-21T13:00:11.015482+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.730577+0000","last_change":"2024-03-21T12:59:51.730551+0000","last_active":"2024-03-21T13:00:10.730577+0000","last_peered":"2024-03-21T13:00:10.730577+0000","last_clean":"2024-03-21T13:00:10.730577+0000","last_became_active":"2024-03-21T12:58:43.182126+0000","last_became_peered":"2024-03-21T12:58:43.182126+0000","last_unstale":"2024-03-21T13:00:10.730577+0000","last_undegraded":"2024-03-21T13:00:10.730577+0000","last_fullsized":"2024-03-21T13:00:10.730577+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T12:59:51.730413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.791726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:12.122039+0000","last_change":"2024-03-21T13:00:12.122078+0000","last_active":"2024-03-21T13:00:12.122039+0000","last_peered":"2024-03-21T13:00:12.122039+0000","last_clean":"2024-03-21T13:00:12.122039+0000","last_became_active":"2024-03-21T12:58:41.172904+0000","last_became_peered":"2024-03-21T12:58:41.172904+0000","last_unstale":"2024-03-21T13:00:12.122039+0000","last_undegraded":"2024-03-21T13:00:12.122039+0000","last_fullsized":"2024-03-21T13:00:12.122039+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:12.121970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:12.121970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:31.573284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014672+0000","last_change":"2024-03-21T12:59:47.203592+0000","last_active":"2024-03-21T13:00:11.014672+0000","last_peered":"2024-03-21T13:00:11.014672+0000","last_clean":"2024-03-21T13:00:11.014672+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:00:11.014672+0000","last_undegraded":"2024-03-21T13:00:11.014672+0000","last_fullsized":"2024-03-21T13:00:11.014672+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T12:59:47.203453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.203453+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.098187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'9","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.731186+0000","last_change":"2024-03-21T13:00:06.685183+0000","last_active":"2024-03-21T13:00:10.731186+0000","last_peered":"2024-03-21T13:00:10.731186+0000","last_clean":"2024-03-21T13:00:10.731186+0000","last_became_active":"2024-03-21T12:58:39.752722+0000","last_became_peered":"2024-03-21T12:58:39.752722+0000","last_unstale":"2024-03-21T13:00:10.731186+0000","last_undegraded":"2024-03-21T13:00:10.731186+0000","last_fullsized":"2024-03-21T13:00:10.731186+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:00:06.685071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.685071+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:35.157380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014762+0000","last_change":"2024-03-21T13:00:08.197105+0000","last_active":"2024-03-21T13:00:11.014762+0000","last_peered":"2024-03-21T13:00:11.014762+0000","last_clean":"2024-03-21T13:00:11.014762+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:00:11.014762+0000","last_undegraded":"2024-03-21T13:00:11.014762+0000","last_fullsized":"2024-03-21T13:00:11.014762+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.196979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.196979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:33.771136+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014654+0000","last_change":"2024-03-21T12:58:41.162672+0000","last_active":"2024-03-21T13:00:11.014654+0000","last_peered":"2024-03-21T13:00:11.014654+0000","last_clean":"2024-03-21T13:00:11.014654+0000","last_became_active":"2024-03-21T12:58:41.162343+0000","last_became_peered":"2024-03-21T12:58:41.162343+0000","last_unstale":"2024-03-21T13:00:11.014654+0000","last_undegraded":"2024-03-21T13:00:11.014654+0000","last_fullsized":"2024-03-21T13:00:11.014654+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.783537+0000","last_change":"2024-03-21T12:59:52.319029+0000","last_active":"2024-03-21T13:00:10.783537+0000","last_peered":"2024-03-21T13:00:10.783537+0000","last_clean":"2024-03-21T13:00:10.783537+0000","last_became_active":"2024-03-21T12:58:37.731932+0000","last_became_peered":"2024-03-21T12:58:37.731932+0000","last_unstale":"2024-03-21T13:00:10.783537+0000","last_undegraded":"2024-03-21T13:00:10.783537+0000","last_fullsized":"2024-03-21T13:00:10.783537+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:52.318927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.318927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:20.356137+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'6","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.782202+0000","last_change":"2024-03-21T12:59:46.176940+0000","last_active":"2024-03-21T13:00:10.782202+0000","last_peered":"2024-03-21T13:00:10.782202+0000","last_clean":"2024-03-21T13:00:10.782202+0000","last_became_active":"2024-03-21T12:58:39.755419+0000","last_became_peered":"2024-03-21T12:58:39.755419+0000","last_unstale":"2024-03-21T13:00:10.782202+0000","last_undegraded":"2024-03-21T13:00:10.782202+0000","last_fullsized":"2024-03-21T13:00:10.782202+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T12:59:46.176819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.176819+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:06.815464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014477+0000","last_change":"2024-03-21T13:00:00.622196+0000","last_active":"2024-03-21T13:00:11.014477+0000","last_peered":"2024-03-21T13:00:11.014477+0000","last_clean":"2024-03-21T13:00:11.014477+0000","last_became_active":"2024-03-21T12:58:43.351103+0000","last_became_peered":"2024-03-21T12:58:43.351103+0000","last_unstale":"2024-03-21T13:00:11.014477+0000","last_undegraded":"2024-03-21T13:00:11.014477+0000","last_fullsized":"2024-03-21T13:00:11.014477+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:00.622094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:00.622094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:21.931294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015609+0000","last_change":"2024-03-21T13:00:06.188217+0000","last_active":"2024-03-21T13:00:11.015609+0000","last_peered":"2024-03-21T13:00:11.015609+0000","last_clean":"2024-03-21T13:00:11.015609+0000","last_became_active":"2024-03-21T12:58:37.735037+0000","last_became_peered":"2024-03-21T12:58:37.735037+0000","last_unstale":"2024-03-21T13:00:11.015609+0000","last_undegraded":"2024-03-21T13:00:11.015609+0000","last_fullsized":"2024-03-21T13:00:11.015609+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:00:06.188118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.188118+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.485685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'15","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014066+0000","last_change":"2024-03-21T12:58:39.750902+0000","last_active":"2024-03-21T13:00:11.014066+0000","last_peered":"2024-03-21T13:00:11.014066+0000","last_clean":"2024-03-21T13:00:11.014066+0000","last_became_active":"2024-03-21T12:58:39.750546+0000","last_became_peered":"2024-03-21T12:58:39.750546+0000","last_unstale":"2024-03-21T13:00:11.014066+0000","last_undegraded":"2024-03-21T13:00:11.014066+0000","last_fullsized":"2024-03-21T13:00:11.014066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"71'8","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014976+0000","last_change":"2024-03-21T12:58:41.166431+0000","last_active":"2024-03-21T13:00:11.014976+0000","last_peered":"2024-03-21T13:00:11.014976+0000","last_clean":"2024-03-21T13:00:11.014976+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:00:11.014976+0000","last_undegraded":"2024-03-21T13:00:11.014976+0000","last_fullsized":"2024-03-21T13:00:11.014976+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781606+0000","last_change":"2024-03-21T12:58:43.181756+0000","last_active":"2024-03-21T13:00:10.781606+0000","last_peered":"2024-03-21T13:00:10.781606+0000","last_clean":"2024-03-21T13:00:10.781606+0000","last_became_active":"2024-03-21T12:58:43.181036+0000","last_became_peered":"2024-03-21T12:58:43.181036+0000","last_unstale":"2024-03-21T13:00:10.781606+0000","last_undegraded":"2024-03-21T13:00:10.781606+0000","last_fullsized":"2024-03-21T13:00:10.781606+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015424+0000","last_change":"2024-03-21T12:59:59.136937+0000","last_active":"2024-03-21T13:00:11.015424+0000","last_peered":"2024-03-21T13:00:11.015424+0000","last_clean":"2024-03-21T13:00:11.015424+0000","last_became_active":"2024-03-21T12:58:37.737012+0000","last_became_peered":"2024-03-21T12:58:37.737012+0000","last_unstale":"2024-03-21T13:00:11.015424+0000","last_undegraded":"2024-03-21T13:00:11.015424+0000","last_fullsized":"2024-03-21T13:00:11.015424+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:59.136829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.136829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:03.237208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"71'12","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728637+0000","last_change":"2024-03-21T12:58:39.751648+0000","last_active":"2024-03-21T13:00:10.728637+0000","last_peered":"2024-03-21T13:00:10.728637+0000","last_clean":"2024-03-21T13:00:10.728637+0000","last_became_active":"2024-03-21T12:58:39.751236+0000","last_became_peered":"2024-03-21T12:58:39.751236+0000","last_unstale":"2024-03-21T13:00:10.728637+0000","last_undegraded":"2024-03-21T13:00:10.728637+0000","last_fullsized":"2024-03-21T13:00:10.728637+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729470+0000","last_change":"2024-03-21T12:59:58.733374+0000","last_active":"2024-03-21T13:00:10.729470+0000","last_peered":"2024-03-21T13:00:10.729470+0000","last_clean":"2024-03-21T13:00:10.729470+0000","last_became_active":"2024-03-21T12:58:41.171443+0000","last_became_peered":"2024-03-21T12:58:41.171443+0000","last_unstale":"2024-03-21T13:00:10.729470+0000","last_undegraded":"2024-03-21T13:00:10.729470+0000","last_fullsized":"2024-03-21T13:00:10.729470+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:58.733235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:58.733235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.428021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.013827+0000","last_change":"2024-03-21T13:00:04.200955+0000","last_active":"2024-03-21T13:00:11.013827+0000","last_peered":"2024-03-21T13:00:11.013827+0000","last_clean":"2024-03-21T13:00:11.013827+0000","last_became_active":"2024-03-21T12:58:43.179007+0000","last_became_peered":"2024-03-21T12:58:43.179007+0000","last_unstale":"2024-03-21T13:00:11.013827+0000","last_undegraded":"2024-03-21T13:00:11.013827+0000","last_fullsized":"2024-03-21T13:00:11.013827+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:04.200849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:04.200849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.029947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.730332+0000","last_change":"2024-03-21T13:00:08.625392+0000","last_active":"2024-03-21T13:00:10.730332+0000","last_peered":"2024-03-21T13:00:10.730332+0000","last_clean":"2024-03-21T13:00:10.730332+0000","last_became_active":"2024-03-21T12:58:41.165203+0000","last_became_peered":"2024-03-21T12:58:41.165203+0000","last_unstale":"2024-03-21T13:00:10.730332+0000","last_undegraded":"2024-03-21T13:00:10.730332+0000","last_fullsized":"2024-03-21T13:00:10.730332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.625230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.625230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:13.968472+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015067+0000","last_change":"2024-03-21T12:59:46.169468+0000","last_active":"2024-03-21T13:00:11.015067+0000","last_peered":"2024-03-21T13:00:11.015067+0000","last_clean":"2024-03-21T13:00:11.015067+0000","last_became_active":"2024-03-21T12:58:37.731448+0000","last_became_peered":"2024-03-21T12:58:37.731448+0000","last_unstale":"2024-03-21T13:00:11.015067+0000","last_undegraded":"2024-03-21T13:00:11.015067+0000","last_fullsized":"2024-03-21T13:00:11.015067+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:46.169315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.169315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:51.199764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"71'19","reported_seq":95,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.783798+0000","last_change":"2024-03-21T12:59:59.392987+0000","last_active":"2024-03-21T13:00:10.783798+0000","last_peered":"2024-03-21T13:00:10.783798+0000","last_clean":"2024-03-21T13:00:10.783798+0000","last_became_active":"2024-03-21T12:58:39.754761+0000","last_became_peered":"2024-03-21T12:58:39.754761+0000","last_unstale":"2024-03-21T13:00:10.783798+0000","last_undegraded":"2024-03-21T13:00:10.783798+0000","last_fullsized":"2024-03-21T13:00:10.783798+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T12:59:59.392834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.392834+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.114737+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729588+0000","last_change":"2024-03-21T12:58:43.180724+0000","last_active":"2024-03-21T13:00:10.729588+0000","last_peered":"2024-03-21T13:00:10.729588+0000","last_clean":"2024-03-21T13:00:10.729588+0000","last_became_active":"2024-03-21T12:58:43.180393+0000","last_became_peered":"2024-03-21T12:58:43.180393+0000","last_unstale":"2024-03-21T13:00:10.729588+0000","last_undegraded":"2024-03-21T13:00:10.729588+0000","last_fullsized":"2024-03-21T13:00:10.729588+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.730157+0000","last_change":"2024-03-21T13:00:01.667365+0000","last_active":"2024-03-21T13:00:10.730157+0000","last_peered":"2024-03-21T13:00:10.730157+0000","last_clean":"2024-03-21T13:00:10.730157+0000","last_became_active":"2024-03-21T12:58:41.163843+0000","last_became_peered":"2024-03-21T12:58:41.163843+0000","last_unstale":"2024-03-21T13:00:10.730157+0000","last_undegraded":"2024-03-21T13:00:10.730157+0000","last_fullsized":"2024-03-21T13:00:10.730157+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:01.667285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.667285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:31.061685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014826+0000","last_change":"2024-03-21T12:59:47.676292+0000","last_active":"2024-03-21T13:00:11.014826+0000","last_peered":"2024-03-21T13:00:11.014826+0000","last_clean":"2024-03-21T13:00:11.014826+0000","last_became_active":"2024-03-21T12:58:37.730518+0000","last_became_peered":"2024-03-21T12:58:37.730518+0000","last_unstale":"2024-03-21T13:00:11.014826+0000","last_undegraded":"2024-03-21T13:00:11.014826+0000","last_fullsized":"2024-03-21T13:00:11.014826+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:47.676198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.676198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.156465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"86'13","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.730776+0000","last_change":"2024-03-21T12:59:46.421228+0000","last_active":"2024-03-21T13:00:10.730776+0000","last_peered":"2024-03-21T13:00:10.730776+0000","last_clean":"2024-03-21T13:00:10.730776+0000","last_became_active":"2024-03-21T12:58:39.748144+0000","last_became_peered":"2024-03-21T12:58:39.748144+0000","last_unstale":"2024-03-21T13:00:10.730776+0000","last_undegraded":"2024-03-21T13:00:10.730776+0000","last_fullsized":"2024-03-21T13:00:10.730776+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.421144+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:06.815237+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781924+0000","last_change":"2024-03-21T12:58:43.180710+0000","last_active":"2024-03-21T13:00:10.781924+0000","last_peered":"2024-03-21T13:00:10.781924+0000","last_clean":"2024-03-21T13:00:10.781924+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:00:10.781924+0000","last_undegraded":"2024-03-21T13:00:10.781924+0000","last_fullsized":"2024-03-21T13:00:10.781924+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:10.959635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"67'2","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014329+0000","last_change":"2024-03-21T12:59:55.541552+0000","last_active":"2024-03-21T13:00:11.014329+0000","last_peered":"2024-03-21T13:00:11.014329+0000","last_clean":"2024-03-21T13:00:11.014329+0000","last_became_active":"2024-03-21T12:58:36.724010+0000","last_became_peered":"2024-03-21T12:58:36.724010+0000","last_unstale":"2024-03-21T13:00:11.014329+0000","last_undegraded":"2024-03-21T13:00:11.014329+0000","last_fullsized":"2024-03-21T13:00:11.014329+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-21T12:59:55.541410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.541410+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:17.828687+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728683+0000","last_change":"2024-03-21T12:59:47.840935+0000","last_active":"2024-03-21T13:00:10.728683+0000","last_peered":"2024-03-21T13:00:10.728683+0000","last_clean":"2024-03-21T13:00:10.728683+0000","last_became_active":"2024-03-21T12:58:37.729454+0000","last_became_peered":"2024-03-21T12:58:37.729454+0000","last_unstale":"2024-03-21T13:00:10.728683+0000","last_undegraded":"2024-03-21T13:00:10.728683+0000","last_fullsized":"2024-03-21T13:00:10.728683+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:47.840831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.840831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.579290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"71'28","reported_seq":132,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836992+0000","last_change":"2024-03-21T13:00:01.917692+0000","last_active":"2024-03-21T13:00:10.836992+0000","last_peered":"2024-03-21T13:00:10.836992+0000","last_clean":"2024-03-21T13:00:10.836992+0000","last_became_active":"2024-03-21T12:58:39.756052+0000","last_became_peered":"2024-03-21T12:58:39.756052+0000","last_unstale":"2024-03-21T13:00:10.836992+0000","last_undegraded":"2024-03-21T13:00:10.836992+0000","last_fullsized":"2024-03-21T13:00:10.836992+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'28","last_scrub_stamp":"2024-03-21T13:00:01.917593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.917593+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:26.189859+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.730925+0000","last_change":"2024-03-21T12:58:41.170763+0000","last_active":"2024-03-21T13:00:10.730925+0000","last_peered":"2024-03-21T13:00:10.730925+0000","last_clean":"2024-03-21T13:00:10.730925+0000","last_became_active":"2024-03-21T12:58:41.170460+0000","last_became_peered":"2024-03-21T12:58:41.170460+0000","last_unstale":"2024-03-21T13:00:10.730925+0000","last_undegraded":"2024-03-21T13:00:10.730925+0000","last_fullsized":"2024-03-21T13:00:10.730925+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015221+0000","last_change":"2024-03-21T12:59:54.191245+0000","last_active":"2024-03-21T13:00:11.015221+0000","last_peered":"2024-03-21T13:00:11.015221+0000","last_clean":"2024-03-21T13:00:11.015221+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:00:11.015221+0000","last_undegraded":"2024-03-21T13:00:11.015221+0000","last_fullsized":"2024-03-21T13:00:11.015221+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:54.191150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:54.191150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:15.445044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836570+0000","last_change":"2024-03-21T12:58:37.733226+0000","last_active":"2024-03-21T13:00:10.836570+0000","last_peered":"2024-03-21T13:00:10.836570+0000","last_clean":"2024-03-21T13:00:10.836570+0000","last_became_active":"2024-03-21T12:58:37.731386+0000","last_became_peered":"2024-03-21T12:58:37.731386+0000","last_unstale":"2024-03-21T13:00:10.836570+0000","last_undegraded":"2024-03-21T13:00:10.836570+0000","last_fullsized":"2024-03-21T13:00:10.836570+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729931+0000","last_change":"2024-03-21T12:59:57.734511+0000","last_active":"2024-03-21T13:00:10.729931+0000","last_peered":"2024-03-21T13:00:10.729931+0000","last_clean":"2024-03-21T13:00:10.729931+0000","last_became_active":"2024-03-21T12:58:36.724067+0000","last_became_peered":"2024-03-21T12:58:36.724067+0000","last_unstale":"2024-03-21T13:00:10.729931+0000","last_undegraded":"2024-03-21T13:00:10.729931+0000","last_fullsized":"2024-03-21T13:00:10.729931+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.734379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.734379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.789374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000743678,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'13","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:13.796147+0000","last_change":"2024-03-21T13:00:13.796181+0000","last_active":"2024-03-21T13:00:13.796147+0000","last_peered":"2024-03-21T13:00:13.796147+0000","last_clean":"2024-03-21T13:00:13.796147+0000","last_became_active":"2024-03-21T12:58:39.750068+0000","last_became_peered":"2024-03-21T12:58:39.750068+0000","last_unstale":"2024-03-21T13:00:13.796147+0000","last_undegraded":"2024-03-21T13:00:13.796147+0000","last_fullsized":"2024-03-21T13:00:13.796147+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'13","last_scrub_stamp":"2024-03-21T13:00:13.796084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:13.796084+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.166815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.730758+0000","last_change":"2024-03-21T13:00:03.688398+0000","last_active":"2024-03-21T13:00:10.730758+0000","last_peered":"2024-03-21T13:00:10.730758+0000","last_clean":"2024-03-21T13:00:10.730758+0000","last_became_active":"2024-03-21T12:58:41.164725+0000","last_became_peered":"2024-03-21T12:58:41.164725+0000","last_unstale":"2024-03-21T13:00:10.730758+0000","last_undegraded":"2024-03-21T13:00:10.730758+0000","last_fullsized":"2024-03-21T13:00:10.730758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:03.688311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:03.688311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:03.827045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014656+0000","last_change":"2024-03-21T12:59:52.658473+0000","last_active":"2024-03-21T13:00:11.014656+0000","last_peered":"2024-03-21T13:00:11.014656+0000","last_clean":"2024-03-21T13:00:11.014656+0000","last_became_active":"2024-03-21T12:58:43.183113+0000","last_became_peered":"2024-03-21T12:58:43.183113+0000","last_unstale":"2024-03-21T13:00:11.014656+0000","last_undegraded":"2024-03-21T13:00:11.014656+0000","last_fullsized":"2024-03-21T13:00:11.014656+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:52.658377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.658377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.565790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":316,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.013961+0000","last_change":"2024-03-21T12:59:02.091964+0000","last_active":"2024-03-21T13:00:11.013961+0000","last_peered":"2024-03-21T13:00:11.013961+0000","last_clean":"2024-03-21T13:00:11.013961+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:00:11.013961+0000","last_undegraded":"2024-03-21T13:00:11.013961+0000","last_fullsized":"2024-03-21T13:00:11.013961+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'119","last_scrub_stamp":"2024-03-21T12:59:02.091835+0000","last_deep_scrub":"76'119","last_deep_scrub_stamp":"2024-03-21T12:59:02.091835+0000","last_clean_scrub_stamp":"2024-03-21T12:59:02.091835+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781685+0000","last_change":"2024-03-21T12:58:37.729254+0000","last_active":"2024-03-21T13:00:10.781685+0000","last_peered":"2024-03-21T13:00:10.781685+0000","last_clean":"2024-03-21T13:00:10.781685+0000","last_became_active":"2024-03-21T12:58:37.728911+0000","last_became_peered":"2024-03-21T12:58:37.728911+0000","last_unstale":"2024-03-21T13:00:10.781685+0000","last_undegraded":"2024-03-21T13:00:10.781685+0000","last_fullsized":"2024-03-21T13:00:10.781685+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":106,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.784096+0000","last_change":"2024-03-21T12:59:56.298894+0000","last_active":"2024-03-21T13:00:10.784096+0000","last_peered":"2024-03-21T13:00:10.784096+0000","last_clean":"2024-03-21T13:00:10.784096+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:00:10.784096+0000","last_undegraded":"2024-03-21T13:00:10.784096+0000","last_fullsized":"2024-03-21T13:00:10.784096+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-21T12:59:56.298768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:56.298768+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.767366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":59,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.121578+0000","last_change":"2024-03-21T13:00:11.121619+0000","last_active":"2024-03-21T13:00:11.121578+0000","last_peered":"2024-03-21T13:00:11.121578+0000","last_clean":"2024-03-21T13:00:11.121578+0000","last_became_active":"2024-03-21T12:58:41.168003+0000","last_became_peered":"2024-03-21T12:58:41.168003+0000","last_unstale":"2024-03-21T13:00:11.121578+0000","last_undegraded":"2024-03-21T13:00:11.121578+0000","last_fullsized":"2024-03-21T13:00:11.121578+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:11.121495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:11.121495+0000","last_clean_scrub_stamp":"2024-03-21T13:00:11.121495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.671209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836582+0000","last_change":"2024-03-21T12:58:43.184189+0000","last_active":"2024-03-21T13:00:10.836582+0000","last_peered":"2024-03-21T13:00:10.836582+0000","last_clean":"2024-03-21T13:00:10.836582+0000","last_became_active":"2024-03-21T12:58:43.183860+0000","last_became_peered":"2024-03-21T12:58:43.183860+0000","last_unstale":"2024-03-21T13:00:10.836582+0000","last_undegraded":"2024-03-21T13:00:10.836582+0000","last_fullsized":"2024-03-21T13:00:10.836582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":59,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729849+0000","last_change":"2024-03-21T12:58:37.734459+0000","last_active":"2024-03-21T13:00:10.729849+0000","last_peered":"2024-03-21T13:00:10.729849+0000","last_clean":"2024-03-21T13:00:10.729849+0000","last_became_active":"2024-03-21T12:58:37.734176+0000","last_became_peered":"2024-03-21T12:58:37.734176+0000","last_unstale":"2024-03-21T13:00:10.729849+0000","last_undegraded":"2024-03-21T13:00:10.729849+0000","last_fullsized":"2024-03-21T13:00:10.729849+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":160,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:13.243336+0000","last_change":"2024-03-21T12:58:40.754485+0000","last_active":"2024-03-21T13:00:13.243336+0000","last_peered":"2024-03-21T13:00:13.243336+0000","last_clean":"2024-03-21T13:00:13.243336+0000","last_became_active":"2024-03-21T12:58:36.725735+0000","last_became_peered":"2024-03-21T12:58:36.725735+0000","last_unstale":"2024-03-21T13:00:13.243336+0000","last_undegraded":"2024-03-21T13:00:13.243336+0000","last_fullsized":"2024-03-21T13:00:13.243336+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:58:35.713454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.730976+0000","last_change":"2024-03-21T13:00:01.563940+0000","last_active":"2024-03-21T13:00:10.730976+0000","last_peered":"2024-03-21T13:00:10.730976+0000","last_clean":"2024-03-21T13:00:10.730976+0000","last_became_active":"2024-03-21T12:58:39.747315+0000","last_became_peered":"2024-03-21T12:58:39.747315+0000","last_unstale":"2024-03-21T13:00:10.730976+0000","last_undegraded":"2024-03-21T13:00:10.730976+0000","last_fullsized":"2024-03-21T13:00:10.730976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:00:01.563844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.563844+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:12.072972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014942+0000","last_change":"2024-03-21T13:00:10.646160+0000","last_active":"2024-03-21T13:00:11.014942+0000","last_peered":"2024-03-21T13:00:11.014942+0000","last_clean":"2024-03-21T13:00:11.014942+0000","last_became_active":"2024-03-21T12:58:41.167131+0000","last_became_peered":"2024-03-21T12:58:41.167131+0000","last_unstale":"2024-03-21T13:00:11.014942+0000","last_undegraded":"2024-03-21T13:00:11.014942+0000","last_fullsized":"2024-03-21T13:00:11.014942+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:10.646083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.646083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.616835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.888250+0000","last_change":"2024-03-21T13:00:11.888272+0000","last_active":"2024-03-21T13:00:11.888250+0000","last_peered":"2024-03-21T13:00:11.888250+0000","last_clean":"2024-03-21T13:00:11.888250+0000","last_became_active":"2024-03-21T12:58:43.351138+0000","last_became_peered":"2024-03-21T12:58:43.351138+0000","last_unstale":"2024-03-21T13:00:11.888250+0000","last_undegraded":"2024-03-21T13:00:11.888250+0000","last_fullsized":"2024-03-21T13:00:11.888250+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:11.888191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:11.888191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.059106+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.836809+0000","last_change":"2024-03-21T12:59:57.925311+0000","last_active":"2024-03-21T13:00:10.836809+0000","last_peered":"2024-03-21T13:00:10.836809+0000","last_clean":"2024-03-21T13:00:10.836809+0000","last_became_active":"2024-03-21T12:58:37.729818+0000","last_became_peered":"2024-03-21T12:58:37.729818+0000","last_unstale":"2024-03-21T13:00:10.836809+0000","last_undegraded":"2024-03-21T13:00:10.836809+0000","last_fullsized":"2024-03-21T13:00:10.836809+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.925179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.925179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.845352+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"71'19","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.730643+0000","last_change":"2024-03-21T12:58:39.751068+0000","last_active":"2024-03-21T13:00:10.730643+0000","last_peered":"2024-03-21T13:00:10.730643+0000","last_clean":"2024-03-21T13:00:10.730643+0000","last_became_active":"2024-03-21T12:58:39.750216+0000","last_became_peered":"2024-03-21T12:58:39.750216+0000","last_unstale":"2024-03-21T13:00:10.730643+0000","last_undegraded":"2024-03-21T13:00:10.730643+0000","last_fullsized":"2024-03-21T13:00:10.730643+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.783878+0000","last_change":"2024-03-21T13:00:06.485807+0000","last_active":"2024-03-21T13:00:10.783878+0000","last_peered":"2024-03-21T13:00:10.783878+0000","last_clean":"2024-03-21T13:00:10.783878+0000","last_became_active":"2024-03-21T12:58:41.166495+0000","last_became_peered":"2024-03-21T12:58:41.166495+0000","last_unstale":"2024-03-21T13:00:10.783878+0000","last_undegraded":"2024-03-21T13:00:10.783878+0000","last_fullsized":"2024-03-21T13:00:10.783878+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:06.485726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.485726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.711578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729398+0000","last_change":"2024-03-21T13:00:05.703953+0000","last_active":"2024-03-21T13:00:10.729398+0000","last_peered":"2024-03-21T13:00:10.729398+0000","last_clean":"2024-03-21T13:00:10.729398+0000","last_became_active":"2024-03-21T12:58:43.177409+0000","last_became_peered":"2024-03-21T12:58:43.177409+0000","last_unstale":"2024-03-21T13:00:10.729398+0000","last_undegraded":"2024-03-21T13:00:10.729398+0000","last_fullsized":"2024-03-21T13:00:10.729398+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:05.703839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:05.703839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:29.696086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015352+0000","last_change":"2024-03-21T12:58:37.730839+0000","last_active":"2024-03-21T13:00:11.015352+0000","last_peered":"2024-03-21T13:00:11.015352+0000","last_clean":"2024-03-21T13:00:11.015352+0000","last_became_active":"2024-03-21T12:58:37.730547+0000","last_became_peered":"2024-03-21T12:58:37.730547+0000","last_unstale":"2024-03-21T13:00:11.015352+0000","last_undegraded":"2024-03-21T13:00:11.015352+0000","last_fullsized":"2024-03-21T13:00:11.015352+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"86'15","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729864+0000","last_change":"2024-03-21T12:59:45.653329+0000","last_active":"2024-03-21T13:00:10.729864+0000","last_peered":"2024-03-21T13:00:10.729864+0000","last_clean":"2024-03-21T13:00:10.729864+0000","last_became_active":"2024-03-21T12:59:01.160456+0000","last_became_peered":"2024-03-21T12:59:01.160456+0000","last_unstale":"2024-03-21T13:00:10.729864+0000","last_undegraded":"2024-03-21T13:00:10.729864+0000","last_fullsized":"2024-03-21T13:00:10.729864+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.653197+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.205091+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.730045+0000","last_change":"2024-03-21T12:59:48.409996+0000","last_active":"2024-03-21T13:00:10.730045+0000","last_peered":"2024-03-21T13:00:10.730045+0000","last_clean":"2024-03-21T13:00:10.730045+0000","last_became_active":"2024-03-21T12:58:41.171770+0000","last_became_peered":"2024-03-21T12:58:41.171770+0000","last_unstale":"2024-03-21T13:00:10.730045+0000","last_undegraded":"2024-03-21T13:00:10.730045+0000","last_fullsized":"2024-03-21T13:00:10.730045+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:48.409905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.409905+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.077210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014648+0000","last_change":"2024-03-21T12:58:43.353379+0000","last_active":"2024-03-21T13:00:11.014648+0000","last_peered":"2024-03-21T13:00:11.014648+0000","last_clean":"2024-03-21T13:00:11.014648+0000","last_became_active":"2024-03-21T12:58:43.353049+0000","last_became_peered":"2024-03-21T12:58:43.353049+0000","last_unstale":"2024-03-21T13:00:11.014648+0000","last_undegraded":"2024-03-21T13:00:11.014648+0000","last_fullsized":"2024-03-21T13:00:11.014648+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.731035+0000","last_change":"2024-03-21T12:58:37.735715+0000","last_active":"2024-03-21T13:00:10.731035+0000","last_peered":"2024-03-21T13:00:10.731035+0000","last_clean":"2024-03-21T13:00:10.731035+0000","last_became_active":"2024-03-21T12:58:37.735421+0000","last_became_peered":"2024-03-21T12:58:37.735421+0000","last_unstale":"2024-03-21T13:00:10.731035+0000","last_undegraded":"2024-03-21T13:00:10.731035+0000","last_fullsized":"2024-03-21T13:00:10.731035+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"87'21","reported_seq":92,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728506+0000","last_change":"2024-03-21T12:59:52.916371+0000","last_active":"2024-03-21T13:00:10.728506+0000","last_peered":"2024-03-21T13:00:10.728506+0000","last_clean":"2024-03-21T13:00:10.728506+0000","last_became_active":"2024-03-21T12:58:39.743592+0000","last_became_peered":"2024-03-21T12:58:39.743592+0000","last_unstale":"2024-03-21T13:00:10.728506+0000","last_undegraded":"2024-03-21T13:00:10.728506+0000","last_fullsized":"2024-03-21T13:00:10.728506+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'21","last_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.916260+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.636952+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"71'8","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781529+0000","last_change":"2024-03-21T12:58:41.174471+0000","last_active":"2024-03-21T13:00:10.781529+0000","last_peered":"2024-03-21T13:00:10.781529+0000","last_clean":"2024-03-21T13:00:10.781529+0000","last_became_active":"2024-03-21T12:58:41.174137+0000","last_became_peered":"2024-03-21T12:58:41.174137+0000","last_unstale":"2024-03-21T13:00:10.781529+0000","last_undegraded":"2024-03-21T13:00:10.781529+0000","last_fullsized":"2024-03-21T13:00:10.781529+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728509+0000","last_change":"2024-03-21T13:00:08.823941+0000","last_active":"2024-03-21T13:00:10.728509+0000","last_peered":"2024-03-21T13:00:10.728509+0000","last_clean":"2024-03-21T13:00:10.728509+0000","last_became_active":"2024-03-21T12:58:43.180939+0000","last_became_peered":"2024-03-21T12:58:43.180939+0000","last_unstale":"2024-03-21T13:00:10.728509+0000","last_undegraded":"2024-03-21T13:00:10.728509+0000","last_fullsized":"2024-03-21T13:00:10.728509+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.823841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:28.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.782151+0000","last_change":"2024-03-21T12:59:43.240826+0000","last_active":"2024-03-21T13:00:10.782151+0000","last_peered":"2024-03-21T13:00:10.782151+0000","last_clean":"2024-03-21T13:00:10.782151+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:00:10.782151+0000","last_undegraded":"2024-03-21T13:00:10.782151+0000","last_fullsized":"2024-03-21T13:00:10.782151+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T12:59:43.240674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.075915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"71'18","reported_seq":96,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781462+0000","last_change":"2024-03-21T12:59:48.179286+0000","last_active":"2024-03-21T13:00:10.781462+0000","last_peered":"2024-03-21T13:00:10.781462+0000","last_clean":"2024-03-21T13:00:10.781462+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:00:10.781462+0000","last_undegraded":"2024-03-21T13:00:10.781462+0000","last_fullsized":"2024-03-21T13:00:10.781462+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T12:59:48.179188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.179188+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:15.921161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728351+0000","last_change":"2024-03-21T12:58:41.165071+0000","last_active":"2024-03-21T13:00:10.728351+0000","last_peered":"2024-03-21T13:00:10.728351+0000","last_clean":"2024-03-21T13:00:10.728351+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:00:10.728351+0000","last_undegraded":"2024-03-21T13:00:10.728351+0000","last_fullsized":"2024-03-21T13:00:10.728351+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.015144+0000","last_change":"2024-03-21T12:59:57.599220+0000","last_active":"2024-03-21T13:00:11.015144+0000","last_peered":"2024-03-21T13:00:11.015144+0000","last_clean":"2024-03-21T13:00:11.015144+0000","last_became_active":"2024-03-21T12:58:43.183614+0000","last_became_peered":"2024-03-21T12:58:43.183614+0000","last_unstale":"2024-03-21T13:00:11.015144+0000","last_undegraded":"2024-03-21T13:00:11.015144+0000","last_fullsized":"2024-03-21T13:00:11.015144+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.599076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.599076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:00.058326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729352+0000","last_change":"2024-03-21T12:58:41.172355+0000","last_active":"2024-03-21T13:00:10.729352+0000","last_peered":"2024-03-21T13:00:10.729352+0000","last_clean":"2024-03-21T13:00:10.729352+0000","last_became_active":"2024-03-21T12:58:41.172019+0000","last_became_peered":"2024-03-21T12:58:41.172019+0000","last_unstale":"2024-03-21T13:00:10.729352+0000","last_undegraded":"2024-03-21T13:00:10.729352+0000","last_fullsized":"2024-03-21T13:00:10.729352+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781381+0000","last_change":"2024-03-21T12:59:41.231430+0000","last_active":"2024-03-21T13:00:10.781381+0000","last_peered":"2024-03-21T13:00:10.781381+0000","last_clean":"2024-03-21T13:00:10.781381+0000","last_became_active":"2024-03-21T12:58:37.731327+0000","last_became_peered":"2024-03-21T12:58:37.731327+0000","last_unstale":"2024-03-21T13:00:10.781381+0000","last_undegraded":"2024-03-21T13:00:10.781381+0000","last_fullsized":"2024-03-21T13:00:10.781381+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:41.231340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:41.231340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.247538+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.837206+0000","last_change":"2024-03-21T12:58:39.755619+0000","last_active":"2024-03-21T13:00:10.837206+0000","last_peered":"2024-03-21T13:00:10.837206+0000","last_clean":"2024-03-21T13:00:10.837206+0000","last_became_active":"2024-03-21T12:58:39.755326+0000","last_became_peered":"2024-03-21T12:58:39.755326+0000","last_unstale":"2024-03-21T13:00:10.837206+0000","last_undegraded":"2024-03-21T13:00:10.837206+0000","last_fullsized":"2024-03-21T13:00:10.837206+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781606+0000","last_change":"2024-03-21T12:58:43.351417+0000","last_active":"2024-03-21T13:00:10.781606+0000","last_peered":"2024-03-21T13:00:10.781606+0000","last_clean":"2024-03-21T13:00:10.781606+0000","last_became_active":"2024-03-21T12:58:43.350921+0000","last_became_peered":"2024-03-21T12:58:43.350921+0000","last_unstale":"2024-03-21T13:00:10.781606+0000","last_undegraded":"2024-03-21T13:00:10.781606+0000","last_fullsized":"2024-03-21T13:00:10.781606+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.014889+0000","last_change":"2024-03-21T13:00:08.639131+0000","last_active":"2024-03-21T13:00:11.014889+0000","last_peered":"2024-03-21T13:00:11.014889+0000","last_clean":"2024-03-21T13:00:11.014889+0000","last_became_active":"2024-03-21T12:58:41.168784+0000","last_became_peered":"2024-03-21T12:58:41.168784+0000","last_unstale":"2024-03-21T13:00:11.014889+0000","last_undegraded":"2024-03-21T13:00:11.014889+0000","last_fullsized":"2024-03-21T13:00:11.014889+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.639041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.639041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:17.524978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.728385+0000","last_change":"2024-03-21T12:58:37.731156+0000","last_active":"2024-03-21T13:00:10.728385+0000","last_peered":"2024-03-21T13:00:10.728385+0000","last_clean":"2024-03-21T13:00:10.728385+0000","last_became_active":"2024-03-21T12:58:37.730858+0000","last_became_peered":"2024-03-21T12:58:37.730858+0000","last_unstale":"2024-03-21T13:00:10.728385+0000","last_undegraded":"2024-03-21T13:00:10.728385+0000","last_fullsized":"2024-03-21T13:00:10.728385+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'11","reported_seq":75,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729188+0000","last_change":"2024-03-21T12:58:39.754932+0000","last_active":"2024-03-21T13:00:10.729188+0000","last_peered":"2024-03-21T13:00:10.729188+0000","last_clean":"2024-03-21T13:00:10.729188+0000","last_became_active":"2024-03-21T12:58:39.754337+0000","last_became_peered":"2024-03-21T12:58:39.754337+0000","last_unstale":"2024-03-21T13:00:10.729188+0000","last_undegraded":"2024-03-21T13:00:10.729188+0000","last_fullsized":"2024-03-21T13:00:10.729188+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.731021+0000","last_change":"2024-03-21T12:58:43.183848+0000","last_active":"2024-03-21T13:00:10.731021+0000","last_peered":"2024-03-21T13:00:10.731021+0000","last_clean":"2024-03-21T13:00:10.731021+0000","last_became_active":"2024-03-21T12:58:43.183380+0000","last_became_peered":"2024-03-21T12:58:43.183380+0000","last_unstale":"2024-03-21T13:00:10.731021+0000","last_undegraded":"2024-03-21T13:00:10.731021+0000","last_fullsized":"2024-03-21T13:00:10.731021+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:11.881657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.837068+0000","last_change":"2024-03-21T12:59:50.039168+0000","last_active":"2024-03-21T13:00:10.837068+0000","last_peered":"2024-03-21T13:00:10.837068+0000","last_clean":"2024-03-21T13:00:10.837068+0000","last_became_active":"2024-03-21T12:58:41.163145+0000","last_became_peered":"2024-03-21T12:58:41.163145+0000","last_unstale":"2024-03-21T13:00:10.837068+0000","last_undegraded":"2024-03-21T13:00:10.837068+0000","last_fullsized":"2024-03-21T13:00:10.837068+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.039091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.039091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.361130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.783929+0000","last_change":"2024-03-21T12:59:49.204557+0000","last_active":"2024-03-21T13:00:10.783929+0000","last_peered":"2024-03-21T13:00:10.783929+0000","last_clean":"2024-03-21T13:00:10.783929+0000","last_became_active":"2024-03-21T12:58:37.729792+0000","last_became_peered":"2024-03-21T12:58:37.729792+0000","last_unstale":"2024-03-21T13:00:10.783929+0000","last_undegraded":"2024-03-21T13:00:10.783929+0000","last_fullsized":"2024-03-21T13:00:10.783929+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:49.204419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:49.204419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.549632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":84,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729058+0000","last_change":"2024-03-21T12:58:39.746134+0000","last_active":"2024-03-21T13:00:10.729058+0000","last_peered":"2024-03-21T13:00:10.729058+0000","last_clean":"2024-03-21T13:00:10.729058+0000","last_became_active":"2024-03-21T12:58:39.745807+0000","last_became_peered":"2024-03-21T12:58:39.745807+0000","last_unstale":"2024-03-21T13:00:10.729058+0000","last_undegraded":"2024-03-21T13:00:10.729058+0000","last_fullsized":"2024-03-21T13:00:10.729058+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.729305+0000","last_change":"2024-03-21T12:58:43.185167+0000","last_active":"2024-03-21T13:00:10.729305+0000","last_peered":"2024-03-21T13:00:10.729305+0000","last_clean":"2024-03-21T13:00:10.729305+0000","last_became_active":"2024-03-21T12:58:43.184576+0000","last_became_peered":"2024-03-21T12:58:43.184576+0000","last_unstale":"2024-03-21T13:00:10.729305+0000","last_undegraded":"2024-03-21T13:00:10.729305+0000","last_fullsized":"2024-03-21T13:00:10.729305+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"71'8","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:11.614182+0000","last_change":"2024-03-21T13:00:11.614182+0000","last_active":"2024-03-21T13:00:11.614182+0000","last_peered":"2024-03-21T13:00:11.614182+0000","last_clean":"2024-03-21T13:00:11.614182+0000","last_became_active":"2024-03-21T12:58:41.165560+0000","last_became_peered":"2024-03-21T12:58:41.165560+0000","last_unstale":"2024-03-21T13:00:11.614182+0000","last_undegraded":"2024-03-21T13:00:11.614182+0000","last_fullsized":"2024-03-21T13:00:11.614182+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:00:11.614084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:11.614084+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:28.325286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.781836+0000","last_change":"2024-03-21T12:58:37.732264+0000","last_active":"2024-03-21T13:00:10.781836+0000","last_peered":"2024-03-21T13:00:10.781836+0000","last_clean":"2024-03-21T13:00:10.781836+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:00:10.781836+0000","last_undegraded":"2024-03-21T13:00:10.781836+0000","last_fullsized":"2024-03-21T13:00:10.781836+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T13:00:10.785829+0000","last_change":"2024-03-21T13:00:10.785895+0000","last_active":"2024-03-21T13:00:10.785829+0000","last_peered":"2024-03-21T13:00:10.785829+0000","last_clean":"2024-03-21T13:00:10.785829+0000","last_became_active":"2024-03-21T12:58:39.749956+0000","last_became_peered":"2024-03-21T12:58:39.749956+0000","last_unstale":"2024-03-21T13:00:10.785829+0000","last_undegraded":"2024-03-21T13:00:10.785829+0000","last_fullsized":"2024-03-21T13:00:10.785829+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:00:10.785752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.785752+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:30.125302+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":415,"ondisk_log_size":415,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168600,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1388544,"data_stored":946614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462942,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32644,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712508,"statfs":{"total":95995035648,"available":95961608192,"internally_reserved":0,"allocated":1376256,"data_stored":946011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31980,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713172,"statfs":{"total":95995035648,"available":95962288128,"internally_reserved":0,"allocated":823296,"data_stored":355738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1388544,"data_stored":945445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31940,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713212,"statfs":{"total":95995035648,"available":95962329088,"internally_reserved":0,"allocated":782336,"data_stored":355164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":815104,"data_stored":355623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018919,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":802816,"data_stored":355799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724578,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":778240,"data_stored":354622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:00:20.669 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-21T13:00:20.670 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd dump 2024-03-21T13:00:20.864 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:20 smithi163 podman[76112]: 2024-03-21 13:00:20.520428124 +0000 UTC m=+4.746451986 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-21T13:00:20.864 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:20 smithi163 podman[76112]: 2024-03-21 13:00:20.758388882 +0000 UTC m=+4.984412736 container create ad11847ba7c4e03477d31c86aee11c3ac9dc24a199d5159f69ae0e4c6f40add8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:00:20.864 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[29621]: 3.15 scrub starts 2024-03-21T13:00:20.864 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[29621]: 3.15 scrub ok 2024-03-21T13:00:20.864 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[29621]: 6.f deep-scrub starts 2024-03-21T13:00:20.864 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[29621]: 6.f deep-scrub ok 2024-03-21T13:00:20.864 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[29621]: 4.15 scrub starts 2024-03-21T13:00:20.865 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[29621]: 4.15 scrub ok 2024-03-21T13:00:20.865 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[29621]: 4.18 deep-scrub starts 2024-03-21T13:00:20.865 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[40675]: 3.15 scrub starts 2024-03-21T13:00:20.865 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[40675]: 3.15 scrub ok 2024-03-21T13:00:20.865 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[40675]: 6.f deep-scrub starts 2024-03-21T13:00:20.865 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[40675]: 6.f deep-scrub ok 2024-03-21T13:00:20.865 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[40675]: 4.15 scrub starts 2024-03-21T13:00:20.865 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[40675]: 4.15 scrub ok 2024-03-21T13:00:20.865 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:20 smithi163 ceph-mon[40675]: 4.18 deep-scrub starts 2024-03-21T13:00:21.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:20 smithi204 ceph-mon[33153]: 3.15 scrub starts 2024-03-21T13:00:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:20 smithi204 ceph-mon[33153]: 3.15 scrub ok 2024-03-21T13:00:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:20 smithi204 ceph-mon[33153]: 6.f deep-scrub starts 2024-03-21T13:00:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:20 smithi204 ceph-mon[33153]: 6.f deep-scrub ok 2024-03-21T13:00:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:20 smithi204 ceph-mon[33153]: 4.15 scrub starts 2024-03-21T13:00:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:20 smithi204 ceph-mon[33153]: 4.15 scrub ok 2024-03-21T13:00:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:20 smithi204 ceph-mon[33153]: 4.18 deep-scrub starts 2024-03-21T13:00:21.167 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:21.261 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:20 smithi163 podman[76112]: 2024-03-21 13:00:20.994016951 +0000 UTC m=+5.220040820 container init ad11847ba7c4e03477d31c86aee11c3ac9dc24a199d5159f69ae0e4c6f40add8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:00:21.261 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:20 smithi163 podman[76112]: 2024-03-21 13:00:20.99674975 +0000 UTC m=+5.222773607 container start ad11847ba7c4e03477d31c86aee11c3ac9dc24a199d5159f69ae0e4c6f40add8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:00:21.261 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.000Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-21T13:00:21.261 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.000Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-21T13:00:21.261 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-21T13:00:21.261 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T13:00:21.262 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-21T13:00:21.262 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-21T13:00:21.262 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-21T13:00:21.262 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=arp 2024-03-21T13:00:21.262 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-21T13:00:21.262 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-21T13:00:21.262 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-21T13:00:21.262 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-21T13:00:21.262 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-21T13:00:21.262 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-21T13:00:21.263 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-21T13:00:21.263 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-21T13:00:21.263 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=edac 2024-03-21T13:00:21.263 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-21T13:00:21.263 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-21T13:00:21.263 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-21T13:00:21.263 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-21T13:00:21.263 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-21T13:00:21.264 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-21T13:00:21.264 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-21T13:00:21.264 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-21T13:00:21.264 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-21T13:00:21.264 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-21T13:00:21.264 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-21T13:00:21.264 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-21T13:00:21.264 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-21T13:00:21.264 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-21T13:00:21.264 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-21T13:00:21.264 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-21T13:00:21.264 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=os 2024-03-21T13:00:21.265 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-21T13:00:21.265 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-21T13:00:21.265 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-21T13:00:21.265 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-21T13:00:21.265 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-21T13:00:21.265 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-21T13:00:21.265 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-21T13:00:21.265 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=stat 2024-03-21T13:00:21.265 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-21T13:00:21.265 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-21T13:00:21.265 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-21T13:00:21.265 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=time 2024-03-21T13:00:21.266 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-21T13:00:21.266 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=uname 2024-03-21T13:00:21.266 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-21T13:00:21.266 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-21T13:00:21.266 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.001Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-21T13:00:21.266 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.002Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-21T13:00:21.266 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a[76357]: ts=2024-03-21T13:00:21.002Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-21T13:00:21.266 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 bash[76112]: ad11847ba7c4e03477d31c86aee11c3ac9dc24a199d5159f69ae0e4c6f40add8 2024-03-21T13:00:21.266 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:00:21 smithi163 systemd[1]: Started Ceph node-exporter.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T13:00:21.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:00:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: 4.18 deep-scrub ok 2024-03-21T13:00:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: 5.9 deep-scrub starts 2024-03-21T13:00:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: 5.9 deep-scrub ok 2024-03-21T13:00:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: 3.5 scrub starts 2024-03-21T13:00:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: 3.5 scrub ok 2024-03-21T13:00:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: 6.e scrub starts 2024-03-21T13:00:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: 6.e scrub ok 2024-03-21T13:00:22.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: 4.3 scrub starts 2024-03-21T13:00:22.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: 4.3 scrub ok 2024-03-21T13:00:22.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: 6.4 scrub starts 2024-03-21T13:00:22.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: 6.4 scrub ok 2024-03-21T13:00:22.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:22.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:22.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:22.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: Deploying daemon node-exporter.b on smithi204 2024-03-21T13:00:22.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:21 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:22.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: 4.18 deep-scrub ok 2024-03-21T13:00:22.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: 5.9 deep-scrub starts 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: 5.9 deep-scrub ok 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: 3.5 scrub starts 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: 3.5 scrub ok 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: 6.e scrub starts 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: 6.e scrub ok 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: 4.3 scrub starts 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: 4.3 scrub ok 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: 6.4 scrub starts 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: 6.4 scrub ok 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:22.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: Deploying daemon node-exporter.b on smithi204 2024-03-21T13:00:22.014 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:22.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: 4.18 deep-scrub ok 2024-03-21T13:00:22.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: 5.9 deep-scrub starts 2024-03-21T13:00:22.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: 5.9 deep-scrub ok 2024-03-21T13:00:22.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:22.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: 3.5 scrub starts 2024-03-21T13:00:22.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: 3.5 scrub ok 2024-03-21T13:00:22.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: 6.e scrub starts 2024-03-21T13:00:22.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: 6.e scrub ok 2024-03-21T13:00:22.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: 4.3 scrub starts 2024-03-21T13:00:22.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: 4.3 scrub ok 2024-03-21T13:00:22.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: 6.4 scrub starts 2024-03-21T13:00:22.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: 6.4 scrub ok 2024-03-21T13:00:22.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:22.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:22.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:22.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: Deploying daemon node-exporter.b on smithi204 2024-03-21T13:00:22.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:21 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:23.373 INFO:teuthology.orchestra.run.smithi163.stdout:epoch 89 2024-03-21T13:00:23.373 INFO:teuthology.orchestra.run.smithi163.stdout:fsid d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T13:00:23.373 INFO:teuthology.orchestra.run.smithi163.stdout:created 2024-03-21T12:52:56.031419+0000 2024-03-21T13:00:23.373 INFO:teuthology.orchestra.run.smithi163.stdout:modified 2024-03-21T13:00:10.722630+0000 2024-03-21T13:00:23.373 INFO:teuthology.orchestra.run.smithi163.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-21T13:00:23.373 INFO:teuthology.orchestra.run.smithi163.stdout:crush_version 18 2024-03-21T13:00:23.373 INFO:teuthology.orchestra.run.smithi163.stdout:full_ratio 0.95 2024-03-21T13:00:23.373 INFO:teuthology.orchestra.run.smithi163.stdout:backfillfull_ratio 0.9 2024-03-21T13:00:23.373 INFO:teuthology.orchestra.run.smithi163.stdout:nearfull_ratio 0.85 2024-03-21T13:00:23.373 INFO:teuthology.orchestra.run.smithi163.stdout:require_min_compat_client luminous 2024-03-21T13:00:23.374 INFO:teuthology.orchestra.run.smithi163.stdout:min_compat_client luminous 2024-03-21T13:00:23.374 INFO:teuthology.orchestra.run.smithi163.stdout:require_osd_release squid 2024-03-21T13:00:23.374 INFO:teuthology.orchestra.run.smithi163.stdout:stretch_mode_enabled false 2024-03-21T13:00:23.374 INFO:teuthology.orchestra.run.smithi163.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-21T13:00:23.374 INFO:teuthology.orchestra.run.smithi163.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 67 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-21T13:00:23.374 INFO:teuthology.orchestra.run.smithi163.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 65 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-21T13:00:23.374 INFO:teuthology.orchestra.run.smithi163.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 67 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-21T13:00:23.374 INFO:teuthology.orchestra.run.smithi163.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 69 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-21T13:00:23.374 INFO:teuthology.orchestra.run.smithi163.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-21T13:00:23.374 INFO:teuthology.orchestra.run.smithi163.stdout:max_osd 8 2024-03-21T13:00:23.374 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0 up in weight 1 up_from 8 up_thru 69 down_at 0 last_clean_interval [0,0) v2:172.21.15.163:6801/3086410981 v2:172.21.15.163:6802/3086410981 exists,up c34a3b25-9e02-46e6-8f9a-d0af2615258f 2024-03-21T13:00:23.374 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1 up in weight 1 up_from 13 up_thru 69 down_at 0 last_clean_interval [0,0) v2:172.21.15.163:6805/3666062784 v2:172.21.15.163:6806/3666062784 exists,up 36e05f89-c238-41f7-be65-1553b529e8ce 2024-03-21T13:00:23.374 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2 up in weight 1 up_from 18 up_thru 75 down_at 0 last_clean_interval [0,0) v2:172.21.15.163:6809/4009213507 v2:172.21.15.163:6810/4009213507 exists,up 968ec78d-25b8-4905-b328-d490c38ca85f 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3 up in weight 1 up_from 25 up_thru 75 down_at 0 last_clean_interval [0,0) v2:172.21.15.163:6813/3981878189 v2:172.21.15.163:6814/3981878189 exists,up 67dfa4f6-1709-48f6-971d-c6287233492d 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4 up in weight 1 up_from 30 up_thru 75 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6800/719276054 v2:172.21.15.204:6801/719276054 exists,up 7fd311a9-f3c2-456d-8ab1-341cf5b33471 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5 up in weight 1 up_from 39 up_thru 69 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6804/1346947668 v2:172.21.15.204:6805/1346947668 exists,up a87759c8-251c-415d-9387-a49f7d2f36d1 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6 up in weight 1 up_from 47 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6808/200077988 v2:172.21.15.204:6809/200077988 exists,up 5eb135cd-f677-4d0c-b23a-738f31615b6b 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7 up in weight 1 up_from 56 up_thru 69 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6812/315664888 v2:172.21.15.204:6813/315664888 exists,up 4f528e10-2c93-4bc2-a54c-06ca4132ed49 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:pg_upmap_items 4.2 [1,2] 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:pg_upmap_items 4.18 [1,2] 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:pg_upmap_items 4.1d [1,2] 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/3453855427 expires 2024-03-22T12:59:18.599967+0000 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/2571244897 expires 2024-03-22T12:59:18.599967+0000 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:6800/3073378854 expires 2024-03-22T12:59:18.599967+0000 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/1334591766 expires 2024-03-22T12:59:18.599967+0000 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/116934309 expires 2024-03-22T12:54:00.224135+0000 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/3890775846 expires 2024-03-22T12:59:18.599967+0000 2024-03-21T13:00:23.375 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/1783921524 expires 2024-03-22T12:54:00.224135+0000 2024-03-21T13:00:23.376 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:6800/2553440211 expires 2024-03-22T12:53:32.790116+0000 2024-03-21T13:00:23.376 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/1812694077 expires 2024-03-22T12:53:32.790116+0000 2024-03-21T13:00:23.376 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/1070540283 expires 2024-03-22T12:53:32.790116+0000 2024-03-21T13:00:23.376 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/2354157786 expires 2024-03-22T12:53:32.790116+0000 2024-03-21T13:00:23.376 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/1591817392 expires 2024-03-22T12:59:18.599967+0000 2024-03-21T13:00:23.376 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/4152198951 expires 2024-03-22T12:54:00.224135+0000 2024-03-21T13:00:23.376 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:6800/1798115633 expires 2024-03-22T12:54:00.224135+0000 2024-03-21T13:00:23.503 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:23 smithi204 bash[64295]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-21T13:00:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:23 smithi204 ceph-mon[33153]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-03-21T13:00:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:23 smithi204 ceph-mon[33153]: 6.9 scrub starts 2024-03-21T13:00:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:23 smithi204 ceph-mon[33153]: 6.9 scrub ok 2024-03-21T13:00:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:23 smithi204 ceph-mon[33153]: 3.0 scrub starts 2024-03-21T13:00:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:23 smithi204 ceph-mon[33153]: 3.0 scrub ok 2024-03-21T13:00:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:23 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/276649258' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-21T13:00:24.003 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:23 smithi204 bash[64295]: Getting image source signatures 2024-03-21T13:00:24.003 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:23 smithi204 bash[64295]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-21T13:00:24.003 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:23 smithi204 bash[64295]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-21T13:00:24.003 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:23 smithi204 bash[64295]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-21T13:00:24.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:23 smithi163 ceph-mon[29621]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-03-21T13:00:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:23 smithi163 ceph-mon[29621]: 6.9 scrub starts 2024-03-21T13:00:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:23 smithi163 ceph-mon[29621]: 6.9 scrub ok 2024-03-21T13:00:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:23 smithi163 ceph-mon[29621]: 3.0 scrub starts 2024-03-21T13:00:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:23 smithi163 ceph-mon[29621]: 3.0 scrub ok 2024-03-21T13:00:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:23 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/276649258' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-21T13:00:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:23 smithi163 ceph-mon[40675]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-03-21T13:00:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:23 smithi163 ceph-mon[40675]: 6.9 scrub starts 2024-03-21T13:00:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:23 smithi163 ceph-mon[40675]: 6.9 scrub ok 2024-03-21T13:00:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:23 smithi163 ceph-mon[40675]: 3.0 scrub starts 2024-03-21T13:00:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:23 smithi163 ceph-mon[40675]: 3.0 scrub ok 2024-03-21T13:00:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:23 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/276649258' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-21T13:00:24.045 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 69 down_at 0 last_clean_interval [0,0) v2:172.21.15.163:6801/3086410981 v2:172.21.15.163:6802/3086410981 exists,up c34a3b25-9e02-46e6-8f9a-d0af2615258f', 'osd.1 up in weight 1 up_from 13 up_thru 69 down_at 0 last_clean_interval [0,0) v2:172.21.15.163:6805/3666062784 v2:172.21.15.163:6806/3666062784 exists,up 36e05f89-c238-41f7-be65-1553b529e8ce', 'osd.2 up in weight 1 up_from 18 up_thru 75 down_at 0 last_clean_interval [0,0) v2:172.21.15.163:6809/4009213507 v2:172.21.15.163:6810/4009213507 exists,up 968ec78d-25b8-4905-b328-d490c38ca85f', 'osd.3 up in weight 1 up_from 25 up_thru 75 down_at 0 last_clean_interval [0,0) v2:172.21.15.163:6813/3981878189 v2:172.21.15.163:6814/3981878189 exists,up 67dfa4f6-1709-48f6-971d-c6287233492d', 'osd.4 up in weight 1 up_from 30 up_thru 75 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6800/719276054 v2:172.21.15.204:6801/719276054 exists,up 7fd311a9-f3c2-456d-8ab1-341cf5b33471', 'osd.5 up in weight 1 up_from 39 up_thru 69 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6804/1346947668 v2:172.21.15.204:6805/1346947668 exists,up a87759c8-251c-415d-9387-a49f7d2f36d1', 'osd.6 up in weight 1 up_from 47 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6808/200077988 v2:172.21.15.204:6809/200077988 exists,up 5eb135cd-f677-4d0c-b23a-738f31615b6b', 'osd.7 up in weight 1 up_from 56 up_thru 69 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6812/315664888 v2:172.21.15.204:6813/315664888 exists,up 4f528e10-2c93-4bc2-a54c-06ca4132ed49'] 2024-03-21T13:00:24.046 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon mon.a config show 2024-03-21T13:00:24.289 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:24 smithi204 ceph-mon[33153]: 5.1e scrub starts 2024-03-21T13:00:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:24 smithi204 ceph-mon[33153]: 5.1e scrub ok 2024-03-21T13:00:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:24 smithi204 ceph-mon[33153]: 5.e scrub starts 2024-03-21T13:00:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:24 smithi204 ceph-mon[33153]: 5.e scrub ok 2024-03-21T13:00:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:24 smithi204 ceph-mon[33153]: 4.f scrub starts 2024-03-21T13:00:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:24 smithi204 ceph-mon[33153]: 4.f scrub ok 2024-03-21T13:00:25.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:24 smithi163 ceph-mon[29621]: 5.1e scrub starts 2024-03-21T13:00:25.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:24 smithi163 ceph-mon[29621]: 5.1e scrub ok 2024-03-21T13:00:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:24 smithi163 ceph-mon[29621]: 5.e scrub starts 2024-03-21T13:00:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:24 smithi163 ceph-mon[29621]: 5.e scrub ok 2024-03-21T13:00:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:24 smithi163 ceph-mon[29621]: 4.f scrub starts 2024-03-21T13:00:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:24 smithi163 ceph-mon[29621]: 4.f scrub ok 2024-03-21T13:00:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:24 smithi163 ceph-mon[40675]: 5.1e scrub starts 2024-03-21T13:00:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:24 smithi163 ceph-mon[40675]: 5.1e scrub ok 2024-03-21T13:00:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:24 smithi163 ceph-mon[40675]: 5.e scrub starts 2024-03-21T13:00:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:24 smithi163 ceph-mon[40675]: 5.e scrub ok 2024-03-21T13:00:25.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:24 smithi163 ceph-mon[40675]: 4.f scrub starts 2024-03-21T13:00:25.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:24 smithi163 ceph-mon[40675]: 4.f scrub ok 2024-03-21T13:00:25.823 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:25 smithi163 ceph-mon[29621]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:25.823 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:25 smithi163 ceph-mon[29621]: 6.b scrub starts 2024-03-21T13:00:25.823 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:25 smithi163 ceph-mon[29621]: 6.b scrub ok 2024-03-21T13:00:25.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:25 smithi163 ceph-mon[40675]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:25.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:25 smithi163 ceph-mon[40675]: 6.b scrub starts 2024-03-21T13:00:25.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:25 smithi163 ceph-mon[40675]: 6.b scrub ok 2024-03-21T13:00:25.823 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:00:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:00:26.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:25 smithi204 ceph-mon[33153]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:26.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:25 smithi204 ceph-mon[33153]: 6.b scrub starts 2024-03-21T13:00:26.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:25 smithi204 ceph-mon[33153]: 6.b scrub ok 2024-03-21T13:00:26.003 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:25 smithi204 bash[64295]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-21T13:00:26.003 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:25 smithi204 bash[64295]: Writing manifest to image destination 2024-03-21T13:00:26.081 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:00:26.082 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mon.a", 2024-03-21T13:00:26.082 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster": "ceph", 2024-03-21T13:00:26.082 INFO:teuthology.orchestra.run.smithi163.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-21T13:00:26.082 INFO:teuthology.orchestra.run.smithi163.stdout: "admin_socket_mode": "", 2024-03-21T13:00:26.082 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-21T13:00:26.082 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_client_required": "cephx, none", 2024-03-21T13:00:26.082 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_cluster_required": "cephx", 2024-03-21T13:00:26.082 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_debug": "true", 2024-03-21T13:00:26.082 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-21T13:00:26.082 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-21T13:00:26.082 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_service_required": "cephx", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_supported": "", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_aio": "true", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_aio_poll_ms": "250", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_aio_reap_max": "16", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_async_discard_threads": "0", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_block_size": "4096", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_debug_aio": "false", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_debug_inflight_ios": "false", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_enable_discard": "false", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_flock_retry": "3", 2024-03-21T13:00:26.083 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-21T13:00:26.084 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_inject_crash": "0", 2024-03-21T13:00:26.084 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-21T13:00:26.084 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_ioring": "false", 2024-03-21T13:00:26.084 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_ioring_hipri": "false", 2024-03-21T13:00:26.084 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-21T13:00:26.084 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-21T13:00:26.084 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-21T13:00:26.084 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-21T13:00:26.084 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_type": "", 2024-03-21T13:00:26.084 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_alloc_size": "1048576", 2024-03-21T13:00:26.084 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_allocator": "hybrid", 2024-03-21T13:00:26.084 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_buffered_io": "true", 2024-03-21T13:00:26.084 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_check_for_zeros": "false", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_compact_log_sync": "false", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_max_log_runway": "4194304", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_max_prefetch": "1048576", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_min_flush_size": "524288", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_min_log_runway": "1048576", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_replay_recovery": "false", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-21T13:00:26.085 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_sync_write": "false", 2024-03-21T13:00:26.086 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-21T13:00:26.086 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-21T13:00:26.086 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-21T13:00:26.086 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_allocation_from_file": "true", 2024-03-21T13:00:26.086 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_allocator": "hybrid", 2024-03-21T13:00:26.086 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-21T13:00:26.086 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-21T13:00:26.086 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-21T13:00:26.086 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-21T13:00:26.086 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-21T13:00:26.086 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-21T13:00:26.086 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-21T13:00:26.086 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_create": "true", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_db_create": "false", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_db_path": "", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_db_size": "0", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_path": "", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_preallocate_file": "false", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_size": "107374182400", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_wal_create": "false", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_wal_path": "", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_wal_size": "100663296", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_bluefs": "true", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-21T13:00:26.087 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_autotune": "true", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_size": "0", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_type": "2q", 2024-03-21T13:00:26.088 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_clone_cow": "true", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_mode": "none", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_csum_type": "crc32c", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-21T13:00:26.089 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_freelist": "false", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_small_allocations": "0", 2024-03-21T13:00:26.090 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_default_buffered_read": "true", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_default_buffered_write": "false", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fail_eio": "false", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-21T13:00:26.091 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-21T13:00:26.092 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-21T13:00:26.092 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-21T13:00:26.092 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-21T13:00:26.092 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-21T13:00:26.092 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_on_mount": "false", 2024-03-21T13:00:26.092 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-21T13:00:26.092 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_on_umount": "false", 2024-03-21T13:00:26.092 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-21T13:00:26.092 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-21T13:00:26.092 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-21T13:00:26.092 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-21T13:00:26.092 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-21T13:00:26.092 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_ignore_data_csum": "false", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_log_op_age": "5.000000", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_max_alloc_size": "0", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_max_blob_size": "0", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_max_deferred_txc": "32", 2024-03-21T13:00:26.093 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_min_alloc_size": "0", 2024-03-21T13:00:26.094 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-21T13:00:26.094 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-21T13:00:26.094 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_nid_prealloc": "1024", 2024-03-21T13:00:26.094 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-21T13:00:26.094 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-21T13:00:26.094 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-21T13:00:26.094 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-21T13:00:26.094 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_retry_disk_reads": "3", 2024-03-21T13:00:26.094 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_rocksdb_cf": "true", 2024-03-21T13:00:26.094 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-21T13:00:26.094 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-21T13:00:26.094 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-21T13:00:26.094 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_spdk_mem": "512", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_tracing": "false", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-21T13:00:26.095 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_zero_block_detection": "false", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-21T13:00:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_cluster_require_signatures": "false", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_cluster_require_version": "2", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_require_signatures": "false", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_require_version": "2", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_service_require_signatures": "false", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_service_require_version": "2", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_sign_messages": "true", 2024-03-21T13:00:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: "chdir": "", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_acl_type": "", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_asio_thread_count": "2", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_cache_mid": "0.750000", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_cache_size": "16384", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_caps_release_delay": "5", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_check_pool_perm": "true", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_debug_force_sync_read": "false", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_debug_getattr_caps": "false", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_debug_inject_tick_delay": "0", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_die_on_failed_remount": "false", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_dirsize_rbytes": "true", 2024-03-21T13:00:26.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_force_lazyio": "false", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_fs": "", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_inject_release_failure": "false", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_max_inline_size": "4096", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_mds_namespace": "", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_metadata": "", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_mount_gid": "-1", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_mount_timeout": "300", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_mount_uid": "-1", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_mountpoint": "/", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_notify_timeout": "10", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_oc": "true", 2024-03-21T13:00:26.099 INFO:teuthology.orchestra.run.smithi163.stdout: "client_oc_max_dirty": "104857600", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_oc_max_objects": "1000", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_oc_size": "209715200", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_oc_target_dirty": "8388608", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_permissions": "true", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_quota": "true", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_quota_df": "true", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_readahead_max_bytes": "0", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_readahead_max_periods": "4", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_readahead_min": "131072", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_reconnect_stale": "false", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_shutdown_timeout": "30", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_snapdir": ".snap", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_tick_interval": "1", 2024-03-21T13:00:26.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_trace": "", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "client_try_dentry_invalidate": "false", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "client_use_faked_inos": "false", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "client_use_random_mds": "false", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_graylog": "false", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_graylog_port": "12201", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_monitors": "default=true", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_syslog": "false", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_syslog_level": "info", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "-", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_network": "", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_network_interface": "", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "compressor_zlib_isal": "false", 2024-03-21T13:00:26.101 INFO:teuthology.orchestra.run.smithi163.stdout: "compressor_zlib_level": "5", 2024-03-21T13:00:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: "compressor_zlib_winsize": "-15", 2024-03-21T13:00:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: "compressor_zstd_level": "1", 2024-03-21T13:00:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T13:00:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-21T13:00:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: "crimson_alien_op_num_threads": "6", 2024-03-21T13:00:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-21T13:00:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-21T13:00:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-21T13:00:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: "crimson_seastar_cpu_cores": "", 2024-03-21T13:00:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_location": "", 2024-03-21T13:00:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_location_hook": "", 2024-03-21T13:00:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_location_hook_timeout": "10", 2024-03-21T13:00:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: "daemonize": "false", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "daos_pool": "tank", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_alienstore": "0/5", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_allow_any_pool_priority": "false", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_asok": "1/5", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_asok_assert_abort": "false", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_asserts_on_shutdown": "true", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_auth": "1/5", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_bdev": "1/3", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_bluefs": "1/5", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_bluestore": "1/5", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_buffer": "0/1", 2024-03-21T13:00:26.103 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_ceph_exporter": "1/5", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_cephfs_mirror": "0/5", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_cephsqlite": "0/5", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_client": "0/5", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_compressor": "1/5", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_context": "0/1", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_crimson_interrupt": "0/5", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_crush": "1/1", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_crypto": "1/5", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_cyanstore": "0/5", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_deliberately_leak_memory": "false", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_disable_randomized_ping": "false", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_dpdk": "1/5", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_eventtrace": "1/5", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_filer": "0/1", 2024-03-21T13:00:26.104 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_filestore": "1/3", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_finisher": "1/1", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_fuse": "1/5", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_heartbeat_testing_span": "0", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_heartbeatmap": "1/5", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_javaclient": "1/5", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_journal": "1/3", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_journaler": "0/5", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_kstore": "1/5", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_lockdep": "0/1", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mclock": "1/5", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mds": "1/5", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mds_balancer": "1/5", 2024-03-21T13:00:26.105 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mds_locker": "1/5", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mds_log": "1/5", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mds_log_expire": "1/5", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mds_migrator": "1/5", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mds_quiesce": "3/5", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_memstore": "1/5", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mgr": "2/5", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mgrc": "1/5", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mon": "20/20", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_monc": "0/10", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_ms": "1/1", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_none": "0/5", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_objclass": "0/5", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_objectcacher": "0/5", 2024-03-21T13:00:26.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_objecter": "0/1", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_optracker": "0/5", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_osd": "1/5", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_paxos": "20/20", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_perfcounter": "1/5", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_prioritycache": "1/5", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rados": "0/5", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rbd": "0/5", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rbd_mirror": "0/5", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rbd_pwl": "0/5", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rbd_replay": "0/5", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_refs": "0/0", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_reserver": "1/1", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rgw": "1/5", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rgw_access": "1/5", 2024-03-21T13:00:26.107 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rgw_datacache": "1/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rgw_dbstore": "1/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rgw_flight": "1/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rgw_sync": "1/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rocksdb": "4/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore": "0/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore_backref": "0/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore_cache": "0/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore_cleaner": "0/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore_device": "0/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore_epm": "0/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore_journal": "0/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore_lba": "0/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore_odata": "0/5", 2024-03-21T13:00:26.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore_omap": "0/5", 2024-03-21T13:00:26.109 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore_onode": "0/5", 2024-03-21T13:00:26.109 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore_t": "0/5", 2024-03-21T13:00:26.109 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_seastore_tm": "0/5", 2024-03-21T13:00:26.109 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_striper": "0/1", 2024-03-21T13:00:26.109 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_test": "0/5", 2024-03-21T13:00:26.110 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_throttle": "1/1", 2024-03-21T13:00:26.110 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_timer": "0/1", 2024-03-21T13:00:26.110 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_tp": "0/5", 2024-03-21T13:00:26.110 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_trace": "1/5", 2024-03-21T13:00:26.110 INFO:teuthology.orchestra.run.smithi163.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-03-21T13:00:26.110 INFO:teuthology.orchestra.run.smithi163.stdout: "device_failure_prediction_mode": "none", 2024-03-21T13:00:26.110 INFO:teuthology.orchestra.run.smithi163.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-21T13:00:26.110 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-21T13:00:26.110 INFO:teuthology.orchestra.run.smithi163.stdout: "err_to_graylog": "false", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "err_to_journald": "false", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "err_to_stderr": "true", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "err_to_syslog": "false", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "event_tracing": "false", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "exporter_addr": "0.0.0.0", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "exporter_http_port": "9926", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "exporter_prio_limit": "5", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "exporter_sort_metrics": "true", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "exporter_stats_period": "5", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "fake_statfs_for_testing": "0", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "fatal_signal_handlers": "true", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "filer_max_purge_ops": "10", 2024-03-21T13:00:26.111 INFO:teuthology.orchestra.run.smithi163.stdout: "filer_max_truncate_ops": "128", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_apply_finisher_threads": "1", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_blackhole": "false", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_btrfs_clone_range": "true", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_btrfs_snap": "true", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_caller_concurrency": "10", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_collect_device_partition_information": "true", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_commit_timeout": "600.000000", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_debug_inject_read_err": "false", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_debug_omap_check": "false", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_debug_verify_split": "false", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_dump_file": "", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fadvise": "true", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fail_eio": "true", 2024-03-21T13:00:26.112 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fd_cache_shards": "16", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fd_cache_size": "128", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fiemap": "false", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fiemap_threshold": "4096", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_inject_stall": "0", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_journal_parallel": "false", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_journal_trailing": "false", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_journal_writeahead": "false", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_kill_at": "0", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-21T13:00:26.113 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattrs": "0", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_xattr_value_size": "0", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_merge_threshold": "-10", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_odsync_write": "false", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_omap_backend": "rocksdb", 2024-03-21T13:00:26.114 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_omap_backend_path": "", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_op_thread_timeout": "60", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_op_threads": "2", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_punch_hole": "false", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-21T13:00:26.115 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-21T13:00:26.116 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_max_ops": "50", 2024-03-21T13:00:26.116 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-21T13:00:26.116 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_seek_data_hole": "false", 2024-03-21T13:00:26.116 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_sloppy_crc": "false", 2024-03-21T13:00:26.116 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-21T13:00:26.116 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_splice": "false", 2024-03-21T13:00:26.116 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_split_multiple": "2", 2024-03-21T13:00:26.116 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_split_rand_factor": "20", 2024-03-21T13:00:26.116 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_update_to": "1000", 2024-03-21T13:00:26.116 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-21T13:00:26.116 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-21T13:00:26.116 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-21T13:00:26.116 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_enable": "true", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_xfs_extsize": "false", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_zfs_snap": "false", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "fio_dir": "/tmp/fio", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "d4bf79c0-e781-11ee-95cd-87774f69a715", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_allow_other": "true", 2024-03-21T13:00:26.117 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_atomic_o_trunc": "true", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_big_writes": "true", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_debug": "false", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_default_permissions": "false", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_disable_pagecache": "false", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_max_write": "0", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_multithreaded": "true", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_require_active_mds": "true", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_set_user_groups": "true", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_splice_move": "true", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_splice_read": "true", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_splice_write": "true", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_use_invalidate_cb": "true", 2024-03-21T13:00:26.118 INFO:teuthology.orchestra.run.smithi163.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "gss_target_name": "ceph", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_file": "", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_inject_failure": "0", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_interval": "5", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "host": "smithi163", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-21T13:00:26.119 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "inject_early_sigterm": "false", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "jaeger_agent_port": "6799", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "jaeger_tracing_enable": "false", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_aio": "true", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_align_min_size": "65536", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_block_align": "true", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_block_size": "4096", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_dio": "true", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_discard": "false", 2024-03-21T13:00:26.120 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_force_aio": "false", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_ignore_corruption": "false", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_max_write_bytes": "10485760", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_max_write_entries": "100", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_replay_from": "0", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_write_header_frequency": "0", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_zero_on_create": "false", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "journaler_prefetch_periods": "10", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "journaler_prezero_periods": "5", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "journaler_write_head_interval": "15", 2024-03-21T13:00:26.121 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "keyfile": "", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_backend": "rocksdb", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_default_stripe_size": "65536", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_fsck_on_mount": "false", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_max_bytes": "67108864", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_max_ops": "512", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_nid_prealloc": "1024", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_onode_map_size": "1024", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_sync_submit_transaction": "false", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_sync_transaction": "false", 2024-03-21T13:00:26.122 INFO:teuthology.orchestra.run.smithi163.stdout: "librados_thread_count": "2", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "lockdep": "false", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "lockdep_force_backtrace": "false", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_coarse_timestamps": "true", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_flush_on_exit": "false", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_graylog_host": "127.0.0.1", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_graylog_port": "12201", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_max_new": "1000", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_max_recent": "10000", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_stderr_prefix": "", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_stop_at_utilization": "0.970000", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": "true", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_graylog": "false", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_journald": "false", 2024-03-21T13:00:26.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_stderr": "false", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_syslog": "false", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "max_rotating_auth_attempts": "10", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_action_on_write_error": "1", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_alternate_name_max": "8192", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_asio_thread_count": "2", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_export_pin": "true", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_fragment_dirs": "true", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_fragment_interval": "5", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_interval": "10", 2024-03-21T13:00:26.124 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_max": "-1", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_max_until": "-1", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_merge_size": "50", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_midchunk": "0.300000", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_min_start": "0.200000", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_minchunk": "0.001000", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_mode": "0", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_need_max": "1.200000", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_need_min": "0.800000", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_overload_epochs": "2", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_split_bits": "3", 2024-03-21T13:00:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_split_size": "10000", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_target_decay": "10.000000", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_beacon_grace": "15.000000", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_beacon_interval": "4.000000", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_mid": "0.700000", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_release_free_interval": "10", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_reservation": "0.050000", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_trim_interval": "1", 2024-03-21T13:00:26.126 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_trim_threshold": "262144", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_client_prealloc_inos": "1000", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_connect_bootstrapping": "false", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_damage_table_max_entries": "10000", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_debug_auth_pins": "false", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_debug_frag": "false", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_debug_scatterstat": "false", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_debug_subtrees": "false", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_decay_halflife": "5.000000", 2024-03-21T13:00:26.127 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_default_dir_hash": "2", 2024-03-21T13:00:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_defer_session_stale": "true", 2024-03-21T13:00:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_deny_all_reconnect": "false", 2024-03-21T13:00:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dir_keys_per_op": "16384", 2024-03-21T13:00:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dir_max_commit_size": "10", 2024-03-21T13:00:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dir_max_entries": "0", 2024-03-21T13:00:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dir_prefetch": "true", 2024-03-21T13:00:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-21T13:00:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-21T13:00:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dump_cache_on_map": "false", 2024-03-21T13:00:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-21T13:00:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-21T13:00:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_early_reply": "true", 2024-03-21T13:00:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_enable_op_tracker": "true", 2024-03-21T13:00:26.129 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_enforce_unique_name": "true", 2024-03-21T13:00:26.129 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-21T13:00:26.129 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-21T13:00:26.129 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_export_ephemeral_random": "true", 2024-03-21T13:00:26.129 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-21T13:00:26.129 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-21T13:00:26.129 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-21T13:00:26.129 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-21T13:00:26.129 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-21T13:00:26.129 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_health_summarize_threshold": "10", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_inject_health_dummy": "false", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_inject_migrator_session_race": "false", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_join_fs": "", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_journal_format": "1", 2024-03-21T13:00:26.130 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_create_at": "0", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_export_at": "0", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_import_at": "0", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_journal_at": "0", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_journal_expire_at": "0", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_journal_replay_at": "0", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_link_at": "0", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_mdstable_at": "0", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_openc_at": "0", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_rename_at": "0", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_shutdown_at": "0", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_event_large_threshold": "524288", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_events_per_segment": "1024", 2024-03-21T13:00:26.131 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-21T13:00:26.132 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_max_events": "-1", 2024-03-21T13:00:26.132 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_max_segments": "128", 2024-03-21T13:00:26.132 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_pause": "false", 2024-03-21T13:00:26.132 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_segment_size": "0", 2024-03-21T13:00:26.132 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-21T13:00:26.132 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-21T13:00:26.132 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-21T13:00:26.132 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_trim_threshold": "128", 2024-03-21T13:00:26.132 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-21T13:00:26.132 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_warn_factor": "2.000000", 2024-03-21T13:00:26.132 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_caps_per_client": "1048576", 2024-03-21T13:00:26.132 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_completed_flushes": "100000", 2024-03-21T13:00:26.132 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_completed_requests": "100000", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_export_size": "20971520", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_file_recover": "32", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_purge_files": "64", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_purge_ops": "8192", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_snaps_per_dir": "100", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_metrics_update_interval": "2", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_min_caps_per_client": "100", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_min_caps_working_set": "10000", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_numa_node": "-1", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_op_complaint_time": "30.000000", 2024-03-21T13:00:26.133 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_op_history_duration": "600", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_op_history_size": "20", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_op_history_slow_op_size": "20", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_op_log_threshold": "5", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_ping_grace": "15", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_ping_interval": "5", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_recall_max_caps": "30000", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_recall_warning_threshold": "262144", 2024-03-21T13:00:26.134 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_replay_interval": "1.000000", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_root_ino_gid": "0", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_root_ino_uid": "0", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-21T13:00:26.135 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_shutdown_check": "0", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_skip_ino": "0", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_snap_max_uid": "4294967294", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_snap_min_uid": "0", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_snap_rstat": "false", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_standby_replay_damaged": "false", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_symlink_recovery": "true", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_thrash_exports": "0", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_thrash_fragments": "0", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_tick_interval": "5.000000", 2024-03-21T13:00:26.136 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_valgrind_exit": "false", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_verify_backtrace": "1", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_verify_scatter": "false", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_wipe_sessions": "false", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "mempool_debug": "false", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "memstore_device_bytes": "1073741824", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "memstore_page_set": "false", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "memstore_page_size": "65536", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_client_bytes": "134217728", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_client_messages": "512", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-21T13:00:26.137 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-21T13:00:26.138 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_disabled_modules": "", 2024-03-21T13:00:26.138 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-21T13:00:26.138 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_max_pg_num_change": "128", 2024-03-21T13:00:26.138 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_mds_bytes": "134217728", 2024-03-21T13:00:26.138 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_mds_messages": "128", 2024-03-21T13:00:26.138 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-21T13:00:26.138 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_mon_bytes": "134217728", 2024-03-21T13:00:26.138 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_mon_messages": "128", 2024-03-21T13:00:26.138 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_osd_bytes": "536870912", 2024-03-21T13:00:26.138 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_osd_messages": "8192", 2024-03-21T13:00:26.138 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_pool": "true", 2024-03-21T13:00:26.138 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-21T13:00:26.138 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_standby_modules": "true", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_stats_period": "5", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_stats_threshold": "5", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_tick_period": "2", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_allow_pool_delete": "true", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_allow_pool_size_one": "false", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_auth_validate_all_caps": "true", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_bytes": "104857600", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_directed_command_retry": "2", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-21T13:00:26.139 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_hunt_parallel": "3", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_log_interval": "1.000000", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_ping_interval": "10.000000", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_target_rank": "-1", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_file_level": "debug", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_file": "true", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-21T13:00:26.140 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_journald": "true", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_compact_on_bootstrap": "false", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_compact_on_start": "false", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_compact_on_trim": "true", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cpu_threads": "4", 2024-03-21T13:00:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_crush_min_required_version": "hammer", 2024-03-21T13:00:26.142 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_daemon_bytes": "419430400", 2024-03-21T13:00:26.142 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-21T13:00:26.142 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_data_avail_crit": "5", 2024-03-21T13:00:26.142 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_data_avail_warn": "5", 2024-03-21T13:00:26.142 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_data_size_warn": "16106127360", 2024-03-21T13:00:26.142 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-21T13:00:26.142 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-21T13:00:26.142 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_dump_json": "false", 2024-03-21T13:00:26.142 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-21T13:00:26.142 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_dump_transactions": "false", 2024-03-21T13:00:26.142 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_extra_checks": "false", 2024-03-21T13:00:26.142 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-21T13:00:26.142 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_no_require_reef": "false", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_no_require_squid": "false", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_down_mkfs_grace": "60", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_election_default_strategy": "1", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_election_timeout": "5.000000", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_elector_ping_divisor": "2", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_enable_op_tracker": "true", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_fake_pool_delete": "false", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_force_quorum_join": "false", 2024-03-21T13:00:26.143 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_globalid_prealloc": "10000", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_health_detail_to_clog": "true", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_health_log_update_period": "5", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_health_max_detail": "50", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_health_to_clog": "true", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_health_to_clog_interval": "600", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_host": "[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0]", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_host_override": "", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_initial_members": "", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-21T13:00:26.144 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_lease": "5.000000", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_log_full_interval": "50", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_log_max": "10000", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_log_max_summary": "50", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_log_epochs": "500", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_osd": "10000", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_pg_per_osd": "10000", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_pool_pg_num": "65536", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-21T13:00:26.145 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mds_force_trim_to": "0", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mds_skip_sanity": "false", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_memory_autotune": "true", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_memory_target": "2147483648", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mgr_beacon_grace": "30", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mgr_digest_period": "5", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mgr_inactive_grace": "60", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_min_osdmap_epochs": "50", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_op_complaint_time": "30", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_op_history_duration": "600", 2024-03-21T13:00:26.146 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_op_history_size": "20", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_op_history_slow_op_size": "20", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_op_log_threshold": "5", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_auto_mark_in": "false", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_cache_size": "500", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-21T13:00:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_down_out_interval": "600", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_force_trim_to": "0", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_min_down_reporters": "2", 2024-03-21T13:00:26.148 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_report_timeout": "900", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-21T13:00:26.149 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pg_stuck_threshold": "60", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_probe_timeout": "2.000000", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_reweight_max_change": "0.050000", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_reweight_max_osds": "4", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-21T13:00:26.150 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_scrub_interval": "86400", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_scrub_max_keys": "100", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_scrub_timeout": "300", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_session_timeout": "300", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_smart_report_timeout": "5", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_stat_smooth_intervals": "6", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_stretch_pool_min_size": "2", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_stretch_pool_size": "4", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_sync_debug": "false", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-21T13:00:26.151 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_sync_provider_kill_at": "0", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_sync_requester_kill_at": "0", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_sync_timeout": "60.000000", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_target_pg_per_osd": "100", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_tick_interval": "5", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_timecheck_interval": "300.000000", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_use_min_delay_socket": "false", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_older_version_delay": "604800", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-21T13:00:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_misplaced": "false", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_older_version": "true", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "monmap": "", 2024-03-21T13:00:26.153 INFO:teuthology.orchestra.run.smithi163.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "motr_tracing_enabled": "false", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_op_threads": "3", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_cm": "false", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_device_name": "", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_dscp": "96", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-21T13:00:26.154 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_local_gid": "", 2024-03-21T13:00:26.155 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-21T13:00:26.155 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_port_num": "1", 2024-03-21T13:00:26.155 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-21T13:00:26.155 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-21T13:00:26.155 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-21T13:00:26.155 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-21T13:00:26.155 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_sl": "3", 2024-03-21T13:00:26.155 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_support_srq": "true", 2024-03-21T13:00:26.155 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_type": "ib", 2024-03-21T13:00:26.155 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_reap_threshold": "5", 2024-03-21T13:00:26.155 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_before_connect": "false", 2024-03-21T13:00:26.155 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_ipv4": "true", 2024-03-21T13:00:26.155 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_ipv6": "false", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_msgr1": "false", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_msgr2": "true", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_port_max": "7568", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_port_min": "6800", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_retry_count": "3", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_retry_delay": "5", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_blackhole_client": "false", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_blackhole_mds": "false", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_blackhole_mgr": "false", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_blackhole_mon": "false", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_blackhole_osd": "false", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_client_mode": "crc secure", 2024-03-21T13:00:26.156 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_cluster_mode": "crc secure", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_cluster_type": "", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_compress_secure": "false", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_connection_idle_timeout": "900", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_connection_ready_timeout": "10", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_crc_data": "true", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_crc_header": "true", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_die_on_bad_msg": "false", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_die_on_bug": "true", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_die_on_old_message": "true", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_die_on_skipped_message": "false", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_coremask": "0xF", 2024-03-21T13:00:26.157 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-21T13:00:26.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-21T13:00:26.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_enable_tso": "true", 2024-03-21T13:00:26.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-21T13:00:26.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-21T13:00:26.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_hugepages": "", 2024-03-21T13:00:26.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-21T13:00:26.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-21T13:00:26.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_lro": "true", 2024-03-21T13:00:26.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_memory_channel": "4", 2024-03-21T13:00:26.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-21T13:00:26.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_pmd": "", 2024-03-21T13:00:26.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_port_id": "0", 2024-03-21T13:00:26.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dump_on_send": "false", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_initial_backoff": "0.200000", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_inject_delay_max": "1.000000", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_inject_delay_type": "", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_inject_network_congestion": "0", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_inject_socket_failures": "0", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_learn_addr_from_peer": "true", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_max_accept_failures": "4", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_max_backoff": "15.000000", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_mon_client_mode": "secure crc", 2024-03-21T13:00:26.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_mon_service_mode": "secure crc", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_osd_compress_min_size": "1024", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_osd_compress_mode": "none", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_pq_min_cost": "65536", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_public_type": "", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_service_mode": "crc secure", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_tcp_listen_backlog": "512", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_tcp_nodelay": "true", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_tcp_rcvbuf": "0", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_type": "async+posix", 2024-03-21T13:00:26.160 INFO:teuthology.orchestra.run.smithi163.stdout: "no_config_file": "false", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_completion_locks_per_session": "32", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_inflight_ops": "1024", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_tick_interval": "5.000000", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_timeout": "10.000000", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "objectstore_blackhole": "false", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "openssl_engine_opts": "", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_delay_time": "5.000000", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_hist_halflife": "1000", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_max_low_ops": "2", 2024-03-21T13:00:26.161 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_max_ops": "4", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_slop": "0.020000", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_asio_thread_count": "2", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_async_recovery_min_cost": "100", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_backfill_scan_max": "512", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_backfill_scan_min": "64", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_backoff_on_degraded": "false", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_backoff_on_peering": "false", 2024-03-21T13:00:26.162 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_backoff_on_unfound": "true", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_beacon_report_interval": "300", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_bench_duration": "30", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_bench_max_block_size": "67108864", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_blkin_trace_all": "false", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_check_for_log_corruption": "false", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-21T13:00:26.163 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_class_update_on_start": "true", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_client_message_cap": "256", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_client_message_size_cap": "524288000", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_client_op_priority": "63", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_client_watch_timeout": "30", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_command_max_records": "256", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_compact_on_start": "false", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_crush_update_on_start": "true", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_crush_update_weight_set": "true", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-21T13:00:26.164 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_feed_pullee": "-1", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_misdirected_ops": "false", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_no_acting_change": "false", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_no_purge_strays": "false", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_op_order": "false", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-21T13:00:26.165 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-21T13:00:26.166 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_shutdown": "false", 2024-03-21T13:00:26.166 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-21T13:00:26.166 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-21T13:00:26.166 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_trim_objects": "false", 2024-03-21T13:00:26.166 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-21T13:00:26.166 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-21T13:00:26.166 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_verify_snaps": "false", 2024-03-21T13:00:26.166 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-21T13:00:26.166 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-21T13:00:26.166 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_keys": "1024", 2024-03-21T13:00:26.166 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-21T13:00:26.166 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-21T13:00:26.166 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_stride": "524288", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_default_notify_timeout": "30", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_delete_sleep": "0.000000", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_discard_disconnected_ops": "true", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_enable_op_tracker": "true", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-21T13:00:26.167 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_fast_info": "true", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_fast_shutdown": "true", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_function_tracing": "false", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_grace": "20", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_interval": "6", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_min_peers": "10", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_min_size": "2000", 2024-03-21T13:00:26.168 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_stale": "600", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_hit_set_max_size": "100000", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_hit_set_min_size": "1000", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_journal_size": "5120", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_kill_backfill_at": "0", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_map_cache_size": "50", 2024-03-21T13:00:26.169 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_map_dedup": "true", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_map_message_max": "40", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_map_share_max_epochs": "40", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_attr_name_len": "100", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_attr_size": "0", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_backfills": "1", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_markdown_count": "5", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_markdown_period": "600", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_object_name_len": "2048", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_object_namespace_len": "256", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_object_size": "134217728", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-21T13:00:26.170 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_pg_blocked_by": "16", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_pg_log_entries": "10000", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_pgls": "1024", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_push_cost": "8388608", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_push_objects": "10", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_scrubs": "3", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_trimming_pgs": "2", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_write_op_reply_len": "64", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_write_size": "90", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-21T13:00:26.171 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_profile": "balanced", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-21T13:00:26.172 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_base": "805306368", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_cache_min": "134217728", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_target": "4294967296", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_target_autotune": "false", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_min_pg_log_entries": "250", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_min_recovery_priority": "0", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mon_report_interval": "5", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-21T13:00:26.173 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_num_cache_shards": "32", 2024-03-21T13:00:26.174 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_num_op_tracker_shard": "32", 2024-03-21T13:00:26.174 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_numa_auto_affinity": "true", 2024-03-21T13:00:26.174 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_numa_node": "-1", 2024-03-21T13:00:26.174 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_numa_prefer_iface": "true", 2024-03-21T13:00:26.174 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-21T13:00:26.174 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_objecter_finishers": "1", 2024-03-21T13:00:26.174 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_objectstore": "bluestore", 2024-03-21T13:00:26.174 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_objectstore_fuse": "false", 2024-03-21T13:00:26.174 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_objectstore_tracing": "false", 2024-03-21T13:00:26.174 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_complaint_time": "30.000000", 2024-03-21T13:00:26.174 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_history_duration": "600", 2024-03-21T13:00:26.174 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_history_size": "20", 2024-03-21T13:00:26.174 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_history_slow_op_size": "20", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_log_threshold": "5", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_num_shards": "0", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_num_shards_hdd": "5", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_num_shards_ssd": "8", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_pq_min_cost": "65536", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_queue_cut_off": "high", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_thread_timeout": "15", 2024-03-21T13:00:26.175 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_open_classes_on_start": "true", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_os_flags": "0", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_peering_op_priority": "255", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_delete_cost": "1048576", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_delete_priority": "5", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_log_trim_max": "10000", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_log_trim_min": "100", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-21T13:00:26.176 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_crimson": "false", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_flags": "0", 2024-03-21T13:00:26.177 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-21T13:00:26.178 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_min_size": "0", 2024-03-21T13:00:26.178 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-21T13:00:26.178 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_pg_num": "32", 2024-03-21T13:00:26.178 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_pgp_num": "0", 2024-03-21T13:00:26.178 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-21T13:00:26.178 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_read_ratio": "70", 2024-03-21T13:00:26.178 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_size": "3", 2024-03-21T13:00:26.178 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_type": "replicated", 2024-03-21T13:00:26.178 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-21T13:00:26.178 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-21T13:00:26.178 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_push_per_object_cost": "1000", 2024-03-21T13:00:26.179 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-21T13:00:26.179 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recover_clone_overlap": "true", 2024-03-21T13:00:26.179 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-21T13:00:26.179 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_cost": "20971520", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_max_active": "0", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_max_single_start": "1", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_op_priority": "3", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_priority": "5", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_sleep": "0.000000", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-21T13:00:26.180 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_repair_during_recovery": "false", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_requested_scrub_priority": "120", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_auto_repair": "false", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_begin_hour": "0", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_begin_week_day": "0", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_chunk_max": "25", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_chunk_min": "5", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_cost": "52428800", 2024-03-21T13:00:26.181 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-21T13:00:26.182 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_during_recovery": "false", 2024-03-21T13:00:26.182 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_end_hour": "0", 2024-03-21T13:00:26.182 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_end_week_day": "0", 2024-03-21T13:00:26.182 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_event_cost": "4096", 2024-03-21T13:00:26.182 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-21T13:00:26.182 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-21T13:00:26.182 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_invalid_stats": "true", 2024-03-21T13:00:26.182 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-21T13:00:26.182 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-21T13:00:26.182 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_max_preemptions": "5", 2024-03-21T13:00:26.182 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-21T13:00:26.182 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_priority": "5", 2024-03-21T13:00:26.182 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_sleep": "0.000000", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_skip_data_digest": "false", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_smart_report_timeout": "5", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_snap_trim_cost": "1048576", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_snap_trim_priority": "5", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-21T13:00:26.183 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_target_transaction_size": "30", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tracing": "false", 2024-03-21T13:00:26.184 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_use_stale_snap": "false", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "osdc_blkin_trace_all": "false", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_kill_at": "0", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_max_join_drift": "10", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_min": "500", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_min_wait": "0.050000", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_propose_interval": "1.000000", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_service_trim_max": "500", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_service_trim_min": "10", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_stash_full_interval": "25", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_trim_max": "500", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_trim_min": "250", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "perf": "true", 2024-03-21T13:00:26.185 INFO:teuthology.orchestra.run.smithi163.stdout: "pid_file": "", 2024-03-21T13:00:26.186 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-21T13:00:26.186 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-21T13:00:26.186 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "-", 2024-03-21T13:00:26.186 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrv": "", 2024-03-21T13:00:26.186 INFO:teuthology.orchestra.run.smithi163.stdout: "public_bind_addr": "-", 2024-03-21T13:00:26.186 INFO:teuthology.orchestra.run.smithi163.stdout: "public_network": "", 2024-03-21T13:00:26.186 INFO:teuthology.orchestra.run.smithi163.stdout: "public_network_interface": "", 2024-03-21T13:00:26.186 INFO:teuthology.orchestra.run.smithi163.stdout: "qat_compressor_enabled": "false", 2024-03-21T13:00:26.186 INFO:teuthology.orchestra.run.smithi163.stdout: "qat_compressor_session_max_number": "256", 2024-03-21T13:00:26.186 INFO:teuthology.orchestra.run.smithi163.stdout: "rados_mon_op_timeout": "0", 2024-03-21T13:00:26.186 INFO:teuthology.orchestra.run.smithi163.stdout: "rados_osd_op_timeout": "0", 2024-03-21T13:00:26.186 INFO:teuthology.orchestra.run.smithi163.stdout: "rados_tracing": "false", 2024-03-21T13:00:26.186 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_atime_update_interval": "60", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_balance_parent_reads": "false", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_balance_snap_reads": "false", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_blkin_trace_all": "false", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache": "true", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_policy": "writearound", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_size": "33554432", 2024-03-21T13:00:26.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_clone_copy_on_read": "false", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_compression_hint": "none", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_concurrent_management_ops": "10", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_clone_format": "auto", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_data_pool": "", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_features": "61", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_format": "2", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_map_options": "", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_order": "22", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_pool": "rbd", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_stripe_count": "0", 2024-03-21T13:00:26.188 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_stripe_unit": "0", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_enable_alloc_hint": "true", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_io_scheduler": "simple", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-21T13:00:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-21T13:00:26.190 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_order": "24", 2024-03-21T13:00:26.190 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_pool": "", 2024-03-21T13:00:26.190 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_splay_width": "4", 2024-03-21T13:00:26.190 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_localize_parent_reads": "false", 2024-03-21T13:00:26.190 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_localize_snap_reads": "false", 2024-03-21T13:00:26.190 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-21T13:00:26.190 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-21T13:00:26.190 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-21T13:00:26.190 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-21T13:00:26.190 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-21T13:00:26.190 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-21T13:00:26.190 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-21T13:00:26.190 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-21T13:00:26.191 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mtime_update_interval": "60", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_non_blocking_aio": "true", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_op_thread_timeout": "60", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_op_threads": "1", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_parent_cache_enabled": "false", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-21T13:00:26.192 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_plugins": "", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_bps_burst": "0", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_bps_limit": "0", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_exclude_ops": "0", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_iops_burst": "0", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_iops_limit": "0", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-21T13:00:26.193 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_read_from_replica_policy": "default", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-21T13:00:26.194 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_skip_partial_discard": "true", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_tracing": "false", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_validate_names": "true", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_validate_pool": "true", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "restapi_base_url": "", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "restapi_log_level": "", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_acl_grants_max_num": "100", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_admin_entry": "admin", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_backend_store": "rados", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_barbican_url": "", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_beast_enable_async": "true", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_counters_cache": "false", 2024-03-21T13:00:26.195 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_cache_enabled": "true", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_cache_expiry_interval": "900", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_cache_lru_size": "10000", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_config_store": "rados", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_content_length_compat": "false", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_copy_obj_progress": "true", 2024-03-21T13:00:26.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_cors_rules_max_num": "100", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_cross_domain_policy": "", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_addr": "", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_password": "", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_username": "", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_require_ssl": "true", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-21T13:00:26.197 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-21T13:00:26.198 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-21T13:00:26.198 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-21T13:00:26.198 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-21T13:00:26.198 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-21T13:00:26.198 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-21T13:00:26.198 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-21T13:00:26.198 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-21T13:00:26.198 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-21T13:00:26.198 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-21T13:00:26.198 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-21T13:00:26.198 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-21T13:00:26.198 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-21T13:00:26.198 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_addr": "", 2024-03-21T13:00:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_auth": "token", 2024-03-21T13:00:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_namespace": "", 2024-03-21T13:00:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_prefix": "", 2024-03-21T13:00:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-21T13:00:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-21T13:00:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-21T13:00:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-21T13:00:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_token_file": "", 2024-03-21T13:00:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-21T13:00:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_curl_buffersize": "524288", 2024-03-21T13:00:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-21T13:00:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_curl_low_speed_time": "300", 2024-03-21T13:00:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_d4n_port": "6379", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data_log_changes_size": "1000", 2024-03-21T13:00:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data_log_num_shards": "128", 2024-03-21T13:00:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-21T13:00:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data_log_window": "30", 2024-03-21T13:00:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-21T13:00:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-21T13:00:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-21T13:00:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-21T13:00:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-21T13:00:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-21T13:00:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-21T13:00:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-21T13:00:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-21T13:00:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-21T13:00:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-21T13:00:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-21T13:00:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-21T13:00:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-21T13:00:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-21T13:00:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-21T13:00:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-21T13:00:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-21T13:00:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-21T13:00:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-21T13:00:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-21T13:00:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-21T13:00:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-21T13:00:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dns_name": "", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dns_s3website_name": "", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dynamic_resharding": "true", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_gc_threads": "true", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_lc_threads": "true", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_ops_log": "false", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_quota_threads": "true", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_static_website": "false", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_usage_log": "false", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enforce_swift_acls": "true", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_exit_timeout_secs": "120", 2024-03-21T13:00:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_expose_bucket": "false", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_extended_http_attrs": "", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_filter": "none", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_frontends": "beast port=7480", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_max_deferred": "50", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_max_objs": "32", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_processor_period": "3600", 2024-03-21T13:00:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_init_timeout": "300", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_kafka_connection_idle": "300", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_kafka_message_timeout": "5000", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_domain": "", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_password": "", 2024-03-21T13:00:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_password_path": "", 2024-03-21T13:00:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_project": "", 2024-03-21T13:00:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_tenant": "", 2024-03-21T13:00:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_token": "", 2024-03-21T13:00:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_token_path": "", 2024-03-21T13:00:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_user": "", 2024-03-21T13:00:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_api_version": "2", 2024-03-21T13:00:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_barbican_domain": "", 2024-03-21T13:00:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_barbican_password": "", 2024-03-21T13:00:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_barbican_project": "", 2024-03-21T13:00:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-21T13:00:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_barbican_user": "", 2024-03-21T13:00:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-21T13:00:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_url": "", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_debug_interval": "-1", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_lock_max_time": "90", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_max_objs": "32", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_max_rules": "1000", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_max_worker": "3", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_thread_delay": "0", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-21T13:00:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ldap_dnattr": "uid", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ldap_searchfilter": "", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_log_http_headers": "", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_log_object_name_utc": "false", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-21T13:00:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_attr_name_len": "0", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_attr_size": "0", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_chunk_size": "4194304", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_listing_results": "1000", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_notify_retries": "10", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_put_param_size": "1048576", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_put_size": "5368709120", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_slo_entries": "1000", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_md_log_max_shards": "64", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-21T13:00:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_mp_lock_max_time": "600", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_max_gc": "300", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-21T13:00:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_num_async_rados_threads": "32", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_num_control_oids": "8", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_numa_node": "-1", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_objexp_chunk_size": "100", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_objexp_gc_interval": "600", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-21T13:00:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_op_thread_timeout": "600", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_op_tracing": "false", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_opa_token": "", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_opa_url": "", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_opa_verify_ssl": "true", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ops_log_rados": "false", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ops_log_socket_path": "", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-21T13:00:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_period_push_interval": "2.000000", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_posix_cache_lanes": "3", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_posix_cache_partitions": "3", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_print_continue": "true", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-21T13:00:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_rados_tracing": "false", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_realm": "", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_realm_id": "", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_region": "", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_request_uri": "", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_reshard_batch_size": "64", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-21T13:00:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_reshard_max_aio": "128", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_reshard_num_logs": "16", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_reshard_thread_interval": "600", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_resolve_cname": "false", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_run_sync_thread": "true", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-21T13:00:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_scheduler_type": "throttler", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_script_uri": "", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_service_provider_name": "", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_client_id": "", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_client_secret": "", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_entry": "sts", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_key": "sts", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_min_session_duration": "900", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_token_introspection_url": "", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_account_in_url": "false", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_auth_entry": "auth", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_auth_url": "", 2024-03-21T13:00:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_custom_header": "", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_need_stats": "true", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_tenant_name": "", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_token_expiration": "86400", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_url": "", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_url_prefix": "swift", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_lease_period": "120", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-21T13:00:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_thread_pool_size": "512", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_comment": "", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_createby": "", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_encoding": "", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_flag": "false", 2024-03-21T13:00:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_origin": "", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_tracker": "", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_trust_forwarded_https": "false", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_usage_max_shards": "32", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_usage_max_user_shards": "1", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_use_opa_authz": "false", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_counters_cache": "false", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_max_buckets": "1000", 2024-03-21T13:00:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_policies_max_num": "100", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_unique_email": "true", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_verify_ssl": "true", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_zone": "", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_zone_id": "", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_zonegroup": "", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_zonegroup_id": "", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgwlc_auto_session_clear": "true", 2024-03-21T13:00:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_block_size": "4096", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cache_size": "536870912", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-21T13:00:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_index_type": "binary_search", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_partition_filters": "false", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_perf": "false", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "run_dir": "/var/run/ceph", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_block_create": "true", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_cache_lru_size": "67108864", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-21T13:00:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_default_max_object_size": "16777216", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_device_size": "53687091200", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_journal_batch_capacity": "16", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_main_device_type": "SSD", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "seastore_segment_size": "67108864", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "set_keepcaps": "false", 2024-03-21T13:00:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "setgroup": "ceph", 2024-03-21T13:00:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "setuser": "ceph", 2024-03-21T13:00:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "setuser_match_path": "", 2024-03-21T13:00:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-21T13:00:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "thp": "false", 2024-03-21T13:00:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "threadpool_default_timeout": "60", 2024-03-21T13:00:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-21T13:00:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "throttler_perf_counter": "true" 2024-03-21T13:00:26.224 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:00:26.405 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 podman[64295]: 2024-03-21 13:00:26.111734522 +0000 UTC m=+2.803070396 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-21T13:00:26.405 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 podman[64295]: 2024-03-21 13:00:26.204117691 +0000 UTC m=+2.895453597 container create 0be4949f37bf2656b78bb5202b6304bf8df71c6218d56b03fae6c287fd729966 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:00:26.715 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 podman[64295]: 2024-03-21 13:00:26.40615407 +0000 UTC m=+3.097489948 container init 0be4949f37bf2656b78bb5202b6304bf8df71c6218d56b03fae6c287fd729966 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:00:26.715 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 podman[64295]: 2024-03-21 13:00:26.408778311 +0000 UTC m=+3.100114177 container start 0be4949f37bf2656b78bb5202b6304bf8df71c6218d56b03fae6c287fd729966 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:00:26.715 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.412Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-21T13:00:26.715 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.412Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-21T13:00:26.715 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.412Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-21T13:00:26.715 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.412Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-21T13:00:26.716 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-21T13:00:26.716 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T13:00:26.716 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-21T13:00:26.716 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=arp 2024-03-21T13:00:26.716 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-21T13:00:26.716 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-21T13:00:26.716 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-21T13:00:26.716 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-21T13:00:26.716 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-21T13:00:26.716 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-21T13:00:26.717 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-21T13:00:26.717 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-21T13:00:26.717 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=edac 2024-03-21T13:00:26.717 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-21T13:00:26.717 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-21T13:00:26.717 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-21T13:00:26.717 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-21T13:00:26.717 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-21T13:00:26.717 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-21T13:00:26.717 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-21T13:00:26.718 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-21T13:00:26.718 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-21T13:00:26.718 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-21T13:00:26.718 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-21T13:00:26.718 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-21T13:00:26.718 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-21T13:00:26.718 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-21T13:00:26.718 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-21T13:00:26.718 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-21T13:00:26.718 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=os 2024-03-21T13:00:26.718 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-21T13:00:26.718 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=stat 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=time 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=uname 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-21T13:00:26.719 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.413Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-21T13:00:26.720 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.414Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-21T13:00:26.720 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b[64345]: ts=2024-03-21T13:00:26.414Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-21T13:00:26.720 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 bash[64295]: 0be4949f37bf2656b78bb5202b6304bf8df71c6218d56b03fae6c287fd729966 2024-03-21T13:00:26.720 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:00:26 smithi204 systemd[1]: Started Ceph node-exporter.b for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T13:00:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:00:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:00:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-mon[33153]: 5.19 scrub starts 2024-03-21T13:00:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-mon[33153]: 5.19 scrub ok 2024-03-21T13:00:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-mon[33153]: 3.b scrub starts 2024-03-21T13:00:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-mon[33153]: 3.b scrub ok 2024-03-21T13:00:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:26 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:27.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:00:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:00:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[29621]: 5.19 scrub starts 2024-03-21T13:00:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[29621]: 5.19 scrub ok 2024-03-21T13:00:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[29621]: 3.b scrub starts 2024-03-21T13:00:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[29621]: 3.b scrub ok 2024-03-21T13:00:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:00:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:00:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[40675]: 5.19 scrub starts 2024-03-21T13:00:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[40675]: 5.19 scrub ok 2024-03-21T13:00:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[40675]: 3.b scrub starts 2024-03-21T13:00:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[40675]: 3.b scrub ok 2024-03-21T13:00:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:26 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:27.151 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-21T13:00:28.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:27 smithi204 ceph-mon[33153]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:28.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:27 smithi204 ceph-mon[33153]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T13:00:28.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:27 smithi163 ceph-mon[40675]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:28.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:27 smithi163 ceph-mon[40675]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T13:00:28.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:27 smithi163 ceph-mon[29621]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:28.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:27 smithi163 ceph-mon[29621]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T13:00:28.052 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:28.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[29621]: 4.8 scrub starts 2024-03-21T13:00:28.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[29621]: 4.8 scrub ok 2024-03-21T13:00:28.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:00:28.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:00:28.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[29621]: 6.c scrub starts 2024-03-21T13:00:28.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[29621]: 6.c scrub ok 2024-03-21T13:00:28.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:28.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[40675]: 4.8 scrub starts 2024-03-21T13:00:28.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[40675]: 4.8 scrub ok 2024-03-21T13:00:28.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:00:28.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:00:28.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[40675]: 6.c scrub starts 2024-03-21T13:00:28.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[40675]: 6.c scrub ok 2024-03-21T13:00:28.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:28 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:29.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:28 smithi204 ceph-mon[33153]: 4.8 scrub starts 2024-03-21T13:00:29.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:28 smithi204 ceph-mon[33153]: 4.8 scrub ok 2024-03-21T13:00:29.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:28 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:00:29.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:28 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:00:29.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:28 smithi204 ceph-mon[33153]: 6.c scrub starts 2024-03-21T13:00:29.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:28 smithi204 ceph-mon[33153]: 6.c scrub ok 2024-03-21T13:00:29.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:28 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:29.831 INFO:teuthology.orchestra.run.smithi163.stdout:mon.a: {} 2024-03-21T13:00:29.832 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:29 smithi163 ceph-mon[29621]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:29.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:29 smithi163 ceph-mon[40675]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:29.842 INFO:teuthology.orchestra.run.smithi163.stdout:mon.b: {} 2024-03-21T13:00:29.854 INFO:teuthology.orchestra.run.smithi163.stdout:mon.c: {} 2024-03-21T13:00:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-21T13:00:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-21T13:00:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-21T13:00:30.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:29 smithi204 ceph-mon[33153]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:30.576 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-21T13:00:30.585 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-21T13:00:30.586 INFO:tasks.radosbench:Beginning radosbench... 2024-03-21T13:00:30.586 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-21T13:00:30.587 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-21T13:00:30.587 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-21T13:00:30.587 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:00:30.587 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:00:30.587 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-21T13:00:30.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-21T13:00:30.588 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-21T13:00:30.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:30.588 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-21T13:00:30.589 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:00:30.589 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-21T13:00:30.589 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-21T13:00:30.589 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-21T13:00:30.590 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:00:30.615 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:30.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:30.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:30.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:30.741 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:30.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:30 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4238357336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:00:30.842 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:30.873 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:30.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:30.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:30.927 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:30 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4238357336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:00:30.927 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:30 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4238357336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:00:30.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:30.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:30.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:31.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:31.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:31.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:31.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:31.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:31.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:31.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:31.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:31.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:31.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:31.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:31.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:31.500 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:31.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:31.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:31.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:00:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:00:31.515 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:31.518 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:31.520 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:31.530 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:31.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:31.625 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:31.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:31.727 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:31.751 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:31.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:31.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:31.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:31.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:31.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:31.978 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:32.001 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:32.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:32.103 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:32.127 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:32.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:32.229 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:32.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[29621]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:32.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[29621]: 6.17 deep-scrub starts 2024-03-21T13:00:32.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[29621]: 6.17 deep-scrub ok 2024-03-21T13:00:32.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:32.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:32.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[29621]: 4.19 scrub starts 2024-03-21T13:00:32.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[29621]: 4.19 scrub ok 2024-03-21T13:00:32.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:32.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[40675]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:32.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[40675]: 6.17 deep-scrub starts 2024-03-21T13:00:32.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[40675]: 6.17 deep-scrub ok 2024-03-21T13:00:32.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:32.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:32.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[40675]: 4.19 scrub starts 2024-03-21T13:00:32.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[40675]: 4.19 scrub ok 2024-03-21T13:00:32.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:31 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:32.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:31 smithi204 ceph-mon[33153]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:32.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:31 smithi204 ceph-mon[33153]: 6.17 deep-scrub starts 2024-03-21T13:00:32.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:31 smithi204 ceph-mon[33153]: 6.17 deep-scrub ok 2024-03-21T13:00:32.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:31 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:32.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:31 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:32.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:31 smithi204 ceph-mon[33153]: 4.19 scrub starts 2024-03-21T13:00:32.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:31 smithi204 ceph-mon[33153]: 4.19 scrub ok 2024-03-21T13:00:32.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:31 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:32.254 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:32.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:32.355 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:32.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:32.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:32.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:32.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:32.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:32.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:32.631 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:32.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:32.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:32.732 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:32.756 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:32.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:32.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:32.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:32.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:32.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:32.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:32.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:33.007 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:33.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:33.090 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:33.109 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:33.132 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:33.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:33.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:33.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:33.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:33.258 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:33.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:33.359 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:33.382 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:33.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:33.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:33.484 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:33.508 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:33.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:33.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:33.610 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:33.634 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:33.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:33.736 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:33.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[29621]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[29621]: 5.16 scrub starts 2024-03-21T13:00:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[29621]: 5.16 scrub ok 2024-03-21T13:00:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[29621]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T13:00:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[29621]: Reconfiguring daemon alertmanager.a on smithi163 2024-03-21T13:00:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[29621]: 3.17 scrub starts 2024-03-21T13:00:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[29621]: 3.17 scrub ok 2024-03-21T13:00:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[40675]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[40675]: 5.16 scrub starts 2024-03-21T13:00:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[40675]: 5.16 scrub ok 2024-03-21T13:00:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[40675]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T13:00:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[40675]: Reconfiguring daemon alertmanager.a on smithi163 2024-03-21T13:00:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[40675]: 3.17 scrub starts 2024-03-21T13:00:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:33 smithi163 ceph-mon[40675]: 3.17 scrub ok 2024-03-21T13:00:33.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:33.763 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:33.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:33.864 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:33.888 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:33.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:33.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:33 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:33 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:33 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:33 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:33 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:33 smithi204 ceph-mon[33153]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:33 smithi204 ceph-mon[33153]: 5.16 scrub starts 2024-03-21T13:00:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:33 smithi204 ceph-mon[33153]: 5.16 scrub ok 2024-03-21T13:00:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:33 smithi204 ceph-mon[33153]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T13:00:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:33 smithi204 ceph-mon[33153]: Reconfiguring daemon alertmanager.a on smithi163 2024-03-21T13:00:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:33 smithi204 ceph-mon[33153]: 3.17 scrub starts 2024-03-21T13:00:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:33 smithi204 ceph-mon[33153]: 3.17 scrub ok 2024-03-21T13:00:34.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:34.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:34.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:34.116 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:34.140 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:34.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:34.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:34.242 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:34.265 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:34.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:34.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:34.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:34.394 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:34.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:34.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:34.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:34.523 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:34.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:34.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:34.625 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:34.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:34.651 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:34.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:34.752 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:34.780 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:34.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:34.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:34.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:34.909 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:34.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:34.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:34.924 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:34 smithi163 ceph-mon[29621]: 4.9 scrub starts 2024-03-21T13:00:34.924 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:34 smithi163 ceph-mon[29621]: 4.9 scrub ok 2024-03-21T13:00:34.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:34 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:34.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:34 smithi163 ceph-mon[40675]: 4.9 scrub starts 2024-03-21T13:00:34.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:34 smithi163 ceph-mon[40675]: 4.9 scrub ok 2024-03-21T13:00:34.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:34 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:35.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:34 smithi204 ceph-mon[33153]: 4.9 scrub starts 2024-03-21T13:00:35.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:34 smithi204 ceph-mon[33153]: 4.9 scrub ok 2024-03-21T13:00:35.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:34 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:35.011 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:35.035 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:35.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:35.137 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:35.162 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:35.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:35.263 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:35.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:35.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:35.393 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:35.417 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:35.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:35.518 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:35.546 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:35.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:35.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:35.648 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:35.671 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:35.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:35.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:35.801 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:35.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:35.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:35.931 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:35.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:35.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:35 smithi204 ceph-mon[33153]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:35 smithi204 ceph-mon[33153]: 6.1f scrub starts 2024-03-21T13:00:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:35 smithi204 ceph-mon[33153]: 6.1f scrub ok 2024-03-21T13:00:36.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:00:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:00:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:35 smithi163 ceph-mon[29621]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:35 smithi163 ceph-mon[29621]: 6.1f scrub starts 2024-03-21T13:00:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:35 smithi163 ceph-mon[29621]: 6.1f scrub ok 2024-03-21T13:00:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:35 smithi163 ceph-mon[40675]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:35 smithi163 ceph-mon[40675]: 6.1f scrub starts 2024-03-21T13:00:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:35 smithi163 ceph-mon[40675]: 6.1f scrub ok 2024-03-21T13:00:36.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:36.060 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:36.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:36.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:36.162 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:36.189 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:36.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:36.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:36.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:36.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:36.420 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:36.444 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:36.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:36.546 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:36.570 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:36.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:36.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:36.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:36.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:36.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:36.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:36 smithi163 ceph-mon[29621]: 3.12 scrub starts 2024-03-21T13:00:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:36 smithi163 ceph-mon[29621]: 3.12 scrub ok 2024-03-21T13:00:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:36 smithi163 ceph-mon[40675]: 3.12 scrub starts 2024-03-21T13:00:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:36 smithi163 ceph-mon[40675]: 3.12 scrub ok 2024-03-21T13:00:36.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:36.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:36.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:36.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:36.954 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:36.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:36.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:36 smithi204 ceph-mon[33153]: 3.12 scrub starts 2024-03-21T13:00:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:36 smithi204 ceph-mon[33153]: 3.12 scrub ok 2024-03-21T13:00:37.056 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:37.079 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:37.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:37.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:37.181 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:37.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:37.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:37.311 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:37.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:37.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:37.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:37.438 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:37.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:37.463 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:37.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:37.563 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:37.587 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:37.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:37.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:37.689 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:37.714 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:37.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:37.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:37.815 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:37.840 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:37.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:37.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:37.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:37.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:38.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:37 smithi204 ceph-mon[33153]: 4.1a deep-scrub starts 2024-03-21T13:00:38.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:37 smithi204 ceph-mon[33153]: 4.1a deep-scrub ok 2024-03-21T13:00:38.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:37 smithi204 ceph-mon[33153]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:38.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:37 smithi204 ceph-mon[33153]: 5.5 scrub starts 2024-03-21T13:00:38.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:37 smithi204 ceph-mon[33153]: 5.5 scrub ok 2024-03-21T13:00:38.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:37 smithi163 ceph-mon[40675]: 4.1a deep-scrub starts 2024-03-21T13:00:38.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:37 smithi163 ceph-mon[40675]: 4.1a deep-scrub ok 2024-03-21T13:00:38.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:37 smithi163 ceph-mon[40675]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:38.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:37 smithi163 ceph-mon[40675]: 5.5 scrub starts 2024-03-21T13:00:38.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:37 smithi163 ceph-mon[40675]: 5.5 scrub ok 2024-03-21T13:00:38.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:37 smithi163 ceph-mon[29621]: 4.1a deep-scrub starts 2024-03-21T13:00:38.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:37 smithi163 ceph-mon[29621]: 4.1a deep-scrub ok 2024-03-21T13:00:38.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:37 smithi163 ceph-mon[29621]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:38.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:37 smithi163 ceph-mon[29621]: 5.5 scrub starts 2024-03-21T13:00:38.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:37 smithi163 ceph-mon[29621]: 5.5 scrub ok 2024-03-21T13:00:38.066 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:38.090 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:38.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:38.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:38.192 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:38.215 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:38.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:38.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:38.317 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:38.344 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:38.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:38.446 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:38.470 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:38.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:38.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:38.600 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:38.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:38.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:38.701 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:38.726 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:38.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:38.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:38.827 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:38.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:38.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:38.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:38 smithi163 ceph-mon[29621]: 3.2 scrub starts 2024-03-21T13:00:38.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:38 smithi163 ceph-mon[29621]: 3.2 scrub ok 2024-03-21T13:00:38.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:38 smithi163 ceph-mon[40675]: 3.2 scrub starts 2024-03-21T13:00:38.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:38 smithi163 ceph-mon[40675]: 3.2 scrub ok 2024-03-21T13:00:38.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:38.984 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:38.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:38.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:39.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:38 smithi204 ceph-mon[33153]: 3.2 scrub starts 2024-03-21T13:00:39.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:38 smithi204 ceph-mon[33153]: 3.2 scrub ok 2024-03-21T13:00:39.086 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:39.109 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:39.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:39.211 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:39.239 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:39.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:39.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:39.340 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:39.364 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:39.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:39.466 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:39.490 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:39.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:39.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:39.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:39.619 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:39.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:39.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:39.720 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:39.744 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:39.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:39.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:39.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:39 smithi163 ceph-mon[29621]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-03-21T13:00:39.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:39 smithi163 ceph-mon[29621]: 4.e scrub starts 2024-03-21T13:00:39.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:39 smithi163 ceph-mon[29621]: 4.e scrub ok 2024-03-21T13:00:39.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:39 smithi163 ceph-mon[29621]: 5.13 scrub starts 2024-03-21T13:00:39.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:39 smithi163 ceph-mon[29621]: 5.13 scrub ok 2024-03-21T13:00:39.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:39 smithi163 ceph-mon[40675]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-03-21T13:00:39.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:39 smithi163 ceph-mon[40675]: 4.e scrub starts 2024-03-21T13:00:39.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:39 smithi163 ceph-mon[40675]: 4.e scrub ok 2024-03-21T13:00:39.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:39 smithi163 ceph-mon[40675]: 5.13 scrub starts 2024-03-21T13:00:39.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:39 smithi163 ceph-mon[40675]: 5.13 scrub ok 2024-03-21T13:00:39.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:39.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:39.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:39.912 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:00:39.912 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:00:39.912 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 90, 2024-03-21T13:00:39.912 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "d4bf79c0-e781-11ee-95cd-87774f69a715", 2024-03-21T13:00:39.913 INFO:teuthology.orchestra.run.smithi163.stdout: "created": "2024-03-21T12:52:56.031419+0000", 2024-03-21T13:00:39.913 INFO:teuthology.orchestra.run.smithi163.stdout: "modified": "2024-03-21T13:00:26.711890+0000", 2024-03-21T13:00:39.913 INFO:teuthology.orchestra.run.smithi163.stdout: "last_up_change": "2024-03-21T12:58:23.775302+0000", 2024-03-21T13:00:39.913 INFO:teuthology.orchestra.run.smithi163.stdout: "last_in_change": "2024-03-21T12:58:05.763916+0000", 2024-03-21T13:00:39.913 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-21T13:00:39.913 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_num": 5799936, 2024-03-21T13:00:39.913 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_set": [ 2024-03-21T13:00:39.913 INFO:teuthology.orchestra.run.smithi163.stdout: "pglog_hardlimit", 2024-03-21T13:00:39.913 INFO:teuthology.orchestra.run.smithi163.stdout: "purged_snapdirs", 2024-03-21T13:00:39.913 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_deletes", 2024-03-21T13:00:39.913 INFO:teuthology.orchestra.run.smithi163.stdout: "sortbitwise" 2024-03-21T13:00:39.913 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:00:39.913 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_version": 18, 2024-03-21T13:00:39.914 INFO:teuthology.orchestra.run.smithi163.stdout: "full_ratio": 0.94999998807907104, 2024-03-21T13:00:39.914 INFO:teuthology.orchestra.run.smithi163.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-21T13:00:39.914 INFO:teuthology.orchestra.run.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-21T13:00:39.914 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_snapshot": "", 2024-03-21T13:00:39.914 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_max": 6, 2024-03-21T13:00:39.914 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd": 8, 2024-03-21T13:00:39.914 INFO:teuthology.orchestra.run.smithi163.stdout: "require_min_compat_client": "luminous", 2024-03-21T13:00:39.914 INFO:teuthology.orchestra.run.smithi163.stdout: "min_compat_client": "luminous", 2024-03-21T13:00:39.914 INFO:teuthology.orchestra.run.smithi163.stdout: "require_osd_release": "squid", 2024-03-21T13:00:39.914 INFO:teuthology.orchestra.run.smithi163.stdout: "allow_crimson": false, 2024-03-21T13:00:39.914 INFO:teuthology.orchestra.run.smithi163.stdout: "pools": [ 2024-03-21T13:00:39.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.914 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 1, 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".mgr", 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:56:20.557490+0000", 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:00:39.915 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 1, 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 1, 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 1, 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 1, 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 1, 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.916 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "21", 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:00:39.917 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:00:39.918 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:00:39.918 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:00:39.918 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:00:39.918 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:00:39.918 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:00:39.918 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:00:39.918 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:00:39.918 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:00:39.918 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:00:39.918 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:00:39.918 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:00:39.918 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:00:39.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_max": 32, 2024-03-21T13:00:39.919 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_min": 1 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": {} 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 7.8899998664855957, 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 7.8899998664855957, 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.37999999523162842, 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 3, 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 3, 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 1, 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 1, 2024-03-21T13:00:39.920 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 1 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 2, 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "datapool", 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:35.713383+0000", 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:00:39.921 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 3, 2024-03-21T13:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 3, 2024-03-21T13:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 3, 2024-03-21T13:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 3, 2024-03-21T13:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 3, 2024-03-21T13:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "67", 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 2, 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 67, 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:00:39.923 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:00:39.924 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:00:39.924 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:00:39.924 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:00:39.924 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:00:39.924 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:00:39.924 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:00:39.924 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:00:39.924 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:00:39.924 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:00:39.924 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:00:39.924 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:00:39.924 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:00:39.924 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:00:39.925 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:00:39.926 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:00:39.926 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:00:39.926 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:00:39.926 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:00:39.926 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd": {} 2024-03-21T13:00:39.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.926 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:00:39.926 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:00:39.926 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.6500000953674316, 2024-03-21T13:00:39.926 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.6500000953674316, 2024-03-21T13:00:39.926 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.87999999523162842, 2024-03-21T13:00:39.926 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-21T13:00:39.926 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 1, 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 1, 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 1 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 3, 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".rgw.root", 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:36.436384+0000", 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:00:39.927 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:00:39.928 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:00:39.928 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:00:39.928 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:00:39.928 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:00:39.928 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:00:39.928 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:00:39.928 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:00:39.928 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:00:39.928 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:00:39.928 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:00:39.928 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:00:39.928 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:00:39.928 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:00:39.929 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:00:39.929 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:00:39.929 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:00:39.929 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:00:39.929 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:00:39.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.929 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "65", 2024-03-21T13:00:39.930 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:00:39.930 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:00:39.930 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:00:39.930 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:00:39.930 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:00:39.930 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:00:39.931 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:00:39.932 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:00:39.932 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:00:39.932 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:00:39.932 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:00:39.932 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:00:39.932 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:00:39.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.932 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:00:39.932 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:00:39.932 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:00:39.932 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:00:39.932 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:00:39.932 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 1.5, 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 1.5, 2024-03-21T13:00:39.933 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 1, 2024-03-21T13:00:39.934 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.5, 2024-03-21T13:00:39.934 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.5, 2024-03-21T13:00:39.934 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 1, 2024-03-21T13:00:39.934 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 1, 2024-03-21T13:00:39.934 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 1 2024-03-21T13:00:39.934 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.934 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 4, 2024-03-21T13:00:39.934 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.log", 2024-03-21T13:00:39.934 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:38.049304+0000", 2024-03-21T13:00:39.934 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:00:39.935 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:00:39.936 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:00:39.936 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:00:39.936 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:00:39.936 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:00:39.936 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:00:39.936 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:00:39.936 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:00:39.936 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:00:39.936 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:00:39.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.936 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "67", 2024-03-21T13:00:39.936 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:00:39.936 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:00:39.937 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:00:39.937 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:00:39.937 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:00:39.937 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:00:39.937 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:00:39.937 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:00:39.937 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:00:39.937 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:00:39.937 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:00:39.937 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:00:39.937 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:00:39.937 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:00:39.937 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:00:39.938 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:00:39.938 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:00:39.938 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:00:39.938 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:00:39.938 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:00:39.938 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:00:39.938 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:00:39.938 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:00:39.938 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:00:39.938 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:00:39.938 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:00:39.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.938 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:00:39.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.940 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:00:39.940 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:00:39.940 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.25, 2024-03-21T13:00:39.940 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.25, 2024-03-21T13:00:39.940 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 1, 2024-03-21T13:00:39.940 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 2.25, 2024-03-21T13:00:39.940 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 2.25, 2024-03-21T13:00:39.940 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 1, 2024-03-21T13:00:39.940 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 1, 2024-03-21T13:00:39.940 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 1 2024-03-21T13:00:39.940 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 5, 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.control", 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:39.990512+0000", 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:00:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:00:39.942 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:00:39.942 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:00:39.942 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:00:39.942 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:00:39.942 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:00:39.942 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:00:39.942 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:00:39.942 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:00:39.942 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:00:39.942 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:00:39.942 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:00:39.942 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:00:39.942 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:00:39.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.943 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "69", 2024-03-21T13:00:39.943 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:00:39.943 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:00:39.943 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:00:39.943 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:00:39.943 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:00:39.943 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:00:39.943 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:00:39.943 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:00:39.943 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:00:39.943 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:00:39.943 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:00:39.944 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:00:39.945 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:00:39.946 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:00:39.946 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:00:39.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.946 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:00:39.946 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:00:39.946 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 1.25, 2024-03-21T13:00:39.946 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 1.25, 2024-03-21T13:00:39.946 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 1, 2024-03-21T13:00:39.946 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.25, 2024-03-21T13:00:39.946 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.25, 2024-03-21T13:00:39.946 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 1, 2024-03-21T13:00:39.946 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 1, 2024-03-21T13:00:39.946 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 1 2024-03-21T13:00:39.947 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.947 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 6, 2024-03-21T13:00:39.947 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.meta", 2024-03-21T13:00:39.947 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:41.463686+0000", 2024-03-21T13:00:39.947 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:00:39.947 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:00:39.947 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:00:39.947 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:00:39.947 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:00:39.947 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:00:39.947 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:00:39.948 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "71", 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:00:39.949 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:00:39.950 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:00:39.951 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:00:39.951 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:00:39.951 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:00:39.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.951 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:00:39.951 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:00:39.951 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:00:39.951 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:00:39.951 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:00:39.951 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:00:39.951 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:00:39.951 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:00:39.951 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_bias": 4 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 1.75, 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 1.75, 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 1, 2024-03-21T13:00:39.952 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.75, 2024-03-21T13:00:39.953 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.75, 2024-03-21T13:00:39.953 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 1, 2024-03-21T13:00:39.953 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 1, 2024-03-21T13:00:39.953 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 1 2024-03-21T13:00:39.953 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.953 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.953 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:00:39.953 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:00:39.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.953 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-03-21T13:00:39.953 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f", 2024-03-21T13:00:39.953 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:00:39.953 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:00:39.954 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:00:39.954 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:00:39.954 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:00:39.954 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:00:39.954 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 8, 2024-03-21T13:00:39.954 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 69, 2024-03-21T13:00:39.954 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:00:39.954 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:00:39.954 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:00:39.954 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.954 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.954 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6801", 2024-03-21T13:00:39.955 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:00:39.955 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.955 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.955 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:00:39.955 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.955 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.955 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6802", 2024-03-21T13:00:39.955 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:00:39.955 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.955 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6804", 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.956 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6803", 2024-03-21T13:00:39.957 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:00:39.957 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.957 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.957 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6801/3086410981", 2024-03-21T13:00:39.957 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6802/3086410981", 2024-03-21T13:00:39.957 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6804/3086410981", 2024-03-21T13:00:39.957 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6803/3086410981", 2024-03-21T13:00:39.957 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:00:39.957 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:00:39.957 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:00:39.957 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.958 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-03-21T13:00:39.958 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "36e05f89-c238-41f7-be65-1553b529e8ce", 2024-03-21T13:00:39.958 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:00:39.958 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:00:39.958 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:00:39.958 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:00:39.958 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:00:39.958 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:00:39.958 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 13, 2024-03-21T13:00:39.958 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 69, 2024-03-21T13:00:39.958 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:00:39.958 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6805", 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.959 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6806", 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6808", 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.960 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6807", 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6805/3666062784", 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6806/3666062784", 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6808/3666062784", 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6807/3666062784", 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:00:39.961 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:00:39.962 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:00:39.962 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.962 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-03-21T13:00:39.962 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f", 2024-03-21T13:00:39.962 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:00:39.962 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:00:39.962 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:00:39.962 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:00:39.962 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:00:39.962 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:00:39.962 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 18, 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 75, 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6809", 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:00:39.963 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6810", 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6812", 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:00:39.964 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6811", 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6809/4009213507", 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6810/4009213507", 2024-03-21T13:00:39.965 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6812/4009213507", 2024-03-21T13:00:39.966 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6811/4009213507", 2024-03-21T13:00:39.966 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:00:39.966 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:00:39.966 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:00:39.966 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.966 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-03-21T13:00:39.966 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d", 2024-03-21T13:00:39.966 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:00:39.966 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:00:39.966 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:00:39.966 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 25, 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 75, 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6813", 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.967 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6814", 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.968 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.969 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6816", 2024-03-21T13:00:39.969 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:00:39.969 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.969 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.969 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:00:39.969 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.969 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.969 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6815", 2024-03-21T13:00:39.969 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:00:39.969 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.969 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6813/3981878189", 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6814/3981878189", 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6816/3981878189", 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6815/3981878189", 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471", 2024-03-21T13:00:39.970 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:00:39.971 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:00:39.971 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:00:39.971 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:00:39.971 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:00:39.971 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:00:39.971 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 30, 2024-03-21T13:00:39.971 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 75, 2024-03-21T13:00:39.971 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:00:39.971 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:00:39.971 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:00:39.971 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.971 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6800", 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6801", 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:00:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6803", 2024-03-21T13:00:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:00:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:00:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.974 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6802", 2024-03-21T13:00:39.974 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:00:39.974 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.974 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.974 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6800/719276054", 2024-03-21T13:00:39.974 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6801/719276054", 2024-03-21T13:00:39.974 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6803/719276054", 2024-03-21T13:00:39.974 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6802/719276054", 2024-03-21T13:00:39.974 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:00:39.974 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:00:39.974 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:00:39.974 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.975 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-03-21T13:00:39.975 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1", 2024-03-21T13:00:39.975 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:00:39.975 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:00:39.975 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:00:39.975 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:00:39.975 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:00:39.975 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:00:39.975 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 39, 2024-03-21T13:00:39.975 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 69, 2024-03-21T13:00:39.975 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6804", 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.976 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6805", 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6807", 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.978 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:00:39.978 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.978 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.978 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6806", 2024-03-21T13:00:39.978 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:00:39.978 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.978 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.978 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6804/1346947668", 2024-03-21T13:00:39.978 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6805/1346947668", 2024-03-21T13:00:39.978 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6807/1346947668", 2024-03-21T13:00:39.978 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6806/1346947668", 2024-03-21T13:00:39.979 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:00:39.979 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:00:39.979 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:00:39.979 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.979 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-03-21T13:00:39.979 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b", 2024-03-21T13:00:39.979 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:00:39.979 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:00:39.979 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:00:39.979 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:00:39.979 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 47, 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 67, 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6808", 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.981 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:00:39.981 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.981 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.981 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6809", 2024-03-21T13:00:39.981 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:00:39.981 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.981 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.981 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:00:39.981 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6811", 2024-03-21T13:00:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:00:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:00:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6810", 2024-03-21T13:00:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:00:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.983 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.983 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6808/200077988", 2024-03-21T13:00:39.983 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6809/200077988", 2024-03-21T13:00:39.983 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6811/200077988", 2024-03-21T13:00:39.983 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6810/200077988", 2024-03-21T13:00:39.983 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:00:39.983 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:00:39.983 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:00:39.983 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.983 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-03-21T13:00:39.984 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49", 2024-03-21T13:00:39.984 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:00:39.984 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:00:39.984 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:00:39.984 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:00:39.984 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:00:39.984 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:00:39.984 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 56, 2024-03-21T13:00:39.984 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 69, 2024-03-21T13:00:39.984 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:00:39.984 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:00:39.984 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:00:39.984 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.985 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.985 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6812", 2024-03-21T13:00:39.985 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:00:39.985 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.985 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.985 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:00:39.985 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.985 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:39.986 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.987 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6813", 2024-03-21T13:00:39.987 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:00:39.987 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.987 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.987 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:00:39.987 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.987 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.987 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6815", 2024-03-21T13:00:39.987 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:00:39.987 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.987 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.988 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:00:39.988 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:00:39.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.988 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:00:39.988 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6814", 2024-03-21T13:00:39.988 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:00:39.988 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.988 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.988 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6812/315664888", 2024-03-21T13:00:39.988 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6813/315664888", 2024-03-21T13:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6815/315664888", 2024-03-21T13:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6814/315664888", 2024-03-21T13:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_xinfo": [ 2024-03-21T13:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-03-21T13:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:00:39.990 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:00:39.990 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:00:39.990 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:00:39.990 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T12:59:31.170335+0000", 2024-03-21T13:00:39.990 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:00:39.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.990 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-03-21T13:00:39.990 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:00:39.990 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:00:39.990 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:00:39.990 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:00:39.990 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T12:59:36.599420+0000", 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T12:59:52.444082+0000", 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.992 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-03-21T13:00:39.992 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:00:39.992 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:00:39.992 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:00:39.992 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:00:39.992 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:00:39.992 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:00:25.915761+0000", 2024-03-21T13:00:39.992 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:00:39.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.992 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-03-21T13:00:39.992 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:00:39.992 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T12:59:51.277908+0000", 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:00:39.993 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:00:09.825968+0000", 2024-03-21T13:00:39.994 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:00:39.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.994 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-03-21T13:00:39.994 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:00:39.994 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:00:39.994 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:00:39.994 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:00:39.994 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:00:39.994 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T12:59:25.395685+0000", 2024-03-21T13:00:39.994 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:00:39.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.995 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-03-21T13:00:39.995 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:00:39.995 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:00:39.995 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:00:39.995 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:00:39.995 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:00:39.995 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T12:59:50.547901+0000", 2024-03-21T13:00:39.995 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:00:39.995 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.995 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:00:39.995 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap": [], 2024-03-21T13:00:39.995 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_items": [ 2024-03-21T13:00:39.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.996 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.2", 2024-03-21T13:00:39.996 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:00:39.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.996 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-03-21T13:00:39.996 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-03-21T13:00:39.996 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.996 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.996 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.18", 2024-03-21T13:00:39.996 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:00:39.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.997 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-03-21T13:00:39.997 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-03-21T13:00:39.997 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.997 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.997 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.1d", 2024-03-21T13:00:39.997 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:00:39.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:00:39.997 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-03-21T13:00:39.997 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-03-21T13:00:39.997 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.997 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:00:39.998 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:39.998 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:00:39.998 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_primaries": [], 2024-03-21T13:00:39.998 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_temp": [], 2024-03-21T13:00:39.998 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_temp": [], 2024-03-21T13:00:39.998 INFO:teuthology.orchestra.run.smithi163.stdout: "blocklist": { 2024-03-21T13:00:39.998 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3453855427": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:00:39.998 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/2571244897": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:00:39.998 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/3073378854": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:00:39.998 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1334591766": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:00:39.998 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/116934309": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:00:39.998 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3890775846": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1783921524": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2553440211": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1812694077": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1070540283": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/2354157786": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1591817392": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/4152198951": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/1798115633": "2024-03-22T12:54:00.224135+0000" 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: "range_blocklist": {}, 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profiles": { 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: "default": { 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-03-21T13:00:39.999 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-03-21T13:00:40.000 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-03-21T13:00:40.000 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-03-21T13:00:40.000 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van" 2024-03-21T13:00:40.000 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:40.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:00:40.000 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps_queue": [], 2024-03-21T13:00:40.000 INFO:teuthology.orchestra.run.smithi163.stdout: "new_removed_snaps": [], 2024-03-21T13:00:40.000 INFO:teuthology.orchestra.run.smithi163.stdout: "new_purged_snaps": [], 2024-03-21T13:00:40.000 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_node_flags": {}, 2024-03-21T13:00:40.000 INFO:teuthology.orchestra.run.smithi163.stdout: "device_class_flags": {}, 2024-03-21T13:00:40.000 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode": { 2024-03-21T13:00:40.000 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_enabled": false, 2024-03-21T13:00:40.000 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_bucket_count": 0, 2024-03-21T13:00:40.001 INFO:teuthology.orchestra.run.smithi163.stdout: "degraded_stretch_mode": 0, 2024-03-21T13:00:40.001 INFO:teuthology.orchestra.run.smithi163.stdout: "recovering_stretch_mode": 0, 2024-03-21T13:00:40.001 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_bucket": 0 2024-03-21T13:00:40.001 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:00:40.001 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:00:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:39 smithi204 ceph-mon[33153]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-03-21T13:00:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:39 smithi204 ceph-mon[33153]: 4.e scrub starts 2024-03-21T13:00:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:39 smithi204 ceph-mon[33153]: 4.e scrub ok 2024-03-21T13:00:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:39 smithi204 ceph-mon[33153]: 5.13 scrub starts 2024-03-21T13:00:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:39 smithi204 ceph-mon[33153]: 5.13 scrub ok 2024-03-21T13:00:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:40.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:40.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:40.154 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:40.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:40.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:40.256 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:40.279 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:40.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:40.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:40.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:40.409 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:40.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:40.511 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:40.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:40.536 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:40.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:40.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:40.645 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:00:40.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:40.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:40.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:40.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:40 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4079142445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:00:40.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:40 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2400826757' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:00:40.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:40 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4079142445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:00:40.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:40 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2400826757' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:00:40.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:40.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:40.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:40.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:40.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:40.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:40.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:41.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:40 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4079142445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:00:41.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:40 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2400826757' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:00:41.024 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:41.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:41.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:41.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:41.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:41.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:41.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:41.282 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:41.306 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:41.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:41.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:41.436 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:41.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:41.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:41.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:00:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:00:41.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:41.565 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:41.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:41.667 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:41.692 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:41.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:41.698 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'unique_pool_0' created 2024-03-21T13:00:41.785 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:41 smithi204 ceph-mon[33153]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:41.785 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:41 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:00:41.785 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:41 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2400826757' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:00:41.785 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:41 smithi204 ceph-mon[33153]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T13:00:41.785 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:41 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:41.785 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:41 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1798503126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-21T13:00:41.794 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:41.819 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:41.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:41.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:41.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:41.945 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:41.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:42.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:41 smithi163 ceph-mon[40675]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:42.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:41 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:00:42.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:41 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2400826757' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:00:42.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:41 smithi163 ceph-mon[40675]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T13:00:42.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:41 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:42.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:41 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1798503126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-21T13:00:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:41 smithi163 ceph-mon[29621]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:41 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:00:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:41 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2400826757' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:00:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:41 smithi163 ceph-mon[29621]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T13:00:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:41 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:41 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1798503126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-21T13:00:42.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:42.070 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:42.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:42.171 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:42.195 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:42.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:42.296 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:42.320 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:42.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:42.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:42.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:42.445 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:42.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:42.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:42.449 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-21T13:00:42.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:42.571 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:42.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:42.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:42.696 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:42.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:42.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:42 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-21T13:00:42.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:42 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-21T13:00:42.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:42 smithi163 ceph-mon[29621]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T13:00:42.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:42 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-21T13:00:42.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:42 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-21T13:00:42.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:42 smithi163 ceph-mon[40675]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T13:00:42.798 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:42.821 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:42.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:42.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:42.920 INFO:tasks.thrashosds.thrasher:j is {'epoch': 90, 'fsid': 'd4bf79c0-e781-11ee-95cd-87774f69a715', 'created': '2024-03-21T12:52:56.031419+0000', 'modified': '2024-03-21T13:00:26.711890+0000', 'last_up_change': '2024-03-21T12:58:23.775302+0000', 'last_in_change': '2024-03-21T12:58:05.763916+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T12:56:20.557490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-21T12:58:35.713383+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-21T12:58:36.436384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-21T12:58:38.049304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-21T12:58:39.990512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-21T12:58:41.463686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'c34a3b25-9e02-46e6-8f9a-d0af2615258f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6801', 'nonce': 3086410981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6802', 'nonce': 3086410981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6804', 'nonce': 3086410981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6803', 'nonce': 3086410981}]}, 'public_addr': '172.21.15.163:6801/3086410981', 'cluster_addr': '172.21.15.163:6802/3086410981', 'heartbeat_back_addr': '172.21.15.163:6804/3086410981', 'heartbeat_front_addr': '172.21.15.163:6803/3086410981', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '36e05f89-c238-41f7-be65-1553b529e8ce', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6805', 'nonce': 3666062784}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6806', 'nonce': 3666062784}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6808', 'nonce': 3666062784}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6807', 'nonce': 3666062784}]}, 'public_addr': '172.21.15.163:6805/3666062784', 'cluster_addr': '172.21.15.163:6806/3666062784', 'heartbeat_back_addr': '172.21.15.163:6808/3666062784', 'heartbeat_front_addr': '172.21.15.163:6807/3666062784', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '968ec78d-25b8-4905-b328-d490c38ca85f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 75, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6809', 'nonce': 4009213507}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6810', 'nonce': 4009213507}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6812', 'nonce': 4009213507}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6811', 'nonce': 4009213507}]}, 'public_addr': '172.21.15.163:6809/4009213507', 'cluster_addr': '172.21.15.163:6810/4009213507', 'heartbeat_back_addr': '172.21.15.163:6812/4009213507', 'heartbeat_front_addr': '172.21.15.163:6811/4009213507', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '67dfa4f6-1709-48f6-971d-c6287233492d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 75, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6813', 'nonce': 3981878189}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6814', 'nonce': 3981878189}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6816', 'nonce': 3981878189}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6815', 'nonce': 3981878189}]}, 'public_addr': '172.21.15.163:6813/3981878189', 'cluster_addr': '172.21.15.163:6814/3981878189', 'heartbeat_back_addr': '172.21.15.163:6816/3981878189', 'heartbeat_front_addr': '172.21.15.163:6815/3981878189', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7fd311a9-f3c2-456d-8ab1-341cf5b33471', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 75, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 719276054}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6801', 'nonce': 719276054}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6803', 'nonce': 719276054}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 719276054}]}, 'public_addr': '172.21.15.204:6800/719276054', 'cluster_addr': '172.21.15.204:6801/719276054', 'heartbeat_back_addr': '172.21.15.204:6803/719276054', 'heartbeat_front_addr': '172.21.15.204:6802/719276054', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a87759c8-251c-415d-9387-a49f7d2f36d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 1346947668}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6805', 'nonce': 1346947668}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6807', 'nonce': 1346947668}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 1346947668}]}, 'public_addr': '172.21.15.204:6804/1346947668', 'cluster_addr': '172.21.15.204:6805/1346947668', 'heartbeat_back_addr': '172.21.15.204:6807/1346947668', 'heartbeat_front_addr': '172.21.15.204:6806/1346947668', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5eb135cd-f677-4d0c-b23a-738f31615b6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 200077988}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6809', 'nonce': 200077988}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6811', 'nonce': 200077988}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 200077988}]}, 'public_addr': '172.21.15.204:6808/200077988', 'cluster_addr': '172.21.15.204:6809/200077988', 'heartbeat_back_addr': '172.21.15.204:6811/200077988', 'heartbeat_front_addr': '172.21.15.204:6810/200077988', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4f528e10-2c93-4bc2-a54c-06ca4132ed49', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 315664888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6813', 'nonce': 315664888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6815', 'nonce': 315664888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 315664888}]}, 'public_addr': '172.21.15.204:6812/315664888', 'cluster_addr': '172.21.15.204:6813/315664888', 'heartbeat_back_addr': '172.21.15.204:6815/315664888', 'heartbeat_front_addr': '172.21.15.204:6814/315664888', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T12:59:31.170335+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T12:59:36.599420+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T12:59:52.444082+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:00:25.915761+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T12:59:51.277908+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:00:09.825968+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T12:59:25.395685+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T12:59:50.547901+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.163:0/3453855427': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/2571244897': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:6800/3073378854': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/1334591766': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/116934309': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:0/3890775846': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/1783921524': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:6800/2553440211': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1812694077': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1070540283': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/2354157786': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1591817392': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/4152198951': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:6800/1798115633': '2024-03-22T12:54:00.224135+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-21T13:00:42.921 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-21T13:00:42.923 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:42.946 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:42.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:42.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:43.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:42 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-21T13:00:43.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:42 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-21T13:00:43.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:42 smithi204 ceph-mon[33153]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T13:00:43.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:43.071 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:43.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:43.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:43.173 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:43.197 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:43.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:43.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:43.299 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:43.324 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:43.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:43.425 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:43.448 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:43.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:43.550 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:43.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:43.575 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:43.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:43.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:43.701 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:43.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:43.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:43.826 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:43.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:43.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:43.839 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:00:43.862 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:43 smithi163 ceph-mon[29621]: pgmap v58: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:00:43.863 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:43 smithi163 ceph-mon[29621]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:00:43.863 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:43 smithi163 ceph-mon[29621]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T13:00:43.863 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:43 smithi163 ceph-mon[40675]: pgmap v58: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:00:43.863 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:43 smithi163 ceph-mon[40675]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:00:43.863 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:43 smithi163 ceph-mon[40675]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T13:00:43.915 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:00:43.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:43.935 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:00:43.951 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:43.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:43.954 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:00:43.974 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:00:43.994 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:00:44.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:43 smithi204 ceph-mon[33153]: pgmap v58: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:00:44.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:43 smithi204 ceph-mon[33153]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:00:44.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:43 smithi204 ceph-mon[33153]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T13:00:44.014 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:00:44.044 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:00:44.053 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:44.069 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:00:44.071 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:00:44.071 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:00:44.071 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:00:44.071 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:00:44.071 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:00:44.072 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:00:44.072 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:00:44.072 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:00:44.077 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:44.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:44.178 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:44.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:44.202 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:44.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:44.303 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:44.328 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:44.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:44.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:44.430 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:44.454 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:44.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:44.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:44.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:44.581 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:44.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:44.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:44.683 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:44.708 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:44.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:44.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:44.810 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:44.834 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:44.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:44.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:44.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:44 smithi163 ceph-mon[29621]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T13:00:44.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3416150847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:00:44.848 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:44 smithi163 ceph-mon[40675]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T13:00:44.848 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3416150847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:00:44.936 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:44.960 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:44.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:44.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:45.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:44 smithi204 ceph-mon[33153]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T13:00:45.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3416150847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:00:45.062 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:45.085 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:45.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:45.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:45.186 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:45.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:45.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:45.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:45.315 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:45.339 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:45.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:45.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:45.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:45.449 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:00:45.449 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:00:45.468 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:45.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:45.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:45.569 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:45.597 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:45.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:45.699 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:45.723 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:45.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:45.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:45.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:45.853 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:45.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:45.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:45.904 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:45 smithi163 ceph-mon[29621]: pgmap v61: 148 pgs: 6 unknown, 142 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:45.904 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:45 smithi163 ceph-mon[40675]: pgmap v61: 148 pgs: 6 unknown, 142 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:45.905 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:00:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:00:45.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:45.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:45.980 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:45.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:46.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:45 smithi204 ceph-mon[33153]: pgmap v61: 148 pgs: 6 unknown, 142 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:46.081 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:46.104 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:46.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:46.206 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:46.229 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:46.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:46.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:46.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:46.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:46.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:46.487 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:46.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:46.589 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:46.614 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:46.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:46.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:46.716 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:46.740 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:46.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:46.760 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:46 smithi163 systemd[1]: Stopping Ceph alertmanager.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:00:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:46.841 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:46.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:46.865 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:46.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:46.966 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:46.989 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:46.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:46.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:47.091 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:47.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:47.115 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:47.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:47.216 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:47.245 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:47.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:47.250 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-21T13:00:47.278 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:47.345 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:47.368 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:47.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:47.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:47.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:47.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:47.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:47.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:47.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:47.622 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:47.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:47.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:47.724 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:47.748 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:47.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:47.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:47.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:47.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:47.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:47.921 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:00:47.921 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:00:47.932 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:00:47.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:48.008 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:48.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:48.109 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:48.128 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:48.145 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:48.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:48.247 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:48.274 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:48.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:48.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:48.376 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:48.400 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:48.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:48.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:48.428 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:48 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[75154]: ts=2024-03-21T13:00:48.107Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-21T13:00:48.428 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:48 smithi163 podman[79578]: 2024-03-21 13:00:48.128196918 +0000 UTC m=+0.148934455 container died 846b4d858ab0fecb43a36280744e3bb3e0fb3170315f16e7de3bfa260c63c1fe (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:00:48.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:48 smithi163 ceph-mon[29621]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-03-21T13:00:48.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:48 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:00:48.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:48 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:00:48.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:48 smithi163 ceph-mon[29621]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T13:00:48.429 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:48 smithi163 ceph-mon[40675]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-03-21T13:00:48.429 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:48 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:00:48.429 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:48 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:00:48.429 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:48 smithi163 ceph-mon[40675]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T13:00:48.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:48.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:48 smithi204 ceph-mon[33153]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-03-21T13:00:48.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:48 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:00:48.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:48 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:00:48.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:48 smithi204 ceph-mon[33153]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T13:00:48.530 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:48.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:48.631 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:48.658 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:48.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:48.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:48.734 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:48 smithi163 podman[79578]: 2024-03-21 13:00:48.611883702 +0000 UTC m=+0.632621291 container cleanup 846b4d858ab0fecb43a36280744e3bb3e0fb3170315f16e7de3bfa260c63c1fe (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:00:48.735 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:48 smithi163 bash[79578]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T13:00:48.760 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:48.783 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:48.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:48.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:48.914 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:48.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:48.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:49.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:49.043 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:49.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:49.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:49.145 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:49.170 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:49.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:49.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:49.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:49.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:49.402 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:49.430 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:49.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:49.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:49.562 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:49.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:49.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:49.664 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:49.689 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:49.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:49.790 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:49.819 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:49.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:49.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:49.920 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:49.945 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:49.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:50.021 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:49 smithi163 podman[79631]: 2024-03-21 13:00:49.928941362 +0000 UTC m=+1.794680968 container remove 846b4d858ab0fecb43a36280744e3bb3e0fb3170315f16e7de3bfa260c63c1fe (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:00:50.021 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:49 smithi163 ceph-mon[29621]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:50.021 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:49 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:50.021 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:49 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:50.021 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:49 smithi163 ceph-mon[40675]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:50.022 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:49 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:50.022 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:49 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:50.048 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:50.072 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:50.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:50.173 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:50.201 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:50.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:50.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:49 smithi204 ceph-mon[33153]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:50.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:49 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:00:50.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:49 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:50.286 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:50 smithi163 podman[79631]: 2024-03-21 13:00:50.021410477 +0000 UTC m=+1.887150083 volume remove d7f5bf520d3e515760b26142a3c7e5f250aa7d579e3e6fea79fa6f641f700197 2024-03-21T13:00:50.303 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:50.327 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:50.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:50.429 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:50.452 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:50.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:50.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:50.547 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:00:50.554 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:50.586 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:50.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:50.688 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:50.717 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:50.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:50.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:50.819 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:50.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:50.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:50.869 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:50 smithi163 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T13:00:50.869 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:50 smithi163 systemd[1]: Stopped Ceph alertmanager.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T13:00:50.869 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:50 smithi163 systemd[1]: Starting Ceph alertmanager.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:00:50.948 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:50.972 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:50.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:50.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:51.073 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:51.097 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:51.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:51.198 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:51.222 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:51.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:51.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:51.324 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:51.348 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:51.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:51.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:51.449 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:51.474 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:51.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:51.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:00:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:00:51.575 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:51.603 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:51.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:51.704 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:51.732 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:51.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:51.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:51.774 INFO:teuthology.orchestra.run.smithi163.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-21T13:00:51.834 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:51.859 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:51.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:51.960 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:51.988 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:51.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:52.090 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:52.113 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:52.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:52.215 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:52.218 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:51 smithi204 ceph-mon[33153]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:00:52.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:51 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4283673734' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:00:52.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:51 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:52.239 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:52.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:52.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:51 smithi163 ceph-mon[29621]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:00:52.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:51 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4283673734' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:00:52.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:51 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:52.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:51 smithi163 ceph-mon[40675]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:00:52.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:51 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4283673734' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:00:52.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:51 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:52.341 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:52.368 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:52.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:52.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:52.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:52.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:52.599 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:52.624 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:52.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:52.727 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:52.762 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:52.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:52.864 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:52.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:52.892 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:52.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:52.993 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:53.020 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:53.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:53.091 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:52 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4283673734' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:00:53.091 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:52 smithi163 ceph-mon[29621]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T13:00:53.091 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:52 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:00:53.091 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:52 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:00:53.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:52 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4283673734' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:00:53.092 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:52 smithi163 ceph-mon[40675]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T13:00:53.092 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:52 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:00:53.092 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:52 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:00:53.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:53.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:53.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:53.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:52 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4283673734' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:00:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:52 smithi204 ceph-mon[33153]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T13:00:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:52 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:00:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:52 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:00:53.280 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:53.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:53.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:53.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:53.409 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:53.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:53.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:53.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:53.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:53.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:53.640 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:53.665 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:53.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:53.767 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:53.792 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:53.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:53.894 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:53.919 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:53.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:53.974 DEBUG:teuthology.orchestra.run.smithi163:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-21T13:00:53.975 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-21T13:00:53.975 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-21T13:00:53.975 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:00:53.978 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-21T13:00:53.978 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-21T13:00:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:53 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:00:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:53 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:00:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:53 smithi163 ceph-mon[29621]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:00:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:53 smithi163 ceph-mon[29621]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:00:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:53 smithi163 ceph-mon[29621]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T13:00:54.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:53 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:00:54.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:53 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:00:54.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:53 smithi163 ceph-mon[40675]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:00:54.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:53 smithi163 ceph-mon[40675]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:00:54.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:53 smithi163 ceph-mon[40675]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T13:00:54.020 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:54.044 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:54.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:54.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:54.074 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:hints = 1 2024-03-21T13:00:54.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-21T13:00:54.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Object prefix: benchmark_data_smithi163_80312 2024-03-21T13:00:54.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:00:54.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 0 0 0 0 0 0 - 0 2024-03-21T13:00:54.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:54.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:54.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:54.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:53 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:00:54.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:53 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:00:54.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:53 smithi204 ceph-mon[33153]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:00:54.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:53 smithi204 ceph-mon[33153]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:00:54.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:53 smithi204 ceph-mon[33153]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T13:00:54.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:54.307 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:54.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:54.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:54.409 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:54.453 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:54.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:54.554 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:54.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:54.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:54.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:54.686 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:54.744 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:54.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:54.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:54.835 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:54 smithi163 podman[80479]: 2024-03-21 13:00:54.683794974 +0000 UTC m=+0.115894848 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-21T13:00:54.845 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:54.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:54.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:54.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:54.977 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:55.026 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:55.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:55.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:55.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 1 16 2911 2895 180.934 180.938 0.00411332 0.005502 2024-03-21T13:00:55.086 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:00:55.086 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:00:55.128 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:55.158 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:55.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:55.221 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:00:55.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:55.267 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:00:55.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:55.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:55.309 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:00:55.356 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:00:55.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:55.394 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:00:55.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:55.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:55.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:55.434 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:00:55.480 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:00:55.525 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:00:55.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:55.532 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:00:55.532 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:00:55.532 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:00:55.532 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:00:55.532 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:00:55.532 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:00:55.532 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:00:55.532 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:00:55.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:55.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:55.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:55.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:55.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:55.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:55.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:00:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:00:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:00:55.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:55.821 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:55.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:55.923 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:55.947 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:55.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:56.049 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:56.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 2 16 5403 5387 168.335 155.75 0.00389609 0.00592399 2024-03-21T13:00:56.086 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:56.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:56.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:56.188 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:56.220 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:56.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:56.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:55 smithi204 ceph-mon[33153]: pgmap v69: 148 pgs: 148 active+clean; 73 MiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 9.7 MiB/s wr, 156 op/s 2024-03-21T13:00:56.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:55 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2697735749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:00:56.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:55 smithi163 ceph-mon[29621]: pgmap v69: 148 pgs: 148 active+clean; 73 MiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 9.7 MiB/s wr, 156 op/s 2024-03-21T13:00:56.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:55 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2697735749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:00:56.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:55 smithi163 ceph-mon[40675]: pgmap v69: 148 pgs: 148 active+clean; 73 MiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 9.7 MiB/s wr, 156 op/s 2024-03-21T13:00:56.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:55 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2697735749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:00:56.322 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:56.347 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:56.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:56.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:56.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:56.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:56.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:56.585 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:56.624 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:56.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:56.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:56.726 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:56.766 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:56.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:56.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:56.911 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:56.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:56.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:57.013 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:57.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:57.055 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:57.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:57.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 3 16 8010 7994 166.531 162.938 0.00328237 0.00599451 2024-03-21T13:00:57.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:57.189 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:57.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:57.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:57.260 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:56 smithi163 podman[80479]: 2024-03-21 13:00:56.82150075 +0000 UTC m=+2.253600599 volume create 53557f7936d28c2bf92aa378bb2f0e2757f2a32d7b7397f26f140d856a4c33c2 2024-03-21T13:00:57.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:57.322 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:57.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:57.423 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:57.446 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:57.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:57.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:57.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:57.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:57.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:57.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:57.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:57.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:00:57.713 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-21T13:00:57.812 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:00:57.850 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:57.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:57.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:00:57.951 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:57.996 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:57.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:58.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 4 16 10507 10491 163.908 156.062 0.00444829 0.00609402 2024-03-21T13:00:58.097 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:58.139 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:58.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:58.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:58.240 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:58.269 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:58.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:58.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:58.402 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:58.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:58.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:58.463 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:58 smithi163 ceph-mon[29621]: pgmap v70: 148 pgs: 148 active+clean; 101 MiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 13 MiB/s wr, 202 op/s 2024-03-21T13:00:58.463 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:58 smithi163 ceph-mon[40675]: pgmap v70: 148 pgs: 148 active+clean; 101 MiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 13 MiB/s wr, 202 op/s 2024-03-21T13:00:58.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:58 smithi204 ceph-mon[33153]: pgmap v70: 148 pgs: 148 active+clean; 101 MiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 13 MiB/s wr, 202 op/s 2024-03-21T13:00:58.504 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:00:58.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:58.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:58.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:00:58.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:58.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:58.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:58.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:58.760 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:00:58 smithi163 podman[80479]: 2024-03-21 13:00:58.463302286 +0000 UTC m=+3.895402141 container create 68f866c321577c22fe67c34073adfedea9d496cc88e981d15e00587341ed1b7a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:00:58.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:00:58.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:58.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:58.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:00:58.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:00:58.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:58.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:58.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:00:59.034 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:59.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 5 15 13052 13037 162.947 159.125 0.0051439 0.00612963 2024-03-21T13:00:59.085 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:59.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:59.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:59.225 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:00:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:59.236 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:59.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:59.325 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:59 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3942732796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:00:59.325 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:00:59 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:00:59.325 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:59 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3942732796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:00:59.325 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:00:59 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:00:59.337 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:59.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:59.387 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:59.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:59.488 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:00:59.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:59 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3942732796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:00:59.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:00:59 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:00:59.528 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:59.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:00:59.630 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:00:59.669 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:59.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:00:59.771 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:00:59.813 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:00:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:59.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:00:59.914 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:00:59.946 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:00:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:00:59.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:00.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:00.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 6 16 15717 15701 163.537 166.5 0.00441205 0.00610782 2024-03-21T13:01:00.079 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:00.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:00.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:00.180 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:00.223 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:00.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:00.325 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:00.365 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:00.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:00.365 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-mon[29621]: pgmap v71: 148 pgs: 148 active+clean; 101 MiB data, 301 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 13 MiB/s wr, 202 op/s 2024-03-21T13:01:00.366 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:01:00.366 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:01:00.366 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-mon[29621]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T13:01:00.366 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-mon[40675]: pgmap v71: 148 pgs: 148 active+clean; 101 MiB data, 301 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 13 MiB/s wr, 202 op/s 2024-03-21T13:01:00.366 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:01:00.367 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:01:00.367 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-mon[40675]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T13:01:00.367 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:00 smithi163 podman[80479]: 2024-03-21 13:01:00.280897253 +0000 UTC m=+5.712997124 container init 68f866c321577c22fe67c34073adfedea9d496cc88e981d15e00587341ed1b7a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:01:00.367 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:00 smithi163 podman[80479]: 2024-03-21 13:01:00.289754172 +0000 UTC m=+5.721854024 container start 68f866c321577c22fe67c34073adfedea9d496cc88e981d15e00587341ed1b7a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:01:00.367 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[80908]: ts=2024-03-21T13:01:00.312Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-21T13:01:00.367 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[80908]: ts=2024-03-21T13:01:00.312Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-21T13:01:00.367 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[80908]: ts=2024-03-21T13:01:00.313Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.163 port=9094 2024-03-21T13:01:00.367 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[80908]: ts=2024-03-21T13:01:00.315Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-21T13:01:00.367 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[80908]: ts=2024-03-21T13:01:00.360Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T13:01:00.367 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[80908]: ts=2024-03-21T13:01:00.361Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T13:01:00.407 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:00.407 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:00.466 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:00.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:00 smithi204 ceph-mon[33153]: pgmap v71: 148 pgs: 148 active+clean; 101 MiB data, 301 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 13 MiB/s wr, 202 op/s 2024-03-21T13:01:00.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:00 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:01:00.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:00 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:01:00.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:00 smithi204 ceph-mon[33153]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T13:01:00.519 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:00.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:00.621 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:00.665 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:00.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:00.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:00.693 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[80908]: ts=2024-03-21T13:01:00.365Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-21T13:01:00.694 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:00 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[80908]: ts=2024-03-21T13:01:00.366Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-21T13:01:00.694 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:00 smithi163 bash[80479]: 68f866c321577c22fe67c34073adfedea9d496cc88e981d15e00587341ed1b7a 2024-03-21T13:01:00.694 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:00 smithi163 systemd[1]: Started Ceph alertmanager.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T13:01:00.767 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:00.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:00.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:00.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:00.900 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:00.941 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:00.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:00.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:01.043 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:01.074 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:01:01.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 7 16 18045 18029 160.958 145.5 0.00591194 0.00620365 2024-03-21T13:01:01.083 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:01.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:01.186 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:01.232 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:01.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:01.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:01.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:01.384 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:01.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:01.486 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:01.523 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:01.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:01.584 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:01:01 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:01:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:01 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:01 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:01 smithi204 ceph-mon[33153]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T13:01:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:01 smithi204 ceph-mon[33153]: pgmap v73: 148 pgs: 148 active+clean; 906 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-21T13:01:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:01 smithi204 ceph-mon[33153]: Reconfiguring daemon prometheus.a on smithi204 2024-03-21T13:01:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:01 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:01.625 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:01.655 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:01.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:01.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:01.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:01.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:01 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:01.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:01 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:01.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:01 smithi163 ceph-mon[29621]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T13:01:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:01 smithi163 ceph-mon[29621]: pgmap v73: 148 pgs: 148 active+clean; 906 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-21T13:01:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:01 smithi163 ceph-mon[29621]: Reconfiguring daemon prometheus.a on smithi204 2024-03-21T13:01:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:01 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:01 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:01 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:01 smithi163 ceph-mon[40675]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T13:01:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:01 smithi163 ceph-mon[40675]: pgmap v73: 148 pgs: 148 active+clean; 906 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-21T13:01:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:01 smithi163 ceph-mon[40675]: Reconfiguring daemon prometheus.a on smithi204 2024-03-21T13:01:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:01 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:01.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:01.790 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:01.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:01.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:01.924 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:01.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:02.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:02.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:02.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:02.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:02.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 8 16 20427 20411 159.446 148.875 0.00808825 0.00626572 2024-03-21T13:01:02.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:02.166 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:01:02.196 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:02.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:02.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:02.297 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:02.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:02.344 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:02.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:02.445 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:02.464 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:02 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[80908]: ts=2024-03-21T13:01:02.316Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000296382s 2024-03-21T13:01:02.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:02.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:02.550 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:02.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:02.605 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:02.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:02.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:02.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:02.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:02.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:02.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:02.839 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:02.879 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:02.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:02.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:03.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:03.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:03.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:03.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 9 16 22637 22621 157.076 138.125 0.00821432 0.00636061 2024-03-21T13:01:03.116 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:03.147 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:03.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:03.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:03.291 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:03.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:03.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:03.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:03.423 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:03.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:03.525 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:03.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:03.571 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:03.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:03.672 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:03.711 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:03.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:03.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:03.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:03.843 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:03.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:03.945 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:03.964 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:03 smithi163 ceph-mon[29621]: pgmap v74: 148 pgs: 148 active+clean; 906 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T13:01:03.965 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:03 smithi163 ceph-mon[40675]: pgmap v74: 148 pgs: 148 active+clean; 906 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T13:01:03.987 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:03.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:03.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:03 smithi204 ceph-mon[33153]: pgmap v74: 148 pgs: 148 active+clean; 906 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T13:01:04.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 10 16 24805 24789 154.917 135.5 0.00865771 0.00644893 2024-03-21T13:01:04.089 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:04.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:04.127 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:04.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:04.227 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:04.267 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:04.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:04.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:04.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:04.400 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:04.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:04.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:04.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:04.535 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:04.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:04.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:04.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:04.673 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:04.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:04.674 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:04.752 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 systemd[1]: Stopping Ceph prometheus.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:01:04.774 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:04.820 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:04.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:04.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:04.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:04.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:04.958 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:04.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:05.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:04 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:05.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:04 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:05.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:04 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:05.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:04 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:05.059 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:05.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 11 16 26968 26952 153.123 135.188 0.00161845 0.00650887 2024-03-21T13:01:05.087 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:05.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:05.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:05.188 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:05.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:05.240 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:05.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:05.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:05.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:05.252 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T13:01:04.938Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T13:01:05.253 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T13:01:04.939Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-21T13:01:05.253 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T13:01:04.939Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-21T13:01:05.253 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T13:01:04.939Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T13:01:05.253 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T13:01:04.939Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T13:01:05.253 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T13:01:04.939Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-21T13:01:05.253 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T13:01:04.939Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-21T13:01:05.253 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T13:01:04.939Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-21T13:01:05.253 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T13:01:04.940Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-21T13:01:05.253 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T13:01:04.941Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T13:01:05.253 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T13:01:04.941Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-21T13:01:05.253 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[62039]: ts=2024-03-21T13:01:04.941Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-21T13:01:05.254 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:04 smithi204 podman[67579]: 2024-03-21 13:01:04.950944284 +0000 UTC m=+0.149391169 container died 2f6d58b7a4bb63eb7ee5530b319a26e688a8f817921465a965f8557846e0f820 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:01:05.341 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:05.370 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:05.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:05.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:05.501 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:05.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:05.603 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:05.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:05.653 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:05.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:05.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:05.781 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:05.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:05.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:05.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:05.912 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:05.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:05.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:06.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:05 smithi163 ceph-mon[29621]: pgmap v75: 148 pgs: 148 active+clean; 1.4 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 137 MiB/s wr, 2.20k op/s 2024-03-21T13:01:06.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:05 smithi163 ceph-mon[40675]: pgmap v75: 148 pgs: 148 active+clean; 1.4 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 137 MiB/s wr, 2.20k op/s 2024-03-21T13:01:06.014 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:06.027 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:01:06.027 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-03-21T13:01:06.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-03-21T13:01:06.027 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:01:06.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.027 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:5211 7.e e97/92 7:7072655b:::benchmark_data_smithi163_80312_object5210:head v 97'328, mlcod=97'326)", 2024-03-21T13:01:06.027 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:55.908011+0000", 2024-03-21T13:01:06.028 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.114951863, 2024-03-21T13:01:06.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090978528000000003, 2024-03-21T13:01:06.028 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.028 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.028 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908011+0000", 2024-03-21T13:01:06.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908011+0000", 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908012+0000", 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0999999999999997e-07 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908055+0000", 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3532000000000003e-05 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908066+0000", 2024-03-21T13:01:06.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0866999999999999e-05 2024-03-21T13:01:06.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908074+0000", 2024-03-21T13:01:06.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.362e-06 2024-03-21T13:01:06.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908213+0000", 2024-03-21T13:01:06.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013936499999999999 2024-03-21T13:01:06.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908244+0000", 2024-03-21T13:01:06.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0994000000000002e-05 2024-03-21T13:01:06.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.998941+0000", 2024-03-21T13:01:06.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090696820999999997 2024-03-21T13:01:06.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.998990+0000", 2024-03-21T13:01:06.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8776999999999997e-05 2024-03-21T13:01:06.032 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.032 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.032 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.032 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:5216 7.e e97/92 7:7a5cce51:::benchmark_data_smithi163_80312_object5215:head v 97'329, mlcod=97'326)", 2024-03-21T13:01:06.032 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:55.908093+0000", 2024-03-21T13:01:06.032 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.114870064, 2024-03-21T13:01:06.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090928917999999997, 2024-03-21T13:01:06.032 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908093+0000", 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908093+0000", 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908093+0000", 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2000000000000001e-07 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908112+0000", 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9264000000000001e-05 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908119+0000", 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2419999999999997e-06 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.908123+0000", 2024-03-21T13:01:06.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7999999999999998e-06 2024-03-21T13:01:06.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910249+0000", 2024-03-21T13:01:06.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021251170000000002 2024-03-21T13:01:06.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910272+0000", 2024-03-21T13:01:06.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3180000000000002e-05 2024-03-21T13:01:06.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.998995+0000", 2024-03-21T13:01:06.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.088723234999999998 2024-03-21T13:01:06.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999022+0000", 2024-03-21T13:01:06.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6760000000000001e-05 2024-03-21T13:01:06.036 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.036 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.036 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.037 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5223 7.6 7:6297f9f5:::benchmark_data_smithi163_80312_object5222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.037 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:55.910785+0000", 2024-03-21T13:01:06.037 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.112178351000001, 2024-03-21T13:01:06.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.088361527999999995, 2024-03-21T13:01:06.037 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.037 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.037 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.037 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.037 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.037 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5223 2024-03-21T13:01:06.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.038 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910785+0000", 2024-03-21T13:01:06.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910785+0000", 2024-03-21T13:01:06.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910785+0000", 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1100000000000005e-07 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910807+0000", 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1523e-05 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910813+0000", 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2360000000000001e-06 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910819+0000", 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0290000000000002e-06 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910866+0000", 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7293999999999998e-05 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910959+0000", 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2492999999999997e-05 2024-03-21T13:01:06.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911110+0000", 2024-03-21T13:01:06.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015101000000000001 2024-03-21T13:01:06.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.998700+0000", 2024-03-21T13:01:06.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.087589745999999996 2024-03-21T13:01:06.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.998928+0000", 2024-03-21T13:01:06.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000228707 2024-03-21T13:01:06.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999042+0000", 2024-03-21T13:01:06.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000113705 2024-03-21T13:01:06.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999092+0000", 2024-03-21T13:01:06.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0231000000000001e-05 2024-03-21T13:01:06.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999146+0000", 2024-03-21T13:01:06.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3742999999999997e-05 2024-03-21T13:01:06.043 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.043 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.043 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5224 7.6 7:6e1ead02:::benchmark_data_smithi163_80312_object5223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:55.910831+0000", 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.11213148, 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090457315999999996, 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5224 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910831+0000", 2024-03-21T13:01:06.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910831+0000", 2024-03-21T13:01:06.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910832+0000", 2024-03-21T13:01:06.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9699999999999996e-07 2024-03-21T13:01:06.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910859+0000", 2024-03-21T13:01:06.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6743000000000001e-05 2024-03-21T13:01:06.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910865+0000", 2024-03-21T13:01:06.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9179999999999999e-06 2024-03-21T13:01:06.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910871+0000", 2024-03-21T13:01:06.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4880000000000004e-06 2024-03-21T13:01:06.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911833+0000", 2024-03-21T13:01:06.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00096174400000000005 2024-03-21T13:01:06.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911936+0000", 2024-03-21T13:01:06.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000103213 2024-03-21T13:01:06.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912085+0000", 2024-03-21T13:01:06.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000149228 2024-03-21T13:01:06.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.998975+0000", 2024-03-21T13:01:06.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.086889653999999997 2024-03-21T13:01:06.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999153+0000", 2024-03-21T13:01:06.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000178407 2024-03-21T13:01:06.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.001194+0000", 2024-03-21T13:01:06.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002041101 2024-03-21T13:01:06.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.001248+0000", 2024-03-21T13:01:06.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3838999999999997e-05 2024-03-21T13:01:06.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.001289+0000", 2024-03-21T13:01:06.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0484000000000001e-05 2024-03-21T13:01:06.050 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.050 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.050 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.050 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5227 7.f 7:f5d74172:::benchmark_data_smithi163_80312_object5226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.050 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:55.910880+0000", 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.112083134000001, 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.088822389000000002, 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5227 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910880+0000", 2024-03-21T13:01:06.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910880+0000", 2024-03-21T13:01:06.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910880+0000", 2024-03-21T13:01:06.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6899999999999997e-07 2024-03-21T13:01:06.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910914+0000", 2024-03-21T13:01:06.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3201999999999998e-05 2024-03-21T13:01:06.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910920+0000", 2024-03-21T13:01:06.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5719999999999999e-06 2024-03-21T13:01:06.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910926+0000", 2024-03-21T13:01:06.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7899999999999996e-06 2024-03-21T13:01:06.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910971+0000", 2024-03-21T13:01:06.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5265000000000001e-05 2024-03-21T13:01:06.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911099+0000", 2024-03-21T13:01:06.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012815099999999999 2024-03-21T13:01:06.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911230+0000", 2024-03-21T13:01:06.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013102599999999999 2024-03-21T13:01:06.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.055 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912848+0000", 2024-03-21T13:01:06.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016177050000000001 2024-03-21T13:01:06.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.055 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999577+0000", 2024-03-21T13:01:06.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.086729290000000001 2024-03-21T13:01:06.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.056 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.056 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999619+0000", 2024-03-21T13:01:06.056 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1767000000000002e-05 2024-03-21T13:01:06.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.056 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.056 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999660+0000", 2024-03-21T13:01:06.056 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1251999999999997e-05 2024-03-21T13:01:06.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.056 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.056 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999702+0000", 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.18e-05 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5228 7.f 7:fef257da:::benchmark_data_smithi163_80312_object5227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:55.910941+0000", 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.112021990000001, 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.12004672600000001, 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.057 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.058 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.058 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5228 2024-03-21T13:01:06.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.058 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910941+0000", 2024-03-21T13:01:06.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910941+0000", 2024-03-21T13:01:06.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910941+0000", 2024-03-21T13:01:06.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6800000000000001e-07 2024-03-21T13:01:06.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910970+0000", 2024-03-21T13:01:06.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8487999999999999e-05 2024-03-21T13:01:06.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910977+0000", 2024-03-21T13:01:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8419999999999999e-06 2024-03-21T13:01:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910983+0000", 2024-03-21T13:01:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1850000000000001e-06 2024-03-21T13:01:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.998855+0000", 2024-03-21T13:01:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.087871749999999998 2024-03-21T13:01:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.061 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999013+0000", 2024-03-21T13:01:06.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015872299999999999 2024-03-21T13:01:06.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.061 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999168+0000", 2024-03-21T13:01:06.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015443400000000001 2024-03-21T13:01:06.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.000350+0000", 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011821290000000001 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.000381+0000", 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0716999999999997e-05 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.003758+0000", 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00337762 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.063 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.030928+0000", 2024-03-21T13:01:06.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027170069000000002 2024-03-21T13:01:06.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.063 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.063 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.030988+0000", 2024-03-21T13:01:06.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9301e-05 2024-03-21T13:01:06.063 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.063 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.063 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.063 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5229 7.6 7:646e45d9:::benchmark_data_smithi163_80312_object5228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:55.910993+0000", 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.111969818, 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.120508013, 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5229 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910993+0000", 2024-03-21T13:01:06.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910993+0000", 2024-03-21T13:01:06.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.910994+0000", 2024-03-21T13:01:06.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9299999999999998e-07 2024-03-21T13:01:06.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911014+0000", 2024-03-21T13:01:06.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0047999999999998e-05 2024-03-21T13:01:06.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911020+0000", 2024-03-21T13:01:06.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0090000000000004e-06 2024-03-21T13:01:06.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911037+0000", 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7464000000000001e-05 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912443+0000", 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001405945 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912549+0000", 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105994 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912707+0000", 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015772200000000001 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999158+0000", 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.086451330000000007 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999717+0000", 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00055864700000000001 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.031404+0000", 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.031687424999999998 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.031466+0000", 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1247999999999994e-05 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.031501+0000", 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5587999999999999e-05 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.069 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.070 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.070 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:5214 7.7 e97/92 7:e085b7b9:::benchmark_data_smithi163_80312_object5213:head v 97'352, mlcod=97'350)", 2024-03-21T13:01:06.070 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:55.911115+0000", 2024-03-21T13:01:06.070 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.111848105, 2024-03-21T13:01:06.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.088627834000000003, 2024-03-21T13:01:06.070 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.070 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.070 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911115+0000", 2024-03-21T13:01:06.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911115+0000", 2024-03-21T13:01:06.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911116+0000", 2024-03-21T13:01:06.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0100000000000004e-07 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911156+0000", 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0837999999999998e-05 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911162+0000", 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0480000000000004e-06 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.911168+0000", 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3419999999999998e-06 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912407+0000", 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012393700000000001 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912426+0000", 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8919e-05 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999711+0000", 2024-03-21T13:01:06.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.087284579000000001 2024-03-21T13:01:06.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999743+0000", 2024-03-21T13:01:06.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2036999999999998e-05 2024-03-21T13:01:06.074 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.074 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.074 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.074 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5231 7.4 7:2bf7cb90:::benchmark_data_smithi163_80312_object5230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:55.912670+0000", 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.110292725000001, 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.088393069000000005, 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5231 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912670+0000", 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912670+0000", 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912671+0000", 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.223e-06 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.076 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912694+0000", 2024-03-21T13:01:06.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2571000000000001e-05 2024-03-21T13:01:06.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912704+0000", 2024-03-21T13:01:06.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0162e-05 2024-03-21T13:01:06.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912712+0000", 2024-03-21T13:01:06.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8790000000000003e-06 2024-03-21T13:01:06.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912764+0000", 2024-03-21T13:01:06.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2369e-05 2024-03-21T13:01:06.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.998759+0000", 2024-03-21T13:01:06.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085994841000000002 2024-03-21T13:01:06.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,4", 2024-03-21T13:01:06.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.998954+0000", 2024-03-21T13:01:06.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019495600000000001 2024-03-21T13:01:06.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.000823+0000", 2024-03-21T13:01:06.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018685399999999999 2024-03-21T13:01:06.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.000860+0000", 2024-03-21T13:01:06.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6863999999999998e-05 2024-03-21T13:01:06.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.000961+0000", 2024-03-21T13:01:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000101065 2024-03-21T13:01:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.001011+0000", 2024-03-21T13:01:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9846999999999999e-05 2024-03-21T13:01:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.001063+0000", 2024-03-21T13:01:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2751999999999999e-05 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5232 7.f 7:f06f9c18:::benchmark_data_smithi163_80312_object5231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:55.912847+0000", 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.110116304, 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.118292931, 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.081 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.082 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5232 2024-03-21T13:01:06.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.082 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912847+0000", 2024-03-21T13:01:06.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912847+0000", 2024-03-21T13:01:06.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.912847+0000", 2024-03-21T13:01:06.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9500000000000002e-07 2024-03-21T13:01:06.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.913003+0000", 2024-03-21T13:01:06.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015576899999999999 2024-03-21T13:01:06.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999187+0000", 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.086183621000000002 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999196+0000", 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5109999999999992e-06 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999736+0000", 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00054006099999999999 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999845+0000", 2024-03-21T13:01:06.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010888800000000001 2024-03-21T13:01:06.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:06.086 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:06.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:06.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:55.999988+0000", 2024-03-21T13:01:06.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014243599999999999 2024-03-21T13:01:06.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.001880+0000", 2024-03-21T13:01:06.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001891996 2024-03-21T13:01:06.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.001994+0000", 2024-03-21T13:01:06.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114032 2024-03-21T13:01:06.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.031044+0000", 2024-03-21T13:01:06.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.029050141000000002 2024-03-21T13:01:06.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.031098+0000", 2024-03-21T13:01:06.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4289000000000001e-05 2024-03-21T13:01:06.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.031140+0000", 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1492000000000002e-05 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5924 7.f 7:f6b81e0b:::benchmark_data_smithi163_80312_object5923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.233717+0000", 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.7892461839999996, 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090536137000000003, 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.089 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.090 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.090 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5924 2024-03-21T13:01:06.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.090 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.233717+0000", 2024-03-21T13:01:06.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.233717+0000", 2024-03-21T13:01:06.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.233718+0000", 2024-03-21T13:01:06.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0380000000000001e-06 2024-03-21T13:01:06.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.233800+0000", 2024-03-21T13:01:06.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2100000000000003e-05 2024-03-21T13:01:06.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.233808+0000", 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6690000000000006e-06 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.233814+0000", 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7479999999999996e-06 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234164+0000", 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00034922799999999998 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234289+0000", 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000125592 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234467+0000", 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017824400000000001 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236163+0000", 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016957980000000001 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236212+0000", 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9067000000000003e-05 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324156+0000", 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.087943427000000005 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324208+0000", 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2447000000000001e-05 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324253+0000", 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4778999999999999e-05 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5927 7.6 7:64ff54ad:::benchmark_data_smithi163_80312_object5926:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.234450+0000", 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.7885132630000005, 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090060308000000006, 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.095 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5927 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234450+0000", 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234450+0000", 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234451+0000", 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.085e-06 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234471+0000", 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0523999999999999e-05 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234482+0000", 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1090000000000001e-05 2024-03-21T13:01:06.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234492+0000", 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7559999999999994e-06 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234605+0000", 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112693 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234719+0000", 2024-03-21T13:01:06.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114271 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234840+0000", 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120591 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236221+0000", 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001381211 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236239+0000", 2024-03-21T13:01:06.100 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8283000000000001e-05 2024-03-21T13:01:06.100 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.100 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 12 15 28528 28513 148.492 97.5625 0.00679716 0.00672758 2024-03-21T13:01:06.100 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.100 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324430+0000", 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.088190530000000003 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324474+0000", 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4783000000000001e-05 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324510+0000", 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5491000000000003e-05 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.101 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5933 7.f 7:ffedaddd:::benchmark_data_smithi163_80312_object5932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.235134+0000", 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.7878292869999992, 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.089234472999999995, 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5933 2024-03-21T13:01:06.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.103 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235134+0000", 2024-03-21T13:01:06.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235134+0000", 2024-03-21T13:01:06.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235134+0000", 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1999999999999999e-07 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235156+0000", 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1682999999999999e-05 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235171+0000", 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5435999999999999e-05 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235179+0000", 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.34e-06 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235532+0000", 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035337200000000002 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235640+0000", 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000108021 2024-03-21T13:01:06.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235775+0000", 2024-03-21T13:01:06.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013473700000000001 2024-03-21T13:01:06.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236427+0000", 2024-03-21T13:01:06.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000652543 2024-03-21T13:01:06.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237219+0000", 2024-03-21T13:01:06.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00079210000000000001 2024-03-21T13:01:06.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324288+0000", 2024-03-21T13:01:06.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.087068735999999994 2024-03-21T13:01:06.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324336+0000", 2024-03-21T13:01:06.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8022999999999997e-05 2024-03-21T13:01:06.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.108 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324368+0000", 2024-03-21T13:01:06.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1862000000000001e-05 2024-03-21T13:01:06.108 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.108 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.108 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.108 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5940 7.9 7:94720b83:::benchmark_data_smithi163_80312_object5939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.108 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.236961+0000", 2024-03-21T13:01:06.108 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.7860015390000008, 2024-03-21T13:01:06.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.088995576000000007, 2024-03-21T13:01:06.109 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.109 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.109 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.109 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.109 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.109 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5940 2024-03-21T13:01:06.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.109 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.109 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.109 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.109 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236961+0000", 2024-03-21T13:01:06.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236961+0000", 2024-03-21T13:01:06.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236962+0000", 2024-03-21T13:01:06.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3099999999999996e-07 2024-03-21T13:01:06.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236980+0000", 2024-03-21T13:01:06.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8133000000000001e-05 2024-03-21T13:01:06.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.111 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.111 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.111 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236988+0000", 2024-03-21T13:01:06.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.362e-06 2024-03-21T13:01:06.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.111 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.111 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.111 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236993+0000", 2024-03-21T13:01:06.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2560000000000002e-06 2024-03-21T13:01:06.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.111 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.112 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.112 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237060+0000", 2024-03-21T13:01:06.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7149000000000004e-05 2024-03-21T13:01:06.112 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.112 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.112 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.112 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237174+0000", 2024-03-21T13:01:06.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011353000000000001 2024-03-21T13:01:06.112 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.112 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.112 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,4", 2024-03-21T13:01:06.112 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237327+0000", 2024-03-21T13:01:06.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015376800000000001 2024-03-21T13:01:06.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.113 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.113 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324065+0000", 2024-03-21T13:01:06.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.086737152999999997 2024-03-21T13:01:06.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.113 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.113 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324107+0000", 2024-03-21T13:01:06.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2880000000000003e-05 2024-03-21T13:01:06.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.113 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.114 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.325859+0000", 2024-03-21T13:01:06.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001751977 2024-03-21T13:01:06.114 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.114 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.114 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.114 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.325915+0000", 2024-03-21T13:01:06.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5594999999999999e-05 2024-03-21T13:01:06.114 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.114 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.114 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.114 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.325957+0000", 2024-03-21T13:01:06.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1941999999999999e-05 2024-03-21T13:01:06.115 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.115 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.115 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.115 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5946 7.f 7:fb0a2d9a:::benchmark_data_smithi163_80312_object5945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.115 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.324255+0000", 2024-03-21T13:01:06.115 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.6987083920000003, 2024-03-21T13:01:06.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.115065446, 2024-03-21T13:01:06.115 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.115 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.115 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.116 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.116 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.116 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5946 2024-03-21T13:01:06.116 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.116 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.116 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324255+0000", 2024-03-21T13:01:06.116 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.116 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.116 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324255+0000", 2024-03-21T13:01:06.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.117 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.117 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324255+0000", 2024-03-21T13:01:06.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9699999999999996e-07 2024-03-21T13:01:06.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.117 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.117 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324279+0000", 2024-03-21T13:01:06.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3728000000000001e-05 2024-03-21T13:01:06.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324285+0000", 2024-03-21T13:01:06.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4010000000000002e-06 2024-03-21T13:01:06.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.118 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324291+0000", 2024-03-21T13:01:06.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6840000000000001e-06 2024-03-21T13:01:06.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.118 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324396+0000", 2024-03-21T13:01:06.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010553200000000001 2024-03-21T13:01:06.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.119 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.119 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324528+0000", 2024-03-21T13:01:06.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013143900000000001 2024-03-21T13:01:06.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.119 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.119 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.324671+0000", 2024-03-21T13:01:06.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143411 2024-03-21T13:01:06.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.325764+0000", 2024-03-21T13:01:06.120 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010924109999999999 2024-03-21T13:01:06.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.326759+0000", 2024-03-21T13:01:06.120 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00099525700000000004 2024-03-21T13:01:06.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.438513+0000", 2024-03-21T13:01:06.121 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.11175358000000001 2024-03-21T13:01:06.121 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.121 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.121 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439273+0000", 2024-03-21T13:01:06.121 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00075996600000000005 2024-03-21T13:01:06.121 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.121 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.121 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439320+0000", 2024-03-21T13:01:06.121 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.744e-05 2024-03-21T13:01:06.121 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.122 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.122 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.122 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.122 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.122 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5966 7.6 7:6a38029f:::benchmark_data_smithi163_80312_object5965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.122 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.326879+0000", 2024-03-21T13:01:06.122 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.6960836619999995, 2024-03-21T13:01:06.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.109433528, 2024-03-21T13:01:06.122 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.122 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.122 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.122 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.122 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5966 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.326879+0000", 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.326879+0000", 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.326880+0000", 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1600000000000001e-07 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.326901+0000", 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1157000000000001e-05 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.326908+0000", 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4420000000000001e-06 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.124 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.326914+0000", 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1120000000000001e-06 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.326962+0000", 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7170999999999999e-05 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.327094+0000", 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000132834 2024-03-21T13:01:06.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.327231+0000", 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000136484 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.328691+0000", 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014597709999999999 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.329150+0000", 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00045919899999999998 2024-03-21T13:01:06.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.436203+0000", 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.107053417 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.436262+0000", 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8536999999999999e-05 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.436313+0000", 2024-03-21T13:01:06.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0887999999999998e-05 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5967 7.f 7:f9df3479:::benchmark_data_smithi163_80312_object5966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.327262+0000", 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.6957005649999992, 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.112255851, 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.128 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.129 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5967 2024-03-21T13:01:06.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.129 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.327262+0000", 2024-03-21T13:01:06.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.130 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.327262+0000", 2024-03-21T13:01:06.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.130 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.327263+0000", 2024-03-21T13:01:06.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.018e-06 2024-03-21T13:01:06.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.327295+0000", 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1495000000000003e-05 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.327307+0000", 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1636e-05 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.327315+0000", 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1890000000000007e-06 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.328743+0000", 2024-03-21T13:01:06.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014283690000000001 2024-03-21T13:01:06.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.328913+0000", 2024-03-21T13:01:06.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000170314 2024-03-21T13:01:06.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.329222+0000", 2024-03-21T13:01:06.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030884400000000001 2024-03-21T13:01:06.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.330432+0000", 2024-03-21T13:01:06.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012100290000000001 2024-03-21T13:01:06.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.330500+0000", 2024-03-21T13:01:06.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7992999999999998e-05 2024-03-21T13:01:06.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439421+0000", 2024-03-21T13:01:06.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.108920431 2024-03-21T13:01:06.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439472+0000", 2024-03-21T13:01:06.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-03-21T13:01:06.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439518+0000", 2024-03-21T13:01:06.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6190000000000003e-05 2024-03-21T13:01:06.134 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5971 7.f 7:f4177769:::benchmark_data_smithi163_80312_object5970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.328382+0000", 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.6945806389999998, 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.11126900000000001, 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.135 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5971 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.328382+0000", 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.328382+0000", 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.136 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.328383+0000", 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0979999999999999e-06 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.328407+0000", 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3292999999999999e-05 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.328418+0000", 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1428e-05 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.328426+0000", 2024-03-21T13:01:06.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3070000000000003e-06 2024-03-21T13:01:06.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.329714+0000", 2024-03-21T13:01:06.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001287814 2024-03-21T13:01:06.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.329832+0000", 2024-03-21T13:01:06.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011793799999999999 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.330049+0000", 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00021730299999999999 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.331140+0000", 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010900860000000001 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.331152+0000", 2024-03-21T13:01:06.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2169e-05 2024-03-21T13:01:06.140 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.140 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.140 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.140 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439561+0000", 2024-03-21T13:01:06.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.10840931700000001 2024-03-21T13:01:06.140 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.140 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.140 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.140 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439613+0000", 2024-03-21T13:01:06.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2040000000000003e-05 2024-03-21T13:01:06.140 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.140 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439651+0000", 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8207000000000002e-05 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5977 7.f 7:f7fa85a2:::benchmark_data_smithi163_80312_object5976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.329963+0000", 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.6929998800000003, 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.10982272899999999, 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.141 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.142 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.142 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.142 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.142 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5977 2024-03-21T13:01:06.142 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.142 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.142 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.142 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.142 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.329963+0000", 2024-03-21T13:01:06.142 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.142 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.142 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.143 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.329963+0000", 2024-03-21T13:01:06.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.143 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.143 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.329965+0000", 2024-03-21T13:01:06.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4839999999999999e-06 2024-03-21T13:01:06.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.143 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.143 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.329989+0000", 2024-03-21T13:01:06.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4241000000000001e-05 2024-03-21T13:01:06.144 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.144 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.144 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.144 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.329999+0000", 2024-03-21T13:01:06.144 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0293e-05 2024-03-21T13:01:06.144 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.144 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.144 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.144 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.330006+0000", 2024-03-21T13:01:06.144 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4100000000000002e-06 2024-03-21T13:01:06.144 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.144 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.144 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.145 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.330520+0000", 2024-03-21T13:01:06.145 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00051343299999999999 2024-03-21T13:01:06.145 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.145 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.145 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.145 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.330618+0000", 2024-03-21T13:01:06.145 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8498999999999993e-05 2024-03-21T13:01:06.145 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.145 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.145 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.145 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.330751+0000", 2024-03-21T13:01:06.145 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013236300000000001 2024-03-21T13:01:06.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.331251+0000", 2024-03-21T13:01:06.146 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050039699999999997 2024-03-21T13:01:06.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.331581+0000", 2024-03-21T13:01:06.146 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032999400000000001 2024-03-21T13:01:06.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.147 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439697+0000", 2024-03-21T13:01:06.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.108115423 2024-03-21T13:01:06.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.147 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.147 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439750+0000", 2024-03-21T13:01:06.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3683000000000001e-05 2024-03-21T13:01:06.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.147 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.147 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439786+0000", 2024-03-21T13:01:06.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5509000000000002e-05 2024-03-21T13:01:06.148 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.148 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.148 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.148 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5985 7.f 7:fa54d7e4:::benchmark_data_smithi163_80312_object5984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:01:06.148 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.336578+0000", 2024-03-21T13:01:06.148 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.6863848049999994, 2024-03-21T13:01:06.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.103330674, 2024-03-21T13:01:06.148 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:06.148 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:01:06.148 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:06.148 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:06.149 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:06.149 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5985 2024-03-21T13:01:06.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.149 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:06.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.149 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:06.149 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.336578+0000", 2024-03-21T13:01:06.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.149 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:06.149 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.336578+0000", 2024-03-21T13:01:06.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:06.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:06.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.336579+0000", 2024-03-21T13:01:06.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6899999999999996e-07 2024-03-21T13:01:06.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:06.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.336597+0000", 2024-03-21T13:01:06.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8428999999999999e-05 2024-03-21T13:01:06.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:06.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.336608+0000", 2024-03-21T13:01:06.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0336000000000001e-05 2024-03-21T13:01:06.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:06.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.336614+0000", 2024-03-21T13:01:06.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1029999999999998e-06 2024-03-21T13:01:06.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:06.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.336662+0000", 2024-03-21T13:01:06.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8328000000000002e-05 2024-03-21T13:01:06.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:06.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.336752+0000", 2024-03-21T13:01:06.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0245000000000001e-05 2024-03-21T13:01:06.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:01:06.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.336838+0000", 2024-03-21T13:01:06.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5663999999999998e-05 2024-03-21T13:01:06.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.337243+0000", 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040498699999999999 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.337591+0000", 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00034799700000000001 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439827+0000", 2024-03-21T13:01:06.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.10223573699999999 2024-03-21T13:01:06.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:01:06.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439873+0000", 2024-03-21T13:01:06.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6180000000000002e-05 2024-03-21T13:01:06.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:06.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:06.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:01:06.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.439909+0000", 2024-03-21T13:01:06.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5899000000000003e-05 2024-03-21T13:01:06.154 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.154 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.154 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.155 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:06.155 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:06.155 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:01:06.155 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:01:06.185 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:01:06.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:06.216 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:06.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:06.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:06.228 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:01:06.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:05 smithi204 ceph-mon[33153]: pgmap v75: 148 pgs: 148 active+clean; 1.4 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 137 MiB/s wr, 2.20k op/s 2024-03-21T13:01:06.275 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:01:06.319 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:06.319 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:01:06.346 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:06.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:06.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:06.368 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:01:06.416 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:01:06.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:06.459 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:01:06.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:01:06.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:01:06.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:01:06.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:01:06.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:01:06.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:01:06.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:01:06.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:01:06.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:06.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:06.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:06.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:06.606 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:06.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:06.707 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:06.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:06.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:06.843 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:01:06.848 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:01:06.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:06 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:01:06.849 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:06 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:01:06.849 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:06 smithi163 ceph-mon[29621]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T13:01:06.849 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:06 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2132606500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:01:06.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:06 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:01:06.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:06 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:01:06.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:06 smithi163 ceph-mon[40675]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T13:01:06.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:06 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2132606500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:01:06.851 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:06.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:06.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:06.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:06.959 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:06 smithi204 podman[67579]: 2024-03-21 13:01:06.673323446 +0000 UTC m=+1.871770335 container cleanup 2f6d58b7a4bb63eb7ee5530b319a26e688a8f817921465a965f8557846e0f820 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:01:06.959 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:06 smithi204 bash[67579]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T13:01:06.959 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:06 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:01:06.959 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:06 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:01:06.959 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:06 smithi204 ceph-mon[33153]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T13:01:06.959 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:06 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2132606500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:01:06.983 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:07.027 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:07.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:07.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 13 16 29967 29951 143.982 89.875 0.00700231 0.00693867 2024-03-21T13:01:07.128 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:07.152 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:07.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:07.252 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:06 smithi204 podman[67607]: 2024-03-21 13:01:06.95903058 +0000 UTC m=+2.001001292 container remove 2f6d58b7a4bb63eb7ee5530b319a26e688a8f817921465a965f8557846e0f820 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:01:07.252 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T13:01:07.252 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 systemd[1]: Stopped Ceph prometheus.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T13:01:07.252 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.254s CPU time. 2024-03-21T13:01:07.253 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 systemd[1]: Starting Ceph prometheus.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:01:07.254 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:07.294 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:07.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:07.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:07.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:07.439 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:07.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:07.540 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:07.569 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:07.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:07.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:07.647 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 podman[67837]: 2024-03-21 13:01:07.264416168 +0000 UTC m=+0.025755226 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-21T13:01:07.647 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 podman[67837]: 2024-03-21 13:01:07.393763182 +0000 UTC m=+0.155102226 container create 617428b1bc18ebf7103465de0cd8d6697e9e2408141382f534b454e3362eecaa (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:01:07.671 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:07.708 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:07.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:07.810 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:07.834 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:07.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:07.936 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:07.967 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:07.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 podman[67837]: 2024-03-21 13:01:07.64717127 +0000 UTC m=+0.408510333 container init 617428b1bc18ebf7103465de0cd8d6697e9e2408141382f534b454e3362eecaa (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:01:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 podman[67837]: 2024-03-21 13:01:07.652036491 +0000 UTC m=+0.413375541 container start 617428b1bc18ebf7103465de0cd8d6697e9e2408141382f534b454e3362eecaa (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:01:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.686Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-21T13:01:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.686Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-21T13:01:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.686Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi204 (none))" 2024-03-21T13:01:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.686Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T13:01:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.687Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T13:01:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.691Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T13:01:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.692Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-21T13:01:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.693Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-21T13:01:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.693Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-21T13:01:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.695Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T13:01:08.004 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.695Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.568µs 2024-03-21T13:01:08.004 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.696Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T13:01:08.004 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.696Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-21T13:01:08.004 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.697Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-21T13:01:08.004 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.697Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=66.592µs wal_replay_duration=932.049µs wbl_replay_duration=124ns total_replay_duration=1.292744ms 2024-03-21T13:01:08.004 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.698Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T13:01:08.004 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.698Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-21T13:01:08.004 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.698Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T13:01:08.004 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.726Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=27.686043ms db_storage=1.135µs remote_storage=1.761µs web_handler=516ns query_engine=1.091µs scrape=596.092µs scrape_sd=212.241µs notify=26.779µs notify_sd=29.341µs rules=26.095499ms tracing=8.386µs 2024-03-21T13:01:08.004 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.726Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-21T13:01:08.004 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:01:07.726Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-21T13:01:08.004 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 bash[67837]: 617428b1bc18ebf7103465de0cd8d6697e9e2408141382f534b454e3362eecaa 2024-03-21T13:01:08.004 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:01:07 smithi204 systemd[1]: Started Ceph prometheus.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T13:01:08.069 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:08.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 14 16 31394 31378 140.068 89.1875 0.00616044 0.00713364 2024-03-21T13:01:08.095 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:07 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:07] ENGINE Bus STOPPING 2024-03-21T13:01:08.095 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:07 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:01:08.095 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:07 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:07] ENGINE Bus STOPPED 2024-03-21T13:01:08.095 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:07 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:07] ENGINE Bus STARTING 2024-03-21T13:01:08.095 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:07 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:07] ENGINE Serving on http://:::9283 2024-03-21T13:01:08.095 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:07 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:07] ENGINE Bus STARTED 2024-03-21T13:01:08.095 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:07 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:07] ENGINE Bus STOPPING 2024-03-21T13:01:08.095 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:07 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:01:08.095 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:07 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:07] ENGINE Bus STOPPED 2024-03-21T13:01:08.095 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:07 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:07] ENGINE Bus STARTING 2024-03-21T13:01:08.096 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:08] ENGINE Serving on http://:::9283 2024-03-21T13:01:08.096 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:08] ENGINE Bus STARTED 2024-03-21T13:01:08.096 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:08] ENGINE Bus STOPPING 2024-03-21T13:01:08.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:08.109 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:08.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:08.210 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:08.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:08.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:08.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:08.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:08.384 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:01:08.384 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:08] ENGINE Bus STOPPED 2024-03-21T13:01:08.384 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:08] ENGINE Bus STARTING 2024-03-21T13:01:08.384 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:08] ENGINE Serving on http://:::9283 2024-03-21T13:01:08.384 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:01:08] ENGINE Bus STARTED 2024-03-21T13:01:08.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: pgmap v77: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 191 MiB/s wr, 3.05k op/s 2024-03-21T13:01:08.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: from='client.24619 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:01:08.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:01:08.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:01:08.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T13:01:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T13:01:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi163.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T13:01:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T13:01:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T13:01:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T13:01:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T13:01:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:08.386 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: pgmap v77: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 191 MiB/s wr, 3.05k op/s 2024-03-21T13:01:08.386 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: from='client.24619 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:01:08.386 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:01:08.386 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:01:08.386 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.386 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.386 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T13:01:08.386 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T13:01:08.386 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi163.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T13:01:08.386 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.386 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T13:01:08.387 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T13:01:08.387 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.387 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T13:01:08.387 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T13:01:08.387 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.387 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:08.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:08.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:08.418 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: pgmap v77: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 191 MiB/s wr, 3.05k op/s 2024-03-21T13:01:08.418 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: from='client.24619 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:01:08.418 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:01:08.418 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:01:08.418 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T13:01:08.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T13:01:08.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi163.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T13:01:08.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T13:01:08.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T13:01:08.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T13:01:08.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T13:01:08.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:08.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:08.490 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:08.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:08.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:08.632 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:08.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:08.680 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:08.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:08.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:08.822 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:08.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:08.924 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:08.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:08.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:09.069 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:09.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 15 16 33189 33173 138.209 112.188 0.0043313 0.00723063 2024-03-21T13:01:09.104 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:09.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:09.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:09.156 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-21T13:01:09.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:09.258 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:09.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:09.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:09.359 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:09.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:09.426 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:09.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:09.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:09 smithi204 ceph-mon[33153]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T13:01:09.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:09 smithi204 ceph-mon[33153]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi163.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T13:01:09.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:09 smithi204 ceph-mon[33153]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T13:01:09.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:09 smithi204 ceph-mon[33153]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T13:01:09.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:09 smithi204 ceph-mon[33153]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T13:01:09.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:09 smithi204 ceph-mon[33153]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T13:01:09.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:09 smithi163 ceph-mon[29621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T13:01:09.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:09 smithi163 ceph-mon[29621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi163.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T13:01:09.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:09 smithi163 ceph-mon[29621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T13:01:09.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:09 smithi163 ceph-mon[29621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T13:01:09.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:09 smithi163 ceph-mon[29621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T13:01:09.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:09 smithi163 ceph-mon[29621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T13:01:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:09 smithi163 ceph-mon[40675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T13:01:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:09 smithi163 ceph-mon[40675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi163.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T13:01:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:09 smithi163 ceph-mon[40675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T13:01:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:09 smithi163 ceph-mon[40675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T13:01:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:09 smithi163 ceph-mon[40675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T13:01:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:09 smithi163 ceph-mon[40675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T13:01:09.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:09.561 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:09.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:09.663 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:09.703 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:09.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:09.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:09.848 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:09.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:09.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:09.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:09.989 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:09.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:09.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:10.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 16 15 35058 35043 136.875 116.875 0.0060864 0.00729948 2024-03-21T13:01:10.090 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:10.130 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:10.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:10.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:10.232 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:10.264 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:10.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:10.366 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:10.379 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:10 smithi163 ceph-mon[29621]: pgmap v79: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-21T13:01:10.379 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:10 smithi163 ceph-mon[40675]: pgmap v79: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-21T13:01:10.380 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:01:10 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[80908]: ts=2024-03-21T13:01:10.319Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00371428s 2024-03-21T13:01:10.409 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:10.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:10.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:10 smithi204 ceph-mon[33153]: pgmap v79: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-21T13:01:10.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:10.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:10.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:10.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:10.644 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:10.679 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:10.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:10.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:10.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:10.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:10.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:10.942 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:10.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:10.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:11.044 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:11.074 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:11.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:11.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:11.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 17 16 36939 36923 135.734 117.5 0.00394589 0.00736213 2024-03-21T13:01:11.176 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:11.221 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:11.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:11.320 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-03-21T13:01:06.642084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1706891688,"num_objects":26234,"num_object_clones":0,"num_object_copies":78702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26234,"num_whiteouts":0,"num_read":1073,"num_read_kb":919,"num_write":26651,"num_write_kb":1668765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26669,"ondisk_log_size":26669,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5021564,"kb_used_data":4759676,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":744939652,"statfs":{"total":767960285184,"available":762818203648,"internally_reserved":0,"allocated":4873908224,"data_stored":4870828776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":268095024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,7,6,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1600520192,"num_objects":24422,"num_object_clones":0,"num_object_copies":73266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24422,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":24422,"num_write_kb":1563008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003116"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":74,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213024+0000","last_change":"2024-03-21T12:59:58.846777+0000","last_active":"2024-03-21T13:00:59.213024+0000","last_peered":"2024-03-21T13:00:59.213024+0000","last_clean":"2024-03-21T13:00:59.213024+0000","last_became_active":"2024-03-21T12:58:43.182506+0000","last_became_peered":"2024-03-21T12:58:43.182506+0000","last_unstale":"2024-03-21T13:00:59.213024+0000","last_undegraded":"2024-03-21T13:00:59.213024+0000","last_fullsized":"2024-03-21T13:00:59.213024+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:58.846634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:58.846634+0000","last_clean_scrub_stamp":"2024-03-21T12:59:58.846634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.412490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.218690+0000","last_change":"2024-03-21T13:00:25.947183+0000","last_active":"2024-03-21T13:00:59.218690+0000","last_peered":"2024-03-21T13:00:59.218690+0000","last_clean":"2024-03-21T13:00:59.218690+0000","last_became_active":"2024-03-21T12:58:41.160342+0000","last_became_peered":"2024-03-21T12:58:41.160342+0000","last_unstale":"2024-03-21T13:00:59.218690+0000","last_undegraded":"2024-03-21T13:00:59.218690+0000","last_fullsized":"2024-03-21T13:00:59.218690+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:25.947092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:25.947092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:45.873585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":86,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.214475+0000","last_change":"2024-03-21T12:59:53.478218+0000","last_active":"2024-03-21T13:00:59.214475+0000","last_peered":"2024-03-21T13:00:59.214475+0000","last_clean":"2024-03-21T13:00:59.214475+0000","last_became_active":"2024-03-21T12:58:37.731393+0000","last_became_peered":"2024-03-21T12:58:37.731393+0000","last_unstale":"2024-03-21T13:00:59.214475+0000","last_undegraded":"2024-03-21T13:00:59.214475+0000","last_fullsized":"2024-03-21T13:00:59.214475+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:53.478141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:53.478141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"89'13","reported_seq":118,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.212934+0000","last_change":"2024-03-21T13:00:19.789926+0000","last_active":"2024-03-21T13:00:59.212934+0000","last_peered":"2024-03-21T13:00:59.212934+0000","last_clean":"2024-03-21T13:00:59.212934+0000","last_became_active":"2024-03-21T12:59:01.161098+0000","last_became_peered":"2024-03-21T12:59:01.161098+0000","last_unstale":"2024-03-21T13:00:59.212934+0000","last_undegraded":"2024-03-21T13:00:59.212934+0000","last_fullsized":"2024-03-21T13:00:59.212934+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-03-21T13:00:19.789748+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:00:19.789748+0000","last_clean_scrub_stamp":"2024-03-21T13:00:19.789748+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:35.174452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":74,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.223626+0000","last_change":"2024-03-21T13:00:05.173374+0000","last_active":"2024-03-21T13:00:59.223626+0000","last_peered":"2024-03-21T13:00:59.223626+0000","last_clean":"2024-03-21T13:00:59.223626+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:00:59.223626+0000","last_undegraded":"2024-03-21T13:00:59.223626+0000","last_fullsized":"2024-03-21T13:00:59.223626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:00:05.173276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:14.939464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213003+0000","last_change":"2024-03-21T13:00:16.731547+0000","last_active":"2024-03-21T13:00:59.213003+0000","last_peered":"2024-03-21T13:00:59.213003+0000","last_clean":"2024-03-21T13:00:59.213003+0000","last_became_active":"2024-03-21T12:58:41.167710+0000","last_became_peered":"2024-03-21T12:58:41.167710+0000","last_unstale":"2024-03-21T13:00:59.213003+0000","last_undegraded":"2024-03-21T13:00:59.213003+0000","last_fullsized":"2024-03-21T13:00:59.213003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:00:16.731434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:35.557020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":85,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.223378+0000","last_change":"2024-03-21T12:58:37.728682+0000","last_active":"2024-03-21T13:00:59.223378+0000","last_peered":"2024-03-21T13:00:59.223378+0000","last_clean":"2024-03-21T13:00:59.223378+0000","last_became_active":"2024-03-21T12:58:37.728383+0000","last_became_peered":"2024-03-21T12:58:37.728383+0000","last_unstale":"2024-03-21T13:00:59.223378+0000","last_undegraded":"2024-03-21T13:00:59.223378+0000","last_fullsized":"2024-03-21T13:00:59.223378+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:58:36.715100+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.068574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"71'15","reported_seq":107,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.224818+0000","last_change":"2024-03-21T13:00:31.008873+0000","last_active":"2024-03-21T13:00:59.224818+0000","last_peered":"2024-03-21T13:00:59.224818+0000","last_clean":"2024-03-21T13:00:59.224818+0000","last_became_active":"2024-03-21T12:58:39.750396+0000","last_became_peered":"2024-03-21T12:58:39.750396+0000","last_unstale":"2024-03-21T13:00:59.224818+0000","last_undegraded":"2024-03-21T13:00:59.224818+0000","last_fullsized":"2024-03-21T13:00:59.224818+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:00:31.008733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:31.008733+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:35.686270+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":73,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221888+0000","last_change":"2024-03-21T13:00:02.599805+0000","last_active":"2024-03-21T13:00:59.221888+0000","last_peered":"2024-03-21T13:00:59.221888+0000","last_clean":"2024-03-21T13:00:59.221888+0000","last_became_active":"2024-03-21T12:58:43.179920+0000","last_became_peered":"2024-03-21T12:58:43.179920+0000","last_unstale":"2024-03-21T13:00:59.221888+0000","last_undegraded":"2024-03-21T13:00:59.221888+0000","last_fullsized":"2024-03-21T13:00:59.221888+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:02.599679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:02.599679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:10.324225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221375+0000","last_change":"2024-03-21T13:00:03.644301+0000","last_active":"2024-03-21T13:00:59.221375+0000","last_peered":"2024-03-21T13:00:59.221375+0000","last_clean":"2024-03-21T13:00:59.221375+0000","last_became_active":"2024-03-21T12:58:41.168978+0000","last_became_peered":"2024-03-21T12:58:41.168978+0000","last_unstale":"2024-03-21T13:00:59.221375+0000","last_undegraded":"2024-03-21T13:00:59.221375+0000","last_fullsized":"2024-03-21T13:00:59.221375+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:03.644209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:03.644209+0000","last_clean_scrub_stamp":"2024-03-21T13:00:03.644209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:18.843767+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":86,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221598+0000","last_change":"2024-03-21T12:59:44.661395+0000","last_active":"2024-03-21T13:00:59.221598+0000","last_peered":"2024-03-21T13:00:59.221598+0000","last_clean":"2024-03-21T13:00:59.221598+0000","last_became_active":"2024-03-21T12:58:37.733705+0000","last_became_peered":"2024-03-21T12:58:37.733705+0000","last_unstale":"2024-03-21T13:00:59.221598+0000","last_undegraded":"2024-03-21T13:00:59.221598+0000","last_fullsized":"2024-03-21T13:00:59.221598+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:44.661289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:44.661289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:59.754984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":102,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213206+0000","last_change":"2024-03-21T13:00:35.880283+0000","last_active":"2024-03-21T13:00:59.213206+0000","last_peered":"2024-03-21T13:00:59.213206+0000","last_clean":"2024-03-21T13:00:59.213206+0000","last_became_active":"2024-03-21T12:58:39.749010+0000","last_became_peered":"2024-03-21T12:58:39.749010+0000","last_unstale":"2024-03-21T13:00:59.213206+0000","last_undegraded":"2024-03-21T13:00:59.213206+0000","last_fullsized":"2024-03-21T13:00:59.213206+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.880161+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:54.305321+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":74,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221149+0000","last_change":"2024-03-21T13:00:02.683946+0000","last_active":"2024-03-21T13:00:59.221149+0000","last_peered":"2024-03-21T13:00:59.221149+0000","last_clean":"2024-03-21T13:00:59.221149+0000","last_became_active":"2024-03-21T12:58:43.181494+0000","last_became_peered":"2024-03-21T12:58:43.181494+0000","last_unstale":"2024-03-21T13:00:59.221149+0000","last_undegraded":"2024-03-21T13:00:59.221149+0000","last_fullsized":"2024-03-21T13:00:59.221149+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:02.683841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:02.683841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:10.243871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":76,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.208196+0000","last_change":"2024-03-21T12:59:41.158247+0000","last_active":"2024-03-21T13:00:59.208196+0000","last_peered":"2024-03-21T13:00:59.208196+0000","last_clean":"2024-03-21T13:00:59.208196+0000","last_became_active":"2024-03-21T12:58:41.163517+0000","last_became_peered":"2024-03-21T12:58:41.163517+0000","last_unstale":"2024-03-21T13:00:59.208196+0000","last_undegraded":"2024-03-21T13:00:59.208196+0000","last_fullsized":"2024-03-21T13:00:59.208196+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:41.158105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:41.158105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.139595+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":86,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221490+0000","last_change":"2024-03-21T12:59:59.594026+0000","last_active":"2024-03-21T13:00:59.221490+0000","last_peered":"2024-03-21T13:00:59.221490+0000","last_clean":"2024-03-21T13:00:59.221490+0000","last_became_active":"2024-03-21T12:58:37.731058+0000","last_became_peered":"2024-03-21T12:58:37.731058+0000","last_unstale":"2024-03-21T13:00:59.221490+0000","last_undegraded":"2024-03-21T13:00:59.221490+0000","last_fullsized":"2024-03-21T13:00:59.221490+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:59.593927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.593927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:14.344496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":94,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213675+0000","last_change":"2024-03-21T12:59:53.952540+0000","last_active":"2024-03-21T13:00:59.213675+0000","last_peered":"2024-03-21T13:00:59.213675+0000","last_clean":"2024-03-21T13:00:59.213675+0000","last_became_active":"2024-03-21T12:58:39.749211+0000","last_became_peered":"2024-03-21T12:58:39.749211+0000","last_unstale":"2024-03-21T13:00:59.213675+0000","last_undegraded":"2024-03-21T13:00:59.213675+0000","last_fullsized":"2024-03-21T13:00:59.213675+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T12:59:53.952401+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.263330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213521+0000","last_change":"2024-03-21T12:59:55.899438+0000","last_active":"2024-03-21T13:00:59.213521+0000","last_peered":"2024-03-21T13:00:59.213521+0000","last_clean":"2024-03-21T13:00:59.213521+0000","last_became_active":"2024-03-21T12:58:43.351768+0000","last_became_peered":"2024-03-21T12:58:43.351768+0000","last_unstale":"2024-03-21T13:00:59.213521+0000","last_undegraded":"2024-03-21T13:00:59.213521+0000","last_fullsized":"2024-03-21T13:00:59.213521+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:55.899344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.899344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:59.849673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.219866+0000","last_change":"2024-03-21T12:59:55.014569+0000","last_active":"2024-03-21T13:00:59.219866+0000","last_peered":"2024-03-21T13:00:59.219866+0000","last_clean":"2024-03-21T13:00:59.219866+0000","last_became_active":"2024-03-21T12:58:41.165783+0000","last_became_peered":"2024-03-21T12:58:41.165783+0000","last_unstale":"2024-03-21T13:00:59.219866+0000","last_undegraded":"2024-03-21T13:00:59.219866+0000","last_fullsized":"2024-03-21T13:00:59.219866+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.014453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:10.470418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":85,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.216000+0000","last_change":"2024-03-21T13:00:14.717611+0000","last_active":"2024-03-21T13:00:59.216000+0000","last_peered":"2024-03-21T13:00:59.216000+0000","last_clean":"2024-03-21T13:00:59.216000+0000","last_became_active":"2024-03-21T12:58:37.733064+0000","last_became_peered":"2024-03-21T12:58:37.733064+0000","last_unstale":"2024-03-21T13:00:59.216000+0000","last_undegraded":"2024-03-21T13:00:59.216000+0000","last_fullsized":"2024-03-21T13:00:59.216000+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:14.717533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:14.717533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:39.916345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":108,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.212724+0000","last_change":"2024-03-21T12:59:55.664752+0000","last_active":"2024-03-21T13:00:59.212724+0000","last_peered":"2024-03-21T13:00:59.212724+0000","last_clean":"2024-03-21T13:00:59.212724+0000","last_became_active":"2024-03-21T12:58:39.749130+0000","last_became_peered":"2024-03-21T12:58:39.749130+0000","last_unstale":"2024-03-21T13:00:59.212724+0000","last_undegraded":"2024-03-21T13:00:59.212724+0000","last_fullsized":"2024-03-21T13:00:59.212724+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T12:59:55.664649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.664649+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:11.693713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.224620+0000","last_change":"2024-03-21T13:00:35.036444+0000","last_active":"2024-03-21T13:00:59.224620+0000","last_peered":"2024-03-21T13:00:59.224620+0000","last_clean":"2024-03-21T13:00:59.224620+0000","last_became_active":"2024-03-21T12:58:43.352510+0000","last_became_peered":"2024-03-21T12:58:43.352510+0000","last_unstale":"2024-03-21T13:00:59.224620+0000","last_undegraded":"2024-03-21T13:00:59.224620+0000","last_fullsized":"2024-03-21T13:00:59.224620+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:35.036335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.036335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:46.110420+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213604+0000","last_change":"2024-03-21T12:59:50.904504+0000","last_active":"2024-03-21T13:00:59.213604+0000","last_peered":"2024-03-21T13:00:59.213604+0000","last_clean":"2024-03-21T13:00:59.213604+0000","last_became_active":"2024-03-21T12:58:41.167961+0000","last_became_peered":"2024-03-21T12:58:41.167961+0000","last_unstale":"2024-03-21T13:00:59.213604+0000","last_undegraded":"2024-03-21T13:00:59.213604+0000","last_fullsized":"2024-03-21T13:00:59.213604+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.904419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.904419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.905923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213252+0000","last_change":"2024-03-21T12:59:54.948197+0000","last_active":"2024-03-21T13:00:59.213252+0000","last_peered":"2024-03-21T13:00:59.213252+0000","last_clean":"2024-03-21T13:00:59.213252+0000","last_became_active":"2024-03-21T12:58:37.732355+0000","last_became_peered":"2024-03-21T12:58:37.732355+0000","last_unstale":"2024-03-21T13:00:59.213252+0000","last_undegraded":"2024-03-21T13:00:59.213252+0000","last_fullsized":"2024-03-21T13:00:59.213252+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_clean_scrub_stamp":"2024-03-21T12:59:54.948081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'12","reported_seq":115,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.223493+0000","last_change":"2024-03-21T12:59:01.168089+0000","last_active":"2024-03-21T13:00:59.223493+0000","last_peered":"2024-03-21T13:00:59.223493+0000","last_clean":"2024-03-21T13:00:59.223493+0000","last_became_active":"2024-03-21T12:59:01.161520+0000","last_became_peered":"2024-03-21T12:59:01.161520+0000","last_unstale":"2024-03-21T13:00:59.223493+0000","last_undegraded":"2024-03-21T13:00:59.223493+0000","last_fullsized":"2024-03-21T13:00:59.223493+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.068607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":73,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.208835+0000","last_change":"2024-03-21T13:00:00.168621+0000","last_active":"2024-03-21T13:00:59.208835+0000","last_peered":"2024-03-21T13:00:59.208835+0000","last_clean":"2024-03-21T13:00:59.208835+0000","last_became_active":"2024-03-21T12:58:43.184473+0000","last_became_peered":"2024-03-21T12:58:43.184473+0000","last_unstale":"2024-03-21T13:00:59.208835+0000","last_undegraded":"2024-03-21T13:00:59.208835+0000","last_fullsized":"2024-03-21T13:00:59.208835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:00:00.168509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:00.168509+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.057928+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":120,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.212681+0000","last_change":"2024-03-21T12:59:55.277279+0000","last_active":"2024-03-21T13:00:59.212681+0000","last_peered":"2024-03-21T13:00:59.212681+0000","last_clean":"2024-03-21T13:00:59.212681+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:00:59.212681+0000","last_undegraded":"2024-03-21T13:00:59.212681+0000","last_fullsized":"2024-03-21T13:00:59.212681+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:55.277165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.277165+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:19.101757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.218823+0000","last_change":"2024-03-21T13:00:10.838057+0000","last_active":"2024-03-21T13:00:59.218823+0000","last_peered":"2024-03-21T13:00:59.218823+0000","last_clean":"2024-03-21T13:00:59.218823+0000","last_became_active":"2024-03-21T12:58:37.736455+0000","last_became_peered":"2024-03-21T12:58:37.736455+0000","last_unstale":"2024-03-21T13:00:59.218823+0000","last_undegraded":"2024-03-21T13:00:59.218823+0000","last_fullsized":"2024-03-21T13:00:59.218823+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:00:10.837949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.837949+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:29.483399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"71'10","reported_seq":97,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.214981+0000","last_change":"2024-03-21T12:59:50.414098+0000","last_active":"2024-03-21T13:00:59.214981+0000","last_peered":"2024-03-21T13:00:59.214981+0000","last_clean":"2024-03-21T13:00:59.214981+0000","last_became_active":"2024-03-21T12:58:39.754548+0000","last_became_peered":"2024-03-21T12:58:39.754548+0000","last_unstale":"2024-03-21T13:00:59.214981+0000","last_undegraded":"2024-03-21T13:00:59.214981+0000","last_fullsized":"2024-03-21T13:00:59.214981+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T12:59:50.413983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.413983+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.575091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":74,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.219695+0000","last_change":"2024-03-21T13:00:14.812018+0000","last_active":"2024-03-21T13:00:59.219695+0000","last_peered":"2024-03-21T13:00:59.219695+0000","last_clean":"2024-03-21T13:00:59.219695+0000","last_became_active":"2024-03-21T12:58:43.186289+0000","last_became_peered":"2024-03-21T12:58:43.186289+0000","last_unstale":"2024-03-21T13:00:59.219695+0000","last_undegraded":"2024-03-21T13:00:59.219695+0000","last_fullsized":"2024-03-21T13:00:59.219695+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:14.811940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:14.811940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:26.184428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221133+0000","last_change":"2024-03-21T13:00:23.746678+0000","last_active":"2024-03-21T13:00:59.221133+0000","last_peered":"2024-03-21T13:00:59.221133+0000","last_clean":"2024-03-21T13:00:59.221133+0000","last_became_active":"2024-03-21T12:58:41.167443+0000","last_became_peered":"2024-03-21T12:58:41.167443+0000","last_unstale":"2024-03-21T13:00:59.221133+0000","last_undegraded":"2024-03-21T13:00:59.221133+0000","last_fullsized":"2024-03-21T13:00:59.221133+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:23.746600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:23.746600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:34.558908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":86,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.220622+0000","last_change":"2024-03-21T12:59:49.178866+0000","last_active":"2024-03-21T13:00:59.220622+0000","last_peered":"2024-03-21T13:00:59.220622+0000","last_clean":"2024-03-21T13:00:59.220622+0000","last_became_active":"2024-03-21T12:58:37.728914+0000","last_became_peered":"2024-03-21T12:58:37.728914+0000","last_unstale":"2024-03-21T13:00:59.220622+0000","last_undegraded":"2024-03-21T13:00:59.220622+0000","last_fullsized":"2024-03-21T13:00:59.220622+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:49.178785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:49.178785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.644002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'11","reported_seq":102,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.212578+0000","last_change":"2024-03-21T13:00:01.396865+0000","last_active":"2024-03-21T13:00:59.212578+0000","last_peered":"2024-03-21T13:00:59.212578+0000","last_clean":"2024-03-21T13:00:59.212578+0000","last_became_active":"2024-03-21T12:58:39.755294+0000","last_became_peered":"2024-03-21T12:58:39.755294+0000","last_unstale":"2024-03-21T13:00:59.212578+0000","last_undegraded":"2024-03-21T13:00:59.212578+0000","last_fullsized":"2024-03-21T13:00:59.212578+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:00:01.396749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.396749+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.505311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":73,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.208233+0000","last_change":"2024-03-21T12:59:50.209251+0000","last_active":"2024-03-21T13:00:59.208233+0000","last_peered":"2024-03-21T13:00:59.208233+0000","last_clean":"2024-03-21T13:00:59.208233+0000","last_became_active":"2024-03-21T12:58:43.183475+0000","last_became_peered":"2024-03-21T12:58:43.183475+0000","last_unstale":"2024-03-21T13:00:59.208233+0000","last_undegraded":"2024-03-21T13:00:59.208233+0000","last_fullsized":"2024-03-21T13:00:59.208233+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.209152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.209152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:18.566644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.211452+0000","last_change":"2024-03-21T13:00:03.421382+0000","last_active":"2024-03-21T13:00:59.211452+0000","last_peered":"2024-03-21T13:00:59.211452+0000","last_clean":"2024-03-21T13:00:59.211452+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:00:59.211452+0000","last_undegraded":"2024-03-21T13:00:59.211452+0000","last_fullsized":"2024-03-21T13:00:59.211452+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:03.421305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:03.421305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.521044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":86,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.214707+0000","last_change":"2024-03-21T13:00:32.790166+0000","last_active":"2024-03-21T13:00:59.214707+0000","last_peered":"2024-03-21T13:00:59.214707+0000","last_clean":"2024-03-21T13:00:59.214707+0000","last_became_active":"2024-03-21T12:58:37.732144+0000","last_became_peered":"2024-03-21T12:58:37.732144+0000","last_unstale":"2024-03-21T13:00:59.214707+0000","last_undegraded":"2024-03-21T13:00:59.214707+0000","last_fullsized":"2024-03-21T13:00:59.214707+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:32.789993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.789993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:46.315695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":81,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.220688+0000","last_change":"2024-03-21T12:58:39.755658+0000","last_active":"2024-03-21T13:00:59.220688+0000","last_peered":"2024-03-21T13:00:59.220688+0000","last_clean":"2024-03-21T13:00:59.220688+0000","last_became_active":"2024-03-21T12:58:39.755336+0000","last_became_peered":"2024-03-21T12:58:39.755336+0000","last_unstale":"2024-03-21T13:00:59.220688+0000","last_undegraded":"2024-03-21T13:00:59.220688+0000","last_fullsized":"2024-03-21T13:00:59.220688+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.068540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":74,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.224073+0000","last_change":"2024-03-21T13:00:10.110413+0000","last_active":"2024-03-21T13:00:59.224073+0000","last_peered":"2024-03-21T13:00:59.224073+0000","last_clean":"2024-03-21T13:00:59.224073+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:00:59.224073+0000","last_undegraded":"2024-03-21T13:00:59.224073+0000","last_fullsized":"2024-03-21T13:00:59.224073+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:10.110335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.110335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:30.400061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":77,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.208470+0000","last_change":"2024-03-21T13:00:18.163092+0000","last_active":"2024-03-21T13:00:59.208470+0000","last_peered":"2024-03-21T13:00:59.208470+0000","last_clean":"2024-03-21T13:00:59.208470+0000","last_became_active":"2024-03-21T12:58:41.164784+0000","last_became_peered":"2024-03-21T12:58:41.164784+0000","last_unstale":"2024-03-21T13:00:59.208470+0000","last_undegraded":"2024-03-21T13:00:59.208470+0000","last_fullsized":"2024-03-21T13:00:59.208470+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:18.162988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:18.162988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:35.655138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221881+0000","last_change":"2024-03-21T13:00:09.658052+0000","last_active":"2024-03-21T13:00:59.221881+0000","last_peered":"2024-03-21T13:00:59.221881+0000","last_clean":"2024-03-21T13:00:59.221881+0000","last_became_active":"2024-03-21T12:58:37.736127+0000","last_became_peered":"2024-03-21T12:58:37.736127+0000","last_unstale":"2024-03-21T13:00:59.221881+0000","last_undegraded":"2024-03-21T13:00:59.221881+0000","last_fullsized":"2024-03-21T13:00:59.221881+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:09.657971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:09.657971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:30.873328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"71'11","reported_seq":102,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.224764+0000","last_change":"2024-03-21T12:59:52.181828+0000","last_active":"2024-03-21T13:00:59.224764+0000","last_peered":"2024-03-21T13:00:59.224764+0000","last_clean":"2024-03-21T13:00:59.224764+0000","last_became_active":"2024-03-21T12:58:39.753014+0000","last_became_peered":"2024-03-21T12:58:39.753014+0000","last_unstale":"2024-03-21T13:00:59.224764+0000","last_undegraded":"2024-03-21T13:00:59.224764+0000","last_fullsized":"2024-03-21T13:00:59.224764+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T12:59:52.181697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.181697+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.248453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":66,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.217521+0000","last_change":"2024-03-21T12:58:43.178915+0000","last_active":"2024-03-21T13:00:59.217521+0000","last_peered":"2024-03-21T13:00:59.217521+0000","last_clean":"2024-03-21T13:00:59.217521+0000","last_became_active":"2024-03-21T12:58:43.178589+0000","last_became_peered":"2024-03-21T12:58:43.178589+0000","last_unstale":"2024-03-21T13:00:59.217521+0000","last_undegraded":"2024-03-21T13:00:59.217521+0000","last_fullsized":"2024-03-21T13:00:59.217521+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:06.829865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.223571+0000","last_change":"2024-03-21T13:00:38.995307+0000","last_active":"2024-03-21T13:00:59.223571+0000","last_peered":"2024-03-21T13:00:59.223571+0000","last_clean":"2024-03-21T13:00:59.223571+0000","last_became_active":"2024-03-21T12:58:41.167371+0000","last_became_peered":"2024-03-21T12:58:41.167371+0000","last_unstale":"2024-03-21T13:00:59.223571+0000","last_undegraded":"2024-03-21T13:00:59.223571+0000","last_fullsized":"2024-03-21T13:00:59.223571+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:38.995215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.995215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:07.081472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":85,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.209059+0000","last_change":"2024-03-21T13:00:19.153249+0000","last_active":"2024-03-21T13:00:59.209059+0000","last_peered":"2024-03-21T13:00:59.209059+0000","last_clean":"2024-03-21T13:00:59.209059+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:00:59.209059+0000","last_undegraded":"2024-03-21T13:00:59.209059+0000","last_fullsized":"2024-03-21T13:00:59.209059+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:19.153095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:19.153095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:25.434748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"86'13","reported_seq":103,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.219728+0000","last_change":"2024-03-21T12:59:48.085870+0000","last_active":"2024-03-21T13:00:59.219728+0000","last_peered":"2024-03-21T13:00:59.219728+0000","last_clean":"2024-03-21T13:00:59.219728+0000","last_became_active":"2024-03-21T12:58:39.753224+0000","last_became_peered":"2024-03-21T12:58:39.753224+0000","last_unstale":"2024-03-21T13:00:59.219728+0000","last_undegraded":"2024-03-21T13:00:59.219728+0000","last_fullsized":"2024-03-21T13:00:59.219728+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.085774+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:00:59.953375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":67,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.224404+0000","last_change":"2024-03-21T12:58:43.184148+0000","last_active":"2024-03-21T13:00:59.224404+0000","last_peered":"2024-03-21T13:00:59.224404+0000","last_clean":"2024-03-21T13:00:59.224404+0000","last_became_active":"2024-03-21T12:58:43.183712+0000","last_became_peered":"2024-03-21T12:58:43.183712+0000","last_unstale":"2024-03-21T13:00:59.224404+0000","last_undegraded":"2024-03-21T13:00:59.224404+0000","last_fullsized":"2024-03-21T13:00:59.224404+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.219527+0000","last_change":"2024-03-21T12:59:45.088950+0000","last_active":"2024-03-21T13:00:59.219527+0000","last_peered":"2024-03-21T13:00:59.219527+0000","last_clean":"2024-03-21T13:00:59.219527+0000","last_became_active":"2024-03-21T12:58:41.164463+0000","last_became_peered":"2024-03-21T12:58:41.164463+0000","last_unstale":"2024-03-21T13:00:59.219527+0000","last_undegraded":"2024-03-21T13:00:59.219527+0000","last_fullsized":"2024-03-21T13:00:59.219527+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:45.088861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.088861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:11.118145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213350+0000","last_change":"2024-03-21T12:59:45.833435+0000","last_active":"2024-03-21T13:00:59.213350+0000","last_peered":"2024-03-21T13:00:59.213350+0000","last_clean":"2024-03-21T13:00:59.213350+0000","last_became_active":"2024-03-21T12:58:37.731659+0000","last_became_peered":"2024-03-21T12:58:37.731659+0000","last_unstale":"2024-03-21T13:00:59.213350+0000","last_undegraded":"2024-03-21T13:00:59.213350+0000","last_fullsized":"2024-03-21T13:00:59.213350+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.833304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.590100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":94,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213592+0000","last_change":"2024-03-21T12:58:39.746514+0000","last_active":"2024-03-21T13:00:59.213592+0000","last_peered":"2024-03-21T13:00:59.213592+0000","last_clean":"2024-03-21T13:00:59.213592+0000","last_became_active":"2024-03-21T12:58:39.746182+0000","last_became_peered":"2024-03-21T12:58:39.746182+0000","last_unstale":"2024-03-21T13:00:59.213592+0000","last_undegraded":"2024-03-21T13:00:59.213592+0000","last_fullsized":"2024-03-21T13:00:59.213592+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":65,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.214623+0000","last_change":"2024-03-21T12:58:43.181024+0000","last_active":"2024-03-21T13:00:59.214623+0000","last_peered":"2024-03-21T13:00:59.214623+0000","last_clean":"2024-03-21T13:00:59.214623+0000","last_became_active":"2024-03-21T12:58:43.180773+0000","last_became_peered":"2024-03-21T12:58:43.180773+0000","last_unstale":"2024-03-21T13:00:59.214623+0000","last_undegraded":"2024-03-21T13:00:59.214623+0000","last_fullsized":"2024-03-21T13:00:59.214623+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:00.883276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":120,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:02.799651+0000","last_change":"2024-03-21T12:59:50.724334+0000","last_active":"2024-03-21T13:01:02.799651+0000","last_peered":"2024-03-21T13:01:02.799651+0000","last_clean":"2024-03-21T13:01:02.799651+0000","last_became_active":"2024-03-21T12:58:41.167956+0000","last_became_peered":"2024-03-21T12:58:41.167956+0000","last_unstale":"2024-03-21T13:01:02.799651+0000","last_undegraded":"2024-03-21T13:01:02.799651+0000","last_fullsized":"2024-03-21T13:01:02.799651+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:50.724242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.724242+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":85,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.208335+0000","last_change":"2024-03-21T13:00:14.161251+0000","last_active":"2024-03-21T13:00:59.208335+0000","last_peered":"2024-03-21T13:00:59.208335+0000","last_clean":"2024-03-21T13:00:59.208335+0000","last_became_active":"2024-03-21T12:58:37.731988+0000","last_became_peered":"2024-03-21T12:58:37.731988+0000","last_unstale":"2024-03-21T13:00:59.208335+0000","last_undegraded":"2024-03-21T13:00:59.208335+0000","last_fullsized":"2024-03-21T13:00:59.208335+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:14.161136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:14.161136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:26.647081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"71'10","reported_seq":98,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.223915+0000","last_change":"2024-03-21T13:00:06.198017+0000","last_active":"2024-03-21T13:00:59.223915+0000","last_peered":"2024-03-21T13:00:59.223915+0000","last_clean":"2024-03-21T13:00:59.223915+0000","last_became_active":"2024-03-21T12:58:39.757806+0000","last_became_peered":"2024-03-21T12:58:39.757806+0000","last_unstale":"2024-03-21T13:00:59.223915+0000","last_undegraded":"2024-03-21T13:00:59.223915+0000","last_fullsized":"2024-03-21T13:00:59.223915+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:00:06.197903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.197903+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:24.909636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":74,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213664+0000","last_change":"2024-03-21T13:00:30.855570+0000","last_active":"2024-03-21T13:00:59.213664+0000","last_peered":"2024-03-21T13:00:59.213664+0000","last_clean":"2024-03-21T13:00:59.213664+0000","last_became_active":"2024-03-21T12:58:43.186155+0000","last_became_peered":"2024-03-21T12:58:43.186155+0000","last_unstale":"2024-03-21T13:00:59.213664+0000","last_undegraded":"2024-03-21T13:00:59.213664+0000","last_fullsized":"2024-03-21T13:00:59.213664+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:30.855482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:30.855482+0000","last_clean_scrub_stamp":"2024-03-21T13:00:30.855482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:32.496620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":109,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:02.800319+0000","last_change":"2024-03-21T12:58:41.168116+0000","last_active":"2024-03-21T13:01:02.800319+0000","last_peered":"2024-03-21T13:01:02.800319+0000","last_clean":"2024-03-21T13:01:02.800319+0000","last_became_active":"2024-03-21T12:58:41.167826+0000","last_became_peered":"2024-03-21T12:58:41.167826+0000","last_unstale":"2024-03-21T13:01:02.800319+0000","last_undegraded":"2024-03-21T13:01:02.800319+0000","last_fullsized":"2024-03-21T13:01:02.800319+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":86,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.215480+0000","last_change":"2024-03-21T13:00:35.801717+0000","last_active":"2024-03-21T13:00:59.215480+0000","last_peered":"2024-03-21T13:00:59.215480+0000","last_clean":"2024-03-21T13:00:59.215480+0000","last_became_active":"2024-03-21T12:58:37.732166+0000","last_became_peered":"2024-03-21T12:58:37.732166+0000","last_unstale":"2024-03-21T13:00:59.215480+0000","last_undegraded":"2024-03-21T13:00:59.215480+0000","last_fullsized":"2024-03-21T13:00:59.215480+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:35.801638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.801638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:43.319587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'9","reported_seq":99,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221863+0000","last_change":"2024-03-21T13:00:19.687048+0000","last_active":"2024-03-21T13:00:59.221863+0000","last_peered":"2024-03-21T13:00:59.221863+0000","last_clean":"2024-03-21T13:00:59.221863+0000","last_became_active":"2024-03-21T12:58:39.750820+0000","last_became_peered":"2024-03-21T12:58:39.750820+0000","last_unstale":"2024-03-21T13:00:59.221863+0000","last_undegraded":"2024-03-21T13:00:59.221863+0000","last_fullsized":"2024-03-21T13:00:59.221863+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:00:19.686939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:19.686939+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:49.122148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213064+0000","last_change":"2024-03-21T12:59:51.730551+0000","last_active":"2024-03-21T13:00:59.213064+0000","last_peered":"2024-03-21T13:00:59.213064+0000","last_clean":"2024-03-21T13:00:59.213064+0000","last_became_active":"2024-03-21T12:58:43.182126+0000","last_became_peered":"2024-03-21T12:58:43.182126+0000","last_unstale":"2024-03-21T13:00:59.213064+0000","last_undegraded":"2024-03-21T13:00:59.213064+0000","last_fullsized":"2024-03-21T13:00:59.213064+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T12:59:51.730413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.791726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.225150+0000","last_change":"2024-03-21T13:00:12.122078+0000","last_active":"2024-03-21T13:00:59.225150+0000","last_peered":"2024-03-21T13:00:59.225150+0000","last_clean":"2024-03-21T13:00:59.225150+0000","last_became_active":"2024-03-21T12:58:41.172904+0000","last_became_peered":"2024-03-21T12:58:41.172904+0000","last_unstale":"2024-03-21T13:00:59.225150+0000","last_undegraded":"2024-03-21T13:00:59.225150+0000","last_fullsized":"2024-03-21T13:00:59.225150+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:12.121970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:12.121970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:31.573284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":91,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.208215+0000","last_change":"2024-03-21T12:59:47.203592+0000","last_active":"2024-03-21T13:00:59.208215+0000","last_peered":"2024-03-21T13:00:59.208215+0000","last_clean":"2024-03-21T13:00:59.208215+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:00:59.208215+0000","last_undegraded":"2024-03-21T13:00:59.208215+0000","last_fullsized":"2024-03-21T13:00:59.208215+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T12:59:47.203453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.203453+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.098187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'9","reported_seq":98,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.222269+0000","last_change":"2024-03-21T13:00:06.685183+0000","last_active":"2024-03-21T13:00:59.222269+0000","last_peered":"2024-03-21T13:00:59.222269+0000","last_clean":"2024-03-21T13:00:59.222269+0000","last_became_active":"2024-03-21T12:58:39.752722+0000","last_became_peered":"2024-03-21T12:58:39.752722+0000","last_unstale":"2024-03-21T13:00:59.222269+0000","last_undegraded":"2024-03-21T13:00:59.222269+0000","last_fullsized":"2024-03-21T13:00:59.222269+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:00:06.685071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.685071+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:35.157380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":72,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.208888+0000","last_change":"2024-03-21T13:00:08.197105+0000","last_active":"2024-03-21T13:00:59.208888+0000","last_peered":"2024-03-21T13:00:59.208888+0000","last_clean":"2024-03-21T13:00:59.208888+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:00:59.208888+0000","last_undegraded":"2024-03-21T13:00:59.208888+0000","last_fullsized":"2024-03-21T13:00:59.208888+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.196979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.196979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:33.771136+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221785+0000","last_change":"2024-03-21T13:00:32.639997+0000","last_active":"2024-03-21T13:00:59.221785+0000","last_peered":"2024-03-21T13:00:59.221785+0000","last_clean":"2024-03-21T13:00:59.221785+0000","last_became_active":"2024-03-21T12:58:41.162343+0000","last_became_peered":"2024-03-21T12:58:41.162343+0000","last_unstale":"2024-03-21T13:00:59.221785+0000","last_undegraded":"2024-03-21T13:00:59.221785+0000","last_fullsized":"2024-03-21T13:00:59.221785+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:32.639920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.639920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:58.933596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.210999+0000","last_change":"2024-03-21T12:59:52.319029+0000","last_active":"2024-03-21T13:00:59.210999+0000","last_peered":"2024-03-21T13:00:59.210999+0000","last_clean":"2024-03-21T13:00:59.210999+0000","last_became_active":"2024-03-21T12:58:37.731932+0000","last_became_peered":"2024-03-21T12:58:37.731932+0000","last_unstale":"2024-03-21T13:00:59.210999+0000","last_undegraded":"2024-03-21T13:00:59.210999+0000","last_fullsized":"2024-03-21T13:00:59.210999+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:52.318927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.318927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:20.356137+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'6","reported_seq":92,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.225012+0000","last_change":"2024-03-21T12:59:46.176940+0000","last_active":"2024-03-21T13:00:59.225012+0000","last_peered":"2024-03-21T13:00:59.225012+0000","last_clean":"2024-03-21T13:00:59.225012+0000","last_became_active":"2024-03-21T12:58:39.755419+0000","last_became_peered":"2024-03-21T12:58:39.755419+0000","last_unstale":"2024-03-21T13:00:59.225012+0000","last_undegraded":"2024-03-21T13:00:59.225012+0000","last_fullsized":"2024-03-21T13:00:59.225012+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T12:59:46.176819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.176819+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:06.815464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"98'1567","reported_seq":1589,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.618117+0000","last_change":"2024-03-21T13:00:42.710144+0000","last_active":"2024-03-21T13:01:04.618117+0000","last_peered":"2024-03-21T13:01:04.618117+0000","last_clean":"2024-03-21T13:01:04.618117+0000","last_became_active":"2024-03-21T13:00:42.709816+0000","last_became_peered":"2024-03-21T13:00:42.709816+0000","last_unstale":"2024-03-21T13:01:04.618117+0000","last_undegraded":"2024-03-21T13:01:04.618117+0000","last_fullsized":"2024-03-21T13:01:04.618117+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1567,"log_dups_size":0,"ondisk_log_size":1567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:03.188475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102694912,"num_objects":1567,"num_object_clones":0,"num_object_copies":4701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1567,"num_write_kb":100288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221451+0000","last_change":"2024-03-21T13:00:00.622196+0000","last_active":"2024-03-21T13:00:59.221451+0000","last_peered":"2024-03-21T13:00:59.221451+0000","last_clean":"2024-03-21T13:00:59.221451+0000","last_became_active":"2024-03-21T12:58:43.351103+0000","last_became_peered":"2024-03-21T12:58:43.351103+0000","last_unstale":"2024-03-21T13:00:59.221451+0000","last_undegraded":"2024-03-21T13:00:59.221451+0000","last_fullsized":"2024-03-21T13:00:59.221451+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:00.622094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:00.622094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:21.931294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":97,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.209511+0000","last_change":"2024-03-21T13:00:06.188217+0000","last_active":"2024-03-21T13:00:59.209511+0000","last_peered":"2024-03-21T13:00:59.209511+0000","last_clean":"2024-03-21T13:00:59.209511+0000","last_became_active":"2024-03-21T12:58:37.735037+0000","last_became_peered":"2024-03-21T12:58:37.735037+0000","last_unstale":"2024-03-21T13:00:59.209511+0000","last_undegraded":"2024-03-21T13:00:59.209511+0000","last_fullsized":"2024-03-21T13:00:59.209511+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:00:06.188118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.188118+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.485685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'15","reported_seq":108,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221168+0000","last_change":"2024-03-21T13:00:27.640501+0000","last_active":"2024-03-21T13:00:59.221168+0000","last_peered":"2024-03-21T13:00:59.221168+0000","last_clean":"2024-03-21T13:00:59.221168+0000","last_became_active":"2024-03-21T12:58:39.750546+0000","last_became_peered":"2024-03-21T12:58:39.750546+0000","last_unstale":"2024-03-21T13:00:59.221168+0000","last_undegraded":"2024-03-21T13:00:59.221168+0000","last_fullsized":"2024-03-21T13:00:59.221168+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:00:27.640392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:27.640392+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:49.270291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":116,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.208380+0000","last_change":"2024-03-21T13:00:20.176757+0000","last_active":"2024-03-21T13:00:59.208380+0000","last_peered":"2024-03-21T13:00:59.208380+0000","last_clean":"2024-03-21T13:00:59.208380+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:00:59.208380+0000","last_undegraded":"2024-03-21T13:00:59.208380+0000","last_fullsized":"2024-03-21T13:00:59.208380+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.176653+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:46.820452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"98'1605","reported_seq":1627,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.613274+0000","last_change":"2024-03-21T13:00:42.712078+0000","last_active":"2024-03-21T13:01:04.613274+0000","last_peered":"2024-03-21T13:01:04.613274+0000","last_clean":"2024-03-21T13:01:04.613274+0000","last_became_active":"2024-03-21T13:00:42.711762+0000","last_became_peered":"2024-03-21T13:00:42.711762+0000","last_unstale":"2024-03-21T13:01:04.613274+0000","last_undegraded":"2024-03-21T13:01:04.613274+0000","last_fullsized":"2024-03-21T13:01:04.613274+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1605,"log_dups_size":0,"ondisk_log_size":1605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:06.070955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105185280,"num_objects":1605,"num_object_clones":0,"num_object_copies":4815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1605,"num_write_kb":102720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.223747+0000","last_change":"2024-03-21T13:00:24.988879+0000","last_active":"2024-03-21T13:00:59.223747+0000","last_peered":"2024-03-21T13:00:59.223747+0000","last_clean":"2024-03-21T13:00:59.223747+0000","last_became_active":"2024-03-21T12:58:43.181036+0000","last_became_peered":"2024-03-21T12:58:43.181036+0000","last_unstale":"2024-03-21T13:00:59.223747+0000","last_undegraded":"2024-03-21T13:00:59.223747+0000","last_fullsized":"2024-03-21T13:00:59.223747+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:24.988792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:24.988792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:49.295998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":85,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.209260+0000","last_change":"2024-03-21T12:59:59.136937+0000","last_active":"2024-03-21T13:00:59.209260+0000","last_peered":"2024-03-21T13:00:59.209260+0000","last_clean":"2024-03-21T13:00:59.209260+0000","last_became_active":"2024-03-21T12:58:37.737012+0000","last_became_peered":"2024-03-21T12:58:37.737012+0000","last_unstale":"2024-03-21T13:00:59.209260+0000","last_undegraded":"2024-03-21T13:00:59.209260+0000","last_fullsized":"2024-03-21T13:00:59.209260+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:59.136829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.136829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:03.237208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"71'12","reported_seq":106,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213402+0000","last_change":"2024-03-21T13:00:32.864701+0000","last_active":"2024-03-21T13:00:59.213402+0000","last_peered":"2024-03-21T13:00:59.213402+0000","last_clean":"2024-03-21T13:00:59.213402+0000","last_became_active":"2024-03-21T12:58:39.751236+0000","last_became_peered":"2024-03-21T12:58:39.751236+0000","last_unstale":"2024-03-21T13:00:59.213402+0000","last_undegraded":"2024-03-21T13:00:59.213402+0000","last_fullsized":"2024-03-21T13:00:59.213402+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:00:32.864608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.864608+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:50.366753+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":80,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.212503+0000","last_change":"2024-03-21T12:59:58.733374+0000","last_active":"2024-03-21T13:00:59.212503+0000","last_peered":"2024-03-21T13:00:59.212503+0000","last_clean":"2024-03-21T13:00:59.212503+0000","last_became_active":"2024-03-21T12:58:41.171443+0000","last_became_peered":"2024-03-21T12:58:41.171443+0000","last_unstale":"2024-03-21T13:00:59.212503+0000","last_undegraded":"2024-03-21T13:00:59.212503+0000","last_fullsized":"2024-03-21T13:00:59.212503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:58.733235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:58.733235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.428021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"98'1703","reported_seq":1725,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.632420+0000","last_change":"2024-03-21T13:00:42.711466+0000","last_active":"2024-03-21T13:01:04.632420+0000","last_peered":"2024-03-21T13:01:04.632420+0000","last_clean":"2024-03-21T13:01:04.632420+0000","last_became_active":"2024-03-21T13:00:42.711148+0000","last_became_peered":"2024-03-21T13:00:42.711148+0000","last_unstale":"2024-03-21T13:01:04.632420+0000","last_undegraded":"2024-03-21T13:01:04.632420+0000","last_fullsized":"2024-03-21T13:01:04.632420+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1703,"log_dups_size":0,"ondisk_log_size":1703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:52.430414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111607808,"num_objects":1703,"num_object_clones":0,"num_object_copies":5109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1703,"num_write_kb":108992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":73,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.208080+0000","last_change":"2024-03-21T13:00:04.200955+0000","last_active":"2024-03-21T13:00:59.208080+0000","last_peered":"2024-03-21T13:00:59.208080+0000","last_clean":"2024-03-21T13:00:59.208080+0000","last_became_active":"2024-03-21T12:58:43.179007+0000","last_became_peered":"2024-03-21T12:58:43.179007+0000","last_unstale":"2024-03-21T13:00:59.208080+0000","last_undegraded":"2024-03-21T13:00:59.208080+0000","last_fullsized":"2024-03-21T13:00:59.208080+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:04.200849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:04.200849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:07.029947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":80,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.214177+0000","last_change":"2024-03-21T13:00:08.625392+0000","last_active":"2024-03-21T13:00:59.214177+0000","last_peered":"2024-03-21T13:00:59.214177+0000","last_clean":"2024-03-21T13:00:59.214177+0000","last_became_active":"2024-03-21T12:58:41.165203+0000","last_became_peered":"2024-03-21T12:58:41.165203+0000","last_unstale":"2024-03-21T13:00:59.214177+0000","last_undegraded":"2024-03-21T13:00:59.214177+0000","last_fullsized":"2024-03-21T13:00:59.214177+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.625230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.625230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:13.968472+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":85,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.209177+0000","last_change":"2024-03-21T12:59:46.169468+0000","last_active":"2024-03-21T13:00:59.209177+0000","last_peered":"2024-03-21T13:00:59.209177+0000","last_clean":"2024-03-21T13:00:59.209177+0000","last_became_active":"2024-03-21T12:58:37.731448+0000","last_became_peered":"2024-03-21T12:58:37.731448+0000","last_unstale":"2024-03-21T13:00:59.209177+0000","last_undegraded":"2024-03-21T13:00:59.209177+0000","last_fullsized":"2024-03-21T13:00:59.209177+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:46.169315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.169315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"71'19","reported_seq":114,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.211144+0000","last_change":"2024-03-21T12:59:59.392987+0000","last_active":"2024-03-21T13:00:59.211144+0000","last_peered":"2024-03-21T13:00:59.211144+0000","last_clean":"2024-03-21T13:00:59.211144+0000","last_became_active":"2024-03-21T12:58:39.754761+0000","last_became_peered":"2024-03-21T12:58:39.754761+0000","last_unstale":"2024-03-21T13:00:59.211144+0000","last_undegraded":"2024-03-21T13:00:59.211144+0000","last_fullsized":"2024-03-21T13:00:59.211144+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T12:59:59.392834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:59.392834+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.114737+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"98'1656","reported_seq":1678,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.629736+0000","last_change":"2024-03-21T13:00:42.709181+0000","last_active":"2024-03-21T13:01:04.629736+0000","last_peered":"2024-03-21T13:01:04.629736+0000","last_clean":"2024-03-21T13:01:04.629736+0000","last_became_active":"2024-03-21T13:00:42.708899+0000","last_became_peered":"2024-03-21T13:00:42.708899+0000","last_unstale":"2024-03-21T13:01:04.629736+0000","last_undegraded":"2024-03-21T13:01:04.629736+0000","last_fullsized":"2024-03-21T13:01:04.629736+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1656,"log_dups_size":0,"ondisk_log_size":1656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:04.762840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":4968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1656,"num_write_kb":105984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":74,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213936+0000","last_change":"2024-03-21T13:00:22.771885+0000","last_active":"2024-03-21T13:00:59.213936+0000","last_peered":"2024-03-21T13:00:59.213936+0000","last_clean":"2024-03-21T13:00:59.213936+0000","last_became_active":"2024-03-21T12:58:43.180393+0000","last_became_peered":"2024-03-21T12:58:43.180393+0000","last_unstale":"2024-03-21T13:00:59.213936+0000","last_undegraded":"2024-03-21T13:00:59.213936+0000","last_fullsized":"2024-03-21T13:00:59.213936+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:22.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:22.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:43.185059+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":80,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213087+0000","last_change":"2024-03-21T13:00:01.667365+0000","last_active":"2024-03-21T13:00:59.213087+0000","last_peered":"2024-03-21T13:00:59.213087+0000","last_clean":"2024-03-21T13:00:59.213087+0000","last_became_active":"2024-03-21T12:58:41.163843+0000","last_became_peered":"2024-03-21T12:58:41.163843+0000","last_unstale":"2024-03-21T13:00:59.213087+0000","last_undegraded":"2024-03-21T13:00:59.213087+0000","last_fullsized":"2024-03-21T13:00:59.213087+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:01.667285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.667285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:31.061685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221501+0000","last_change":"2024-03-21T12:59:47.676292+0000","last_active":"2024-03-21T13:00:59.221501+0000","last_peered":"2024-03-21T13:00:59.221501+0000","last_clean":"2024-03-21T13:00:59.221501+0000","last_became_active":"2024-03-21T12:58:37.730518+0000","last_became_peered":"2024-03-21T12:58:37.730518+0000","last_unstale":"2024-03-21T13:00:59.221501+0000","last_undegraded":"2024-03-21T13:00:59.221501+0000","last_fullsized":"2024-03-21T13:00:59.221501+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:47.676198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.676198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.156465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"86'13","reported_seq":102,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.217440+0000","last_change":"2024-03-21T12:59:46.421228+0000","last_active":"2024-03-21T13:00:59.217440+0000","last_peered":"2024-03-21T13:00:59.217440+0000","last_clean":"2024-03-21T13:00:59.217440+0000","last_became_active":"2024-03-21T12:58:39.748144+0000","last_became_peered":"2024-03-21T12:58:39.748144+0000","last_unstale":"2024-03-21T13:00:59.217440+0000","last_undegraded":"2024-03-21T13:00:59.217440+0000","last_fullsized":"2024-03-21T13:00:59.217440+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.421144+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:06.815237+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"98'1647","reported_seq":1669,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.625985+0000","last_change":"2024-03-21T13:00:42.707087+0000","last_active":"2024-03-21T13:01:04.625985+0000","last_peered":"2024-03-21T13:01:04.625985+0000","last_clean":"2024-03-21T13:01:04.625985+0000","last_became_active":"2024-03-21T13:00:42.706774+0000","last_became_peered":"2024-03-21T13:00:42.706774+0000","last_unstale":"2024-03-21T13:01:04.625985+0000","last_undegraded":"2024-03-21T13:01:04.625985+0000","last_fullsized":"2024-03-21T13:01:04.625985+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1647,"log_dups_size":0,"ondisk_log_size":1647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:41.958747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107937792,"num_objects":1647,"num_object_clones":0,"num_object_copies":4941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1647,"num_write_kb":105408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":67,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.223614+0000","last_change":"2024-03-21T12:58:43.180710+0000","last_active":"2024-03-21T13:00:59.223614+0000","last_peered":"2024-03-21T13:00:59.223614+0000","last_clean":"2024-03-21T13:00:59.223614+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:00:59.223614+0000","last_undegraded":"2024-03-21T13:00:59.223614+0000","last_fullsized":"2024-03-21T13:00:59.223614+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:58:42.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.068636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"67'2","reported_seq":98,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221289+0000","last_change":"2024-03-21T12:59:55.541552+0000","last_active":"2024-03-21T13:00:59.221289+0000","last_peered":"2024-03-21T13:00:59.221289+0000","last_clean":"2024-03-21T13:00:59.221289+0000","last_became_active":"2024-03-21T12:58:36.724010+0000","last_became_peered":"2024-03-21T12:58:36.724010+0000","last_unstale":"2024-03-21T13:00:59.221289+0000","last_undegraded":"2024-03-21T13:00:59.221289+0000","last_fullsized":"2024-03-21T13:00:59.221289+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-21T12:59:55.541410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:59:55.541410+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:17.828687+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213192+0000","last_change":"2024-03-21T12:59:47.840935+0000","last_active":"2024-03-21T13:00:59.213192+0000","last_peered":"2024-03-21T13:00:59.213192+0000","last_clean":"2024-03-21T13:00:59.213192+0000","last_became_active":"2024-03-21T12:58:37.729454+0000","last_became_peered":"2024-03-21T12:58:37.729454+0000","last_unstale":"2024-03-21T13:00:59.213192+0000","last_undegraded":"2024-03-21T13:00:59.213192+0000","last_fullsized":"2024-03-21T13:00:59.213192+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:47.840831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:47.840831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.579290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"71'28","reported_seq":161,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.219363+0000","last_change":"2024-03-21T13:00:01.917692+0000","last_active":"2024-03-21T13:00:59.219363+0000","last_peered":"2024-03-21T13:00:59.219363+0000","last_clean":"2024-03-21T13:00:59.219363+0000","last_became_active":"2024-03-21T12:58:39.756052+0000","last_became_peered":"2024-03-21T12:58:39.756052+0000","last_unstale":"2024-03-21T13:00:59.219363+0000","last_undegraded":"2024-03-21T13:00:59.219363+0000","last_fullsized":"2024-03-21T13:00:59.219363+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'28","last_scrub_stamp":"2024-03-21T13:00:01.917593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.917593+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:26.189859+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":78,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213617+0000","last_change":"2024-03-21T13:00:36.760348+0000","last_active":"2024-03-21T13:00:59.213617+0000","last_peered":"2024-03-21T13:00:59.213617+0000","last_clean":"2024-03-21T13:00:59.213617+0000","last_became_active":"2024-03-21T12:58:41.170460+0000","last_became_peered":"2024-03-21T12:58:41.170460+0000","last_unstale":"2024-03-21T13:00:59.213617+0000","last_undegraded":"2024-03-21T13:00:59.213617+0000","last_fullsized":"2024-03-21T13:00:59.213617+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:36.760263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:36.760263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:54.384592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"98'1635","reported_seq":1657,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.644134+0000","last_change":"2024-03-21T13:00:42.710971+0000","last_active":"2024-03-21T13:01:04.644134+0000","last_peered":"2024-03-21T13:01:04.644134+0000","last_clean":"2024-03-21T13:01:04.644134+0000","last_became_active":"2024-03-21T13:00:42.710669+0000","last_became_peered":"2024-03-21T13:00:42.710669+0000","last_unstale":"2024-03-21T13:01:04.644134+0000","last_undegraded":"2024-03-21T13:01:04.644134+0000","last_fullsized":"2024-03-21T13:01:04.644134+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1635,"log_dups_size":0,"ondisk_log_size":1635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:02.881793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107151360,"num_objects":1635,"num_object_clones":0,"num_object_copies":4905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1635,"num_write_kb":104640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":73,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.209337+0000","last_change":"2024-03-21T12:59:54.191245+0000","last_active":"2024-03-21T13:00:59.209337+0000","last_peered":"2024-03-21T13:00:59.209337+0000","last_clean":"2024-03-21T13:00:59.209337+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:00:59.209337+0000","last_undegraded":"2024-03-21T13:00:59.209337+0000","last_fullsized":"2024-03-21T13:00:59.209337+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:54.191150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:54.191150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:15.445044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.218694+0000","last_change":"2024-03-21T13:00:22.916758+0000","last_active":"2024-03-21T13:00:59.218694+0000","last_peered":"2024-03-21T13:00:59.218694+0000","last_clean":"2024-03-21T13:00:59.218694+0000","last_became_active":"2024-03-21T12:58:37.731386+0000","last_became_peered":"2024-03-21T12:58:37.731386+0000","last_unstale":"2024-03-21T13:00:59.218694+0000","last_undegraded":"2024-03-21T13:00:59.218694+0000","last_fullsized":"2024-03-21T13:00:59.218694+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:22.916635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:22.916635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:51.600542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.212709+0000","last_change":"2024-03-21T12:59:57.734511+0000","last_active":"2024-03-21T13:00:59.212709+0000","last_peered":"2024-03-21T13:00:59.212709+0000","last_clean":"2024-03-21T13:00:59.212709+0000","last_became_active":"2024-03-21T12:58:36.724067+0000","last_became_peered":"2024-03-21T12:58:36.724067+0000","last_unstale":"2024-03-21T13:00:59.212709+0000","last_undegraded":"2024-03-21T13:00:59.212709+0000","last_fullsized":"2024-03-21T13:00:59.212709+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.734379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.734379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.789374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000743678,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'13","reported_seq":110,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.218507+0000","last_change":"2024-03-21T13:00:13.796181+0000","last_active":"2024-03-21T13:00:59.218507+0000","last_peered":"2024-03-21T13:00:59.218507+0000","last_clean":"2024-03-21T13:00:59.218507+0000","last_became_active":"2024-03-21T12:58:39.750068+0000","last_became_peered":"2024-03-21T12:58:39.750068+0000","last_unstale":"2024-03-21T13:00:59.218507+0000","last_undegraded":"2024-03-21T13:00:59.218507+0000","last_fullsized":"2024-03-21T13:00:59.218507+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'13","last_scrub_stamp":"2024-03-21T13:00:13.796084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:13.796084+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.166815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":80,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.214334+0000","last_change":"2024-03-21T13:00:03.688398+0000","last_active":"2024-03-21T13:00:59.214334+0000","last_peered":"2024-03-21T13:00:59.214334+0000","last_clean":"2024-03-21T13:00:59.214334+0000","last_became_active":"2024-03-21T12:58:41.164725+0000","last_became_peered":"2024-03-21T12:58:41.164725+0000","last_unstale":"2024-03-21T13:00:59.214334+0000","last_undegraded":"2024-03-21T13:00:59.214334+0000","last_fullsized":"2024-03-21T13:00:59.214334+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:03.688311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:03.688311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:03.827045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"98'1599","reported_seq":1621,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.647821+0000","last_change":"2024-03-21T13:00:42.709233+0000","last_active":"2024-03-21T13:01:04.647821+0000","last_peered":"2024-03-21T13:01:04.647821+0000","last_clean":"2024-03-21T13:01:04.647821+0000","last_became_active":"2024-03-21T13:00:42.708929+0000","last_became_peered":"2024-03-21T13:00:42.708929+0000","last_unstale":"2024-03-21T13:01:04.647821+0000","last_undegraded":"2024-03-21T13:01:04.647821+0000","last_fullsized":"2024-03-21T13:01:04.647821+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1599,"log_dups_size":0,"ondisk_log_size":1599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:03.242722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104792064,"num_objects":1599,"num_object_clones":0,"num_object_copies":4797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1599,"num_write_kb":102336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221775+0000","last_change":"2024-03-21T12:59:52.658473+0000","last_active":"2024-03-21T13:00:59.221775+0000","last_peered":"2024-03-21T13:00:59.221775+0000","last_clean":"2024-03-21T13:00:59.221775+0000","last_became_active":"2024-03-21T12:58:43.183113+0000","last_became_peered":"2024-03-21T12:58:43.183113+0000","last_unstale":"2024-03-21T13:00:59.221775+0000","last_undegraded":"2024-03-21T13:00:59.221775+0000","last_fullsized":"2024-03-21T13:00:59.221775+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:52.658377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.658377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.565790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":342,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.208202+0000","last_change":"2024-03-21T13:00:16.180260+0000","last_active":"2024-03-21T13:00:59.208202+0000","last_peered":"2024-03-21T13:00:59.208202+0000","last_clean":"2024-03-21T13:00:59.208202+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:00:59.208202+0000","last_undegraded":"2024-03-21T13:00:59.208202+0000","last_fullsized":"2024-03-21T13:00:59.208202+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:00:16.180126+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:00:16.180126+0000","last_clean_scrub_stamp":"2024-03-21T13:00:16.180126+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:37.638456+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.223243+0000","last_change":"2024-03-21T13:00:38.007563+0000","last_active":"2024-03-21T13:00:59.223243+0000","last_peered":"2024-03-21T13:00:59.223243+0000","last_clean":"2024-03-21T13:00:59.223243+0000","last_became_active":"2024-03-21T12:58:37.728911+0000","last_became_peered":"2024-03-21T12:58:37.728911+0000","last_unstale":"2024-03-21T13:00:59.223243+0000","last_undegraded":"2024-03-21T13:00:59.223243+0000","last_fullsized":"2024-03-21T13:00:59.223243+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:38.007486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.007486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:54.428020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":135,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.211828+0000","last_change":"2024-03-21T12:59:56.298894+0000","last_active":"2024-03-21T13:00:59.211828+0000","last_peered":"2024-03-21T13:00:59.211828+0000","last_clean":"2024-03-21T13:00:59.211828+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:00:59.211828+0000","last_undegraded":"2024-03-21T13:00:59.211828+0000","last_fullsized":"2024-03-21T13:00:59.211828+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-21T12:59:56.298768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:56.298768+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.767366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":77,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.208840+0000","last_change":"2024-03-21T13:00:11.121619+0000","last_active":"2024-03-21T13:00:59.208840+0000","last_peered":"2024-03-21T13:00:59.208840+0000","last_clean":"2024-03-21T13:00:59.208840+0000","last_became_active":"2024-03-21T12:58:41.168003+0000","last_became_peered":"2024-03-21T12:58:41.168003+0000","last_unstale":"2024-03-21T13:00:59.208840+0000","last_undegraded":"2024-03-21T13:00:59.208840+0000","last_fullsized":"2024-03-21T13:00:59.208840+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:11.121495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:11.121495+0000","last_clean_scrub_stamp":"2024-03-21T13:00:11.121495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:23.671209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"98'1610","reported_seq":1630,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.584741+0000","last_change":"2024-03-21T13:00:42.710083+0000","last_active":"2024-03-21T13:01:04.584741+0000","last_peered":"2024-03-21T13:01:04.584741+0000","last_clean":"2024-03-21T13:01:04.584741+0000","last_became_active":"2024-03-21T13:00:42.709753+0000","last_became_peered":"2024-03-21T13:00:42.709753+0000","last_unstale":"2024-03-21T13:01:04.584741+0000","last_undegraded":"2024-03-21T13:01:04.584741+0000","last_fullsized":"2024-03-21T13:01:04.584741+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1610,"log_dups_size":0,"ondisk_log_size":1610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:53.083369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105512960,"num_objects":1610,"num_object_clones":0,"num_object_copies":4830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1610,"num_write_kb":103040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.219018+0000","last_change":"2024-03-21T13:00:20.855008+0000","last_active":"2024-03-21T13:00:59.219018+0000","last_peered":"2024-03-21T13:00:59.219018+0000","last_clean":"2024-03-21T13:00:59.219018+0000","last_became_active":"2024-03-21T12:58:43.183860+0000","last_became_peered":"2024-03-21T12:58:43.183860+0000","last_unstale":"2024-03-21T13:00:59.219018+0000","last_undegraded":"2024-03-21T13:00:59.219018+0000","last_fullsized":"2024-03-21T13:00:59.219018+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:20.854917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.854917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:45.440847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":86,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.214186+0000","last_change":"2024-03-21T13:00:17.770235+0000","last_active":"2024-03-21T13:00:59.214186+0000","last_peered":"2024-03-21T13:00:59.214186+0000","last_clean":"2024-03-21T13:00:59.214186+0000","last_became_active":"2024-03-21T12:58:37.734176+0000","last_became_peered":"2024-03-21T12:58:37.734176+0000","last_unstale":"2024-03-21T13:00:59.214186+0000","last_undegraded":"2024-03-21T13:00:59.214186+0000","last_fullsized":"2024-03-21T13:00:59.214186+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:17.770139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:17.770139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:32.379247+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":236,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:03.309244+0000","last_change":"2024-03-21T13:00:15.164306+0000","last_active":"2024-03-21T13:01:03.309244+0000","last_peered":"2024-03-21T13:01:03.309244+0000","last_clean":"2024-03-21T13:01:03.309244+0000","last_became_active":"2024-03-21T12:58:36.725735+0000","last_became_peered":"2024-03-21T12:58:36.725735+0000","last_unstale":"2024-03-21T13:01:03.309244+0000","last_undegraded":"2024-03-21T13:01:03.309244+0000","last_fullsized":"2024-03-21T13:01:03.309244+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:00:15.164219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:00:15.164219+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:31.810769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":134,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":100,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.214733+0000","last_change":"2024-03-21T13:00:01.563940+0000","last_active":"2024-03-21T13:00:59.214733+0000","last_peered":"2024-03-21T13:00:59.214733+0000","last_clean":"2024-03-21T13:00:59.214733+0000","last_became_active":"2024-03-21T12:58:39.747315+0000","last_became_peered":"2024-03-21T12:58:39.747315+0000","last_unstale":"2024-03-21T13:00:59.214733+0000","last_undegraded":"2024-03-21T13:00:59.214733+0000","last_fullsized":"2024-03-21T13:00:59.214733+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:00:01.563844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:01.563844+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:12.072972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.222003+0000","last_change":"2024-03-21T13:00:10.646160+0000","last_active":"2024-03-21T13:00:59.222003+0000","last_peered":"2024-03-21T13:00:59.222003+0000","last_clean":"2024-03-21T13:00:59.222003+0000","last_became_active":"2024-03-21T12:58:41.167131+0000","last_became_peered":"2024-03-21T12:58:41.167131+0000","last_unstale":"2024-03-21T13:00:59.222003+0000","last_undegraded":"2024-03-21T13:00:59.222003+0000","last_fullsized":"2024-03-21T13:00:59.222003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:10.646083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.646083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.616835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"98'1562","reported_seq":1584,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.621764+0000","last_change":"2024-03-21T13:00:42.711444+0000","last_active":"2024-03-21T13:01:04.621764+0000","last_peered":"2024-03-21T13:01:04.621764+0000","last_clean":"2024-03-21T13:01:04.621764+0000","last_became_active":"2024-03-21T13:00:42.711150+0000","last_became_peered":"2024-03-21T13:00:42.711150+0000","last_unstale":"2024-03-21T13:01:04.621764+0000","last_undegraded":"2024-03-21T13:01:04.621764+0000","last_fullsized":"2024-03-21T13:01:04.621764+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1562,"log_dups_size":0,"ondisk_log_size":1562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:01.776871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102367232,"num_objects":1562,"num_object_clones":0,"num_object_copies":4686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1562,"num_write_kb":99968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.218592+0000","last_change":"2024-03-21T13:00:11.888272+0000","last_active":"2024-03-21T13:00:59.218592+0000","last_peered":"2024-03-21T13:00:59.218592+0000","last_clean":"2024-03-21T13:00:59.218592+0000","last_became_active":"2024-03-21T12:58:43.351138+0000","last_became_peered":"2024-03-21T12:58:43.351138+0000","last_unstale":"2024-03-21T13:00:59.218592+0000","last_undegraded":"2024-03-21T13:00:59.218592+0000","last_fullsized":"2024-03-21T13:00:59.218592+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:11.888191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:11.888191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.059106+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.219232+0000","last_change":"2024-03-21T12:59:57.925311+0000","last_active":"2024-03-21T13:00:59.219232+0000","last_peered":"2024-03-21T13:00:59.219232+0000","last_clean":"2024-03-21T13:00:59.219232+0000","last_became_active":"2024-03-21T12:58:37.729818+0000","last_became_peered":"2024-03-21T12:58:37.729818+0000","last_unstale":"2024-03-21T13:00:59.219232+0000","last_undegraded":"2024-03-21T13:00:59.219232+0000","last_fullsized":"2024-03-21T13:00:59.219232+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.925179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.925179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:22.845352+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"71'19","reported_seq":118,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.216885+0000","last_change":"2024-03-21T13:00:20.793827+0000","last_active":"2024-03-21T13:00:59.216885+0000","last_peered":"2024-03-21T13:00:59.216885+0000","last_clean":"2024-03-21T13:00:59.216885+0000","last_became_active":"2024-03-21T12:58:39.750216+0000","last_became_peered":"2024-03-21T12:58:39.750216+0000","last_unstale":"2024-03-21T13:00:59.216885+0000","last_undegraded":"2024-03-21T13:00:59.216885+0000","last_fullsized":"2024-03-21T13:00:59.216885+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:00:20.793740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.793740+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:24.045891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.211229+0000","last_change":"2024-03-21T13:00:06.485807+0000","last_active":"2024-03-21T13:00:59.211229+0000","last_peered":"2024-03-21T13:00:59.211229+0000","last_clean":"2024-03-21T13:00:59.211229+0000","last_became_active":"2024-03-21T12:58:41.166495+0000","last_became_peered":"2024-03-21T12:58:41.166495+0000","last_unstale":"2024-03-21T13:00:59.211229+0000","last_undegraded":"2024-03-21T13:00:59.211229+0000","last_fullsized":"2024-03-21T13:00:59.211229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:06.485726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.485726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:16.711578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"98'1630","reported_seq":1652,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.603229+0000","last_change":"2024-03-21T13:00:42.707801+0000","last_active":"2024-03-21T13:01:04.603229+0000","last_peered":"2024-03-21T13:01:04.603229+0000","last_clean":"2024-03-21T13:01:04.603229+0000","last_became_active":"2024-03-21T13:00:42.707521+0000","last_became_peered":"2024-03-21T13:00:42.707521+0000","last_unstale":"2024-03-21T13:01:04.603229+0000","last_undegraded":"2024-03-21T13:01:04.603229+0000","last_fullsized":"2024-03-21T13:01:04.603229+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1630,"log_dups_size":0,"ondisk_log_size":1630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:46.130912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106823680,"num_objects":1630,"num_object_clones":0,"num_object_copies":4890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1630,"num_write_kb":104320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":74,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.217484+0000","last_change":"2024-03-21T13:00:05.703953+0000","last_active":"2024-03-21T13:00:59.217484+0000","last_peered":"2024-03-21T13:00:59.217484+0000","last_clean":"2024-03-21T13:00:59.217484+0000","last_became_active":"2024-03-21T12:58:43.177409+0000","last_became_peered":"2024-03-21T12:58:43.177409+0000","last_unstale":"2024-03-21T13:00:59.217484+0000","last_undegraded":"2024-03-21T13:00:59.217484+0000","last_fullsized":"2024-03-21T13:00:59.217484+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:05.703839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:05.703839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:29.696086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221691+0000","last_change":"2024-03-21T13:00:20.667015+0000","last_active":"2024-03-21T13:00:59.221691+0000","last_peered":"2024-03-21T13:00:59.221691+0000","last_clean":"2024-03-21T13:00:59.221691+0000","last_became_active":"2024-03-21T12:58:37.730547+0000","last_became_peered":"2024-03-21T12:58:37.730547+0000","last_unstale":"2024-03-21T13:00:59.221691+0000","last_undegraded":"2024-03-21T13:00:59.221691+0000","last_fullsized":"2024-03-21T13:00:59.221691+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:20.666893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.666893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:25.453057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"86'15","reported_seq":80,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213297+0000","last_change":"2024-03-21T12:59:45.653329+0000","last_active":"2024-03-21T13:00:59.213297+0000","last_peered":"2024-03-21T13:00:59.213297+0000","last_clean":"2024-03-21T13:00:59.213297+0000","last_became_active":"2024-03-21T12:59:01.160456+0000","last_became_peered":"2024-03-21T12:59:01.160456+0000","last_unstale":"2024-03-21T13:00:59.213297+0000","last_undegraded":"2024-03-21T13:00:59.213297+0000","last_fullsized":"2024-03-21T13:00:59.213297+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T12:59:45.653197+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.205091+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":116,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.215881+0000","last_change":"2024-03-21T12:59:48.409996+0000","last_active":"2024-03-21T13:00:59.215881+0000","last_peered":"2024-03-21T13:00:59.215881+0000","last_clean":"2024-03-21T13:00:59.215881+0000","last_became_active":"2024-03-21T12:58:41.171770+0000","last_became_peered":"2024-03-21T12:58:41.171770+0000","last_unstale":"2024-03-21T13:00:59.215881+0000","last_undegraded":"2024-03-21T13:00:59.215881+0000","last_fullsized":"2024-03-21T13:00:59.215881+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T12:59:48.409905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.409905+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"98'1651","reported_seq":1672,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.612848+0000","last_change":"2024-03-21T13:00:42.713181+0000","last_active":"2024-03-21T13:01:04.612848+0000","last_peered":"2024-03-21T13:01:04.612848+0000","last_clean":"2024-03-21T13:01:04.612848+0000","last_became_active":"2024-03-21T13:00:42.712516+0000","last_became_peered":"2024-03-21T13:00:42.712516+0000","last_unstale":"2024-03-21T13:01:04.612848+0000","last_undegraded":"2024-03-21T13:01:04.612848+0000","last_fullsized":"2024-03-21T13:01:04.612848+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1651,"log_dups_size":0,"ondisk_log_size":1651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:54.438583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108199936,"num_objects":1651,"num_object_clones":0,"num_object_copies":4953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1651,"num_write_kb":105664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":73,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.208612+0000","last_change":"2024-03-21T13:00:17.198453+0000","last_active":"2024-03-21T13:00:59.208612+0000","last_peered":"2024-03-21T13:00:59.208612+0000","last_clean":"2024-03-21T13:00:59.208612+0000","last_became_active":"2024-03-21T12:58:43.353049+0000","last_became_peered":"2024-03-21T12:58:43.353049+0000","last_unstale":"2024-03-21T13:00:59.208612+0000","last_undegraded":"2024-03-21T13:00:59.208612+0000","last_fullsized":"2024-03-21T13:00:59.208612+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:17.198342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:17.198342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:28.551406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.214824+0000","last_change":"2024-03-21T13:00:27.822504+0000","last_active":"2024-03-21T13:00:59.214824+0000","last_peered":"2024-03-21T13:00:59.214824+0000","last_clean":"2024-03-21T13:00:59.214824+0000","last_became_active":"2024-03-21T12:58:37.735421+0000","last_became_peered":"2024-03-21T12:58:37.735421+0000","last_unstale":"2024-03-21T13:00:59.214824+0000","last_undegraded":"2024-03-21T13:00:59.214824+0000","last_fullsized":"2024-03-21T13:00:59.214824+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:00:27.822416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:27.822416+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:32.892997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"87'21","reported_seq":111,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213228+0000","last_change":"2024-03-21T12:59:52.916371+0000","last_active":"2024-03-21T13:00:59.213228+0000","last_peered":"2024-03-21T13:00:59.213228+0000","last_clean":"2024-03-21T13:00:59.213228+0000","last_became_active":"2024-03-21T12:58:39.743592+0000","last_became_peered":"2024-03-21T12:58:39.743592+0000","last_unstale":"2024-03-21T13:00:59.213228+0000","last_undegraded":"2024-03-21T13:00:59.213228+0000","last_fullsized":"2024-03-21T13:00:59.213228+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'21","last_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.916260+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:09.636952+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"71'8","reported_seq":109,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.220856+0000","last_change":"2024-03-21T12:58:41.174471+0000","last_active":"2024-03-21T13:00:59.220856+0000","last_peered":"2024-03-21T13:00:59.220856+0000","last_clean":"2024-03-21T13:00:59.220856+0000","last_became_active":"2024-03-21T12:58:41.174137+0000","last_became_peered":"2024-03-21T12:58:41.174137+0000","last_unstale":"2024-03-21T13:00:59.220856+0000","last_undegraded":"2024-03-21T13:00:59.220856+0000","last_fullsized":"2024-03-21T13:00:59.220856+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.068506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"98'1626","reported_seq":1648,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.622881+0000","last_change":"2024-03-21T13:00:42.711111+0000","last_active":"2024-03-21T13:01:04.622881+0000","last_peered":"2024-03-21T13:01:04.622881+0000","last_clean":"2024-03-21T13:01:04.622881+0000","last_became_active":"2024-03-21T13:00:42.710829+0000","last_became_peered":"2024-03-21T13:00:42.710829+0000","last_unstale":"2024-03-21T13:01:04.622881+0000","last_undegraded":"2024-03-21T13:01:04.622881+0000","last_fullsized":"2024-03-21T13:01:04.622881+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1626,"log_dups_size":0,"ondisk_log_size":1626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:52.633367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106561536,"num_objects":1626,"num_object_clones":0,"num_object_copies":4878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1626,"num_write_kb":104064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213077+0000","last_change":"2024-03-21T13:00:08.823941+0000","last_active":"2024-03-21T13:00:59.213077+0000","last_peered":"2024-03-21T13:00:59.213077+0000","last_clean":"2024-03-21T13:00:59.213077+0000","last_became_active":"2024-03-21T12:58:43.180939+0000","last_became_peered":"2024-03-21T12:58:43.180939+0000","last_unstale":"2024-03-21T13:00:59.213077+0000","last_undegraded":"2024-03-21T13:00:59.213077+0000","last_fullsized":"2024-03-21T13:00:59.213077+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.823841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:28.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.224834+0000","last_change":"2024-03-21T12:59:43.240826+0000","last_active":"2024-03-21T13:00:59.224834+0000","last_peered":"2024-03-21T13:00:59.224834+0000","last_clean":"2024-03-21T13:00:59.224834+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:00:59.224834+0000","last_undegraded":"2024-03-21T13:00:59.224834+0000","last_fullsized":"2024-03-21T13:00:59.224834+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T12:59:43.240674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:05.075915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"71'18","reported_seq":115,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.220741+0000","last_change":"2024-03-21T12:59:48.179286+0000","last_active":"2024-03-21T13:00:59.220741+0000","last_peered":"2024-03-21T13:00:59.220741+0000","last_clean":"2024-03-21T13:00:59.220741+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:00:59.220741+0000","last_undegraded":"2024-03-21T13:00:59.220741+0000","last_fullsized":"2024-03-21T13:00:59.220741+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T12:59:48.179188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.179188+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:15.921161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":71,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213045+0000","last_change":"2024-03-21T12:58:41.165071+0000","last_active":"2024-03-21T13:00:59.213045+0000","last_peered":"2024-03-21T13:00:59.213045+0000","last_clean":"2024-03-21T13:00:59.213045+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:00:59.213045+0000","last_undegraded":"2024-03-21T13:00:59.213045+0000","last_fullsized":"2024-03-21T13:00:59.213045+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:58:40.734631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"98'1649","reported_seq":1671,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.623387+0000","last_change":"2024-03-21T13:00:42.705026+0000","last_active":"2024-03-21T13:01:04.623387+0000","last_peered":"2024-03-21T13:01:04.623387+0000","last_clean":"2024-03-21T13:01:04.623387+0000","last_became_active":"2024-03-21T13:00:42.704712+0000","last_became_peered":"2024-03-21T13:00:42.704712+0000","last_unstale":"2024-03-21T13:01:04.623387+0000","last_undegraded":"2024-03-21T13:01:04.623387+0000","last_fullsized":"2024-03-21T13:01:04.623387+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1649,"log_dups_size":0,"ondisk_log_size":1649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:58.203175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108068864,"num_objects":1649,"num_object_clones":0,"num_object_copies":4947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1649,"num_write_kb":105536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.221624+0000","last_change":"2024-03-21T12:59:57.599220+0000","last_active":"2024-03-21T13:00:59.221624+0000","last_peered":"2024-03-21T13:00:59.221624+0000","last_clean":"2024-03-21T13:00:59.221624+0000","last_became_active":"2024-03-21T12:58:43.183614+0000","last_became_peered":"2024-03-21T12:58:43.183614+0000","last_unstale":"2024-03-21T13:00:59.221624+0000","last_undegraded":"2024-03-21T13:00:59.221624+0000","last_fullsized":"2024-03-21T13:00:59.221624+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:57.599076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:57.599076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:00.058326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":117,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.627297+0000","last_change":"2024-03-21T13:00:23.782106+0000","last_active":"2024-03-21T13:00:59.627297+0000","last_peered":"2024-03-21T13:00:59.627297+0000","last_clean":"2024-03-21T13:00:59.627297+0000","last_became_active":"2024-03-21T12:58:41.172019+0000","last_became_peered":"2024-03-21T12:58:41.172019+0000","last_unstale":"2024-03-21T13:00:59.627297+0000","last_undegraded":"2024-03-21T13:00:59.627297+0000","last_fullsized":"2024-03-21T13:00:59.627297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:00:23.782010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:23.782010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:48.112219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.220721+0000","last_change":"2024-03-21T12:59:41.231430+0000","last_active":"2024-03-21T13:00:59.220721+0000","last_peered":"2024-03-21T13:00:59.220721+0000","last_clean":"2024-03-21T13:00:59.220721+0000","last_became_active":"2024-03-21T12:58:37.731327+0000","last_became_peered":"2024-03-21T12:58:37.731327+0000","last_unstale":"2024-03-21T13:00:59.220721+0000","last_undegraded":"2024-03-21T13:00:59.220721+0000","last_fullsized":"2024-03-21T13:00:59.220721+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:41.231340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:41.231340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:02.247538+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":108,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.218401+0000","last_change":"2024-03-21T13:00:23.922151+0000","last_active":"2024-03-21T13:00:59.218401+0000","last_peered":"2024-03-21T13:00:59.218401+0000","last_clean":"2024-03-21T13:00:59.218401+0000","last_became_active":"2024-03-21T12:58:39.755326+0000","last_became_peered":"2024-03-21T12:58:39.755326+0000","last_unstale":"2024-03-21T13:00:59.218401+0000","last_undegraded":"2024-03-21T13:00:59.218401+0000","last_fullsized":"2024-03-21T13:00:59.218401+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:00:23.922060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:23.922060+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:42.692158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"98'1611","reported_seq":1633,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.608146+0000","last_change":"2024-03-21T13:00:42.710634+0000","last_active":"2024-03-21T13:01:04.608146+0000","last_peered":"2024-03-21T13:01:04.608146+0000","last_clean":"2024-03-21T13:01:04.608146+0000","last_became_active":"2024-03-21T13:00:42.710313+0000","last_became_peered":"2024-03-21T13:00:42.710313+0000","last_unstale":"2024-03-21T13:01:04.608146+0000","last_undegraded":"2024-03-21T13:01:04.608146+0000","last_fullsized":"2024-03-21T13:01:04.608146+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1611,"log_dups_size":0,"ondisk_log_size":1611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:42.012185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105578496,"num_objects":1611,"num_object_clones":0,"num_object_copies":4833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1611,"num_write_kb":103104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":76,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.223149+0000","last_change":"2024-03-21T13:00:27.958557+0000","last_active":"2024-03-21T13:00:59.223149+0000","last_peered":"2024-03-21T13:00:59.223149+0000","last_clean":"2024-03-21T13:00:59.223149+0000","last_became_active":"2024-03-21T12:58:43.350921+0000","last_became_peered":"2024-03-21T12:58:43.350921+0000","last_unstale":"2024-03-21T13:00:59.223149+0000","last_undegraded":"2024-03-21T13:00:59.223149+0000","last_fullsized":"2024-03-21T13:00:59.223149+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:00:27.958396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:27.958396+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:39.354137+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.222003+0000","last_change":"2024-03-21T13:00:08.639131+0000","last_active":"2024-03-21T13:00:59.222003+0000","last_peered":"2024-03-21T13:00:59.222003+0000","last_clean":"2024-03-21T13:00:59.222003+0000","last_became_active":"2024-03-21T12:58:41.168784+0000","last_became_peered":"2024-03-21T12:58:41.168784+0000","last_unstale":"2024-03-21T13:00:59.222003+0000","last_undegraded":"2024-03-21T13:00:59.222003+0000","last_fullsized":"2024-03-21T13:00:59.222003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.639041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.639041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:17.524978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213112+0000","last_change":"2024-03-21T13:00:15.767101+0000","last_active":"2024-03-21T13:00:59.213112+0000","last_peered":"2024-03-21T13:00:59.213112+0000","last_clean":"2024-03-21T13:00:59.213112+0000","last_became_active":"2024-03-21T12:58:37.730858+0000","last_became_peered":"2024-03-21T12:58:37.730858+0000","last_unstale":"2024-03-21T13:00:59.213112+0000","last_undegraded":"2024-03-21T13:00:59.213112+0000","last_fullsized":"2024-03-21T13:00:59.213112+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:15.767011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:15.767011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:36.754493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'11","reported_seq":102,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213785+0000","last_change":"2024-03-21T13:00:38.769934+0000","last_active":"2024-03-21T13:00:59.213785+0000","last_peered":"2024-03-21T13:00:59.213785+0000","last_clean":"2024-03-21T13:00:59.213785+0000","last_became_active":"2024-03-21T12:58:39.754337+0000","last_became_peered":"2024-03-21T12:58:39.754337+0000","last_unstale":"2024-03-21T13:00:59.213785+0000","last_undegraded":"2024-03-21T13:00:59.213785+0000","last_fullsized":"2024-03-21T13:00:59.213785+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:00:38.769807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.769807+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:08.568203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"98'1639","reported_seq":1660,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.629262+0000","last_change":"2024-03-21T13:00:42.710200+0000","last_active":"2024-03-21T13:01:04.629262+0000","last_peered":"2024-03-21T13:01:04.629262+0000","last_clean":"2024-03-21T13:01:04.629262+0000","last_became_active":"2024-03-21T13:00:42.709910+0000","last_became_peered":"2024-03-21T13:00:42.709910+0000","last_unstale":"2024-03-21T13:01:04.629262+0000","last_undegraded":"2024-03-21T13:01:04.629262+0000","last_fullsized":"2024-03-21T13:01:04.629262+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1639,"log_dups_size":0,"ondisk_log_size":1639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:45.263526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107413504,"num_objects":1639,"num_object_clones":0,"num_object_copies":4917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1639,"num_write_kb":104896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":76,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213613+0000","last_change":"2024-03-21T13:00:19.681494+0000","last_active":"2024-03-21T13:00:59.213613+0000","last_peered":"2024-03-21T13:00:59.213613+0000","last_clean":"2024-03-21T13:00:59.213613+0000","last_became_active":"2024-03-21T12:58:43.183380+0000","last_became_peered":"2024-03-21T12:58:43.183380+0000","last_unstale":"2024-03-21T13:00:59.213613+0000","last_undegraded":"2024-03-21T13:00:59.213613+0000","last_fullsized":"2024-03-21T13:00:59.213613+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:00:19.681413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:38.190626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.219513+0000","last_change":"2024-03-21T12:59:50.039168+0000","last_active":"2024-03-21T13:00:59.219513+0000","last_peered":"2024-03-21T13:00:59.219513+0000","last_clean":"2024-03-21T13:00:59.219513+0000","last_became_active":"2024-03-21T12:58:41.163145+0000","last_became_peered":"2024-03-21T12:58:41.163145+0000","last_unstale":"2024-03-21T13:00:59.219513+0000","last_undegraded":"2024-03-21T13:00:59.219513+0000","last_fullsized":"2024-03-21T13:00:59.219513+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.039091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.039091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:08.361130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.211264+0000","last_change":"2024-03-21T12:59:49.204557+0000","last_active":"2024-03-21T13:00:59.211264+0000","last_peered":"2024-03-21T13:00:59.211264+0000","last_clean":"2024-03-21T13:00:59.211264+0000","last_became_active":"2024-03-21T12:58:37.729792+0000","last_became_peered":"2024-03-21T12:58:37.729792+0000","last_unstale":"2024-03-21T13:00:59.211264+0000","last_undegraded":"2024-03-21T13:00:59.211264+0000","last_fullsized":"2024-03-21T13:00:59.211264+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:49.204419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:49.204419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:01.549632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":103,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213629+0000","last_change":"2024-03-21T12:58:39.746134+0000","last_active":"2024-03-21T13:00:59.213629+0000","last_peered":"2024-03-21T13:00:59.213629+0000","last_clean":"2024-03-21T13:00:59.213629+0000","last_became_active":"2024-03-21T12:58:39.745807+0000","last_became_peered":"2024-03-21T12:58:39.745807+0000","last_unstale":"2024-03-21T13:00:59.213629+0000","last_undegraded":"2024-03-21T13:00:59.213629+0000","last_fullsized":"2024-03-21T13:00:59.213629+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:58:38.718880+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:04.590004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"98'1646","reported_seq":1667,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:01:04.644794+0000","last_change":"2024-03-21T13:00:42.708112+0000","last_active":"2024-03-21T13:01:04.644794+0000","last_peered":"2024-03-21T13:01:04.644794+0000","last_clean":"2024-03-21T13:01:04.644794+0000","last_became_active":"2024-03-21T13:00:42.707835+0000","last_became_peered":"2024-03-21T13:00:42.707835+0000","last_unstale":"2024-03-21T13:01:04.644794+0000","last_undegraded":"2024-03-21T13:01:04.644794+0000","last_fullsized":"2024-03-21T13:01:04.644794+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":1646,"log_dups_size":0,"ondisk_log_size":1646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:11.285700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":4938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1646,"num_write_kb":105344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213755+0000","last_change":"2024-03-21T13:00:20.787009+0000","last_active":"2024-03-21T13:00:59.213755+0000","last_peered":"2024-03-21T13:00:59.213755+0000","last_clean":"2024-03-21T13:00:59.213755+0000","last_became_active":"2024-03-21T12:58:43.184576+0000","last_became_peered":"2024-03-21T12:58:43.184576+0000","last_unstale":"2024-03-21T13:00:59.213755+0000","last_undegraded":"2024-03-21T13:00:59.213755+0000","last_fullsized":"2024-03-21T13:00:59.213755+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:20.786860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.786860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:25.230881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"71'8","reported_seq":118,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.637668+0000","last_change":"2024-03-21T13:00:11.614182+0000","last_active":"2024-03-21T13:00:59.637668+0000","last_peered":"2024-03-21T13:00:59.637668+0000","last_clean":"2024-03-21T13:00:59.637668+0000","last_became_active":"2024-03-21T12:58:41.165560+0000","last_became_peered":"2024-03-21T12:58:41.165560+0000","last_unstale":"2024-03-21T13:00:59.637668+0000","last_undegraded":"2024-03-21T13:00:59.637668+0000","last_fullsized":"2024-03-21T13:00:59.637668+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:00:11.614084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:11.614084+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:28.325286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.223147+0000","last_change":"2024-03-21T13:00:25.960463+0000","last_active":"2024-03-21T13:00:59.223147+0000","last_peered":"2024-03-21T13:00:59.223147+0000","last_clean":"2024-03-21T13:00:59.223147+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:00:59.223147+0000","last_undegraded":"2024-03-21T13:00:59.223147+0000","last_fullsized":"2024-03-21T13:00:59.223147+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:25.960383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:25.960383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:50.928615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":98,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-21T13:00:59.213531+0000","last_change":"2024-03-21T13:00:10.785895+0000","last_active":"2024-03-21T13:00:59.213531+0000","last_peered":"2024-03-21T13:00:59.213531+0000","last_clean":"2024-03-21T13:00:59.213531+0000","last_became_active":"2024-03-21T12:58:39.749956+0000","last_became_peered":"2024-03-21T12:58:39.749956+0000","last_unstale":"2024-03-21T13:00:59.213531+0000","last_undegraded":"2024-03-21T13:00:59.213531+0000","last_fullsized":"2024-03-21T13:00:59.213531+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:00:10.785752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.785752+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:01:30.125302+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1706295296,"num_objects":26036,"num_object_clones":0,"num_object_copies":78108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26036,"num_write_kb":1666304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5118230528,"data_stored":5118230528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26036,"ondisk_log_size":26036,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":139,"num_read_kb":134,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":423,"ondisk_log_size":423,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168610,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":543952,"kb_used_data":500296,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93201200,"statfs":{"total":95995035648,"available":95438028800,"internally_reserved":0,"allocated":512303104,"data_stored":511918758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":44693962},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462952,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":544516,"kb_used_data":517500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93200636,"statfs":{"total":95995035648,"available":95437451264,"internally_reserved":0,"allocated":529920000,"data_stored":529547339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":609640,"kb_used_data":582624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93135512,"statfs":{"total":95995035648,"available":95370764288,"internally_reserved":0,"allocated":596606976,"data_stored":596197002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822864,"kb_used_data":787528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":92922288,"statfs":{"total":95995035648,"available":95152422912,"internally_reserved":0,"allocated":806428672,"data_stored":806043157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":605312,"kb_used_data":578296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93139840,"statfs":{"total":95995035648,"available":95375196160,"internally_reserved":0,"allocated":592175104,"data_stored":591805516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":926304,"kb_used_data":890968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":92818848,"statfs":{"total":95995035648,"available":95046500352,"internally_reserved":0,"allocated":912351232,"data_stored":911949335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018929,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":555412,"kb_used_data":524236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93189740,"statfs":{"total":95995035648,"available":95426293760,"internally_reserved":0,"allocated":536817664,"data_stored":536428231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724588,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":413564,"kb_used_data":378228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93331588,"statfs":{"total":95995035648,"available":95571546112,"internally_reserved":0,"allocated":387305472,"data_stored":386939438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":858587136,"data_stored":858587136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":637206528,"data_stored":637206528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":641662976,"data_stored":641662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":957415424,"data_stored":957415424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":540344320,"data_stored":540344320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":417529856,"data_stored":417529856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":531234816,"data_stored":531234816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":534249472,"data_stored":534249472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:01:11.326 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-21T13:01:11.327 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:11.388 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:11.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:11.489 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:11.537 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:11.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:11.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:11.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:11.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:11.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:11.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:11.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:01:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:01:11.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:11.835 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:11.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:11.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:11.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:01:11.910 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:11.937 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:11.969 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:11.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:11.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:12.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:12.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 18 16 38511 38495 133.651 98.25 0.0195581 0.00747491 2024-03-21T13:01:12.095 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:12.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:12.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:12.197 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:12.259 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:12.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:12.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:12.414 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:12.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:12.422 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:12 smithi163 ceph-mon[29621]: pgmap v80: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 154 MiB/s wr, 2.46k op/s 2024-03-21T13:01:12.422 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:12 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:12.422 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:12 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:01:12.422 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:12 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:01:12.422 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:12 smithi163 ceph-mon[40675]: pgmap v80: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 154 MiB/s wr, 2.46k op/s 2024-03-21T13:01:12.422 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:12 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:12.422 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:12 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:01:12.422 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:12 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:01:12.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:12 smithi204 ceph-mon[33153]: pgmap v80: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 154 MiB/s wr, 2.46k op/s 2024-03-21T13:01:12.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:12 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:12.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:12 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:01:12.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:12 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:01:12.517 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:12.567 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:12.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:12.670 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:12.703 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:12.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:12.804 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:12.843 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:12.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:12.945 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:12.978 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:12.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:13.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 19 16 40077 40061 131.767 97.875 0.00719543 0.00758428 2024-03-21T13:01:13.079 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:13.129 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:13.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:13.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:13.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:13.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:13.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:13.261 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:01:13.360 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:13.402 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:13.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:13.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:13 smithi204 ceph-mon[33153]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T13:01:13.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:13 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1976100000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:01:13.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:13 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:01:13.504 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:13.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:13 smithi163 ceph-mon[29621]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T13:01:13.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:13 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1976100000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:01:13.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:13 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:01:13.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:13 smithi163 ceph-mon[40675]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T13:01:13.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:13 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1976100000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:01:13.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:13 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:01:13.542 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:13.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:13.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:13.646 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:13.687 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:13.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:13.788 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:13.831 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:13.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:13.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:13.955 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:13.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:13.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:14.057 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:14.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:01:14.077912+0000 min lat: 0.00109429 max lat: 0.187673 avg lat: 0.00770396 2024-03-21T13:01:14.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:01:14.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 20 16 41525 41509 129.703 90.5 0.00604768 0.00770396 2024-03-21T13:01:14.088 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:14.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:14.190 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:14.229 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:14.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:14.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:14.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:14.356 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:14.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:14.367 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:14 smithi204 ceph-mon[33153]: pgmap v82: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 748 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-03-21T13:01:14.367 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:14 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:01:14.367 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:14 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:01:14.367 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:14 smithi204 ceph-mon[33153]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T13:01:14.368 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:14 smithi204 ceph-mon[33153]: 4.12 scrub starts 2024-03-21T13:01:14.368 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:14 smithi204 ceph-mon[33153]: 4.12 scrub ok 2024-03-21T13:01:14.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:14.494 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:14.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:14.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:14.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:14 smithi163 ceph-mon[29621]: pgmap v82: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 748 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-03-21T13:01:14.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:14 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:01:14.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:14 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:01:14.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:14 smithi163 ceph-mon[29621]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T13:01:14.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:14 smithi163 ceph-mon[29621]: 4.12 scrub starts 2024-03-21T13:01:14.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:14 smithi163 ceph-mon[29621]: 4.12 scrub ok 2024-03-21T13:01:14.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:14 smithi163 ceph-mon[40675]: pgmap v82: 148 pgs: 148 active+clean; 2.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 748 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-03-21T13:01:14.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:14 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:01:14.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:14 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:01:14.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:14 smithi163 ceph-mon[40675]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T13:01:14.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:14 smithi163 ceph-mon[40675]: 4.12 scrub starts 2024-03-21T13:01:14.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:14 smithi163 ceph-mon[40675]: 4.12 scrub ok 2024-03-21T13:01:14.596 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:14.637 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:14.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:14.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:14.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:14.763 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:14.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:14.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:14.865 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:14.900 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:14.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:15.002 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:15.036 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:15.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:15.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 21 16 42905 42889 127.634 86.25 0.0110581 0.00782887 2024-03-21T13:01:15.139 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:15.161 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:15.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:15.263 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:15.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:15.304 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:15.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:15.405 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:15.443 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:15.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:15.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:15 smithi163 ceph-mon[29621]: 6.8 scrub starts 2024-03-21T13:01:15.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:15 smithi163 ceph-mon[29621]: 6.8 scrub ok 2024-03-21T13:01:15.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:15 smithi163 ceph-mon[40675]: 6.8 scrub starts 2024-03-21T13:01:15.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:15 smithi163 ceph-mon[40675]: 6.8 scrub ok 2024-03-21T13:01:15.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:15.569 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:15.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:15.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:15.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:15.703 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:15.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:15.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:15.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:15 smithi204 ceph-mon[33153]: 6.8 scrub starts 2024-03-21T13:01:15.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:15 smithi204 ceph-mon[33153]: 6.8 scrub ok 2024-03-21T13:01:15.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:15.839 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:15.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:15.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:15.941 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:15.964 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:15.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:15.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:16.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:01:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:01:16.065 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:16.078 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 22 16 43913 43897 124.696 63 0.0187014 0.00801243 2024-03-21T13:01:16.100 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:16.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:16.201 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:16.232 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:16.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:16.327 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:01:16.327 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:01:16.327 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-21T13:01:16.327 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.679957 2024-03-21T13:01:16.328 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 4 0.6799568751846906 2024-03-21T13:01:16.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:16.356 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:16.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:16.458 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:16.492 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:16.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:16.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:16 smithi163 ceph-mon[29621]: pgmap v84: 148 pgs: 148 active+clean; 2.5 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 130 MiB/s wr, 2.09k op/s 2024-03-21T13:01:16.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:16 smithi163 ceph-mon[29621]: 3.1c scrub starts 2024-03-21T13:01:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:16 smithi163 ceph-mon[29621]: 3.1c scrub ok 2024-03-21T13:01:16.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:16 smithi163 ceph-mon[40675]: pgmap v84: 148 pgs: 148 active+clean; 2.5 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 130 MiB/s wr, 2.09k op/s 2024-03-21T13:01:16.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:16 smithi163 ceph-mon[40675]: 3.1c scrub starts 2024-03-21T13:01:16.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:16 smithi163 ceph-mon[40675]: 3.1c scrub ok 2024-03-21T13:01:16.594 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:16.651 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:16.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:16.752 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:16 smithi204 ceph-mon[33153]: pgmap v84: 148 pgs: 148 active+clean; 2.5 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 130 MiB/s wr, 2.09k op/s 2024-03-21T13:01:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:16 smithi204 ceph-mon[33153]: 3.1c scrub starts 2024-03-21T13:01:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:16 smithi204 ceph-mon[33153]: 3.1c scrub ok 2024-03-21T13:01:16.776 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:16.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:16.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:16.877 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:16.916 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:16.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:17.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:17.050 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:17.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:17.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 23 16 45180 45164 122.717 79.1875 0.00940557 0.00814344 2024-03-21T13:01:17.151 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:17.176 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:17.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:17.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:17.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:17.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:17.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:17.408 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:17.448 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:17.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:17.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:17.549 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:17.578 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:17.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:17.680 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:17.719 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:17.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:17 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:17 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:17 smithi204 ceph-mon[33153]: 4.d scrub starts 2024-03-21T13:01:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:17 smithi204 ceph-mon[33153]: 4.d scrub ok 2024-03-21T13:01:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:17 smithi204 ceph-mon[33153]: 3.a scrub starts 2024-03-21T13:01:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:17 smithi204 ceph-mon[33153]: 3.a scrub ok 2024-03-21T13:01:17.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:17 smithi204 ceph-mon[33153]: 5.c scrub starts 2024-03-21T13:01:17.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:17 smithi204 ceph-mon[33153]: 5.c scrub ok 2024-03-21T13:01:17.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:17 smithi204 ceph-mon[33153]: 4.1e scrub starts 2024-03-21T13:01:17.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:17 smithi204 ceph-mon[33153]: 4.1e scrub ok 2024-03-21T13:01:17.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[29621]: 4.d scrub starts 2024-03-21T13:01:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[29621]: 4.d scrub ok 2024-03-21T13:01:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[29621]: 3.a scrub starts 2024-03-21T13:01:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[29621]: 3.a scrub ok 2024-03-21T13:01:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[29621]: 5.c scrub starts 2024-03-21T13:01:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[29621]: 5.c scrub ok 2024-03-21T13:01:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[29621]: 4.1e scrub starts 2024-03-21T13:01:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[29621]: 4.1e scrub ok 2024-03-21T13:01:17.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:17.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:17.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[40675]: 4.d scrub starts 2024-03-21T13:01:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[40675]: 4.d scrub ok 2024-03-21T13:01:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[40675]: 3.a scrub starts 2024-03-21T13:01:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[40675]: 3.a scrub ok 2024-03-21T13:01:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[40675]: 5.c scrub starts 2024-03-21T13:01:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[40675]: 5.c scrub ok 2024-03-21T13:01:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[40675]: 4.1e scrub starts 2024-03-21T13:01:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:17 smithi163 ceph-mon[40675]: 4.1e scrub ok 2024-03-21T13:01:17.820 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:17.854 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:17.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:17.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:17.996 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:17.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:17.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:18.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 24 16 46572 46556 121.228 87 0.00374615 0.00824321 2024-03-21T13:01:18.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:18.129 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:18.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:18.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:18.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:18.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:18.255 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:18.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:18.356 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:18.398 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:18.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:18.500 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:18.626 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:18.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:18.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:18.729 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:18 smithi204 ceph-mon[33153]: pgmap v85: 148 pgs: 148 active+clean; 2.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-21T13:01:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:18 smithi204 ceph-mon[33153]: 2.1 scrub starts 2024-03-21T13:01:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:18 smithi204 ceph-mon[33153]: 2.1 scrub ok 2024-03-21T13:01:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:18 smithi204 ceph-mon[33153]: 5.1a scrub starts 2024-03-21T13:01:18.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:18 smithi204 ceph-mon[33153]: 5.1a scrub ok 2024-03-21T13:01:18.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:18 smithi204 ceph-mon[33153]: 5.1 scrub starts 2024-03-21T13:01:18.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:18 smithi204 ceph-mon[33153]: 5.1 scrub ok 2024-03-21T13:01:18.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[29621]: pgmap v85: 148 pgs: 148 active+clean; 2.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-21T13:01:18.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[29621]: 2.1 scrub starts 2024-03-21T13:01:18.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[29621]: 2.1 scrub ok 2024-03-21T13:01:18.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[29621]: 5.1a scrub starts 2024-03-21T13:01:18.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[29621]: 5.1a scrub ok 2024-03-21T13:01:18.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[29621]: 5.1 scrub starts 2024-03-21T13:01:18.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[29621]: 5.1 scrub ok 2024-03-21T13:01:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[40675]: pgmap v85: 148 pgs: 148 active+clean; 2.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-21T13:01:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[40675]: 2.1 scrub starts 2024-03-21T13:01:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[40675]: 2.1 scrub ok 2024-03-21T13:01:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[40675]: 5.1a scrub starts 2024-03-21T13:01:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[40675]: 5.1a scrub ok 2024-03-21T13:01:18.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[40675]: 5.1 scrub starts 2024-03-21T13:01:18.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:18 smithi163 ceph-mon[40675]: 5.1 scrub ok 2024-03-21T13:01:18.767 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:18.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:18.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:18.898 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:18.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:18.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:19.000 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:19.041 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:19.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:19.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:19.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 25 16 48026 48010 120.014 90.875 0.00546264 0.00832692 2024-03-21T13:01:19.143 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:19.170 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:19.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:19.272 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:19.317 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:19.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:19.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:19.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:19.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:19.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:19.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:19.551 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:19.577 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:19.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:19.679 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:19.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:19.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:19 smithi204 ceph-mon[33153]: 3.d scrub starts 2024-03-21T13:01:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:19 smithi204 ceph-mon[33153]: 3.d scrub ok 2024-03-21T13:01:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:19 smithi204 ceph-mon[33153]: 5.14 scrub starts 2024-03-21T13:01:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:19 smithi204 ceph-mon[33153]: 5.14 scrub ok 2024-03-21T13:01:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:19 smithi204 ceph-mon[33153]: 4.a scrub starts 2024-03-21T13:01:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:19 smithi204 ceph-mon[33153]: 4.a scrub ok 2024-03-21T13:01:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:19 smithi204 ceph-mon[33153]: 5.1d scrub starts 2024-03-21T13:01:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:19 smithi204 ceph-mon[33153]: 5.1d scrub ok 2024-03-21T13:01:19.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[29621]: 3.d scrub starts 2024-03-21T13:01:19.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[29621]: 3.d scrub ok 2024-03-21T13:01:19.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[29621]: 5.14 scrub starts 2024-03-21T13:01:19.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[29621]: 5.14 scrub ok 2024-03-21T13:01:19.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[29621]: 4.a scrub starts 2024-03-21T13:01:19.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[29621]: 4.a scrub ok 2024-03-21T13:01:19.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:19.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[29621]: 5.1d scrub starts 2024-03-21T13:01:19.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[29621]: 5.1d scrub ok 2024-03-21T13:01:19.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[40675]: 3.d scrub starts 2024-03-21T13:01:19.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[40675]: 3.d scrub ok 2024-03-21T13:01:19.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[40675]: 5.14 scrub starts 2024-03-21T13:01:19.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[40675]: 5.14 scrub ok 2024-03-21T13:01:19.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[40675]: 4.a scrub starts 2024-03-21T13:01:19.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[40675]: 4.a scrub ok 2024-03-21T13:01:19.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:19.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[40675]: 5.1d scrub starts 2024-03-21T13:01:19.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:19 smithi163 ceph-mon[40675]: 5.1d scrub ok 2024-03-21T13:01:19.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:19.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:19.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:19.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:19.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:19.978 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:19.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:19.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:20.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 26 16 49426 49410 118.763 87.5 0.00798224 0.00841507 2024-03-21T13:01:20.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:20.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:20.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:20.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:20.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:20.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:20.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:20.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:20.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:20.372 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:20.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:20.474 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:20.504 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:20.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:20.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:20.606 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:20.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:20.648 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:20.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:20.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:20 smithi204 ceph-mon[33153]: pgmap v86: 148 pgs: 148 active+clean; 2.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-21T13:01:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:20 smithi204 ceph-mon[33153]: 4.13 scrub starts 2024-03-21T13:01:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:20 smithi204 ceph-mon[33153]: 4.13 scrub ok 2024-03-21T13:01:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:20 smithi204 ceph-mon[33153]: 5.12 scrub starts 2024-03-21T13:01:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:20 smithi204 ceph-mon[33153]: 5.12 scrub ok 2024-03-21T13:01:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:20 smithi204 ceph-mon[33153]: 6.16 scrub starts 2024-03-21T13:01:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:20 smithi204 ceph-mon[33153]: 6.16 scrub ok 2024-03-21T13:01:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[29621]: pgmap v86: 148 pgs: 148 active+clean; 2.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-21T13:01:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[29621]: 4.13 scrub starts 2024-03-21T13:01:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[29621]: 4.13 scrub ok 2024-03-21T13:01:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[29621]: 5.12 scrub starts 2024-03-21T13:01:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[29621]: 5.12 scrub ok 2024-03-21T13:01:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[29621]: 6.16 scrub starts 2024-03-21T13:01:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[29621]: 6.16 scrub ok 2024-03-21T13:01:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[40675]: pgmap v86: 148 pgs: 148 active+clean; 2.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-21T13:01:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[40675]: 4.13 scrub starts 2024-03-21T13:01:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[40675]: 4.13 scrub ok 2024-03-21T13:01:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[40675]: 5.12 scrub starts 2024-03-21T13:01:20.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[40675]: 5.12 scrub ok 2024-03-21T13:01:20.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[40675]: 6.16 scrub starts 2024-03-21T13:01:20.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:20 smithi163 ceph-mon[40675]: 6.16 scrub ok 2024-03-21T13:01:20.778 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:20.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:20.880 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:20.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:20.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:21.067 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:21.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 27 16 50931 50915 117.848 94.0625 0.00676886 0.00848062 2024-03-21T13:01:21.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:21.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:21.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:21.200 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:21.240 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:21.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:21.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:21.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:21.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:21.474 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:21.505 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:21.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:21.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:21.536 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:21.584 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:21.608 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:21.634 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:21.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:21.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:21.682 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[29621]: 3.e scrub starts 2024-03-21T13:01:21.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[29621]: 3.e scrub ok 2024-03-21T13:01:21.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[29621]: 5.11 scrub starts 2024-03-21T13:01:21.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[29621]: 5.11 scrub ok 2024-03-21T13:01:21.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[29621]: 3.14 scrub starts 2024-03-21T13:01:21.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[29621]: pgmap v87: 148 pgs: 148 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-21T13:01:21.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[29621]: 3.14 scrub ok 2024-03-21T13:01:21.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[29621]: 6.1 scrub starts 2024-03-21T13:01:21.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[29621]: 6.1 scrub ok 2024-03-21T13:01:21.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:21.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[40675]: 3.e scrub starts 2024-03-21T13:01:21.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[40675]: 3.e scrub ok 2024-03-21T13:01:21.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[40675]: 5.11 scrub starts 2024-03-21T13:01:21.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[40675]: 5.11 scrub ok 2024-03-21T13:01:21.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[40675]: 3.14 scrub starts 2024-03-21T13:01:21.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[40675]: pgmap v87: 148 pgs: 148 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-21T13:01:21.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[40675]: 3.14 scrub ok 2024-03-21T13:01:21.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[40675]: 6.1 scrub starts 2024-03-21T13:01:21.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[40675]: 6.1 scrub ok 2024-03-21T13:01:21.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:21 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:21.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:21.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:01:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:01:21.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:21 smithi204 ceph-mon[33153]: 3.e scrub starts 2024-03-21T13:01:21.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:21 smithi204 ceph-mon[33153]: 3.e scrub ok 2024-03-21T13:01:21.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:21 smithi204 ceph-mon[33153]: 5.11 scrub starts 2024-03-21T13:01:21.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:21 smithi204 ceph-mon[33153]: 5.11 scrub ok 2024-03-21T13:01:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:21 smithi204 ceph-mon[33153]: 3.14 scrub starts 2024-03-21T13:01:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:21 smithi204 ceph-mon[33153]: pgmap v87: 148 pgs: 148 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-21T13:01:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:21 smithi204 ceph-mon[33153]: 3.14 scrub ok 2024-03-21T13:01:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:21 smithi204 ceph-mon[33153]: 6.1 scrub starts 2024-03-21T13:01:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:21 smithi204 ceph-mon[33153]: 6.1 scrub ok 2024-03-21T13:01:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:21 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:21.766 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:21.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:21.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:21.897 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:21.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:21.999 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:22.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:22.041 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:22.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:22.078 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 28 16 52362 52346 116.833 89.4375 0.0109001 0.00855371 2024-03-21T13:01:22.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:22.184 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:22.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:22.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:22.286 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:22.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:22.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:22.417 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:22.469 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:22.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:22.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:22.603 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:22.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:22.613 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:22 smithi163 ceph-mon[29621]: 4.5 scrub starts 2024-03-21T13:01:22.613 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:22 smithi163 ceph-mon[29621]: 4.5 scrub ok 2024-03-21T13:01:22.613 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:22 smithi163 ceph-mon[29621]: 6.10 scrub starts 2024-03-21T13:01:22.613 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:22 smithi163 ceph-mon[29621]: 6.10 scrub ok 2024-03-21T13:01:22.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:22 smithi163 ceph-mon[40675]: 4.5 scrub starts 2024-03-21T13:01:22.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:22 smithi163 ceph-mon[40675]: 4.5 scrub ok 2024-03-21T13:01:22.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:22 smithi163 ceph-mon[40675]: 6.10 scrub starts 2024-03-21T13:01:22.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:22 smithi163 ceph-mon[40675]: 6.10 scrub ok 2024-03-21T13:01:22.619 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:01:22.619 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:01:22.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:22.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:22.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:22.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:22.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:22 smithi204 ceph-mon[33153]: 4.5 scrub starts 2024-03-21T13:01:22.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:22 smithi204 ceph-mon[33153]: 4.5 scrub ok 2024-03-21T13:01:22.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:22 smithi204 ceph-mon[33153]: 6.10 scrub starts 2024-03-21T13:01:22.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:22 smithi204 ceph-mon[33153]: 6.10 scrub ok 2024-03-21T13:01:22.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:22.875 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:22.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:22.882 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:01:22.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:23.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:23.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:23.078 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 29 16 53817 53801 115.938 90.9375 0.0108192 0.00861931 2024-03-21T13:01:23.107 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:23.177 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:23.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:23.279 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:23.322 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:23.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:23.424 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:23.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:23.477 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:23.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:23.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:23.607 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:23.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:23.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:23.709 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:23.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:23 smithi204 ceph-mon[33153]: 6.1c scrub starts 2024-03-21T13:01:23.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:23 smithi204 ceph-mon[33153]: 6.1c scrub ok 2024-03-21T13:01:23.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:23 smithi204 ceph-mon[33153]: 5.2 scrub starts 2024-03-21T13:01:23.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:23 smithi204 ceph-mon[33153]: 5.2 scrub ok 2024-03-21T13:01:23.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:23 smithi204 ceph-mon[33153]: pgmap v88: 148 pgs: 148 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-21T13:01:23.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:23.758 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:23.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:23 smithi163 ceph-mon[29621]: 6.1c scrub starts 2024-03-21T13:01:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:23 smithi163 ceph-mon[29621]: 6.1c scrub ok 2024-03-21T13:01:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:23 smithi163 ceph-mon[29621]: 5.2 scrub starts 2024-03-21T13:01:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:23 smithi163 ceph-mon[29621]: 5.2 scrub ok 2024-03-21T13:01:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:23 smithi163 ceph-mon[29621]: pgmap v88: 148 pgs: 148 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-21T13:01:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:23 smithi163 ceph-mon[40675]: 6.1c scrub starts 2024-03-21T13:01:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:23 smithi163 ceph-mon[40675]: 6.1c scrub ok 2024-03-21T13:01:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:23 smithi163 ceph-mon[40675]: 5.2 scrub starts 2024-03-21T13:01:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:23 smithi163 ceph-mon[40675]: 5.2 scrub ok 2024-03-21T13:01:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:23 smithi163 ceph-mon[40675]: pgmap v88: 148 pgs: 148 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-21T13:01:23.859 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:23.897 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:23.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:23.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:23.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:24.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:24.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:24.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:24.078 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 30 16 55302 55286 115.167 92.8125 0.0147117 0.00867818 2024-03-21T13:01:24.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:24.165 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:24.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:24.255 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-21T13:01:24.268 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:24.270 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:24.342 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:24.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:24.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:24.479 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:24.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:24.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:24 smithi163 ceph-mon[29621]: 3.1d scrub starts 2024-03-21T13:01:24.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:24 smithi163 ceph-mon[29621]: 3.1d scrub ok 2024-03-21T13:01:24.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:24 smithi163 ceph-mon[29621]: 3.4 scrub starts 2024-03-21T13:01:24.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:24 smithi163 ceph-mon[29621]: 3.4 scrub ok 2024-03-21T13:01:24.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:24 smithi163 ceph-mon[29621]: 3.1f scrub starts 2024-03-21T13:01:24.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:24 smithi163 ceph-mon[29621]: 3.1f scrub ok 2024-03-21T13:01:24.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:24 smithi163 ceph-mon[40675]: 3.1d scrub starts 2024-03-21T13:01:24.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:24 smithi163 ceph-mon[40675]: 3.1d scrub ok 2024-03-21T13:01:24.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:24 smithi163 ceph-mon[40675]: 3.4 scrub starts 2024-03-21T13:01:24.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:24 smithi163 ceph-mon[40675]: 3.4 scrub ok 2024-03-21T13:01:24.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:24 smithi163 ceph-mon[40675]: 3.1f scrub starts 2024-03-21T13:01:24.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:24 smithi163 ceph-mon[40675]: 3.1f scrub ok 2024-03-21T13:01:24.579 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:24.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:24.648 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:24.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:24.749 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:24 smithi204 ceph-mon[33153]: 3.1d scrub starts 2024-03-21T13:01:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:24 smithi204 ceph-mon[33153]: 3.1d scrub ok 2024-03-21T13:01:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:24 smithi204 ceph-mon[33153]: 3.4 scrub starts 2024-03-21T13:01:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:24 smithi204 ceph-mon[33153]: 3.4 scrub ok 2024-03-21T13:01:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:24 smithi204 ceph-mon[33153]: 3.1f scrub starts 2024-03-21T13:01:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:24 smithi204 ceph-mon[33153]: 3.1f scrub ok 2024-03-21T13:01:24.798 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:24.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:24.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:24.900 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:24.947 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:24.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:25.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:25.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:25.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:25.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:25.079 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 31 16 56604 56588 114.077 81.375 0.0173691 0.008761 2024-03-21T13:01:25.179 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:25.224 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:25.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:25.326 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:25.369 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:25.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:25.470 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:25.507 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:25.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:25.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:25.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[29621]: 5.15 scrub starts 2024-03-21T13:01:25.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[29621]: 5.15 scrub ok 2024-03-21T13:01:25.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[29621]: pgmap v89: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-03-21T13:01:25.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[29621]: 6.1e scrub starts 2024-03-21T13:01:25.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[29621]: 6.1e scrub ok 2024-03-21T13:01:25.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[29621]: 5.3 scrub starts 2024-03-21T13:01:25.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[29621]: 5.3 scrub ok 2024-03-21T13:01:25.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[40675]: 5.15 scrub starts 2024-03-21T13:01:25.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[40675]: 5.15 scrub ok 2024-03-21T13:01:25.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[40675]: pgmap v89: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-03-21T13:01:25.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[40675]: 6.1e scrub starts 2024-03-21T13:01:25.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[40675]: 6.1e scrub ok 2024-03-21T13:01:25.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[40675]: 5.3 scrub starts 2024-03-21T13:01:25.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-mon[40675]: 5.3 scrub ok 2024-03-21T13:01:25.610 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:25.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:25.656 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:25.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:25.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:25 smithi204 ceph-mon[33153]: 5.15 scrub starts 2024-03-21T13:01:25.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:25 smithi204 ceph-mon[33153]: 5.15 scrub ok 2024-03-21T13:01:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:25 smithi204 ceph-mon[33153]: pgmap v89: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-03-21T13:01:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:25 smithi204 ceph-mon[33153]: 6.1e scrub starts 2024-03-21T13:01:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:25 smithi204 ceph-mon[33153]: 6.1e scrub ok 2024-03-21T13:01:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:25 smithi204 ceph-mon[33153]: 5.3 scrub starts 2024-03-21T13:01:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:25 smithi204 ceph-mon[33153]: 5.3 scrub ok 2024-03-21T13:01:25.756 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:25.795 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:25.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:25.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:25.899 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:25.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:25.940 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:25.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:26.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:01:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:01:26.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:26.073 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:26.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:26.078 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 32 16 57899 57883 113.041 80.9375 0.00269136 0.00883994 2024-03-21T13:01:26.176 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:26.219 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:26.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:26.320 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:26.361 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.4 primary-affinity to 0.679957 (8445612) 2024-03-21T13:01:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:26.363 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:26.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:26.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:26.501 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:26.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:26.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:26.633 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:26.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:26.735 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:26 smithi204 ceph-mon[33153]: 6.d scrub starts 2024-03-21T13:01:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:26 smithi204 ceph-mon[33153]: 6.d scrub ok 2024-03-21T13:01:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:26 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3045338501' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6799568751846906}]: dispatch 2024-03-21T13:01:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:26 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6799568751846906}]: dispatch 2024-03-21T13:01:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:26 smithi204 ceph-mon[33153]: 5.6 scrub starts 2024-03-21T13:01:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:26 smithi204 ceph-mon[33153]: 5.6 scrub ok 2024-03-21T13:01:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:26 smithi204 ceph-mon[33153]: 5.0 scrub starts 2024-03-21T13:01:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:26 smithi204 ceph-mon[33153]: 5.0 scrub ok 2024-03-21T13:01:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:26 smithi204 ceph-mon[33153]: 4.b scrub starts 2024-03-21T13:01:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:26 smithi204 ceph-mon[33153]: 4.b scrub ok 2024-03-21T13:01:26.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[40675]: 6.d scrub starts 2024-03-21T13:01:26.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[40675]: 6.d scrub ok 2024-03-21T13:01:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3045338501' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6799568751846906}]: dispatch 2024-03-21T13:01:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6799568751846906}]: dispatch 2024-03-21T13:01:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[40675]: 5.6 scrub starts 2024-03-21T13:01:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[40675]: 5.6 scrub ok 2024-03-21T13:01:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[40675]: 5.0 scrub starts 2024-03-21T13:01:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[40675]: 5.0 scrub ok 2024-03-21T13:01:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[40675]: 4.b scrub starts 2024-03-21T13:01:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[40675]: 4.b scrub ok 2024-03-21T13:01:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[29621]: 6.d scrub starts 2024-03-21T13:01:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[29621]: 6.d scrub ok 2024-03-21T13:01:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3045338501' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6799568751846906}]: dispatch 2024-03-21T13:01:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6799568751846906}]: dispatch 2024-03-21T13:01:26.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[29621]: 5.6 scrub starts 2024-03-21T13:01:26.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[29621]: 5.6 scrub ok 2024-03-21T13:01:26.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[29621]: 5.0 scrub starts 2024-03-21T13:01:26.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[29621]: 5.0 scrub ok 2024-03-21T13:01:26.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[29621]: 4.b scrub starts 2024-03-21T13:01:26.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:26 smithi163 ceph-mon[29621]: 4.b scrub ok 2024-03-21T13:01:26.766 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:26.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:26.868 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:26.906 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:26.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:26.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:27.008 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:27.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:27.049 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:27.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:27.078 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 33 16 59134 59118 111.955 77.1875 0.00400551 0.00892716 2024-03-21T13:01:27.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:27.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:27.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:27.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:27.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:27.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:27.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:27.416 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:27.460 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:27.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:27.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:27.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:27.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:27.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:27.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:27 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6799568751846906}]': finished 2024-03-21T13:01:27.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:27 smithi163 ceph-mon[29621]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T13:01:27.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:27 smithi163 ceph-mon[29621]: pgmap v91: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-21T13:01:27.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:27 smithi163 ceph-mon[29621]: 6.18 scrub starts 2024-03-21T13:01:27.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:27 smithi163 ceph-mon[29621]: 6.18 scrub ok 2024-03-21T13:01:27.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:27 smithi163 ceph-mon[29621]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T13:01:27.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:27 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6799568751846906}]': finished 2024-03-21T13:01:27.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:27 smithi163 ceph-mon[40675]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T13:01:27.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:27 smithi163 ceph-mon[40675]: pgmap v91: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-21T13:01:27.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:27 smithi163 ceph-mon[40675]: 6.18 scrub starts 2024-03-21T13:01:27.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:27 smithi163 ceph-mon[40675]: 6.18 scrub ok 2024-03-21T13:01:27.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:27 smithi163 ceph-mon[40675]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T13:01:27.696 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:27.727 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:27.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:27 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6799568751846906}]': finished 2024-03-21T13:01:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:27 smithi204 ceph-mon[33153]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T13:01:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:27 smithi204 ceph-mon[33153]: pgmap v91: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-21T13:01:27.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:27 smithi204 ceph-mon[33153]: 6.18 scrub starts 2024-03-21T13:01:27.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:27 smithi204 ceph-mon[33153]: 6.18 scrub ok 2024-03-21T13:01:27.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:27 smithi204 ceph-mon[33153]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T13:01:27.829 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:27.860 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:27.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:27.962 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:27.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:27.997 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:27.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:28.079 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 34 16 60514 60498 111.198 86.25 0.0149815 0.00898776 2024-03-21T13:01:28.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:28.126 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:28.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:28.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:28.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:28.259 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:28.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:28.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:28.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:28.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:28.490 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:28.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:28.577 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[40675]: 3.11 scrub starts 2024-03-21T13:01:28.577 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[40675]: 3.11 scrub ok 2024-03-21T13:01:28.577 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[40675]: 4.2 scrub starts 2024-03-21T13:01:28.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[40675]: 4.2 scrub ok 2024-03-21T13:01:28.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[40675]: 3.1a scrub starts 2024-03-21T13:01:28.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[40675]: 3.1a scrub ok 2024-03-21T13:01:28.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[40675]: 4.6 scrub starts 2024-03-21T13:01:28.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[40675]: 4.6 scrub ok 2024-03-21T13:01:28.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[29621]: 3.11 scrub starts 2024-03-21T13:01:28.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[29621]: 3.11 scrub ok 2024-03-21T13:01:28.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[29621]: 4.2 scrub starts 2024-03-21T13:01:28.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[29621]: 4.2 scrub ok 2024-03-21T13:01:28.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[29621]: 3.1a scrub starts 2024-03-21T13:01:28.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[29621]: 3.1a scrub ok 2024-03-21T13:01:28.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[29621]: 4.6 scrub starts 2024-03-21T13:01:28.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:28 smithi163 ceph-mon[29621]: 4.6 scrub ok 2024-03-21T13:01:28.635 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:28.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:28.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:28.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:28 smithi204 ceph-mon[33153]: 3.11 scrub starts 2024-03-21T13:01:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:28 smithi204 ceph-mon[33153]: 3.11 scrub ok 2024-03-21T13:01:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:28 smithi204 ceph-mon[33153]: 4.2 scrub starts 2024-03-21T13:01:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:28 smithi204 ceph-mon[33153]: 4.2 scrub ok 2024-03-21T13:01:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:28 smithi204 ceph-mon[33153]: 3.1a scrub starts 2024-03-21T13:01:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:28 smithi204 ceph-mon[33153]: 3.1a scrub ok 2024-03-21T13:01:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:28 smithi204 ceph-mon[33153]: 4.6 scrub starts 2024-03-21T13:01:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:28 smithi204 ceph-mon[33153]: 4.6 scrub ok 2024-03-21T13:01:28.766 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:28.839 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:28.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:28.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:28.925 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:01:28.925 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:01:28.925 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:01:28.925 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:01:28.925 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:01:28.925 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:01:28.925 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:01:28.925 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:01:28.925 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:01:28.925 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:01:28.926 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:01:28.926 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:01:28.926 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:01:28.926 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:01:28.926 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:01:28.926 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:01:28.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:28.984 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:28.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:28.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:29.079 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 35 16 61844 61828 110.396 83.125 0.0181683 0.00905324 2024-03-21T13:01:29.085 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:29.116 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:29.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:29.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:29.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:29.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:29.260 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:29.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:29.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:29.399 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:29.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:29.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:29.533 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:29.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:29.634 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:29.681 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:29.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:29.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:29.748 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:29 smithi204 ceph-mon[33153]: 6.5 scrub starts 2024-03-21T13:01:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:29 smithi204 ceph-mon[33153]: 6.5 scrub ok 2024-03-21T13:01:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:29 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3543053140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:01:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:29 smithi204 ceph-mon[33153]: pgmap v93: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 904 op/s 2024-03-21T13:01:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:29 smithi204 ceph-mon[33153]: 4.3 scrub starts 2024-03-21T13:01:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:29 smithi204 ceph-mon[33153]: 4.3 scrub ok 2024-03-21T13:01:29.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:29 smithi163 ceph-mon[40675]: 6.5 scrub starts 2024-03-21T13:01:29.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:29 smithi163 ceph-mon[40675]: 6.5 scrub ok 2024-03-21T13:01:29.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:29 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3543053140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:01:29.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:29 smithi163 ceph-mon[40675]: pgmap v93: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 904 op/s 2024-03-21T13:01:29.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:29 smithi163 ceph-mon[40675]: 4.3 scrub starts 2024-03-21T13:01:29.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:29 smithi163 ceph-mon[40675]: 4.3 scrub ok 2024-03-21T13:01:29.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:29 smithi163 ceph-mon[29621]: 6.5 scrub starts 2024-03-21T13:01:29.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:29 smithi163 ceph-mon[29621]: 6.5 scrub ok 2024-03-21T13:01:29.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:29 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3543053140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:01:29.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:29 smithi163 ceph-mon[29621]: pgmap v93: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 904 op/s 2024-03-21T13:01:29.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:29 smithi163 ceph-mon[29621]: 4.3 scrub starts 2024-03-21T13:01:29.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:29 smithi163 ceph-mon[29621]: 4.3 scrub ok 2024-03-21T13:01:29.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:29.832 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:29.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:29.933 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:29.982 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:29.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:29.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:30.079 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 36 15 63205 63190 109.694 85.125 0.00620896 0.00911097 2024-03-21T13:01:30.086 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:30.135 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:30.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:30.237 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:30.266 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:30.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:30.368 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:30.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:30.408 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:30.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:30.509 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:30.543 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:30.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:30.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:30.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:30.676 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:30.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:30.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:30.778 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:30.819 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:30.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:30.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:30.921 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:30.961 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:30.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:30 smithi204 ceph-mon[33153]: 4.1 scrub starts 2024-03-21T13:01:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:30 smithi204 ceph-mon[33153]: 4.1 scrub ok 2024-03-21T13:01:31.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:30 smithi163 ceph-mon[29621]: 4.1 scrub starts 2024-03-21T13:01:31.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:30 smithi163 ceph-mon[29621]: 4.1 scrub ok 2024-03-21T13:01:31.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:30 smithi163 ceph-mon[40675]: 4.1 scrub starts 2024-03-21T13:01:31.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:30 smithi163 ceph-mon[40675]: 4.1 scrub ok 2024-03-21T13:01:31.063 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:31.079 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 37 16 64524 64508 108.955 82.375 0.0141135 0.00917313 2024-03-21T13:01:31.103 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:31.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:31.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:31.247 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:31.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:31.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:31.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:31.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:31.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:31.525 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:31.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:31.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:31.627 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:31.643 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:01:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:01:31.657 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:31.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:31.759 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:31.801 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:31.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:31.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:31.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[29621]: 5.1f scrub starts 2024-03-21T13:01:31.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[29621]: 5.1f scrub ok 2024-03-21T13:01:31.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[29621]: pgmap v94: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 511 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-21T13:01:31.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[29621]: 4.10 scrub starts 2024-03-21T13:01:31.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[29621]: 4.10 scrub ok 2024-03-21T13:01:31.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[29621]: 6.0 scrub starts 2024-03-21T13:01:31.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[29621]: 6.0 scrub ok 2024-03-21T13:01:31.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[29621]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T13:01:31.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:31.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[40675]: 5.1f scrub starts 2024-03-21T13:01:31.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[40675]: 5.1f scrub ok 2024-03-21T13:01:31.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[40675]: pgmap v94: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 511 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-21T13:01:31.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[40675]: 4.10 scrub starts 2024-03-21T13:01:31.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[40675]: 4.10 scrub ok 2024-03-21T13:01:31.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[40675]: 6.0 scrub starts 2024-03-21T13:01:31.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[40675]: 6.0 scrub ok 2024-03-21T13:01:31.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[40675]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T13:01:31.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:31 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:31.950 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:31.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:32.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:31 smithi204 ceph-mon[33153]: 5.1f scrub starts 2024-03-21T13:01:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:31 smithi204 ceph-mon[33153]: 5.1f scrub ok 2024-03-21T13:01:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:31 smithi204 ceph-mon[33153]: pgmap v94: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 511 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-21T13:01:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:31 smithi204 ceph-mon[33153]: 4.10 scrub starts 2024-03-21T13:01:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:31 smithi204 ceph-mon[33153]: 4.10 scrub ok 2024-03-21T13:01:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:31 smithi204 ceph-mon[33153]: 6.0 scrub starts 2024-03-21T13:01:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:31 smithi204 ceph-mon[33153]: 6.0 scrub ok 2024-03-21T13:01:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:31 smithi204 ceph-mon[33153]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T13:01:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:31 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:32.051 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:32.079 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 38 16 65841 65825 108.252 82.3125 0.0129958 0.00923304 2024-03-21T13:01:32.097 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:32.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:32.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:32.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:32.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:32.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:32.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:32.329 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:32.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:32.379 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:32.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:32.480 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:32.524 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:32.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:32.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:32.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:32.655 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:32.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:32.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:32.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:32.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:32.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:32.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:32.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:32.923 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:32.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:33.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:32 smithi204 ceph-mon[33153]: 4.7 scrub starts 2024-03-21T13:01:33.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:32 smithi204 ceph-mon[33153]: 4.7 scrub ok 2024-03-21T13:01:33.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:32 smithi204 ceph-mon[33153]: 3.3 scrub starts 2024-03-21T13:01:33.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:32 smithi204 ceph-mon[33153]: 3.3 scrub ok 2024-03-21T13:01:33.008 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:32 smithi163 ceph-mon[29621]: 4.7 scrub starts 2024-03-21T13:01:33.008 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:32 smithi163 ceph-mon[29621]: 4.7 scrub ok 2024-03-21T13:01:33.008 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:32 smithi163 ceph-mon[29621]: 3.3 scrub starts 2024-03-21T13:01:33.008 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:32 smithi163 ceph-mon[29621]: 3.3 scrub ok 2024-03-21T13:01:33.009 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:32 smithi163 ceph-mon[40675]: 4.7 scrub starts 2024-03-21T13:01:33.009 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:32 smithi163 ceph-mon[40675]: 4.7 scrub ok 2024-03-21T13:01:33.009 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:32 smithi163 ceph-mon[40675]: 3.3 scrub starts 2024-03-21T13:01:33.009 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:32 smithi163 ceph-mon[40675]: 3.3 scrub ok 2024-03-21T13:01:33.024 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:33.066 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:33.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:33.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:33.079 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 39 16 66947 66931 107.248 69.125 0.00538363 0.00931889 2024-03-21T13:01:33.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:33.199 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:33.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:33.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:33.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:33.331 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:33.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:33.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:33.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:33.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:33.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:33.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:33.566 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:33.605 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:33.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:33.680 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:01:33.707 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:33.749 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:33.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:33.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[29621]: 6.e scrub starts 2024-03-21T13:01:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[29621]: 6.e scrub ok 2024-03-21T13:01:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[29621]: pgmap v96: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-21T13:01:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2782612347' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:01:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:01:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[29621]: 6.19 scrub starts 2024-03-21T13:01:33.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[29621]: 6.19 scrub ok 2024-03-21T13:01:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[40675]: 6.e scrub starts 2024-03-21T13:01:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[40675]: 6.e scrub ok 2024-03-21T13:01:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[40675]: pgmap v96: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-21T13:01:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2782612347' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:01:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:01:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[40675]: 6.19 scrub starts 2024-03-21T13:01:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:33 smithi163 ceph-mon[40675]: 6.19 scrub ok 2024-03-21T13:01:33.851 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:33.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:33.884 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:33.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:33.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:01:33.986 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:33 smithi204 ceph-mon[33153]: 6.e scrub starts 2024-03-21T13:01:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:33 smithi204 ceph-mon[33153]: 6.e scrub ok 2024-03-21T13:01:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:33 smithi204 ceph-mon[33153]: pgmap v96: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-21T13:01:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:33 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2782612347' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:01:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:33 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:01:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:33 smithi204 ceph-mon[33153]: 6.19 scrub starts 2024-03-21T13:01:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:33 smithi204 ceph-mon[33153]: 6.19 scrub ok 2024-03-21T13:01:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:34.022 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:34.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:34.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:01:34.080730+0000 min lat: 0.00109429 max lat: 0.19616 avg lat: 0.00937912 2024-03-21T13:01:34.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:01:34.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 40 16 68219 68203 106.555 79.5 0.00942688 0.00937912 2024-03-21T13:01:34.122 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:34.180 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:34.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:34.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:34.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:34.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:34.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:34.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:34.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:34.455 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:34.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:34.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:34.557 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:34.603 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:34.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:34.705 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:34.743 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:34.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:34.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:34.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:34.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:34.946 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[29621]: 6.1d scrub starts 2024-03-21T13:01:34.946 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[29621]: 6.1d scrub ok 2024-03-21T13:01:34.946 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:01:34.946 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:01:34.946 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:01:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[29621]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T13:01:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[29621]: 3.c scrub starts 2024-03-21T13:01:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[29621]: 3.c scrub ok 2024-03-21T13:01:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[29621]: 3.1 deep-scrub starts 2024-03-21T13:01:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[29621]: 3.1 deep-scrub ok 2024-03-21T13:01:34.947 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[40675]: 6.1d scrub starts 2024-03-21T13:01:34.947 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[40675]: 6.1d scrub ok 2024-03-21T13:01:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:01:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:01:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:01:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[40675]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T13:01:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[40675]: 3.c scrub starts 2024-03-21T13:01:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[40675]: 3.c scrub ok 2024-03-21T13:01:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[40675]: 3.1 deep-scrub starts 2024-03-21T13:01:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:34 smithi163 ceph-mon[40675]: 3.1 deep-scrub ok 2024-03-21T13:01:34.982 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:34 smithi204 ceph-mon[33153]: 6.1d scrub starts 2024-03-21T13:01:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:34 smithi204 ceph-mon[33153]: 6.1d scrub ok 2024-03-21T13:01:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:34 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:01:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:34 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:01:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:34 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:01:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:34 smithi204 ceph-mon[33153]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T13:01:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:34 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:34 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:34 smithi204 ceph-mon[33153]: 3.c scrub starts 2024-03-21T13:01:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:34 smithi204 ceph-mon[33153]: 3.c scrub ok 2024-03-21T13:01:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:34 smithi204 ceph-mon[33153]: 3.1 deep-scrub starts 2024-03-21T13:01:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:34 smithi204 ceph-mon[33153]: 3.1 deep-scrub ok 2024-03-21T13:01:35.030 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:35.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:35.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 41 16 69498 69482 105.905 79.9375 0.0196448 0.0094371 2024-03-21T13:01:35.133 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:35.177 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:35.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:35.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:35.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:35.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:35.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:35.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:35.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:35.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:35.537 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:35.581 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:35.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:35.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:35.671 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:01:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:01:35.683 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:35.712 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:35.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:35.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:35.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:35.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:35.842 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:35.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:35.943 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:35.972 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:35.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:35.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:35 smithi204 ceph-mon[33153]: 6.14 scrub starts 2024-03-21T13:01:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:35 smithi204 ceph-mon[33153]: 6.14 scrub ok 2024-03-21T13:01:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:35 smithi204 ceph-mon[33153]: pgmap v98: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 560 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-21T13:01:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:35 smithi204 ceph-mon[33153]: 5.7 scrub starts 2024-03-21T13:01:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:35 smithi204 ceph-mon[33153]: 5.7 scrub ok 2024-03-21T13:01:36.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:35 smithi204 ceph-mon[33153]: 6.12 deep-scrub starts 2024-03-21T13:01:36.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:35 smithi204 ceph-mon[33153]: 6.12 deep-scrub ok 2024-03-21T13:01:36.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[29621]: 6.14 scrub starts 2024-03-21T13:01:36.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[29621]: 6.14 scrub ok 2024-03-21T13:01:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[29621]: pgmap v98: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 560 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-21T13:01:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[29621]: 5.7 scrub starts 2024-03-21T13:01:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[29621]: 5.7 scrub ok 2024-03-21T13:01:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[29621]: 6.12 deep-scrub starts 2024-03-21T13:01:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[29621]: 6.12 deep-scrub ok 2024-03-21T13:01:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[40675]: 6.14 scrub starts 2024-03-21T13:01:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[40675]: 6.14 scrub ok 2024-03-21T13:01:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[40675]: pgmap v98: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 560 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-21T13:01:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[40675]: 5.7 scrub starts 2024-03-21T13:01:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[40675]: 5.7 scrub ok 2024-03-21T13:01:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[40675]: 6.12 deep-scrub starts 2024-03-21T13:01:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:35 smithi163 ceph-mon[40675]: 6.12 deep-scrub ok 2024-03-21T13:01:36.074 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:36.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 42 16 70616 70600 105.047 69.875 0.00526345 0.00951391 2024-03-21T13:01:36.115 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:36.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:36.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:36.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:36.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:36.264 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:36.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:36.366 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:36.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:36.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:36.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:36.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:36.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:36.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:36.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:36.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:36.660 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:36.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:36.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:36.788 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:36.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:36.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:36.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:36.941 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:36.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:36 smithi204 ceph-mon[33153]: 6.1a deep-scrub starts 2024-03-21T13:01:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:36 smithi204 ceph-mon[33153]: 6.1a deep-scrub ok 2024-03-21T13:01:37.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:36 smithi204 ceph-mon[33153]: 5.f scrub starts 2024-03-21T13:01:37.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:36 smithi204 ceph-mon[33153]: 5.f scrub ok 2024-03-21T13:01:37.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:36 smithi204 ceph-mon[33153]: osdmap e107: 8 total, 8 up, 8 in 2024-03-21T13:01:37.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:36 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:01:37.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:36 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:01:37.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[29621]: 6.1a deep-scrub starts 2024-03-21T13:01:37.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[29621]: 6.1a deep-scrub ok 2024-03-21T13:01:37.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[29621]: 5.f scrub starts 2024-03-21T13:01:37.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[29621]: 5.f scrub ok 2024-03-21T13:01:37.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[29621]: osdmap e107: 8 total, 8 up, 8 in 2024-03-21T13:01:37.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:01:37.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:01:37.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[40675]: 6.1a deep-scrub starts 2024-03-21T13:01:37.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[40675]: 6.1a deep-scrub ok 2024-03-21T13:01:37.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[40675]: 5.f scrub starts 2024-03-21T13:01:37.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[40675]: 5.f scrub ok 2024-03-21T13:01:37.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[40675]: osdmap e107: 8 total, 8 up, 8 in 2024-03-21T13:01:37.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:01:37.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:36 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:01:37.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:37.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 43 16 71682 71666 104.153 66.625 0.0213966 0.00959439 2024-03-21T13:01:37.113 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:37.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:37.215 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:37.263 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:37.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:37.364 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:37.392 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:37.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:37.445 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:01:37.493 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:37.519 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:37.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:37.622 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:37.649 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:37.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:37.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:37.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[29621]: 4.4 scrub starts 2024-03-21T13:01:37.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[29621]: 4.4 scrub ok 2024-03-21T13:01:37.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[29621]: 4.1c scrub starts 2024-03-21T13:01:37.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[29621]: 4.1c scrub ok 2024-03-21T13:01:37.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[29621]: pgmap v100: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:01:37.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[29621]: 6.17 deep-scrub starts 2024-03-21T13:01:37.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[29621]: 6.17 deep-scrub ok 2024-03-21T13:01:37.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:37.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:37.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[29621]: osdmap e108: 8 total, 8 up, 8 in 2024-03-21T13:01:37.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[40675]: 4.4 scrub starts 2024-03-21T13:01:37.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[40675]: 4.4 scrub ok 2024-03-21T13:01:37.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[40675]: 4.1c scrub starts 2024-03-21T13:01:37.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[40675]: 4.1c scrub ok 2024-03-21T13:01:37.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[40675]: pgmap v100: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:01:37.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[40675]: 6.17 deep-scrub starts 2024-03-21T13:01:37.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[40675]: 6.17 deep-scrub ok 2024-03-21T13:01:37.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:37.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:37.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:37 smithi163 ceph-mon[40675]: osdmap e108: 8 total, 8 up, 8 in 2024-03-21T13:01:37.751 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:37.783 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:37.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:37.884 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:37.906 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:01:37.906 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:01:37.906 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-03-21T13:01:37.906 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-03-21T13:01:37.907 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:01:37.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:37.937 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:37.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:38.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:37 smithi204 ceph-mon[33153]: 4.4 scrub starts 2024-03-21T13:01:38.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:37 smithi204 ceph-mon[33153]: 4.4 scrub ok 2024-03-21T13:01:38.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:37 smithi204 ceph-mon[33153]: 4.1c scrub starts 2024-03-21T13:01:38.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:37 smithi204 ceph-mon[33153]: 4.1c scrub ok 2024-03-21T13:01:38.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:37 smithi204 ceph-mon[33153]: pgmap v100: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:01:38.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:37 smithi204 ceph-mon[33153]: 6.17 deep-scrub starts 2024-03-21T13:01:38.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:37 smithi204 ceph-mon[33153]: 6.17 deep-scrub ok 2024-03-21T13:01:38.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:37 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:38.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:37 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:38.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:37 smithi204 ceph-mon[33153]: osdmap e108: 8 total, 8 up, 8 in 2024-03-21T13:01:38.038 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:38.079 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:38.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:38.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:38.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 44 16 72606 72590 103.099 57.75 0.0176049 0.00969438 2024-03-21T13:01:38.181 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:38.211 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:38.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:38.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:38.240 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:38.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:38.345 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:38.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:38.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:38.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:38.477 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:38.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:38.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:38.579 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:38.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:38.611 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:38.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:38.712 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:38.748 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:38.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:38.850 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:38.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:38.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:38.946 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:01:38.946 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:01:38.947 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-21T13:01:38.947 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.085282 2024-03-21T13:01:38.947 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 0 0.08528226265260952 2024-03-21T13:01:38.978 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:38.979 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[29621]: 3.19 scrub starts 2024-03-21T13:01:38.979 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[29621]: 3.19 scrub ok 2024-03-21T13:01:38.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[29621]: 4.18 deep-scrub starts 2024-03-21T13:01:38.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[29621]: 4.18 deep-scrub ok 2024-03-21T13:01:38.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:38.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:38.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:38.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[40675]: 3.19 scrub starts 2024-03-21T13:01:38.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[40675]: 3.19 scrub ok 2024-03-21T13:01:38.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[40675]: 4.18 deep-scrub starts 2024-03-21T13:01:38.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[40675]: 4.18 deep-scrub ok 2024-03-21T13:01:38.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:38.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:38.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:38 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:39.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:38 smithi204 ceph-mon[33153]: 3.19 scrub starts 2024-03-21T13:01:39.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:38 smithi204 ceph-mon[33153]: 3.19 scrub ok 2024-03-21T13:01:39.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:38 smithi204 ceph-mon[33153]: 4.18 deep-scrub starts 2024-03-21T13:01:39.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:38 smithi204 ceph-mon[33153]: 4.18 deep-scrub ok 2024-03-21T13:01:39.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:38 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:38 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:38 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:01:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:39.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:39.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 45 16 73710 73694 102.341 69 0.013335 0.00976601 2024-03-21T13:01:39.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:39.172 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:39.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:39.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:39.275 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:39.324 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:39.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:39.425 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:39.452 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:39.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:39.458 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-21T13:01:39.554 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:39.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:39.615 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:39.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:39.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:39.745 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:39.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:39.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:39.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:39.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:39.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:39.968 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:39 smithi163 ceph-mon[29621]: pgmap v102: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:01:39.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:39 smithi163 ceph-mon[29621]: 3.1e deep-scrub starts 2024-03-21T13:01:39.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:39 smithi163 ceph-mon[29621]: 3.1e deep-scrub ok 2024-03-21T13:01:39.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:39 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:01:39.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:39 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:01:39.969 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:39 smithi163 ceph-mon[40675]: pgmap v102: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:01:39.969 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:39 smithi163 ceph-mon[40675]: 3.1e deep-scrub starts 2024-03-21T13:01:39.969 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:39 smithi163 ceph-mon[40675]: 3.1e deep-scrub ok 2024-03-21T13:01:39.969 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:39 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:01:39.969 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:39 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:01:39.976 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:39 smithi204 ceph-mon[33153]: pgmap v102: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:01:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:39 smithi204 ceph-mon[33153]: 3.1e deep-scrub starts 2024-03-21T13:01:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:39 smithi204 ceph-mon[33153]: 3.1e deep-scrub ok 2024-03-21T13:01:40.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:39 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:01:40.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:39 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:01:40.008 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:40.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:40.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 46 16 74863 74847 101.683 72.0625 0.0130736 0.00982955 2024-03-21T13:01:40.110 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:40.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:40.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:40.242 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:40.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:40.259 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:40.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:40.332 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:40.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:40.433 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:40.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:40.484 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:40.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:40.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:40.618 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:40.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:40.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:40.754 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:40.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:40.841 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[29621]: 3.9 scrub starts 2024-03-21T13:01:40.841 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[29621]: 3.9 scrub ok 2024-03-21T13:01:40.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[29621]: 5.b scrub starts 2024-03-21T13:01:40.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[29621]: 5.b scrub ok 2024-03-21T13:01:40.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[29621]: 4.1d scrub starts 2024-03-21T13:01:40.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[29621]: 4.1d scrub ok 2024-03-21T13:01:40.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[29621]: 5.1e scrub starts 2024-03-21T13:01:40.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[29621]: 5.1e scrub ok 2024-03-21T13:01:40.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[29621]: 3.f scrub starts 2024-03-21T13:01:40.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[29621]: 3.f scrub ok 2024-03-21T13:01:40.842 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[40675]: 3.9 scrub starts 2024-03-21T13:01:40.842 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[40675]: 3.9 scrub ok 2024-03-21T13:01:40.842 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[40675]: 5.b scrub starts 2024-03-21T13:01:40.843 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[40675]: 5.b scrub ok 2024-03-21T13:01:40.843 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[40675]: 4.1d scrub starts 2024-03-21T13:01:40.843 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[40675]: 4.1d scrub ok 2024-03-21T13:01:40.843 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[40675]: 5.1e scrub starts 2024-03-21T13:01:40.843 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[40675]: 5.1e scrub ok 2024-03-21T13:01:40.843 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[40675]: 3.f scrub starts 2024-03-21T13:01:40.843 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:40 smithi163 ceph-mon[40675]: 3.f scrub ok 2024-03-21T13:01:40.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:40.882 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:40.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:40.985 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:41.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:40 smithi204 ceph-mon[33153]: 3.9 scrub starts 2024-03-21T13:01:41.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:40 smithi204 ceph-mon[33153]: 3.9 scrub ok 2024-03-21T13:01:41.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:40 smithi204 ceph-mon[33153]: 5.b scrub starts 2024-03-21T13:01:41.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:40 smithi204 ceph-mon[33153]: 5.b scrub ok 2024-03-21T13:01:41.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:40 smithi204 ceph-mon[33153]: 4.1d scrub starts 2024-03-21T13:01:41.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:40 smithi204 ceph-mon[33153]: 4.1d scrub ok 2024-03-21T13:01:41.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:40 smithi204 ceph-mon[33153]: 5.1e scrub starts 2024-03-21T13:01:41.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:40 smithi204 ceph-mon[33153]: 5.1e scrub ok 2024-03-21T13:01:41.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:40 smithi204 ceph-mon[33153]: 3.f scrub starts 2024-03-21T13:01:41.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:40 smithi204 ceph-mon[33153]: 3.f scrub ok 2024-03-21T13:01:41.038 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:41.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:41.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 47 16 76028 76012 101.068 72.8125 0.00767645 0.00988898 2024-03-21T13:01:41.140 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:41.166 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:41.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:41.268 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:41.312 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:41.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:41.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:41.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:41.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:41.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:41.571 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:41.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:41.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:41.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:41.695 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:01:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:01:41.713 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:41.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:41.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:41.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:41.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:41.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:41.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:41.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:41.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:41 smithi204 ceph-mon[33153]: 5.8 scrub starts 2024-03-21T13:01:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:41 smithi204 ceph-mon[33153]: 5.8 scrub ok 2024-03-21T13:01:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:41 smithi204 ceph-mon[33153]: pgmap v103: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-21T13:01:42.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:41 smithi204 ceph-mon[33153]: 4.16 scrub starts 2024-03-21T13:01:42.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:41 smithi204 ceph-mon[33153]: 4.16 scrub ok 2024-03-21T13:01:42.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:41 smithi204 ceph-mon[33153]: 5.4 deep-scrub starts 2024-03-21T13:01:42.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:41 smithi204 ceph-mon[33153]: 5.4 deep-scrub ok 2024-03-21T13:01:42.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:41 smithi204 ceph-mon[33153]: 2.2 scrub starts 2024-03-21T13:01:42.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:41 smithi204 ceph-mon[33153]: 2.2 scrub ok 2024-03-21T13:01:42.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:41 smithi204 ceph-mon[33153]: osdmap e109: 8 total, 8 up, 8 in 2024-03-21T13:01:42.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:41 smithi204 ceph-mon[33153]: 3.10 scrub starts 2024-03-21T13:01:42.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:41 smithi204 ceph-mon[33153]: 3.10 scrub ok 2024-03-21T13:01:42.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[29621]: 5.8 scrub starts 2024-03-21T13:01:42.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[29621]: 5.8 scrub ok 2024-03-21T13:01:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[29621]: pgmap v103: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-21T13:01:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[29621]: 4.16 scrub starts 2024-03-21T13:01:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[29621]: 4.16 scrub ok 2024-03-21T13:01:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[29621]: 5.4 deep-scrub starts 2024-03-21T13:01:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[29621]: 5.4 deep-scrub ok 2024-03-21T13:01:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[29621]: 2.2 scrub starts 2024-03-21T13:01:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[29621]: 2.2 scrub ok 2024-03-21T13:01:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[29621]: osdmap e109: 8 total, 8 up, 8 in 2024-03-21T13:01:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[29621]: 3.10 scrub starts 2024-03-21T13:01:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[29621]: 3.10 scrub ok 2024-03-21T13:01:42.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[40675]: 5.8 scrub starts 2024-03-21T13:01:42.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[40675]: 5.8 scrub ok 2024-03-21T13:01:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[40675]: pgmap v103: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-21T13:01:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[40675]: 4.16 scrub starts 2024-03-21T13:01:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[40675]: 4.16 scrub ok 2024-03-21T13:01:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[40675]: 5.4 deep-scrub starts 2024-03-21T13:01:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[40675]: 5.4 deep-scrub ok 2024-03-21T13:01:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[40675]: 2.2 scrub starts 2024-03-21T13:01:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[40675]: 2.2 scrub ok 2024-03-21T13:01:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[40675]: osdmap e109: 8 total, 8 up, 8 in 2024-03-21T13:01:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[40675]: 3.10 scrub starts 2024-03-21T13:01:42.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:41 smithi163 ceph-mon[40675]: 3.10 scrub ok 2024-03-21T13:01:42.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:42.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 48 16 77134 77118 100.402 69.125 0.014123 0.00995479 2024-03-21T13:01:42.115 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:42.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:42.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:42.216 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:42.262 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:42.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:42.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:42.364 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:42.406 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:42.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:42.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:42.508 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:42.547 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:42.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:42.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:42.649 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:42.689 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:42.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:42.792 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:42.829 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:42.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:42.931 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:42.970 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:42.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:43.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:42 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:43.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:42 smithi204 ceph-mon[33153]: 5.d scrub starts 2024-03-21T13:01:43.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:42 smithi204 ceph-mon[33153]: 5.d scrub ok 2024-03-21T13:01:43.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:42 smithi204 ceph-mon[33153]: 5.1b deep-scrub starts 2024-03-21T13:01:43.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:42 smithi204 ceph-mon[33153]: 5.1b deep-scrub ok 2024-03-21T13:01:43.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:42 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:43.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:42 smithi163 ceph-mon[29621]: 5.d scrub starts 2024-03-21T13:01:43.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:42 smithi163 ceph-mon[29621]: 5.d scrub ok 2024-03-21T13:01:43.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:42 smithi163 ceph-mon[29621]: 5.1b deep-scrub starts 2024-03-21T13:01:43.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:42 smithi163 ceph-mon[29621]: 5.1b deep-scrub ok 2024-03-21T13:01:43.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:42 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:43.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:42 smithi163 ceph-mon[40675]: 5.d scrub starts 2024-03-21T13:01:43.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:42 smithi163 ceph-mon[40675]: 5.d scrub ok 2024-03-21T13:01:43.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:42 smithi163 ceph-mon[40675]: 5.1b deep-scrub starts 2024-03-21T13:01:43.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:42 smithi163 ceph-mon[40675]: 5.1b deep-scrub ok 2024-03-21T13:01:43.073 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:43.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 49 16 78190 78174 99.7 66 0.00649042 0.0100192 2024-03-21T13:01:43.101 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:43.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:43.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:43.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:43.232 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:43.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:43.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:43.382 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:43.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:43.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:43.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:43.514 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:43.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:43.616 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:43.665 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:43.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:43.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:43.767 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:43.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:43.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:43.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:43.929 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:43.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:43.970 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:01:43.971 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:43 smithi163 ceph-mon[29621]: pgmap v105: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 791 B/s rd, 52 MiB/s wr, 834 op/s 2024-03-21T13:01:43.971 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:43 smithi163 ceph-mon[29621]: 6.a deep-scrub starts 2024-03-21T13:01:43.971 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:43 smithi163 ceph-mon[29621]: 6.a deep-scrub ok 2024-03-21T13:01:43.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:43 smithi163 ceph-mon[40675]: pgmap v105: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 791 B/s rd, 52 MiB/s wr, 834 op/s 2024-03-21T13:01:43.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:43 smithi163 ceph-mon[40675]: 6.a deep-scrub starts 2024-03-21T13:01:43.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:43 smithi163 ceph-mon[40675]: 6.a deep-scrub ok 2024-03-21T13:01:44.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:43 smithi204 ceph-mon[33153]: pgmap v105: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 791 B/s rd, 52 MiB/s wr, 834 op/s 2024-03-21T13:01:44.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:43 smithi204 ceph-mon[33153]: 6.a deep-scrub starts 2024-03-21T13:01:44.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:43 smithi204 ceph-mon[33153]: 6.a deep-scrub ok 2024-03-21T13:01:44.016 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:01:44.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:44.060 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:01:44.073 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:44.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:44.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:44.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 50 16 79292 79276 99.0834 68.875 0.00582314 0.0100876 2024-03-21T13:01:44.102 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:01:44.142 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:01:44.174 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:44.183 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:01:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:44.215 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:44.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:44.237 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:01:44.274 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:01:44.282 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:01:44.282 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:01:44.282 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:01:44.283 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:01:44.283 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:01:44.283 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:01:44.283 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:01:44.283 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:01:44.315 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:44.344 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:44.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:44.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:44.446 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:44.497 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:44.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:44.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:44.599 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:44.646 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:44.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:44.664 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:44.749 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:44.796 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:44.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:44.897 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:44.935 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:44.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:45.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/501330925' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:01:45.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:44 smithi204 ceph-mon[33153]: 3.15 scrub starts 2024-03-21T13:01:45.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:44 smithi204 ceph-mon[33153]: 3.15 scrub ok 2024-03-21T13:01:45.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:44 smithi204 ceph-mon[33153]: 3.1b scrub starts 2024-03-21T13:01:45.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:44 smithi204 ceph-mon[33153]: 3.1b scrub ok 2024-03-21T13:01:45.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/501330925' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:01:45.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:44 smithi163 ceph-mon[29621]: 3.15 scrub starts 2024-03-21T13:01:45.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:44 smithi163 ceph-mon[29621]: 3.15 scrub ok 2024-03-21T13:01:45.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:44 smithi163 ceph-mon[29621]: 3.1b scrub starts 2024-03-21T13:01:45.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:44 smithi163 ceph-mon[29621]: 3.1b scrub ok 2024-03-21T13:01:45.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/501330925' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:01:45.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:44 smithi163 ceph-mon[40675]: 3.15 scrub starts 2024-03-21T13:01:45.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:44 smithi163 ceph-mon[40675]: 3.15 scrub ok 2024-03-21T13:01:45.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:44 smithi163 ceph-mon[40675]: 3.1b scrub starts 2024-03-21T13:01:45.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:44 smithi163 ceph-mon[40675]: 3.1b scrub ok 2024-03-21T13:01:45.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:45.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:45.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:45.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:45.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 51 16 80408 80392 98.5081 69.75 0.0078156 0.0101462 2024-03-21T13:01:45.168 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:45.209 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:45.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:45.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:45.311 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:45.339 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:45.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:45.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:45.441 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:45.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:45.487 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:45.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:45.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:45.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:45.630 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:45.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:45.719 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:01:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:01:45.732 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:45.779 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:45.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:45.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:45.881 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:45.925 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:45.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:45.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:45.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:45 smithi163 ceph-mon[29621]: pgmap v106: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 935 op/s 2024-03-21T13:01:45.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:45 smithi163 ceph-mon[29621]: 6.9 scrub starts 2024-03-21T13:01:45.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:45 smithi163 ceph-mon[29621]: 6.9 scrub ok 2024-03-21T13:01:45.979 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:45 smithi163 ceph-mon[40675]: pgmap v106: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 935 op/s 2024-03-21T13:01:45.979 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:45 smithi163 ceph-mon[40675]: 6.9 scrub starts 2024-03-21T13:01:45.979 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:45 smithi163 ceph-mon[40675]: 6.9 scrub ok 2024-03-21T13:01:46.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:45 smithi204 ceph-mon[33153]: pgmap v106: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 935 op/s 2024-03-21T13:01:46.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:45 smithi204 ceph-mon[33153]: 6.9 scrub starts 2024-03-21T13:01:46.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:45 smithi204 ceph-mon[33153]: 6.9 scrub ok 2024-03-21T13:01:46.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:46.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:46.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:46.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:46.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 52 16 81540 81524 97.9742 70.75 0.0116725 0.0102011 2024-03-21T13:01:46.161 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:46.203 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:46.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:46.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:46.332 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:46.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:46.434 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:46.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:46.476 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:46.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:46.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:46.609 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:46.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:46.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:46.741 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:46.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:46.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:46.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:46.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:46.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:46.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:47.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:46 smithi204 ceph-mon[33153]: 3.13 scrub starts 2024-03-21T13:01:47.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:46 smithi204 ceph-mon[33153]: 3.13 scrub ok 2024-03-21T13:01:47.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:46 smithi204 ceph-mon[33153]: 4.f scrub starts 2024-03-21T13:01:47.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:46 smithi204 ceph-mon[33153]: 4.f scrub ok 2024-03-21T13:01:47.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:46 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4066681250' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.08528226265260952}]: dispatch 2024-03-21T13:01:47.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:46 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.08528226265260952}]: dispatch 2024-03-21T13:01:47.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:46 smithi204 ceph-mon[33153]: 5.a scrub starts 2024-03-21T13:01:47.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:46 smithi204 ceph-mon[33153]: 5.a scrub ok 2024-03-21T13:01:47.007 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:47.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:47.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:47.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[40675]: 3.13 scrub starts 2024-03-21T13:01:47.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[40675]: 3.13 scrub ok 2024-03-21T13:01:47.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[40675]: 4.f scrub starts 2024-03-21T13:01:47.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[40675]: 4.f scrub ok 2024-03-21T13:01:47.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4066681250' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.08528226265260952}]: dispatch 2024-03-21T13:01:47.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.08528226265260952}]: dispatch 2024-03-21T13:01:47.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[40675]: 5.a scrub starts 2024-03-21T13:01:47.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[40675]: 5.a scrub ok 2024-03-21T13:01:47.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[29621]: 3.13 scrub starts 2024-03-21T13:01:47.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[29621]: 3.13 scrub ok 2024-03-21T13:01:47.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[29621]: 4.f scrub starts 2024-03-21T13:01:47.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[29621]: 4.f scrub ok 2024-03-21T13:01:47.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4066681250' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.08528226265260952}]: dispatch 2024-03-21T13:01:47.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.08528226265260952}]: dispatch 2024-03-21T13:01:47.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[29621]: 5.a scrub starts 2024-03-21T13:01:47.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:46 smithi163 ceph-mon[29621]: 5.a scrub ok 2024-03-21T13:01:47.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 53 16 82634 82618 97.4157 68.375 0.0113632 0.0102606 2024-03-21T13:01:47.109 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:47.152 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:47.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:47.207 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:01:47.215 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.0 primary-affinity to 0.0852823 (855892) 2024-03-21T13:01:47.253 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:47.277 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:47.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:47.380 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:47.404 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:47.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:47.506 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:47.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:47.531 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:47.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:47.632 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:47.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:47.657 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:47.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:47.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:47.782 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:47.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:47.883 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:47.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:47.909 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:47.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:48.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:47 smithi204 ceph-mon[33153]: 7.1 scrub starts 2024-03-21T13:01:48.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:47 smithi204 ceph-mon[33153]: pgmap v107: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-21T13:01:48.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:47 smithi204 ceph-mon[33153]: 6.6 deep-scrub starts 2024-03-21T13:01:48.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:47 smithi204 ceph-mon[33153]: 6.6 deep-scrub ok 2024-03-21T13:01:48.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:47 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.08528226265260952}]': finished 2024-03-21T13:01:48.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:47 smithi204 ceph-mon[33153]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T13:01:48.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:47 smithi204 ceph-mon[33153]: 6.f scrub starts 2024-03-21T13:01:48.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:47 smithi204 ceph-mon[33153]: 6.f scrub ok 2024-03-21T13:01:48.010 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:48.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[29621]: 7.1 scrub starts 2024-03-21T13:01:48.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[29621]: pgmap v107: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-21T13:01:48.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[29621]: 6.6 deep-scrub starts 2024-03-21T13:01:48.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[29621]: 6.6 deep-scrub ok 2024-03-21T13:01:48.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.08528226265260952}]': finished 2024-03-21T13:01:48.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[29621]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T13:01:48.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[29621]: 6.f scrub starts 2024-03-21T13:01:48.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[29621]: 6.f scrub ok 2024-03-21T13:01:48.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[40675]: 7.1 scrub starts 2024-03-21T13:01:48.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[40675]: pgmap v107: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-21T13:01:48.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[40675]: 6.6 deep-scrub starts 2024-03-21T13:01:48.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[40675]: 6.6 deep-scrub ok 2024-03-21T13:01:48.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.08528226265260952}]': finished 2024-03-21T13:01:48.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[40675]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T13:01:48.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[40675]: 6.f scrub starts 2024-03-21T13:01:48.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:47 smithi163 ceph-mon[40675]: 6.f scrub ok 2024-03-21T13:01:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:48.037 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:48.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:48.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 54 16 82798 82782 95.8015 10.25 0.00607116 0.0102682 2024-03-21T13:01:48.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:48.163 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:48.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:48.264 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:48.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:48.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:48.398 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:48.437 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:48.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:48.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:48.538 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:48.591 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:48.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:48.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:48.693 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:48.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:48.732 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:48.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:48 smithi163 ceph-mon[29621]: 7.1 scrub ok 2024-03-21T13:01:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:48 smithi163 ceph-mon[29621]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T13:01:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:48 smithi163 ceph-mon[29621]: 4.1f scrub starts 2024-03-21T13:01:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:48 smithi163 ceph-mon[29621]: 4.1f scrub ok 2024-03-21T13:01:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:48 smithi163 ceph-mon[40675]: 7.1 scrub ok 2024-03-21T13:01:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:48 smithi163 ceph-mon[40675]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T13:01:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:48 smithi163 ceph-mon[40675]: 4.1f scrub starts 2024-03-21T13:01:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:48 smithi163 ceph-mon[40675]: 4.1f scrub ok 2024-03-21T13:01:48.833 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:48.871 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:48.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:48.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:48.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:48 smithi204 ceph-mon[33153]: 7.1 scrub ok 2024-03-21T13:01:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:48 smithi204 ceph-mon[33153]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T13:01:49.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:48 smithi204 ceph-mon[33153]: 4.1f scrub starts 2024-03-21T13:01:49.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:48 smithi204 ceph-mon[33153]: 4.1f scrub ok 2024-03-21T13:01:49.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:49.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:49.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 55 16 83788 83772 95.1842 61.875 0.0103507 0.0105006 2024-03-21T13:01:49.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:49.158 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:49.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:49.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:49.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:49.296 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:49.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:49.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:49.398 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:49.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:49.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:49.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:49.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:49.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:49.676 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:49.716 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:49.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:49.769 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:49.819 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:49.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:49.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:49.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:49.907 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:49 smithi163 ceph-mon[29621]: pgmap v110: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 685 B/s rd, 47 MiB/s wr, 745 op/s 2024-03-21T13:01:49.907 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:49 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:49.907 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:49 smithi163 ceph-mon[40675]: pgmap v110: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 685 B/s rd, 47 MiB/s wr, 745 op/s 2024-03-21T13:01:49.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:49 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:49.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:49.980 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:49.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:50.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:49 smithi204 ceph-mon[33153]: pgmap v110: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 685 B/s rd, 47 MiB/s wr, 745 op/s 2024-03-21T13:01:50.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:49 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:50.082 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 56 16 84845 84829 94.6641 66.0625 0.00846252 0.0105582 2024-03-21T13:01:50.082 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:50.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:50.132 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:50.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:50.233 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:50.278 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:50.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:50.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:50.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:50.408 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:50.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:50.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:50.510 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:50.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:50.555 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:50.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:50.656 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:50.695 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:50.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:50.755 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:01:50.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:50.826 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:50.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:50.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:50.928 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:50.961 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:50.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:50.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:50 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:01:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:50 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:01:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:50 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/827801500' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:01:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:50 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:01:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:50 smithi204 ceph-mon[33153]: 7.e scrub starts 2024-03-21T13:01:51.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:50 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:01:51.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:50 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:01:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:50 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/827801500' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:01:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:50 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:01:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:50 smithi163 ceph-mon[29621]: 7.e scrub starts 2024-03-21T13:01:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:50 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:01:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:50 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:01:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:50 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/827801500' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:01:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:50 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:01:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:50 smithi163 ceph-mon[40675]: 7.e scrub starts 2024-03-21T13:01:51.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:51.082 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 57 16 85948 85932 94.2127 68.9375 0.0112471 0.0106096 2024-03-21T13:01:51.094 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:51.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:51.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:51.197 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:51.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:51.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:51.287 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:01:51.287 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:01:51.328 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:51.372 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:51.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:51.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:51.474 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:51.511 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:51.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:51.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:51.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:51.643 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:51.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:51.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:01:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:01:51.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:51.776 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:51.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:51.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:51.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:51.911 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:51.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:51.919 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[29621]: 6.3 scrub starts 2024-03-21T13:01:51.919 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[29621]: 6.3 scrub ok 2024-03-21T13:01:51.919 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:01:51.919 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:01:51.919 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[29621]: pgmap v111: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 895 B/s rd, 79 MiB/s wr, 1.26k op/s; 16 KiB/s, 0 objects/s recovering 2024-03-21T13:01:51.920 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:01:51.920 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[29621]: 6.11 scrub starts 2024-03-21T13:01:51.920 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[29621]: 6.11 scrub ok 2024-03-21T13:01:51.920 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:01:51.920 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[29621]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T13:01:51.920 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[29621]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T13:01:51.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[40675]: 6.3 scrub starts 2024-03-21T13:01:51.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[40675]: 6.3 scrub ok 2024-03-21T13:01:51.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:01:51.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:01:51.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[40675]: pgmap v111: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 895 B/s rd, 79 MiB/s wr, 1.26k op/s; 16 KiB/s, 0 objects/s recovering 2024-03-21T13:01:51.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:01:51.921 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[40675]: 6.11 scrub starts 2024-03-21T13:01:51.921 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[40675]: 6.11 scrub ok 2024-03-21T13:01:51.921 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:01:51.921 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[40675]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T13:01:51.921 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:51 smithi163 ceph-mon[40675]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T13:01:51.969 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-21T13:01:52.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:51 smithi204 ceph-mon[33153]: 6.3 scrub starts 2024-03-21T13:01:52.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:51 smithi204 ceph-mon[33153]: 6.3 scrub ok 2024-03-21T13:01:52.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:51 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:01:52.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:51 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:01:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:51 smithi204 ceph-mon[33153]: pgmap v111: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 895 B/s rd, 79 MiB/s wr, 1.26k op/s; 16 KiB/s, 0 objects/s recovering 2024-03-21T13:01:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:51 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:01:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:51 smithi204 ceph-mon[33153]: 6.11 scrub starts 2024-03-21T13:01:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:51 smithi204 ceph-mon[33153]: 6.11 scrub ok 2024-03-21T13:01:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:51 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:01:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:51 smithi204 ceph-mon[33153]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T13:01:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:51 smithi204 ceph-mon[33153]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T13:01:52.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:52.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:52.041 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:52.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:52.082 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 58 16 87090 87074 93.8185 71.375 0.0146747 0.0106536 2024-03-21T13:01:52.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:52.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:52.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:52.275 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:52.305 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:52.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:52.407 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:52.455 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:52.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:52.557 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:52.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:52.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:52.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:52.686 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:52.729 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:52.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:52.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:52.830 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:52.869 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:52.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:52.971 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:53.011 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:53.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:53.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:52 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:53.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:52 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:53.082 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 59 16 88154 88138 93.3554 66.5 0.0116679 0.0107066 2024-03-21T13:01:53.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:53.154 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:53.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:53.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:52 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:53.256 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:53.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:53.303 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:53.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:53.404 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:53.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:53.447 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:53.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:53.548 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:53.587 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:53.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:53.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:53.689 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:53.752 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:53.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:53.758 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:53.761 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:01:53.790 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:53 smithi163 ceph-mon[29621]: pgmap v114: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 375 B/s rd, 52 MiB/s wr, 826 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-21T13:01:53.790 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:53 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1858210780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:01:53.792 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:53 smithi163 ceph-mon[40675]: pgmap v114: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 375 B/s rd, 52 MiB/s wr, 826 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-21T13:01:53.792 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:53 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1858210780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:01:53.805 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:01:53.851 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:01:53.854 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:53.903 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:53.903 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:01:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:53.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:53.947 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:01:53.994 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:01:54.005 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:54.039 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:01:54.049 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:01:54.049 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:01:54.050 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-21T13:01:54.050 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-21T13:01:54.050 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-21T13:01:54.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-21T13:01:54.051 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:54.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:54.079 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:01:54.082 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:01:54.083126+0000 min lat: 0.00109429 max lat: 1.05049 avg lat: 0.0107618 2024-03-21T13:01:54.082 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:01:54.082 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 60 16 89192 89176 92.8807 64.875 0.00937124 0.0107618 2024-03-21T13:01:54.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:53 smithi204 ceph-mon[33153]: pgmap v114: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 375 B/s rd, 52 MiB/s wr, 826 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-21T13:01:54.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:53 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1858210780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:01:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:01:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:01:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:01:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:01:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:01:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:01:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:01:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:01:54.153 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:54.183 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:54.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:54.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:54.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:54.326 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:54.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:54.412 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:01:54.427 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:54.457 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:54.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:54.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:54.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:54.603 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:54.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:54.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:54.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:01:54.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:54.735 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:54.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:01:54.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:54.868 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:54.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:54.970 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:55.011 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:55.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:55.082 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 61 16 90218 90202 92.4092 64.125 0.0123963 0.0108166 2024-03-21T13:01:55.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:55.152 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:55.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:55.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:55.282 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:01:55.285 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:55.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:55.387 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:55.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:55.437 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:55.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:55.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:55.565 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:55.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:55.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:55.667 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:55.698 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:55.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:55.800 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:55.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:55.833 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:55.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:55.840 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:55 smithi163 ceph-mon[40675]: pgmap v115: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 20 KiB/s, 0 objects/s recovering 2024-03-21T13:01:55.840 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:55 smithi163 ceph-mon[29621]: pgmap v115: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 20 KiB/s, 0 objects/s recovering 2024-03-21T13:01:55.840 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:01:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:01:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:01:55.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:55.965 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:55.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:56.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:55 smithi204 ceph-mon[33153]: pgmap v115: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 20 KiB/s, 0 objects/s recovering 2024-03-21T13:01:56.067 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:56.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 62 16 91269 91253 91.9755 65.6875 0.0167425 0.0108682 2024-03-21T13:01:56.107 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:56.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:56.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:56.239 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:56.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:56.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:56.341 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:56.372 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:56.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:56.474 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:56.517 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:56.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:56.619 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:56.669 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:56.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:56.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:56.793 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:56.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:56.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:56.895 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:56.937 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:56.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:57.038 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:57.083 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:57.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:57.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:57.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 63 16 92325 92309 91.5631 66 0.013985 0.0109156 2024-03-21T13:01:57.163 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:01:57.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:57.210 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:01:57.210 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:01:57.210 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.210 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:92426 7.5 e113/92 7:a4508a73:::benchmark_data_smithi163_80312_object92425:head v 113'5795, mlcod=113'5793)", 2024-03-21T13:01:57.210 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:57.205702+0000", 2024-03-21T13:01:57.210 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.002899974, 2024-03-21T13:01:57.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00294173, 2024-03-21T13:01:57.210 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:57.210 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.205702+0000", 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.205702+0000", 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:57.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.205703+0000", 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.018e-06 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.205738+0000", 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5589000000000001e-05 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.205757+0000", 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8732999999999998e-05 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:57.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.205767+0000", 2024-03-21T13:01:57.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8130000000000007e-06 2024-03-21T13:01:57.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.213 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:57.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.205873+0000", 2024-03-21T13:01:57.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000106425 2024-03-21T13:01:57.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.213 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:57.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.205897+0000", 2024-03-21T13:01:57.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3898999999999999e-05 2024-03-21T13:01:57.213 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:57.214 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:57.214 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:57.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.214 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:92424 7.2 e113/92)", 2024-03-21T13:01:57.214 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:57.208293+0000", 2024-03-21T13:01:57.214 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.00030839700000000002, 2024-03-21T13:01:57.214 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00041797200000000002, 2024-03-21T13:01:57.214 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:57.214 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "queued for pg", 2024-03-21T13:01:57.214 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:57.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:57.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208293+0000", 2024-03-21T13:01:57.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:57.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:57.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208293+0000", 2024-03-21T13:01:57.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:57.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:57.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208295+0000", 2024-03-21T13:01:57.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1680000000000001e-06 2024-03-21T13:01:57.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:57.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208326+0000", 2024-03-21T13:01:57.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1350000000000003e-05 2024-03-21T13:01:57.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:57.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208344+0000", 2024-03-21T13:01:57.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8071e-05 2024-03-21T13:01:57.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:57.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208353+0000", 2024-03-21T13:01:57.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.721e-06 2024-03-21T13:01:57.217 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:57.217 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:57.217 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:57.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.217 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:92422 7.3 e113/92)", 2024-03-21T13:01:57.217 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:57.208384+0000", 2024-03-21T13:01:57.217 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.00021789199999999999, 2024-03-21T13:01:57.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00037529099999999998, 2024-03-21T13:01:57.217 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:57.217 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "queued for pg", 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208384+0000", 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208384+0000", 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:57.218 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208384+0000", 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9700000000000003e-07 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208414+0000", 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9504000000000001e-05 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208420+0000", 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5980000000000002e-06 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208426+0000", 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2519999999999999e-06 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:92429 7.3 e113/92)", 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:57.208441+0000", 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.00016097099999999999, 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00036344799999999999, 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "queued for pg", 2024-03-21T13:01:57.220 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208441+0000", 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208441+0000", 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208441+0000", 2024-03-21T13:01:57.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7500000000000001e-07 2024-03-21T13:01:57.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:57.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208465+0000", 2024-03-21T13:01:57.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3543000000000001e-05 2024-03-21T13:01:57.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:57.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208470+0000", 2024-03-21T13:01:57.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7130000000000002e-06 2024-03-21T13:01:57.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:57.223 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.208475+0000", 2024-03-21T13:01:57.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4700000000000004e-06 2024-03-21T13:01:57.223 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:57.223 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:57.223 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:57.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.223 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:92418 7.e 7:77ffa720:::benchmark_data_smithi163_80312_object92417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:01:57.223 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:57.193540+0000", 2024-03-21T13:01:57.223 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.015061873999999999, 2024-03-21T13:01:57.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015326529, 2024-03-21T13:01:57.223 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:57.223 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92418 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193540+0000", 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:57.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193540+0000", 2024-03-21T13:01:57.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:57.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:57.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193542+0000", 2024-03-21T13:01:57.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.838e-06 2024-03-21T13:01:57.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:57.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193578+0000", 2024-03-21T13:01:57.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6146000000000003e-05 2024-03-21T13:01:57.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:57.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193597+0000", 2024-03-21T13:01:57.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8913999999999999e-05 2024-03-21T13:01:57.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:57.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193606+0000", 2024-03-21T13:01:57.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2e-06 2024-03-21T13:01:57.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:57.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193735+0000", 2024-03-21T13:01:57.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000128718 2024-03-21T13:01:57.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:57.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193895+0000", 2024-03-21T13:01:57.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016057300000000001 2024-03-21T13:01:57.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-03-21T13:01:57.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.194079+0000", 2024-03-21T13:01:57.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000183449 2024-03-21T13:01:57.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:57.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.203888+0000", 2024-03-21T13:01:57.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0098088749999999999 2024-03-21T13:01:57.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:57.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.203935+0000", 2024-03-21T13:01:57.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7620999999999997e-05 2024-03-21T13:01:57.229 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:57.229 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:57.229 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:57.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.229 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:92419 7.e 7:73282612:::benchmark_data_smithi163_80312_object92418:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:01:57.229 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:57.193631+0000", 2024-03-21T13:01:57.229 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.014970595, 2024-03-21T13:01:57.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015332533000000001, 2024-03-21T13:01:57.229 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92419 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193631+0000", 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:01:57.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193631+0000", 2024-03-21T13:01:57.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:01:57.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:01:57.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193632+0000", 2024-03-21T13:01:57.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9299999999999998e-07 2024-03-21T13:01:57.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:01:57.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193655+0000", 2024-03-21T13:01:57.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3054999999999999e-05 2024-03-21T13:01:57.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.232 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:01:57.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193666+0000", 2024-03-21T13:01:57.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1124999999999999e-05 2024-03-21T13:01:57.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.232 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:01:57.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.193672+0000", 2024-03-21T13:01:57.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4860000000000002e-06 2024-03-21T13:01:57.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.232 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:01:57.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.197317+0000", 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0036449899999999999 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.197446+0000", 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000128486 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.197589+0000", 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014275900000000001 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:01:57.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.203925+0000", 2024-03-21T13:01:57.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00633623 2024-03-21T13:01:57.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:01:57.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:01:57.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:01:57.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:57.203939+0000", 2024-03-21T13:01:57.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4435e-05 2024-03-21T13:01:57.234 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:57.234 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:01:57.234 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:57.234 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:01:57.234 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:01:57.234 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 6 2024-03-21T13:01:57.235 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:01:57.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:57.236 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:57.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:57.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:57.384 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:57.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:57.486 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:57.516 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:57.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:57.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:57.618 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:57.659 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:57.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:57.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:57.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:57.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:57.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:57.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:57.896 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:57.937 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:57.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:57.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:58.039 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:58.071 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:58.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:58.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 64 16 93405 93389 91.1871 67.5 0.0119143 0.0109618 2024-03-21T13:01:58.173 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:58.210 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:58.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:58.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:58.312 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:01:58.330 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:01:58.330 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-21T13:01:58.330 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:01:58.356 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:58.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:01:58.458 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:58.500 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:58.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:58.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:01:58 smithi204 ceph-mon[33153]: pgmap v116: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 895 B/s rd, 77 MiB/s wr, 1.24k op/s; 16 KiB/s, 0 objects/s recovering 2024-03-21T13:01:58.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:01:58 smithi163 ceph-mon[29621]: pgmap v116: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 895 B/s rd, 77 MiB/s wr, 1.24k op/s; 16 KiB/s, 0 objects/s recovering 2024-03-21T13:01:58.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:01:58 smithi163 ceph-mon[40675]: pgmap v116: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 895 B/s rd, 77 MiB/s wr, 1.24k op/s; 16 KiB/s, 0 objects/s recovering 2024-03-21T13:01:58.571 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:01:58.601 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:01:58.633 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:58.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:58.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:01:58.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:58.781 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:58.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:58.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:58.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:01:58.918 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:58.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:01:59.020 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:59.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:59.052 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:59.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:59.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 65 16 94492 94476 90.8293 67.9375 0.0471271 0.0110025 2024-03-21T13:01:59.153 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:59.190 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:59.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:59.292 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:59.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:59.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:59.418 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:01:59.461 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:59.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:01:59.563 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:01:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:59.611 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:59.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:01:59.712 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:01:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:59.745 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:01:59.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:01:59.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:01:59.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:01:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:01:59.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:01:59.980 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:00.024 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:00.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:00.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 66 16 95414 95398 90.3262 57.625 0.00418137 0.011066 2024-03-21T13:02:00.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:00.158 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:00.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:00.259 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:00.295 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:00.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:00.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:00.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:00.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:00.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:00.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:00 smithi204 ceph-mon[33153]: pgmap v117: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 672 op/s 2024-03-21T13:02:00.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:00 smithi163 ceph-mon[29621]: pgmap v117: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 672 op/s 2024-03-21T13:02:00.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:00 smithi163 ceph-mon[40675]: pgmap v117: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 672 op/s 2024-03-21T13:02:00.529 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:00.568 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:00.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:00.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:00.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:00.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:00.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:00.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:00.797 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:00.836 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:00.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:00.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:00.938 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:00.979 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:00.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:00.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:01.081 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:01.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 67 16 96429 96413 89.9248 63.4375 0.0133606 0.0111156 2024-03-21T13:02:01.117 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:01.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:01.219 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:01.251 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:01.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:01.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:01.365 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-21T13:02:01.380 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:01.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:01.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:01.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:01.510 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:01.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:01.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:01.611 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:01.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:01.656 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:01.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:01.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:02:01 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:02:01.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:01.788 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:01.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:01.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:01.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:01.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:01.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:02.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:02.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:02.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:02.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:02.085 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 68 16 97502 97486 89.5885 67.0625 0.0109462 0.0111575 2024-03-21T13:02:02.159 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:02.201 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:02.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:02.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:02 smithi163 ceph-mon[29621]: pgmap v118: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:02:02.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:02 smithi163 ceph-mon[40675]: pgmap v118: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:02:02.302 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:02.332 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:02.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:02.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:02.434 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:02.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:02.502 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:02.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:02.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:02 smithi204 ceph-mon[33153]: pgmap v118: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:02:02.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:02.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:02.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:02.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:02.734 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:02.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:02.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:02.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:02.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:02.911 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:02.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:03.013 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:03.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:03.058 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:03.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:03.085 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 69 16 98572 98556 89.2593 66.875 0.0265419 0.0111987 2024-03-21T13:02:03.118 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:03.121 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:03.159 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:03.220 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:03.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:03.321 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:03.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:03.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:03.452 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:03.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:03.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:03.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:03.500 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:03.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:03.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:03.600 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:03.638 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:03.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:03.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:03.770 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:03.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:03.872 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:03.912 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:03.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:03.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:04.014 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:04.060 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:04.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:04.085 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 70 16 99606 99590 88.9073 64.625 0.0183217 0.0112428 2024-03-21T13:02:04.125 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:02:04.125 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:02:04.125 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:02:04.125 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd out 1 2024-03-21T13:02:04.161 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:04.210 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:04.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:04.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:04.247 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:02:04.311 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:04.342 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:04.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:04.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:04.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:04.477 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:04.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:04.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:04.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:04 smithi163 ceph-mon[29621]: pgmap v119: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 893 B/s rd, 58 MiB/s wr, 921 op/s 2024-03-21T13:02:04.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:04 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3298427025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:02:04.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:04 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:02:04.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:04 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:04.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:04 smithi163 ceph-mon[40675]: pgmap v119: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 893 B/s rd, 58 MiB/s wr, 921 op/s 2024-03-21T13:02:04.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:04 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3298427025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:02:04.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:04 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:02:04.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:04 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:04.579 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:04.619 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:04.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:04.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:04.721 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:04.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:04 smithi204 ceph-mon[33153]: pgmap v119: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 893 B/s rd, 58 MiB/s wr, 921 op/s 2024-03-21T13:02:04.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:04 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3298427025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:02:04.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:04 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:02:04.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:04 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:04.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:04.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:04.856 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:04.894 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:04.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:04.996 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:05.037 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:05.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:05.085 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 71 16 100634 100618 88.5599 64.25 0.014322 0.0112867 2024-03-21T13:02:05.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:05.168 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:05.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:05.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:05.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:05.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:05.404 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:05.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:05.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:05.542 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:05.591 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:05.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:05.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:05 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:02:05.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:05 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3298427025' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:02:05.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:05 smithi163 ceph-mon[29621]: osdmap e114: 8 total, 8 up, 8 in 2024-03-21T13:02:05.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:05 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:02:05.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:05 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3298427025' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:02:05.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:05 smithi163 ceph-mon[40675]: osdmap e114: 8 total, 8 up, 8 in 2024-03-21T13:02:05.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:05.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:05.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:05 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:02:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:05 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3298427025' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:02:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:05 smithi204 ceph-mon[33153]: osdmap e114: 8 total, 8 up, 8 in 2024-03-21T13:02:05.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:05.865 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:05.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:05.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:05.961 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:02:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:02:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:02:05.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:06.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:06.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:06.086 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 72 16 101752 101736 88.3004 69.875 0.0216218 0.0113198 2024-03-21T13:02:06.103 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:06.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:06.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:06.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:06.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:06.266 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:06.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:06.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:06.368 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:06.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:06.405 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:06.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:06.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:06.520 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:06 smithi163 ceph-mon[29621]: pgmap v121: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-21T13:02:06.520 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:06 smithi163 ceph-mon[40675]: pgmap v121: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-21T13:02:06.536 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:06.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:06.638 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:06.700 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:06.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:06.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:06 smithi204 ceph-mon[33153]: pgmap v121: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-21T13:02:06.803 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:06.831 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:06.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:06.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:06.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:06.965 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:06.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:07.067 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:07.086 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 73 16 102670 102654 87.8766 57.375 0.0140997 0.0113741 2024-03-21T13:02:07.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:07.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:07.100 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:02:07.100 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:02:07.101 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-03-21T13:02:07.101 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-03-21T13:02:07.101 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:02:07.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:07.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:07.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:07.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:07.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:07.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:07.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:07.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:07.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:07.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:07.591 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:07.627 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:07.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:07.633 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:07.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:07.720 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:02:07.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:07.761 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:02:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:07.783 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:07.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:07.810 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:02:07.854 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:02:07.884 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:07.896 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:02:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:07.930 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:07.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:07.931 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:02:07.981 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:02:08.009 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:02:08.019 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:02:08.021 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:02:08.021 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:02:08.021 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:02:08.021 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:02:08.021 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:02:08.021 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:02:08.021 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:02:08.022 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:02:08.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:08.085 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 74 16 103520 103504 87.4069 53.125 0.0161945 0.0114352 2024-03-21T13:02:08.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:08.087 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:08.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:08.188 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:08.233 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:08.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:08.333 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-21T13:02:08.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:08.381 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:08.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:08.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:08.483 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:08.512 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:08.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:08.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:08.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:08.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:08.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:08.725 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:08 smithi163 ceph-mon[29621]: pgmap v122: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T13:02:08.725 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:08 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2514356164' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:02:08.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:08 smithi163 ceph-mon[40675]: pgmap v122: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T13:02:08.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:08 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2514356164' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:02:08.744 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:08 smithi204 ceph-mon[33153]: pgmap v122: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T13:02:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:08 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2514356164' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:02:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:08.791 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:08.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:08.892 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:08.922 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:08.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:09.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:09.053 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:09.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:09.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:09.085 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 75 16 104518 104502 87.0731 62.375 0.00684726 0.0114795 2024-03-21T13:02:09.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:09.188 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:09.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:09.290 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:09.334 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:09.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:09.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:09.466 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:09.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:09.568 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:09.601 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:09.612 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:09.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:09.713 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:09.755 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:09.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:09.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:09.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:09.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:09.889 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:09.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:09.990 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:10.028 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:10.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:10.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:10.085 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 76 16 105610 105594 86.8253 68.25 0.0063794 0.0115126 2024-03-21T13:02:10.130 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:10.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:10.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:10.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:10.264 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:10.310 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:10.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:10.412 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:10.450 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:10.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:10.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:10.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:10 smithi204 ceph-mon[33153]: pgmap v123: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T13:02:10.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:10 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:02:10.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:10 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:02:10.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:10 smithi204 ceph-mon[33153]: osdmap e115: 8 total, 8 up, 8 in 2024-03-21T13:02:10.553 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:10.592 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:10.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:10.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:10.600 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:10.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:10.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:10 smithi163 ceph-mon[29621]: pgmap v123: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T13:02:10.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:10 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:02:10.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:10 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:02:10.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:10 smithi163 ceph-mon[29621]: osdmap e115: 8 total, 8 up, 8 in 2024-03-21T13:02:10.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:10 smithi163 ceph-mon[40675]: pgmap v123: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T13:02:10.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:10 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:02:10.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:10 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:02:10.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:10 smithi163 ceph-mon[40675]: osdmap e115: 8 total, 8 up, 8 in 2024-03-21T13:02:10.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:10.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:10.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:10.784 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:02:10.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:10.852 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:10.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:10.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:10.954 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:10.982 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:10.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:11.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:11.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 77 16 106596 106580 86.4974 61.625 0.00830405 0.0115557 2024-03-21T13:02:11.127 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:11.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:11.229 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:11.278 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:11.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:11.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:11.381 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:11.429 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:11.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:11.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:11.531 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:11.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:11.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:11.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:11.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:11.701 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:11.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:11.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:02:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:02:11.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:11.839 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:11.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:11.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:11.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:11.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:12.086 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:12.087 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 78 16 107668 107652 86.2473 67 0.0157513 0.0115891 2024-03-21T13:02:12.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:12.129 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:12.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:12.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:12.264 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:12.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:12.365 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:12.407 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:12.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:12.508 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:12.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:12 smithi163 ceph-mon[29621]: pgmap v125: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-21T13:02:12.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:12 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:02:12.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:12 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:02:12.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:12 smithi163 ceph-mon[40675]: pgmap v125: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-21T13:02:12.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:12 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:02:12.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:12 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:02:12.549 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:12.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:12.651 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:12.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:12.690 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:12.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:12.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:12 smithi204 ceph-mon[33153]: pgmap v125: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-21T13:02:12.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:12 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:02:12.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:12 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:02:12.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:12.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:12.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:12.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:12.925 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:12.967 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:12.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:13.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a[29617]: 2024-03-21T13:02:12.572+0000 7f8e1b000700 -1 mon.a@0(leader).osd e116 definitely_dead 0 2024-03-21T13:02:13.069 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:13.088 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 79 16 108687 108671 85.9597 63.6875 0.0175112 0.0116276 2024-03-21T13:02:13.097 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:13.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:13.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:13.199 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:13.243 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:13.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:13.321 INFO:teuthology.orchestra.run.smithi163.stderr:marked out osd.1. 2024-03-21T13:02:13.345 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:13.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:13.372 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:13.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:13.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:13.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:13.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:13.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:13 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:13.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:13 smithi163 ceph-mon[29621]: osdmap e116: 8 total, 8 up, 8 in 2024-03-21T13:02:13.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:13 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3590024640' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-03-21T13:02:13.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:13 smithi163 ceph-mon[29621]: Client client.admin marked osd.1 out, while it was still marked up 2024-03-21T13:02:13.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:13 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:13.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:13 smithi163 ceph-mon[40675]: osdmap e116: 8 total, 8 up, 8 in 2024-03-21T13:02:13.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:13 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3590024640' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-03-21T13:02:13.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:13 smithi163 ceph-mon[40675]: Client client.admin marked osd.1 out, while it was still marked up 2024-03-21T13:02:13.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:13.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:13.623 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:13.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:13.724 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:13.751 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:13.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:13.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:13 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:13.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:13 smithi204 ceph-mon[33153]: osdmap e116: 8 total, 8 up, 8 in 2024-03-21T13:02:13.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:13 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3590024640' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-03-21T13:02:13.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:13 smithi204 ceph-mon[33153]: Client client.admin marked osd.1 out, while it was still marked up 2024-03-21T13:02:13.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:13.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:13.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:13.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:13.980 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:14.004 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:14.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:14.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:14.088 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:02:14.089192+0000 min lat: 0.00109429 max lat: 1.05049 avg lat: 0.011636 2024-03-21T13:02:14.088 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:02:14.089 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 80 16 108976 108960 85.111 18.0625 0.0030447 0.011636 2024-03-21T13:02:14.106 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:14.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:14.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:14.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:14.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:14.262 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:14.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:14.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:14.364 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:14.396 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:14.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:14.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:14.497 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:14.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:14 smithi163 ceph-mon[29621]: pgmap v127: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 945 op/s 2024-03-21T13:02:14.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:14 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3590024640' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-03-21T13:02:14.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:14 smithi163 ceph-mon[29621]: osdmap e117: 8 total, 8 up, 7 in 2024-03-21T13:02:14.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:14 smithi163 ceph-mon[29621]: osdmap e118: 8 total, 8 up, 7 in 2024-03-21T13:02:14.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:14 smithi163 ceph-mon[40675]: pgmap v127: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 945 op/s 2024-03-21T13:02:14.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:14 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3590024640' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-03-21T13:02:14.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:14 smithi163 ceph-mon[40675]: osdmap e117: 8 total, 8 up, 7 in 2024-03-21T13:02:14.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:14 smithi163 ceph-mon[40675]: osdmap e118: 8 total, 8 up, 7 in 2024-03-21T13:02:14.521 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:14.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:14.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:14.623 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:14.648 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:14.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:14.660 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:14.750 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:14 smithi204 ceph-mon[33153]: pgmap v127: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 945 op/s 2024-03-21T13:02:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:14 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3590024640' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-03-21T13:02:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:14 smithi204 ceph-mon[33153]: osdmap e117: 8 total, 8 up, 7 in 2024-03-21T13:02:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:14 smithi204 ceph-mon[33153]: osdmap e118: 8 total, 8 up, 7 in 2024-03-21T13:02:14.774 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:14.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:14.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:14.875 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:14.902 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:14.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:14.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:15.003 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:15.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:15.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:15.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:15.088 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 81 16 108976 108960 84.0603 0 - 0.011636 2024-03-21T13:02:15.134 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:15.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:15.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:15.266 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:15.290 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:15.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:15.392 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:15.424 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:15.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:15.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:15.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:15.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:15.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:15.660 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:15.700 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:15.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:15.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:15.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:02:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:02:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:02:15.802 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:15.843 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:15.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:15.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:15.945 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:15.978 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:15.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:15.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:16.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:16.088 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 82 16 109640 109624 83.5413 20.75 0.00852872 0.0119652 2024-03-21T13:02:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:16.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:16.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:16.220 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:16.256 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:16.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:16.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:16.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:16.395 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:16.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:16.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:16.496 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:16.538 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:16.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:16.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:16.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:16.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[29621]: pgmap v130: 148 pgs: 4 remapped+peering, 144 active+clean; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.5 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 279 B/s, 9 objects/s recovering 2024-03-21T13:02:16.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[29621]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 116'6769 2024-03-21T13:02:16.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[29621]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 116'6794 2024-03-21T13:02:16.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[29621]: osdmap e119: 8 total, 8 up, 7 in 2024-03-21T13:02:16.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[29621]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 116'6695 2024-03-21T13:02:16.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[29621]: 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 116'6870 2024-03-21T13:02:16.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[29621]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 116'6719 2024-03-21T13:02:16.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[29621]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 116'6891 2024-03-21T13:02:16.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[40675]: pgmap v130: 148 pgs: 4 remapped+peering, 144 active+clean; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.5 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 279 B/s, 9 objects/s recovering 2024-03-21T13:02:16.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[40675]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 116'6769 2024-03-21T13:02:16.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[40675]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 116'6794 2024-03-21T13:02:16.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[40675]: osdmap e119: 8 total, 8 up, 7 in 2024-03-21T13:02:16.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[40675]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 116'6695 2024-03-21T13:02:16.669 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[40675]: 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 116'6870 2024-03-21T13:02:16.669 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[40675]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 116'6719 2024-03-21T13:02:16.669 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:16 smithi163 ceph-mon[40675]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 116'6891 2024-03-21T13:02:16.670 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:16.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:16.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:16 smithi204 ceph-mon[33153]: pgmap v130: 148 pgs: 4 remapped+peering, 144 active+clean; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.5 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 279 B/s, 9 objects/s recovering 2024-03-21T13:02:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:16 smithi204 ceph-mon[33153]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 116'6769 2024-03-21T13:02:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:16 smithi204 ceph-mon[33153]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 116'6794 2024-03-21T13:02:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:16 smithi204 ceph-mon[33153]: osdmap e119: 8 total, 8 up, 7 in 2024-03-21T13:02:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:16 smithi204 ceph-mon[33153]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 116'6695 2024-03-21T13:02:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:16 smithi204 ceph-mon[33153]: 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 116'6870 2024-03-21T13:02:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:16 smithi204 ceph-mon[33153]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 116'6719 2024-03-21T13:02:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:16 smithi204 ceph-mon[33153]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 116'6891 2024-03-21T13:02:16.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:02:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:02:16.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:16.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:16.904 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:16.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:16.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:16.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:17.039 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:17.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:17.076 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:17.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:17.088 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 83 16 110694 110678 83.3284 65.875 0.00361116 0.0119935 2024-03-21T13:02:17.177 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:17.220 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:17.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:17.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:17.322 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:17.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:17.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:17.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:17.486 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:17.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:17.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:17.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:17.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:17.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:17.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:17 smithi163 ceph-mon[29621]: pgmap v132: 148 pgs: 6 remapped+peering, 142 active+clean; 6.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 234 B/s rd, 54 MiB/s wr, 869 op/s; 651 B/s, 25 objects/s recovering 2024-03-21T13:02:17.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:17 smithi163 ceph-mon[40675]: pgmap v132: 148 pgs: 6 remapped+peering, 142 active+clean; 6.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 234 B/s rd, 54 MiB/s wr, 869 op/s; 651 B/s, 25 objects/s recovering 2024-03-21T13:02:17.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:17 smithi204 ceph-mon[33153]: pgmap v132: 148 pgs: 6 remapped+peering, 142 active+clean; 6.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 234 B/s rd, 54 MiB/s wr, 869 op/s; 651 B/s, 25 objects/s recovering 2024-03-21T13:02:17.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:17.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:17.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:17.857 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:17.902 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:17.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:18.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:18.045 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:18.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:18.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:18.088 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 84 15 111664 111649 83.0588 60.6875 0.00982452 0.0120347 2024-03-21T13:02:18.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:18.184 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:18.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:18.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:18.286 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:18.323 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:18.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:18.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:18.368 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:02:18.425 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:18.468 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:18.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:18.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:18.539 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-21T13:02:18.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:18.583 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:18 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2845103770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:02:18.584 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:18 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:02:18.584 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:18 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2845103770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:02:18.584 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:18 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:02:18.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:18.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:18.704 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:18.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:18.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:18.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:18 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2845103770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:02:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:18 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:02:18.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:18.869 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:18.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:18.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:18.971 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:19.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:19.020 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:19.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:19.089 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 85 16 112697 112681 82.8404 64.5 0.00484278 0.0120664 2024-03-21T13:02:19.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:19.154 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:19.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:19.223 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:02:19.223 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:02:19.223 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 3, 4, 5, 6, 7] 2024-03-21T13:02:19.223 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd out 5 2024-03-21T13:02:19.255 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:19.288 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:19.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:19.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:19.422 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:19.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:19.524 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:19.537 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:02:19.537 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:02:19.538 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[40675]: osdmap e120: 8 total, 8 up, 7 in 2024-03-21T13:02:19.538 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:02:19.538 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:02:19.538 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[40675]: pgmap v134: 148 pgs: 6 remapped+peering, 142 active+clean; 6.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 191 B/s rd, 44 MiB/s wr, 707 op/s; 530 B/s, 20 objects/s recovering 2024-03-21T13:02:19.538 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:19.541 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:02:19.541 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:02:19.542 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[29621]: osdmap e120: 8 total, 8 up, 7 in 2024-03-21T13:02:19.542 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:02:19.542 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:02:19.542 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[29621]: pgmap v134: 148 pgs: 6 remapped+peering, 142 active+clean; 6.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 191 B/s rd, 44 MiB/s wr, 707 op/s; 530 B/s, 20 objects/s recovering 2024-03-21T13:02:19.542 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:19.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:19.590 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:19.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:19.675 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:02:19.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:19.705 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:19.720 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:02:19.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:19.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:19.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:19.737 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:19 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:02:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:19 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:02:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:19 smithi204 ceph-mon[33153]: osdmap e120: 8 total, 8 up, 7 in 2024-03-21T13:02:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:19 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:02:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:19 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:02:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:19 smithi204 ceph-mon[33153]: pgmap v134: 148 pgs: 6 remapped+peering, 142 active+clean; 6.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 191 B/s rd, 44 MiB/s wr, 707 op/s; 530 B/s, 20 objects/s recovering 2024-03-21T13:02:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:19.761 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:02:19.809 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:02:19.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:19.847 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:02:19.859 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:19.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:19.893 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:02:19.934 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:02:19.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:19.972 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:02:19.981 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:02:19.981 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:02:19.981 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:02:19.981 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:02:19.981 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:02:19.981 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:02:19.981 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:02:19.981 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:02:19.994 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:19.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:19.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:20.089 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 86 16 113597 113581 82.5312 56.25 0.00665364 0.0121117 2024-03-21T13:02:20.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:20.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:20.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:20.234 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:20.275 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:20.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:20.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:20.411 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:20.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:20.513 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:20.545 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:20.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:20.647 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:20.687 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:20.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:20.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:20 smithi204 ceph-mon[33153]: osdmap e121: 8 total, 8 up, 7 in 2024-03-21T13:02:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:20 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/759188812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:02:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:20 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:02:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:20 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:02:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:20 smithi163 ceph-mon[29621]: osdmap e121: 8 total, 8 up, 7 in 2024-03-21T13:02:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:20 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/759188812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:02:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:20 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:02:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:20 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:02:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:20 smithi163 ceph-mon[40675]: osdmap e121: 8 total, 8 up, 7 in 2024-03-21T13:02:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:20 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/759188812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:02:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:20 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:02:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:20 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:02:20.789 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:20.833 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:20.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:20.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:20.979 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:20.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:21.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:21.089 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 87 16 114666 114650 82.3505 66.8125 0.00969014 0.0121385 2024-03-21T13:02:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:21.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:21.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:21.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:21.255 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:21.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:21.344 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:02:21.356 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:21.449 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:21.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:21.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:21 smithi163 ceph-mon[29621]: osdmap e122: 8 total, 8 up, 7 in 2024-03-21T13:02:21.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:21 smithi163 ceph-mon[29621]: pgmap v137: 148 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 142 active+clean; 6.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 956 B/s rd, 50 MiB/s wr, 797 op/s; 40031/340290 objects misplaced (11.764%); 27 MiB/s, 435 objects/s recovering 2024-03-21T13:02:21.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:21 smithi163 ceph-mon[40675]: osdmap e122: 8 total, 8 up, 7 in 2024-03-21T13:02:21.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:21 smithi163 ceph-mon[40675]: pgmap v137: 148 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 142 active+clean; 6.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 956 B/s rd, 50 MiB/s wr, 797 op/s; 40031/340290 objects misplaced (11.764%); 27 MiB/s, 435 objects/s recovering 2024-03-21T13:02:21.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:21.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:21.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:21.684 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:21.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:21.734 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:21.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:21.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:21 smithi204 ceph-mon[33153]: osdmap e122: 8 total, 8 up, 7 in 2024-03-21T13:02:21.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:21 smithi204 ceph-mon[33153]: pgmap v137: 148 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 142 active+clean; 6.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 956 B/s rd, 50 MiB/s wr, 797 op/s; 40031/340290 objects misplaced (11.764%); 27 MiB/s, 435 objects/s recovering 2024-03-21T13:02:21.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:02:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:02:21.835 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:21.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:21.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:22.024 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:22.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:22.089 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 88 16 115655 115639 82.1171 61.8125 0.0192902 0.012173 2024-03-21T13:02:22.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:22.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:22.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:22.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:22.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:22.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:22.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:22.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:22.430 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:22.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:22.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:22.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:22.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:22.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:22.692 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:22.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:22.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:22.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:22 smithi204 ceph-mon[33153]: 3.16 scrub starts 2024-03-21T13:02:22.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:22 smithi204 ceph-mon[33153]: 3.16 scrub ok 2024-03-21T13:02:22.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:22 smithi204 ceph-mon[33153]: 3.0 scrub starts 2024-03-21T13:02:22.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:22 smithi204 ceph-mon[33153]: 3.0 scrub ok 2024-03-21T13:02:22.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:22 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:22.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:22 smithi204 ceph-mon[33153]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:02:22.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:22 smithi163 ceph-mon[29621]: 3.16 scrub starts 2024-03-21T13:02:22.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:22 smithi163 ceph-mon[29621]: 3.16 scrub ok 2024-03-21T13:02:22.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:22 smithi163 ceph-mon[29621]: 3.0 scrub starts 2024-03-21T13:02:22.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:22 smithi163 ceph-mon[29621]: 3.0 scrub ok 2024-03-21T13:02:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:22 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:22 smithi163 ceph-mon[29621]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:02:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:22 smithi163 ceph-mon[40675]: 3.16 scrub starts 2024-03-21T13:02:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:22 smithi163 ceph-mon[40675]: 3.16 scrub ok 2024-03-21T13:02:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:22 smithi163 ceph-mon[40675]: 3.0 scrub starts 2024-03-21T13:02:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:22 smithi163 ceph-mon[40675]: 3.0 scrub ok 2024-03-21T13:02:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:22 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:22 smithi163 ceph-mon[40675]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:02:22.794 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:22.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:22.819 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:22.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:22.920 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:22.966 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:22.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:22.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:23.068 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:23.090 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 89 16 116499 116483 81.7863 52.75 0.0214263 0.0122216 2024-03-21T13:02:23.106 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:23.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:23.208 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:23.245 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:23.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:23.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:23.347 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:23.386 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:23.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:23.487 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:23.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:23 smithi204 ceph-mon[33153]: 7.e scrub starts 2024-03-21T13:02:23.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:23 smithi204 ceph-mon[33153]: pgmap v138: 148 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 142 active+clean; 6.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 44 MiB/s wr, 711 op/s; 40031/340290 objects misplaced (11.764%); 24 MiB/s, 388 objects/s recovering 2024-03-21T13:02:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:23.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:23.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:23.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:23 smithi163 ceph-mon[29621]: 7.e scrub starts 2024-03-21T13:02:23.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:23 smithi163 ceph-mon[29621]: pgmap v138: 148 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 142 active+clean; 6.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 44 MiB/s wr, 711 op/s; 40031/340290 objects misplaced (11.764%); 24 MiB/s, 388 objects/s recovering 2024-03-21T13:02:23.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:23 smithi163 ceph-mon[40675]: 7.e scrub starts 2024-03-21T13:02:23.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:23 smithi163 ceph-mon[40675]: pgmap v138: 148 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 142 active+clean; 6.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 44 MiB/s wr, 711 op/s; 40031/340290 objects misplaced (11.764%); 24 MiB/s, 388 objects/s recovering 2024-03-21T13:02:23.666 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:23.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:23.768 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:23.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:23.804 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:23.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:23.905 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:23.946 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:23.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:23.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:24.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:24.081 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:24.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:24.090 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Total time run: 90.0059 2024-03-21T13:02:24.090 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Total writes made: 117438 2024-03-21T13:02:24.090 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Write size: 65536 2024-03-21T13:02:24.090 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Object size: 65536 2024-03-21T13:02:24.090 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Bandwidth (MB/sec): 81.5488 2024-03-21T13:02:24.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev Bandwidth: 33.0807 2024-03-21T13:02:24.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max bandwidth (MB/sec): 180.938 2024-03-21T13:02:24.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min bandwidth (MB/sec): 0 2024-03-21T13:02:24.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Average IOPS: 1304 2024-03-21T13:02:24.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev IOPS: 529.291 2024-03-21T13:02:24.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max IOPS: 2895 2024-03-21T13:02:24.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min IOPS: 0 2024-03-21T13:02:24.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Average Latency(s): 0.0122586 2024-03-21T13:02:24.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev Latency(s): 0.0290622 2024-03-21T13:02:24.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max latency(s): 2.23262 2024-03-21T13:02:24.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min latency(s): 0.00109429 2024-03-21T13:02:24.095 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Cleaning up (deleting benchmark objects) 2024-03-21T13:02:24.184 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:24.226 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:24.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:24.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:24.359 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:24.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:24.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:24.461 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:24.511 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:24.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:24.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:24.613 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:24.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:24.664 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:24.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:24.765 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:24.806 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:24.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:24.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:24.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:24.942 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:24.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:24.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:25.044 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:25.076 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:25.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:25.178 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:25.225 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:25.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:25.254 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:25.326 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:25.332 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:02:25.332 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:02:25.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.332 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118546 7.3 7:c0f32d8e:::benchmark_data_smithi163_80312_object1105:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:02:25.332 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.287900+0000", 2024-03-21T13:02:25.332 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.034424692, 2024-03-21T13:02:25.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034458967, 2024-03-21T13:02:25.332 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:02:25.332 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:02:25.332 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:02:25.332 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:02:25.332 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:02:25.333 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118546 2024-03-21T13:02:25.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.333 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:02:25.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:02:25.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.287900+0000", 2024-03-21T13:02:25.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:02:25.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:02:25.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.287900+0000", 2024-03-21T13:02:25.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:02:25.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:02:25.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.287902+0000", 2024-03-21T13:02:25.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2019999999999998e-06 2024-03-21T13:02:25.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:02:25.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.287909+0000", 2024-03-21T13:02:25.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5240000000000003e-06 2024-03-21T13:02:25.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:02:25.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.287924+0000", 2024-03-21T13:02:25.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4856e-05 2024-03-21T13:02:25.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:02:25.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.287933+0000", 2024-03-21T13:02:25.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2250000000000006e-06 2024-03-21T13:02:25.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:02:25.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.289239+0000", 2024-03-21T13:02:25.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013060249999999999 2024-03-21T13:02:25.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:02:25.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.289390+0000", 2024-03-21T13:02:25.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000151094 2024-03-21T13:02:25.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4", 2024-03-21T13:02:25.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.289534+0000", 2024-03-21T13:02:25.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014325699999999999 2024-03-21T13:02:25.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:02:25.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.303411+0000", 2024-03-21T13:02:25.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01387683 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.318151+0000", 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014740145 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118561 7.a 7:5a148583:::benchmark_data_smithi163_80312_object1120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.315569+0000", 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.0067552289999999997, 2024-03-21T13:02:25.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0068967469999999999, 2024-03-21T13:02:25.338 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:02:25.338 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:02:25.338 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:02:25.338 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:02:25.338 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:02:25.338 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118561 2024-03-21T13:02:25.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.338 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:02:25.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:02:25.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315569+0000", 2024-03-21T13:02:25.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315569+0000", 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315571+0000", 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2120000000000002e-06 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:02:25.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315582+0000", 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1e-05 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315601+0000", 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9190000000000001e-05 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315614+0000", 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2975e-05 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.316042+0000", 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00042713199999999999 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.316179+0000", 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000137319 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,5", 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.316350+0000", 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000171142 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.321870+0000", 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055199309999999996 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:118560 7.0 e122/118 7:0a7af37b:::benchmark_data_smithi163_80312_object1119:head v 122'7466, mlcod=122'7464)", 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.315666+0000", 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.006658494, 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0068990229999999998, 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:02:25.342 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315666+0000", 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315666+0000", 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315667+0000", 2024-03-21T13:02:25.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8299999999999996e-07 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315674+0000", 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0310000000000001e-06 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315682+0000", 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2810000000000008e-06 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:02:25.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315688+0000", 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2110000000000003e-06 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315836+0000", 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000148345 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.315863+0000", 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6122000000000002e-05 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:02:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.346 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118562 7.f 7:f49bbb4e:::benchmark_data_smithi163_80312_object1121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:02:25.346 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.318641+0000", 2024-03-21T13:02:25.346 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.003683357, 2024-03-21T13:02:25.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039955249999999998, 2024-03-21T13:02:25.346 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:02:25.346 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:02:25.346 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:02:25.346 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:02:25.346 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:02:25.346 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118562 2024-03-21T13:02:25.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.318641+0000", 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.318641+0000", 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:02:25.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.318643+0000", 2024-03-21T13:02:25.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6509999999999999e-06 2024-03-21T13:02:25.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:02:25.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.318649+0000", 2024-03-21T13:02:25.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9780000000000002e-06 2024-03-21T13:02:25.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:02:25.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.318659+0000", 2024-03-21T13:02:25.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0868e-05 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.318667+0000", 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7640000000000005e-06 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.318990+0000", 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032241300000000002 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:02:25.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.319133+0000", 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143516 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,7", 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.319263+0000", 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013027500000000001 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.321876+0000", 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026124640000000001 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:02:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:118566 7.7 e122/92 7:e0b75b75:::benchmark_data_smithi163_80312_object1125:head v 122'7534, mlcod=122'7532)", 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.322407+0000", 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4294967295.9999175, 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032338700000000001, 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322407+0000", 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:02:25.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322407+0000", 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322408+0000", 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8599999999999996e-07 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322424+0000", 2024-03-21T13:02:25.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6218e-05 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322437+0000", 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2901999999999999e-05 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322444+0000", 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2679999999999997e-06 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:02:25.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322702+0000", 2024-03-21T13:02:25.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025796800000000002 2024-03-21T13:02:25.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:02:25.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322720+0000", 2024-03-21T13:02:25.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7618999999999999e-05 2024-03-21T13:02:25.354 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:02:25.354 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:02:25.354 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:02:25.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.354 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:118567 7.c e122/92 7:3c685d60:::benchmark_data_smithi163_80312_object1126:head v 122'7406, mlcod=122'7404)", 2024-03-21T13:02:25.354 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.322536+0000", 2024-03-21T13:02:25.355 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4294967295.9997878, 2024-03-21T13:02:25.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00026485199999999999, 2024-03-21T13:02:25.355 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:02:25.355 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:02:25.355 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:02:25.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:02:25.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322536+0000", 2024-03-21T13:02:25.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:02:25.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:02:25.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322536+0000", 2024-03-21T13:02:25.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:02:25.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:02:25.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322537+0000", 2024-03-21T13:02:25.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.58e-07 2024-03-21T13:02:25.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:02:25.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322542+0000", 2024-03-21T13:02:25.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.288e-06 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322557+0000", 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4822e-05 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322563+0000", 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3280000000000002e-06 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:02:25.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322626+0000", 2024-03-21T13:02:25.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2612000000000003e-05 2024-03-21T13:02:25.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:02:25.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:02:25.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:02:25.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.322647+0000", 2024-03-21T13:02:25.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0696999999999999e-05 2024-03-21T13:02:25.358 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:02:25.358 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:02:25.358 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:02:25.358 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:02:25.358 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:02:25.358 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 6 2024-03-21T13:02:25.358 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:02:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:25.360 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:25.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:25.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:25.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:25.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:25.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:25.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:25.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a[29617]: 2024-03-21T13:02:25.492+0000 7f8e1b000700 -1 mon.a@0(leader).osd e122 definitely_dead 0 2024-03-21T13:02:25.639 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:25.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:25.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:25.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:25.743 INFO:teuthology.orchestra.run.smithi163.stderr:marked out osd.5. 2024-03-21T13:02:25.777 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:25.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:25.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:25.879 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:25.907 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:25.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:25.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:26.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:25 smithi204 ceph-mon[33153]: pgmap v139: 148 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 142 active+clean; 7.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 66 MiB/s wr, 1.11k op/s; 38718/346614 objects misplaced (11.170%); 44 MiB/s, 711 objects/s recovering 2024-03-21T13:02:26.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:25 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2909259858' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-21T13:02:26.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:25 smithi204 ceph-mon[33153]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-21T13:02:26.009 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:25 smithi163 ceph-mon[29621]: pgmap v139: 148 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 142 active+clean; 7.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 66 MiB/s wr, 1.11k op/s; 38718/346614 objects misplaced (11.170%); 44 MiB/s, 711 objects/s recovering 2024-03-21T13:02:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:25 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2909259858' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-21T13:02:26.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:25 smithi163 ceph-mon[29621]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-21T13:02:26.012 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:02:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:02:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:02:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:25 smithi163 ceph-mon[40675]: pgmap v139: 148 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 142 active+clean; 7.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 66 MiB/s wr, 1.11k op/s; 38718/346614 objects misplaced (11.170%); 44 MiB/s, 711 objects/s recovering 2024-03-21T13:02:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:25 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2909259858' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-21T13:02:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:25 smithi163 ceph-mon[40675]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-21T13:02:26.036 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:26.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:26.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:26.165 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:26.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:26.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:26.265 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:02:26.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:26.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:26.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:26.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:26.422 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:26.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:26.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:26.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:26.547 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:26.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:26.649 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:26.674 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:26.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:26.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:26.775 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:26.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:26.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:26.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:26.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:26.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:26.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:26 smithi204 ceph-mon[33153]: 7.8 scrub starts 2024-03-21T13:02:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:26 smithi204 ceph-mon[33153]: 7.e scrub ok 2024-03-21T13:02:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:26 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2909259858' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-21T13:02:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:26 smithi204 ceph-mon[33153]: osdmap e123: 8 total, 8 up, 6 in 2024-03-21T13:02:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:26 smithi204 ceph-mon[33153]: osdmap e124: 8 total, 8 up, 6 in 2024-03-21T13:02:27.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:26 smithi163 ceph-mon[29621]: 7.8 scrub starts 2024-03-21T13:02:27.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:26 smithi163 ceph-mon[29621]: 7.e scrub ok 2024-03-21T13:02:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:26 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2909259858' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-21T13:02:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:26 smithi163 ceph-mon[29621]: osdmap e123: 8 total, 8 up, 6 in 2024-03-21T13:02:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:26 smithi163 ceph-mon[29621]: osdmap e124: 8 total, 8 up, 6 in 2024-03-21T13:02:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:26 smithi163 ceph-mon[40675]: 7.8 scrub starts 2024-03-21T13:02:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:26 smithi163 ceph-mon[40675]: 7.e scrub ok 2024-03-21T13:02:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:26 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2909259858' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-21T13:02:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:26 smithi163 ceph-mon[40675]: osdmap e123: 8 total, 8 up, 6 in 2024-03-21T13:02:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:26 smithi163 ceph-mon[40675]: osdmap e124: 8 total, 8 up, 6 in 2024-03-21T13:02:27.028 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:27.052 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:27.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:27.153 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:27.178 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:27.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:27.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:27.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:27.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:27.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:27.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:27.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:27.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:27.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:27.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:27.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:27.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:27.676 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:27.720 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:27.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:27.824 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:27.882 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:27.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:27.984 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:28.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:27 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:02:28.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:27 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:02:28.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:27 smithi204 ceph-mon[33153]: 7.8 scrub ok 2024-03-21T13:02:28.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:27 smithi204 ceph-mon[33153]: pgmap v142: 148 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 7.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 975 B/s rd, 42 MiB/s wr, 767 op/s; 38477/351042 objects misplaced (10.961%); 28 MiB/s, 448 objects/s recovering 2024-03-21T13:02:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:27 smithi204 ceph-mon[33153]: 7.b continuing backfill to osd.2 from (100'2100,122'7389] MIN to 122'7389 2024-03-21T13:02:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:27 smithi204 ceph-mon[33153]: 7.b starting backfill to osd.4 from (0'0,0'0] MAX to 122'7389 2024-03-21T13:02:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:27 smithi204 ceph-mon[33153]: 7.0 continuing backfill to osd.2 from (100'2207,122'7487] MIN to 122'7487 2024-03-21T13:02:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:27 smithi204 ceph-mon[33153]: 7.0 starting backfill to osd.6 from (0'0,0'0] MAX to 122'7487 2024-03-21T13:02:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:27 smithi204 ceph-mon[33153]: osdmap e125: 8 total, 8 up, 6 in 2024-03-21T13:02:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:27 smithi204 ceph-mon[33153]: 7.a starting backfill to osd.4 from (0'0,0'0] MAX to 122'7358 2024-03-21T13:02:28.010 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-rgw-foo-a[67126]: 2024-03-21T13:02:27.814+0000 7f2400060700 -1 rgw watcher librados: RGWWatcher::handle_error cookie 94483560959232 err (110) Connection timed out 2024-03-21T13:02:28.010 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-rgw-foo-a[67126]: 2024-03-21T13:02:27.814+0000 7f2400060700 -1 rgw watcher librados: RGWWatcher::handle_error cookie 94483560950016 err (110) Connection timed out 2024-03-21T13:02:28.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:02:28.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:02:28.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[29621]: 7.8 scrub ok 2024-03-21T13:02:28.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[29621]: pgmap v142: 148 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 7.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 975 B/s rd, 42 MiB/s wr, 767 op/s; 38477/351042 objects misplaced (10.961%); 28 MiB/s, 448 objects/s recovering 2024-03-21T13:02:28.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[29621]: 7.b continuing backfill to osd.2 from (100'2100,122'7389] MIN to 122'7389 2024-03-21T13:02:28.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[29621]: 7.b starting backfill to osd.4 from (0'0,0'0] MAX to 122'7389 2024-03-21T13:02:28.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[29621]: 7.0 continuing backfill to osd.2 from (100'2207,122'7487] MIN to 122'7487 2024-03-21T13:02:28.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[29621]: 7.0 starting backfill to osd.6 from (0'0,0'0] MAX to 122'7487 2024-03-21T13:02:28.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[29621]: osdmap e125: 8 total, 8 up, 6 in 2024-03-21T13:02:28.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[29621]: 7.a starting backfill to osd.4 from (0'0,0'0] MAX to 122'7358 2024-03-21T13:02:28.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:02:28.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:02:28.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[40675]: 7.8 scrub ok 2024-03-21T13:02:28.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[40675]: pgmap v142: 148 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 7.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 975 B/s rd, 42 MiB/s wr, 767 op/s; 38477/351042 objects misplaced (10.961%); 28 MiB/s, 448 objects/s recovering 2024-03-21T13:02:28.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[40675]: 7.b continuing backfill to osd.2 from (100'2100,122'7389] MIN to 122'7389 2024-03-21T13:02:28.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[40675]: 7.b starting backfill to osd.4 from (0'0,0'0] MAX to 122'7389 2024-03-21T13:02:28.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[40675]: 7.0 continuing backfill to osd.2 from (100'2207,122'7487] MIN to 122'7487 2024-03-21T13:02:28.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[40675]: 7.0 starting backfill to osd.6 from (0'0,0'0] MAX to 122'7487 2024-03-21T13:02:28.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[40675]: osdmap e125: 8 total, 8 up, 6 in 2024-03-21T13:02:28.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:27 smithi163 ceph-mon[40675]: 7.a starting backfill to osd.4 from (0'0,0'0] MAX to 122'7358 2024-03-21T13:02:28.027 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:28.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:28.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:28.128 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:28.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:28.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:28.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:28.263 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:28.286 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-21T13:02:28.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:28.309 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:28.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:28.323 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:28.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:28.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:28.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:28.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:28.576 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:28.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:28.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:28.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:28.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:28.812 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:28.878 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:28.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:28.979 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:28 smithi163 ceph-mon[29621]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 122'7471 2024-03-21T13:02:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:28 smithi163 ceph-mon[29621]: 4.d scrub starts 2024-03-21T13:02:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:28 smithi163 ceph-mon[29621]: 4.d scrub ok 2024-03-21T13:02:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:28 smithi163 ceph-mon[40675]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 122'7471 2024-03-21T13:02:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:28 smithi163 ceph-mon[40675]: 4.d scrub starts 2024-03-21T13:02:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:28 smithi163 ceph-mon[40675]: 4.d scrub ok 2024-03-21T13:02:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:29.029 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:29.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:29.130 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:29.173 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:29.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:29.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:28 smithi204 ceph-mon[33153]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 122'7471 2024-03-21T13:02:29.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:28 smithi204 ceph-mon[33153]: 4.d scrub starts 2024-03-21T13:02:29.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:28 smithi204 ceph-mon[33153]: 4.d scrub ok 2024-03-21T13:02:29.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:29.321 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:29.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:29.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:29.423 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:29.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:29.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:29.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:29.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:29.595 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:29.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:29.696 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:29.740 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:29.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:29.842 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:29.885 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:29.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:29.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:29 smithi163 ceph-mon[29621]: pgmap v144: 148 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 7.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1023 B/s rd, 44 MiB/s wr, 805 op/s; 38477/351042 objects misplaced (10.961%); 29 MiB/s, 470 objects/s recovering 2024-03-21T13:02:29.976 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:29 smithi163 ceph-mon[40675]: pgmap v144: 148 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 7.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1023 B/s rd, 44 MiB/s wr, 805 op/s; 38477/351042 objects misplaced (10.961%); 29 MiB/s, 470 objects/s recovering 2024-03-21T13:02:29.986 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:30.027 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:30.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:30.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:30.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:30.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:30.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:30.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:29 smithi204 ceph-mon[33153]: pgmap v144: 148 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 7.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1023 B/s rd, 44 MiB/s wr, 805 op/s; 38477/351042 objects misplaced (10.961%); 29 MiB/s, 470 objects/s recovering 2024-03-21T13:02:30.264 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:30.302 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:30.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:30.322 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:30.404 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:30.438 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:30.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:30.540 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:30.581 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:30.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:30.683 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:30.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:30.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:30.784 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:02:30.821 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:30.858 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:30.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:30.960 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:31.000 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:31.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:31.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:31.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:31.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:31.135 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:31.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:31.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:30 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1981116083' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:02:31.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:30 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:02:31.136 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:30 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1981116083' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:02:31.136 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:30 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:02:31.235 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:31.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:30 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1981116083' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:02:31.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:30 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:02:31.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:31.289 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:31.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:31.390 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:31.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:31.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:31.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:31.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:31.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:31.668 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:31.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:31.709 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:31.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:31.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:02:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:02:31.810 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:31.843 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:31.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:31.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:31.944 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:31.987 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:31.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:31.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:32.089 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:32.129 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:32.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:32.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:31 smithi163 ceph-mon[29621]: pgmap v145: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 511 B/s rd, 19 MiB/s wr, 956 op/s; 62604/340119 objects misplaced (18.406%); 35 MiB/s, 568 objects/s recovering 2024-03-21T13:02:32.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:31 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:02:32.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:31 smithi163 ceph-mon[29621]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:02:32.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:31 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:02:32.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:31 smithi163 ceph-mon[29621]: osdmap e126: 8 total, 8 up, 6 in 2024-03-21T13:02:32.138 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:31 smithi163 ceph-mon[40675]: pgmap v145: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 511 B/s rd, 19 MiB/s wr, 956 op/s; 62604/340119 objects misplaced (18.406%); 35 MiB/s, 568 objects/s recovering 2024-03-21T13:02:32.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:31 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:02:32.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:31 smithi163 ceph-mon[40675]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:02:32.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:31 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:02:32.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:31 smithi163 ceph-mon[40675]: osdmap e126: 8 total, 8 up, 6 in 2024-03-21T13:02:32.231 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:32.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:31 smithi204 ceph-mon[33153]: pgmap v145: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 511 B/s rd, 19 MiB/s wr, 956 op/s; 62604/340119 objects misplaced (18.406%); 35 MiB/s, 568 objects/s recovering 2024-03-21T13:02:32.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:31 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:02:32.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:31 smithi204 ceph-mon[33153]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:02:32.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:31 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:02:32.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:31 smithi204 ceph-mon[33153]: osdmap e126: 8 total, 8 up, 6 in 2024-03-21T13:02:32.273 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:32.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:32.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:32.375 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:32.409 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:32.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:32.510 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:32.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:32.658 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:32.699 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:32.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:32.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:32.794 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:02:32.794 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:02:32.794 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-03-21T13:02:32.795 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:02:32.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:32.827 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:32.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:32.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:32.962 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:32.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:33.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:33.097 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:33.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:33.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:33.231 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:33.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:33.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:33.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:32 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:33.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:32 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:33.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:32 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:33.333 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:33.368 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:33.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:33.470 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:33.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:33.508 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:33.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:33.609 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:33.649 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:33.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:33.751 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:33.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:33.783 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:33.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:33.884 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:33.920 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:33.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:34.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:34.044 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:34.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:34.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:34.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:34.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:34.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:33 smithi204 ceph-mon[33153]: pgmap v147: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 474 B/s rd, 0 B/s wr, 563 op/s; 62604/340119 objects misplaced (18.406%); 26 MiB/s, 426 objects/s recovering 2024-03-21T13:02:34.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:33 smithi163 ceph-mon[29621]: pgmap v147: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 474 B/s rd, 0 B/s wr, 563 op/s; 62604/340119 objects misplaced (18.406%); 26 MiB/s, 426 objects/s recovering 2024-03-21T13:02:34.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:33 smithi163 ceph-mon[40675]: pgmap v147: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 474 B/s rd, 0 B/s wr, 563 op/s; 62604/340119 objects misplaced (18.406%); 26 MiB/s, 426 objects/s recovering 2024-03-21T13:02:34.281 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:34.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:34.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:34.417 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:34.458 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:34.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:34.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:34.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:34.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:34.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:34.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:34.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:34.721 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:34.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:34.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:34.823 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:34.859 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:34.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:34.960 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:34.989 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:34.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:35.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:35.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:35.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:35.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:35.176 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:34 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:02:35.176 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:34 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:35.176 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:34 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:02:35.176 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:34 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:35.227 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:35.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:34 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:02:35.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:34 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:35.256 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:35.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:35.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:35.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:35.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:35.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:35.490 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:35.535 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:35.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:35.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:35.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:35.660 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:35.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:35.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:35.762 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:35.804 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:35.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:35.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:35.865 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:02:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:02:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:02:35.906 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:35.948 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:35.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:36.051 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:36.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:36.084 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:36.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:36.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:35 smithi163 ceph-mon[29621]: pgmap v148: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 777 op/s; 60373/332385 objects misplaced (18.164%); 26 MiB/s, 421 objects/s recovering 2024-03-21T13:02:36.151 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:35 smithi163 ceph-mon[40675]: pgmap v148: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 777 op/s; 60373/332385 objects misplaced (18.164%); 26 MiB/s, 421 objects/s recovering 2024-03-21T13:02:36.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:36.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:36.230 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:36.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:36.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:35 smithi204 ceph-mon[33153]: pgmap v148: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 777 op/s; 60373/332385 objects misplaced (18.164%); 26 MiB/s, 421 objects/s recovering 2024-03-21T13:02:36.331 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:36.372 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:36.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:36.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:36.474 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:36.499 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:36.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:36.601 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:36.648 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:36.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:36.749 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:36.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:36.788 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:36.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:36.889 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:36.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:36.932 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:36.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:37.034 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:37.076 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:37.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:37.177 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:37.216 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:37.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:37.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:37.319 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:37.367 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:37.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:37.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:37.504 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:37.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:37.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:37.605 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:37.635 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:37.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:37.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:37.767 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:37.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:37.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:37.869 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:37.900 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:37.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:38.002 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:38.045 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:38.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:38.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:38.142 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:37 smithi163 ceph-mon[40675]: pgmap v149: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 866 B/s rd, 0 B/s wr, 942 op/s; 57503/324333 objects misplaced (17.730%); 29 MiB/s, 464 objects/s recovering 2024-03-21T13:02:38.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:37 smithi163 ceph-mon[29621]: pgmap v149: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 866 B/s rd, 0 B/s wr, 942 op/s; 57503/324333 objects misplaced (17.730%); 29 MiB/s, 464 objects/s recovering 2024-03-21T13:02:38.147 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:38.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:38.190 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:38.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:38.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:37 smithi204 ceph-mon[33153]: pgmap v149: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 866 B/s rd, 0 B/s wr, 942 op/s; 57503/324333 objects misplaced (17.730%); 29 MiB/s, 464 objects/s recovering 2024-03-21T13:02:38.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:38.319 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:38.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:38.360 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:02:38.360 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:02:38.360 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-03-21T13:02:38.360 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-03-21T13:02:38.361 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:02:38.377 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:02:38.378 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:02:38.387 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:02:38.422 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:38.439 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:02:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:38.464 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:38.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:38.488 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:02:38.528 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:02:38.566 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:38.567 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:02:38.607 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:02:38.611 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:38.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:38.618 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:02:38.618 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:02:38.618 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:02:38.619 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:02:38.619 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:02:38.619 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:02:38.619 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:02:38.619 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:02:38.619 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:38.715 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:38.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:38.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:38.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:38.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:38.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:38.987 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:39.020 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:39.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:39.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:39.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:39.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:39.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:39.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:38 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4043723919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:02:39.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:38 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:39.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:38 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4043723919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:02:39.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:38 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:39.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:38 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4043723919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:02:39.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:38 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:39.262 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:02:39.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:39.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:39.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:39.384 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-21T13:02:39.402 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:39.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:39.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:39.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:39.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:39.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:39.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:39.676 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:39.730 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:39.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:39.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:39.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:39.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:39.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:39.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:40.011 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:40.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:40.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:40.175 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:40.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:40.220 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:39 smithi163 ceph-mon[29621]: pgmap v150: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 818 B/s rd, 0 B/s wr, 890 op/s; 57503/324333 objects misplaced (17.730%); 27 MiB/s, 439 objects/s recovering 2024-03-21T13:02:40.221 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:39 smithi163 ceph-mon[40675]: pgmap v150: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 818 B/s rd, 0 B/s wr, 890 op/s; 57503/324333 objects misplaced (17.730%); 27 MiB/s, 439 objects/s recovering 2024-03-21T13:02:40.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:39 smithi204 ceph-mon[33153]: pgmap v150: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 818 B/s rd, 0 B/s wr, 890 op/s; 57503/324333 objects misplaced (17.730%); 27 MiB/s, 439 objects/s recovering 2024-03-21T13:02:40.276 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:40.324 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:40.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:40.425 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:40.470 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:40.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:40.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:40.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:40.618 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:40.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:40.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:40.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:40.756 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:40.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:40.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:40.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:40.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:40.993 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:41.017 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:41.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:41.119 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:41.156 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:41.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:41.258 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:41.295 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:41.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:41.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:41.397 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:41.447 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:41.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:41.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:41.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:41.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:41.684 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:41.707 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:41.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:41.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:41.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:02:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:02:41.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:41.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:41.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:41.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:41.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:41.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:42.081 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:42.112 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:02:42.119 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:42.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:42.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:42.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:41 smithi204 ceph-mon[33153]: pgmap v151: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 52911/309597 objects misplaced (17.090%); 20 MiB/s, 317 objects/s recovering 2024-03-21T13:02:42.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:41 smithi163 ceph-mon[40675]: pgmap v151: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 52911/309597 objects misplaced (17.090%); 20 MiB/s, 317 objects/s recovering 2024-03-21T13:02:42.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:41 smithi163 ceph-mon[29621]: pgmap v151: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 52911/309597 objects misplaced (17.090%); 20 MiB/s, 317 objects/s recovering 2024-03-21T13:02:42.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:42.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:42.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:42.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:42.396 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:42.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:42.498 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:42.527 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:42.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:42.628 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:42.662 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:42.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:42.764 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:42.791 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:42.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:42.892 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:42.919 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:42.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:42.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:43.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:43.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:43.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:43.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:43.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:43.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:43.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:43.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:42 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:43.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:42 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:43.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:42 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:43.282 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:43.324 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:43.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:43.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:43.426 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:43.460 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:43.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:43.561 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:43.593 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:43.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:43.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:43.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:43.721 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:43.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:43.823 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:43.857 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:43.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:43.959 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:44.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:44.001 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:44.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:44.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:44.132 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:44.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:44.159 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:43 smithi163 ceph-mon[40675]: pgmap v152: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 859 B/s rd, 0 B/s wr, 854 op/s; 52911/309597 objects misplaced (17.090%); 17 MiB/s, 266 objects/s recovering 2024-03-21T13:02:44.160 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:43 smithi163 ceph-mon[29621]: pgmap v152: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 859 B/s rd, 0 B/s wr, 854 op/s; 52911/309597 objects misplaced (17.090%); 17 MiB/s, 266 objects/s recovering 2024-03-21T13:02:44.202 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:44.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:44.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:43 smithi204 ceph-mon[33153]: pgmap v152: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 859 B/s rd, 0 B/s wr, 854 op/s; 52911/309597 objects misplaced (17.090%); 17 MiB/s, 266 objects/s recovering 2024-03-21T13:02:44.262 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:44.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:44.364 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:44.390 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:44.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:44.492 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:44.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:44.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:44.629 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:44.670 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:44.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:44.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:44.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:44.795 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:44.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:44.865 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:44.896 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:44.950 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:44.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:44.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:45.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:45.108 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:45.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:45.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:45.210 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:45.246 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:45.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:45.342 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:45.349 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:45.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:45.383 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:45.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:45.485 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:45.530 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:45.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:45.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:45.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:45.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:45.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:45.729 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:02:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:02:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:02:45.773 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:45.816 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:45.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:45.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:45.918 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:45.959 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:45.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:45.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:46.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:45 smithi163 ceph-mon[40675]: pgmap v153: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 50977/301749 objects misplaced (16.894%); 19 MiB/s, 308 objects/s recovering 2024-03-21T13:02:46.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:45 smithi163 ceph-mon[29621]: pgmap v153: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 50977/301749 objects misplaced (16.894%); 19 MiB/s, 308 objects/s recovering 2024-03-21T13:02:46.061 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:46.098 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:46.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:46.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:46.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:46.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:46.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:46.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:45 smithi204 ceph-mon[33153]: pgmap v153: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 50977/301749 objects misplaced (16.894%); 19 MiB/s, 308 objects/s recovering 2024-03-21T13:02:46.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:46.367 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:46.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:46.469 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:46.520 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:46.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:46.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:46.623 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:46.661 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:46.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:46.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:46.763 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:46.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:46.910 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:46.949 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:46.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:47.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:47.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:47.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:47.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:47.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:47.220 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:47.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:47.322 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:47.370 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:47.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:47.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:47.505 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:47.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:47.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:47.640 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:47.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:47.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:47.742 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:47.798 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:47.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:47.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:47.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:47 smithi163 ceph-mon[29621]: pgmap v154: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s; 48146/294189 objects misplaced (16.366%); 22 MiB/s, 358 objects/s recovering 2024-03-21T13:02:47.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:47 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:02:47.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:47 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:02:47.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:47 smithi163 ceph-mon[40675]: pgmap v154: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s; 48146/294189 objects misplaced (16.366%); 22 MiB/s, 358 objects/s recovering 2024-03-21T13:02:47.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:47 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:02:47.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:47 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:02:47.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:47.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:47.933 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:47.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:48.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:48.065 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:48.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:48.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:48.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:48.195 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:48.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:48.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:47 smithi204 ceph-mon[33153]: pgmap v154: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s; 48146/294189 objects misplaced (16.366%); 22 MiB/s, 358 objects/s recovering 2024-03-21T13:02:48.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:47 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:02:48.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:47 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:02:48.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:48.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:48.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:48.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:48.431 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:48.478 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:48.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:48.580 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:48.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:48.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:48.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:48.713 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:48.756 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:48.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:48.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:48.857 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:48.883 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:48.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:48.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:48.985 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:49.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:49.026 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:49.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:49.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:49.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:49.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:49.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:49.259 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:49.297 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:49.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:49.399 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:49.443 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:49.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:49.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:49.597 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:49.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:49.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:49.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:49.744 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:49.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:49.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:49.846 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:49.892 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:49.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:49.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:49.994 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:50.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:49 smithi163 ceph-mon[29621]: pgmap v155: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 837 op/s; 48146/294189 objects misplaced (16.366%); 17 MiB/s, 273 objects/s recovering 2024-03-21T13:02:50.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:49 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:49 smithi163 ceph-mon[40675]: pgmap v155: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 837 op/s; 48146/294189 objects misplaced (16.366%); 17 MiB/s, 273 objects/s recovering 2024-03-21T13:02:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:49 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:50.023 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:50.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:50.125 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:50.171 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:50.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:50.223 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:02:50.232 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:02:50.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:49 smithi204 ceph-mon[33153]: pgmap v155: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 6.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 837 op/s; 48146/294189 objects misplaced (16.366%); 17 MiB/s, 273 objects/s recovering 2024-03-21T13:02:50.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:49 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:50.272 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:50.322 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:50.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:50.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:50.455 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:50.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:50.556 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:50.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:50.595 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:50.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:50.696 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:50.730 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:50.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:50.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:50.767 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:02:50.768 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:02:50.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:50.867 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:50.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:50.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:50.968 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:50 smithi163 ceph-mon[40675]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:02:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:50 smithi163 ceph-mon[29621]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:02:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:51.012 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:51.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:51.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:51.146 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:51.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:51.248 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:51.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:50 smithi204 ceph-mon[33153]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:02:51.294 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:51.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:51.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:51.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:51.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:51.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:51.564 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:02:51.575 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:51.613 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:02:51.619 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:51.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:51.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:51.669 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:02:51.710 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:02:51.723 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:51.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:02:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:02:51.759 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:02:51.763 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:51.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:51.812 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:02:51.858 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:02:51.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:51.895 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:51.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:51.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:51.903 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:02:51.919 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:02:51.919 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:02:51.919 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:02:51.919 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:02:51.919 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:02:51.919 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:02:51.919 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:02:51.919 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:02:51.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:52.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:52.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:52.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:52.131 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:52.176 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:52.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:52.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:52.307 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:52.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:52.408 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:52.453 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:52.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:52.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:52.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:52 smithi204 ceph-mon[33153]: pgmap v156: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 43344/278310 objects misplaced (15.574%); 26 MiB/s, 409 objects/s recovering 2024-03-21T13:02:52.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:52 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/527556234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:02:52.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:52 smithi163 ceph-mon[29621]: pgmap v156: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 43344/278310 objects misplaced (15.574%); 26 MiB/s, 409 objects/s recovering 2024-03-21T13:02:52.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:52 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/527556234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:02:52.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:52 smithi163 ceph-mon[40675]: pgmap v156: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 43344/278310 objects misplaced (15.574%); 26 MiB/s, 409 objects/s recovering 2024-03-21T13:02:52.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:52 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/527556234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:02:52.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:52.586 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:52.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:52.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:52.688 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:52.712 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:52.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:52.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:52.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:52.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:52.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:52.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:52.981 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:52.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:53.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:53.118 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:53.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:53.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:53.220 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:53.252 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:02:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:53.293 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:53.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:53.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:53.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:53.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:53.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:53.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:53.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:53 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:02:53.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:53 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:02:53.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:53 smithi204 ceph-mon[33153]: osdmap e127: 8 total, 8 up, 6 in 2024-03-21T13:02:53.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:53 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4140330268' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:02:53.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:53 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:02:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:53 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:02:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:53 smithi163 ceph-mon[29621]: osdmap e127: 8 total, 8 up, 6 in 2024-03-21T13:02:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:53 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4140330268' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:02:53.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:53.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:53 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:02:53.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:53 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:02:53.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:53 smithi163 ceph-mon[40675]: osdmap e127: 8 total, 8 up, 6 in 2024-03-21T13:02:53.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:53 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4140330268' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:02:53.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:53.562 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:53.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:53.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:53.665 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:53.709 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:53.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:53.767 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-21T13:02:53.810 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:53.842 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:53.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:53.943 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:53.982 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:53.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:54.084 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:54.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:54.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:54.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:54.160 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:02:54.225 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:54.270 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:54.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:54.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:54.315 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-03-21T13:02:48.685654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6414211520,"num_objects":98063,"num_object_clones":0,"num_object_copies":294189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":48146,"num_objects_unfound":0,"num_objects_dirty":98063,"num_whiteouts":0,"num_read":1159,"num_read_kb":1005,"num_write":137628,"num_write_kb":7518494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12968,"num_bytes_recovered":839718831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65378,"ondisk_log_size":65378,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":16471620,"kb_used_data":15789972,"kb_used_omap":9,"kb_used_meta":681206,"kb_avail":545999292,"statfs":{"total":575970213888,"available":559103275008,"internally_reserved":0,"allocated":16168931328,"data_stored":16166207536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":697555622},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-658505728,"num_objects":-10048,"num_object_clones":0,"num_object_copies":-30144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-9357,"num_objects_unfound":0,"num_objects_dirty":-10048,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10048,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3287,"num_bytes_recovered":215416832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004560"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":151,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.770428+0000","last_change":"2024-03-21T13:02:26.223957+0000","last_active":"2024-03-21T13:02:30.770428+0000","last_peered":"2024-03-21T13:02:30.770428+0000","last_clean":"2024-03-21T13:02:30.770428+0000","last_became_active":"2024-03-21T13:02:26.223292+0000","last_became_peered":"2024-03-21T13:02:26.223292+0000","last_unstale":"2024-03-21T13:02:30.770428+0000","last_undegraded":"2024-03-21T13:02:30.770428+0000","last_fullsized":"2024-03-21T13:02:30.770428+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:01:35.694549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:59.241930+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":18,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.776455+0000","last_change":"2024-03-21T13:02:26.234070+0000","last_active":"2024-03-21T13:02:30.776455+0000","last_peered":"2024-03-21T13:02:30.776455+0000","last_clean":"2024-03-21T13:02:30.776455+0000","last_became_active":"2024-03-21T13:02:26.233709+0000","last_became_peered":"2024-03-21T13:02:26.233709+0000","last_unstale":"2024-03-21T13:02:30.776455+0000","last_undegraded":"2024-03-21T13:02:30.776455+0000","last_fullsized":"2024-03-21T13:02:30.776455+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:25.947092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:25.947092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.776711+0000","last_change":"2024-03-21T13:02:26.237862+0000","last_active":"2024-03-21T13:02:30.776711+0000","last_peered":"2024-03-21T13:02:30.776711+0000","last_clean":"2024-03-21T13:02:30.776711+0000","last_became_active":"2024-03-21T13:02:26.237501+0000","last_became_peered":"2024-03-21T13:02:26.237501+0000","last_unstale":"2024-03-21T13:02:30.776711+0000","last_undegraded":"2024-03-21T13:02:30.776711+0000","last_fullsized":"2024-03-21T13:02:30.776711+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:24.141445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:24.141445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:37.951776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"89'13","reported_seq":199,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.770940+0000","last_change":"2024-03-21T13:02:14.370743+0000","last_active":"2024-03-21T13:02:30.770940+0000","last_peered":"2024-03-21T13:02:30.770940+0000","last_clean":"2024-03-21T13:02:30.770940+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:02:30.770940+0000","last_undegraded":"2024-03-21T13:02:30.770940+0000","last_fullsized":"2024-03-21T13:02:30.770940+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:01:37.692932+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:52.141598+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":131,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.784147+0000","last_change":"2024-03-21T13:00:05.173374+0000","last_active":"2024-03-21T13:02:30.784147+0000","last_peered":"2024-03-21T13:02:30.784147+0000","last_clean":"2024-03-21T13:02:30.784147+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:02:30.784147+0000","last_undegraded":"2024-03-21T13:02:30.784147+0000","last_fullsized":"2024-03-21T13:02:30.784147+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:00:05.173276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":62,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.776404+0000","last_change":"2024-03-21T13:02:14.330937+0000","last_active":"2024-03-21T13:02:30.776404+0000","last_peered":"2024-03-21T13:02:30.776404+0000","last_clean":"2024-03-21T13:02:30.776404+0000","last_became_active":"2024-03-21T13:02:14.330467+0000","last_became_peered":"2024-03-21T13:02:14.330467+0000","last_unstale":"2024-03-21T13:02:30.776404+0000","last_undegraded":"2024-03-21T13:02:30.776404+0000","last_fullsized":"2024-03-21T13:02:30.776404+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:00:16.731434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":151,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.785656+0000","last_change":"2024-03-21T13:01:38.766164+0000","last_active":"2024-03-21T13:02:30.785656+0000","last_peered":"2024-03-21T13:02:30.785656+0000","last_clean":"2024-03-21T13:02:30.785656+0000","last_became_active":"2024-03-21T12:58:37.728383+0000","last_became_peered":"2024-03-21T12:58:37.728383+0000","last_unstale":"2024-03-21T13:02:30.785656+0000","last_undegraded":"2024-03-21T13:02:30.785656+0000","last_fullsized":"2024-03-21T13:02:30.785656+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_clean_scrub_stamp":"2024-03-21T13:01:38.765896+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:58.239614+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"71'15","reported_seq":164,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.785288+0000","last_change":"2024-03-21T13:00:31.008873+0000","last_active":"2024-03-21T13:02:30.785288+0000","last_peered":"2024-03-21T13:02:30.785288+0000","last_clean":"2024-03-21T13:02:30.785288+0000","last_became_active":"2024-03-21T12:58:39.750396+0000","last_became_peered":"2024-03-21T12:58:39.750396+0000","last_unstale":"2024-03-21T13:02:30.785288+0000","last_undegraded":"2024-03-21T13:02:30.785288+0000","last_fullsized":"2024-03-21T13:02:30.785288+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:00:31.008733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:31.008733+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":30,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.772867+0000","last_change":"2024-03-21T13:02:14.336859+0000","last_active":"2024-03-21T13:02:30.772867+0000","last_peered":"2024-03-21T13:02:30.772867+0000","last_clean":"2024-03-21T13:02:30.772867+0000","last_became_active":"2024-03-21T13:02:14.334221+0000","last_became_peered":"2024-03-21T13:02:14.334221+0000","last_unstale":"2024-03-21T13:02:30.772867+0000","last_undegraded":"2024-03-21T13:02:30.772867+0000","last_fullsized":"2024-03-21T13:02:30.772867+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:27.137810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:27.137810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.770068+0000","last_change":"2024-03-21T13:02:26.237088+0000","last_active":"2024-03-21T13:02:30.770068+0000","last_peered":"2024-03-21T13:02:30.770068+0000","last_clean":"2024-03-21T13:02:30.770068+0000","last_became_active":"2024-03-21T13:02:26.236707+0000","last_became_peered":"2024-03-21T13:02:26.236707+0000","last_unstale":"2024-03-21T13:02:30.770068+0000","last_undegraded":"2024-03-21T13:02:30.770068+0000","last_fullsized":"2024-03-21T13:02:30.770068+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:01:42.076564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:59.788109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.777837+0000","last_change":"2024-03-21T13:02:26.245694+0000","last_active":"2024-03-21T13:02:30.777837+0000","last_peered":"2024-03-21T13:02:30.777837+0000","last_clean":"2024-03-21T13:02:30.777837+0000","last_became_active":"2024-03-21T13:02:26.245402+0000","last_became_peered":"2024-03-21T13:02:26.245402+0000","last_unstale":"2024-03-21T13:02:30.777837+0000","last_undegraded":"2024-03-21T13:02:30.777837+0000","last_fullsized":"2024-03-21T13:02:30.777837+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:22.892585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:22.892585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:50.535685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":133,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.788227+0000","last_change":"2024-03-21T13:01:27.370969+0000","last_active":"2024-03-21T13:02:30.788227+0000","last_peered":"2024-03-21T13:02:30.788227+0000","last_clean":"2024-03-21T13:02:30.788227+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:02:30.788227+0000","last_undegraded":"2024-03-21T13:02:30.788227+0000","last_fullsized":"2024-03-21T13:02:30.788227+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.880161+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":136,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.784617+0000","last_change":"2024-03-21T13:02:26.229621+0000","last_active":"2024-03-21T13:02:30.784617+0000","last_peered":"2024-03-21T13:02:30.784617+0000","last_clean":"2024-03-21T13:02:30.784617+0000","last_became_active":"2024-03-21T13:02:26.229397+0000","last_became_peered":"2024-03-21T13:02:26.229397+0000","last_unstale":"2024-03-21T13:02:30.784617+0000","last_undegraded":"2024-03-21T13:02:30.784617+0000","last_fullsized":"2024-03-21T13:02:30.784617+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:32.971665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:32.971665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:54.435521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":147,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773526+0000","last_change":"2024-03-21T13:02:14.336087+0000","last_active":"2024-03-21T13:02:30.773526+0000","last_peered":"2024-03-21T13:02:30.773526+0000","last_clean":"2024-03-21T13:02:30.773526+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:02:30.773526+0000","last_undegraded":"2024-03-21T13:02:30.773526+0000","last_fullsized":"2024-03-21T13:02:30.773526+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:17.015065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:17.015065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":148,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.771767+0000","last_change":"2024-03-21T13:02:26.230571+0000","last_active":"2024-03-21T13:02:30.771767+0000","last_peered":"2024-03-21T13:02:30.771767+0000","last_clean":"2024-03-21T13:02:30.771767+0000","last_became_active":"2024-03-21T13:02:26.230057+0000","last_became_peered":"2024-03-21T13:02:26.230057+0000","last_unstale":"2024-03-21T13:02:30.771767+0000","last_undegraded":"2024-03-21T13:02:30.771767+0000","last_fullsized":"2024-03-21T13:02:30.771767+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:14.869934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:14.869934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:36.089057+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":135,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.783165+0000","last_change":"2024-03-21T13:02:14.383093+0000","last_active":"2024-03-21T13:02:30.783165+0000","last_peered":"2024-03-21T13:02:30.783165+0000","last_clean":"2024-03-21T13:02:30.783165+0000","last_became_active":"2024-03-21T13:02:14.349072+0000","last_became_peered":"2024-03-21T13:02:14.349072+0000","last_unstale":"2024-03-21T13:02:30.783165+0000","last_undegraded":"2024-03-21T13:02:30.783165+0000","last_fullsized":"2024-03-21T13:02:30.783165+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T12:59:53.952401+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.771133+0000","last_change":"2024-03-21T13:02:26.227096+0000","last_active":"2024-03-21T13:02:30.771133+0000","last_peered":"2024-03-21T13:02:30.771133+0000","last_clean":"2024-03-21T13:02:30.771133+0000","last_became_active":"2024-03-21T13:02:26.226675+0000","last_became_peered":"2024-03-21T13:02:26.226675+0000","last_unstale":"2024-03-21T13:02:30.771133+0000","last_undegraded":"2024-03-21T13:02:30.771133+0000","last_fullsized":"2024-03-21T13:02:30.771133+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:24.662938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:24.662938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":34,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.771479+0000","last_change":"2024-03-21T13:02:26.232199+0000","last_active":"2024-03-21T13:02:30.771479+0000","last_peered":"2024-03-21T13:02:30.771479+0000","last_clean":"2024-03-21T13:02:30.771479+0000","last_became_active":"2024-03-21T13:02:26.231879+0000","last_became_peered":"2024-03-21T13:02:26.231879+0000","last_unstale":"2024-03-21T13:02:30.771479+0000","last_undegraded":"2024-03-21T13:02:30.771479+0000","last_fullsized":"2024-03-21T13:02:30.771479+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:18.990434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.990434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:34.418295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":42,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773900+0000","last_change":"2024-03-21T13:01:48.210250+0000","last_active":"2024-03-21T13:02:30.773900+0000","last_peered":"2024-03-21T13:02:30.773900+0000","last_clean":"2024-03-21T13:02:30.773900+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:02:30.773900+0000","last_undegraded":"2024-03-21T13:02:30.773900+0000","last_fullsized":"2024-03-21T13:02:30.773900+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:44.005379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:44.005379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:04.704307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":195,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.781216+0000","last_change":"2024-03-21T13:02:14.385705+0000","last_active":"2024-03-21T13:02:30.781216+0000","last_peered":"2024-03-21T13:02:30.781216+0000","last_clean":"2024-03-21T13:02:30.781216+0000","last_became_active":"2024-03-21T13:02:14.350563+0000","last_became_peered":"2024-03-21T13:02:14.350563+0000","last_unstale":"2024-03-21T13:02:30.781216+0000","last_undegraded":"2024-03-21T13:02:30.781216+0000","last_fullsized":"2024-03-21T13:02:30.781216+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:01:36.603903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:36.603903+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:43.472916+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.783389+0000","last_change":"2024-03-21T13:02:26.243438+0000","last_active":"2024-03-21T13:02:30.783389+0000","last_peered":"2024-03-21T13:02:30.783389+0000","last_clean":"2024-03-21T13:02:30.783389+0000","last_became_active":"2024-03-21T13:02:26.242482+0000","last_became_peered":"2024-03-21T13:02:26.242482+0000","last_unstale":"2024-03-21T13:02:30.783389+0000","last_undegraded":"2024-03-21T13:02:30.783389+0000","last_fullsized":"2024-03-21T13:02:30.783389+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:35.036335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.036335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":56,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.788833+0000","last_change":"2024-03-21T13:01:27.367176+0000","last_active":"2024-03-21T13:02:30.788833+0000","last_peered":"2024-03-21T13:02:30.788833+0000","last_clean":"2024-03-21T13:02:30.788833+0000","last_became_active":"2024-03-21T13:01:27.366846+0000","last_became_peered":"2024-03-21T13:01:27.366846+0000","last_unstale":"2024-03-21T13:02:30.788833+0000","last_undegraded":"2024-03-21T13:02:30.788833+0000","last_fullsized":"2024-03-21T13:02:30.788833+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.904419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.904419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":157,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.771262+0000","last_change":"2024-03-21T13:02:14.329782+0000","last_active":"2024-03-21T13:02:30.771262+0000","last_peered":"2024-03-21T13:02:30.771262+0000","last_clean":"2024-03-21T13:02:30.771262+0000","last_became_active":"2024-03-21T13:02:14.329156+0000","last_became_peered":"2024-03-21T13:02:14.329156+0000","last_unstale":"2024-03-21T13:02:30.771262+0000","last_undegraded":"2024-03-21T13:02:30.771262+0000","last_fullsized":"2024-03-21T13:02:30.771262+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:27.704194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_clean_scrub_stamp":"2024-03-21T13:01:27.704194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:32.910952+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'12","reported_seq":198,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.785699+0000","last_change":"2024-03-21T13:02:14.362313+0000","last_active":"2024-03-21T13:02:30.785699+0000","last_peered":"2024-03-21T13:02:30.785699+0000","last_clean":"2024-03-21T13:02:30.785699+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:02:30.785699+0000","last_undegraded":"2024-03-21T13:02:30.785699+0000","last_fullsized":"2024-03-21T13:02:30.785699+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:01:39.755345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.755345+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:02.847177+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":148,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.774428+0000","last_change":"2024-03-21T13:02:26.254057+0000","last_active":"2024-03-21T13:02:30.774428+0000","last_peered":"2024-03-21T13:02:30.774428+0000","last_clean":"2024-03-21T13:02:30.774428+0000","last_became_active":"2024-03-21T13:02:26.237656+0000","last_became_peered":"2024-03-21T13:02:26.237656+0000","last_unstale":"2024-03-21T13:02:30.774428+0000","last_undegraded":"2024-03-21T13:02:30.774428+0000","last_fullsized":"2024-03-21T13:02:30.774428+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:01:22.076398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:22.076398+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:39.312776+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":204,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:37.819332+0000","last_change":"2024-03-21T13:01:30.587787+0000","last_active":"2024-03-21T13:02:37.819332+0000","last_peered":"2024-03-21T13:02:37.819332+0000","last_clean":"2024-03-21T13:02:37.819332+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:02:37.819332+0000","last_undegraded":"2024-03-21T13:02:37.819332+0000","last_fullsized":"2024-03-21T13:02:37.819332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:30.587660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:30.587660+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:44.843810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":40,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.784904+0000","last_change":"2024-03-21T13:02:14.372498+0000","last_active":"2024-03-21T13:02:30.784904+0000","last_peered":"2024-03-21T13:02:30.784904+0000","last_clean":"2024-03-21T13:02:30.784904+0000","last_became_active":"2024-03-21T13:02:14.348666+0000","last_became_peered":"2024-03-21T13:02:14.348666+0000","last_unstale":"2024-03-21T13:02:30.784904+0000","last_undegraded":"2024-03-21T13:02:30.784904+0000","last_fullsized":"2024-03-21T13:02:30.784904+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:01:37.244557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:37.244557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:49.086797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"71'10","reported_seq":76,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.774739+0000","last_change":"2024-03-21T13:01:48.218829+0000","last_active":"2024-03-21T13:02:30.774739+0000","last_peered":"2024-03-21T13:02:30.774739+0000","last_clean":"2024-03-21T13:02:30.774739+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:02:30.774739+0000","last_undegraded":"2024-03-21T13:02:30.774739+0000","last_fullsized":"2024-03-21T13:02:30.774739+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:01:17.088732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:17.088732+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:46.824472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":18,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.771038+0000","last_change":"2024-03-21T13:02:26.230240+0000","last_active":"2024-03-21T13:02:30.771038+0000","last_peered":"2024-03-21T13:02:30.771038+0000","last_clean":"2024-03-21T13:02:30.771038+0000","last_became_active":"2024-03-21T13:02:26.229188+0000","last_became_peered":"2024-03-21T13:02:26.229188+0000","last_unstale":"2024-03-21T13:02:30.771038+0000","last_undegraded":"2024-03-21T13:02:30.771038+0000","last_fullsized":"2024-03-21T13:02:30.771038+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:33.251265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:33.251265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:39.909707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":42,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.774505+0000","last_change":"2024-03-21T13:01:48.218869+0000","last_active":"2024-03-21T13:02:30.774505+0000","last_peered":"2024-03-21T13:02:30.774505+0000","last_clean":"2024-03-21T13:02:30.774505+0000","last_became_active":"2024-03-21T13:01:48.215160+0000","last_became_peered":"2024-03-21T13:01:48.215160+0000","last_unstale":"2024-03-21T13:02:30.774505+0000","last_undegraded":"2024-03-21T13:02:30.774505+0000","last_fullsized":"2024-03-21T13:02:30.774505+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:39.968654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.968654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:44.155517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.784567+0000","last_change":"2024-03-21T13:02:14.344996+0000","last_active":"2024-03-21T13:02:30.784567+0000","last_peered":"2024-03-21T13:02:30.784567+0000","last_clean":"2024-03-21T13:02:30.784567+0000","last_became_active":"2024-03-21T13:02:14.341286+0000","last_became_peered":"2024-03-21T13:02:14.341286+0000","last_unstale":"2024-03-21T13:02:30.784567+0000","last_undegraded":"2024-03-21T13:02:30.784567+0000","last_fullsized":"2024-03-21T13:02:30.784567+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:49.178785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:49.178785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'11","reported_seq":202,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.777076+0000","last_change":"2024-03-21T13:02:26.277298+0000","last_active":"2024-03-21T13:02:30.777076+0000","last_peered":"2024-03-21T13:02:30.777076+0000","last_clean":"2024-03-21T13:02:30.777076+0000","last_became_active":"2024-03-21T13:02:26.254436+0000","last_became_peered":"2024-03-21T13:02:26.254436+0000","last_unstale":"2024-03-21T13:02:30.777076+0000","last_undegraded":"2024-03-21T13:02:30.777076+0000","last_fullsized":"2024-03-21T13:02:30.777076+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:01:48.562468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:48.562468+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:17.230692+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773565+0000","last_change":"2024-03-21T13:01:35.005181+0000","last_active":"2024-03-21T13:02:30.773565+0000","last_peered":"2024-03-21T13:02:30.773565+0000","last_clean":"2024-03-21T13:02:30.773565+0000","last_became_active":"2024-03-21T12:58:43.183475+0000","last_became_peered":"2024-03-21T12:58:43.183475+0000","last_unstale":"2024-03-21T13:02:30.773565+0000","last_undegraded":"2024-03-21T13:02:30.773565+0000","last_fullsized":"2024-03-21T13:02:30.773565+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:01:35.005026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:03.261414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":143,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.777201+0000","last_change":"2024-03-21T13:01:20.591831+0000","last_active":"2024-03-21T13:02:30.777201+0000","last_peered":"2024-03-21T13:02:30.777201+0000","last_clean":"2024-03-21T13:02:30.777201+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:02:30.777201+0000","last_undegraded":"2024-03-21T13:02:30.777201+0000","last_fullsized":"2024-03-21T13:02:30.777201+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.591636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.591636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:47.946660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.787944+0000","last_change":"2024-03-21T13:02:26.243710+0000","last_active":"2024-03-21T13:02:30.787944+0000","last_peered":"2024-03-21T13:02:30.787944+0000","last_clean":"2024-03-21T13:02:30.787944+0000","last_became_active":"2024-03-21T13:02:26.242973+0000","last_became_peered":"2024-03-21T13:02:26.242973+0000","last_unstale":"2024-03-21T13:02:30.787944+0000","last_undegraded":"2024-03-21T13:02:30.787944+0000","last_fullsized":"2024-03-21T13:02:30.787944+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:32.789993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.789993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":157,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.783219+0000","last_change":"2024-03-21T13:02:14.369668+0000","last_active":"2024-03-21T13:02:30.783219+0000","last_peered":"2024-03-21T13:02:30.783219+0000","last_clean":"2024-03-21T13:02:30.783219+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:02:30.783219+0000","last_undegraded":"2024-03-21T13:02:30.783219+0000","last_fullsized":"2024-03-21T13:02:30.783219+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-21T13:01:30.807979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:30.807979+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:55.729376+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":131,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.784428+0000","last_change":"2024-03-21T13:00:10.110413+0000","last_active":"2024-03-21T13:02:30.784428+0000","last_peered":"2024-03-21T13:02:30.784428+0000","last_clean":"2024-03-21T13:02:30.784428+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:02:30.784428+0000","last_undegraded":"2024-03-21T13:02:30.784428+0000","last_fullsized":"2024-03-21T13:02:30.784428+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:10.110335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.110335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":133,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773261+0000","last_change":"2024-03-21T13:00:18.163092+0000","last_active":"2024-03-21T13:02:30.773261+0000","last_peered":"2024-03-21T13:02:30.773261+0000","last_clean":"2024-03-21T13:02:30.773261+0000","last_became_active":"2024-03-21T12:58:41.164784+0000","last_became_peered":"2024-03-21T12:58:41.164784+0000","last_unstale":"2024-03-21T13:02:30.773261+0000","last_undegraded":"2024-03-21T13:02:30.773261+0000","last_fullsized":"2024-03-21T13:02:30.773261+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:18.162988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:18.162988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":140,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.774342+0000","last_change":"2024-03-21T13:02:26.236549+0000","last_active":"2024-03-21T13:02:30.774342+0000","last_peered":"2024-03-21T13:02:30.774342+0000","last_clean":"2024-03-21T13:02:30.774342+0000","last_became_active":"2024-03-21T13:02:26.236235+0000","last_became_peered":"2024-03-21T13:02:26.236235+0000","last_unstale":"2024-03-21T13:02:30.774342+0000","last_undegraded":"2024-03-21T13:02:30.774342+0000","last_fullsized":"2024-03-21T13:02:30.774342+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:02:21.087308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:02:21.087308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:24.336763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"71'11","reported_seq":158,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.785194+0000","last_change":"2024-03-21T12:59:52.181828+0000","last_active":"2024-03-21T13:02:30.785194+0000","last_peered":"2024-03-21T13:02:30.785194+0000","last_clean":"2024-03-21T13:02:30.785194+0000","last_became_active":"2024-03-21T12:58:39.753014+0000","last_became_peered":"2024-03-21T12:58:39.753014+0000","last_unstale":"2024-03-21T13:02:30.785194+0000","last_undegraded":"2024-03-21T13:02:30.785194+0000","last_fullsized":"2024-03-21T13:02:30.785194+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T12:59:52.181697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.181697+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":39,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.775948+0000","last_change":"2024-03-21T13:02:26.253063+0000","last_active":"2024-03-21T13:02:30.775948+0000","last_peered":"2024-03-21T13:02:30.775948+0000","last_clean":"2024-03-21T13:02:30.775948+0000","last_became_active":"2024-03-21T13:02:26.252488+0000","last_became_peered":"2024-03-21T13:02:26.252488+0000","last_unstale":"2024-03-21T13:02:30.775948+0000","last_undegraded":"2024-03-21T13:02:30.775948+0000","last_fullsized":"2024-03-21T13:02:30.775948+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:22.083890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:22.083890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:31.885824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":147,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.783371+0000","last_change":"2024-03-21T13:02:26.233084+0000","last_active":"2024-03-21T13:02:30.783371+0000","last_peered":"2024-03-21T13:02:30.783371+0000","last_clean":"2024-03-21T13:02:30.783371+0000","last_became_active":"2024-03-21T13:02:26.232682+0000","last_became_peered":"2024-03-21T13:02:26.232682+0000","last_unstale":"2024-03-21T13:02:30.783371+0000","last_undegraded":"2024-03-21T13:02:30.783371+0000","last_fullsized":"2024-03-21T13:02:30.783371+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:38.995215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.995215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773828+0000","last_change":"2024-03-21T13:01:43.982673+0000","last_active":"2024-03-21T13:02:30.773828+0000","last_peered":"2024-03-21T13:02:30.773828+0000","last_clean":"2024-03-21T13:02:30.773828+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:02:30.773828+0000","last_undegraded":"2024-03-21T13:02:30.773828+0000","last_fullsized":"2024-03-21T13:02:30.773828+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:43.982553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:43.982553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:11.583365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"86'13","reported_seq":107,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.776484+0000","last_change":"2024-03-21T13:02:14.351413+0000","last_active":"2024-03-21T13:02:30.776484+0000","last_peered":"2024-03-21T13:02:30.776484+0000","last_clean":"2024-03-21T13:02:30.776484+0000","last_became_active":"2024-03-21T13:02:14.330196+0000","last_became_peered":"2024-03-21T13:02:14.330196+0000","last_unstale":"2024-03-21T13:02:30.776484+0000","last_undegraded":"2024-03-21T13:02:30.776484+0000","last_fullsized":"2024-03-21T13:02:30.776484+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:01:14.027717+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:01:14.027717+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:39.505960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.784761+0000","last_change":"2024-03-21T13:02:26.229754+0000","last_active":"2024-03-21T13:02:30.784761+0000","last_peered":"2024-03-21T13:02:30.784761+0000","last_clean":"2024-03-21T13:02:30.784761+0000","last_became_active":"2024-03-21T13:02:26.229425+0000","last_became_peered":"2024-03-21T13:02:26.229425+0000","last_unstale":"2024-03-21T13:02:30.784761+0000","last_undegraded":"2024-03-21T13:02:30.784761+0000","last_fullsized":"2024-03-21T13:02:30.784761+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:50.729022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:50.729022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:13.309163+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":18,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.788333+0000","last_change":"2024-03-21T13:02:26.242867+0000","last_active":"2024-03-21T13:02:30.788333+0000","last_peered":"2024-03-21T13:02:30.788333+0000","last_clean":"2024-03-21T13:02:30.788333+0000","last_became_active":"2024-03-21T13:02:26.242491+0000","last_became_peered":"2024-03-21T13:02:26.242491+0000","last_unstale":"2024-03-21T13:02:30.788333+0000","last_undegraded":"2024-03-21T13:02:30.788333+0000","last_fullsized":"2024-03-21T13:02:30.788333+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.029629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.029629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:36.096098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":151,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.772405+0000","last_change":"2024-03-21T13:01:20.648033+0000","last_active":"2024-03-21T13:02:30.772405+0000","last_peered":"2024-03-21T13:02:30.772405+0000","last_clean":"2024-03-21T13:02:30.772405+0000","last_became_active":"2024-03-21T12:58:37.731659+0000","last_became_peered":"2024-03-21T12:58:37.731659+0000","last_unstale":"2024-03-21T13:02:30.772405+0000","last_undegraded":"2024-03-21T13:02:30.772405+0000","last_fullsized":"2024-03-21T13:02:30.772405+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.647931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.647931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:43.319863+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":194,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.771378+0000","last_change":"2024-03-21T13:02:26.258852+0000","last_active":"2024-03-21T13:02:30.771378+0000","last_peered":"2024-03-21T13:02:30.771378+0000","last_clean":"2024-03-21T13:02:30.771378+0000","last_became_active":"2024-03-21T13:02:26.225627+0000","last_became_peered":"2024-03-21T13:02:26.225627+0000","last_unstale":"2024-03-21T13:02:30.771378+0000","last_undegraded":"2024-03-21T13:02:30.771378+0000","last_fullsized":"2024-03-21T13:02:30.771378+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:01:19.642156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:19.642156+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":42,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.777231+0000","last_change":"2024-03-21T13:01:48.215368+0000","last_active":"2024-03-21T13:02:30.777231+0000","last_peered":"2024-03-21T13:02:30.777231+0000","last_clean":"2024-03-21T13:02:30.777231+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:02:30.777231+0000","last_undegraded":"2024-03-21T13:02:30.777231+0000","last_fullsized":"2024-03-21T13:02:30.777231+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.062842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.062842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"125'10","reported_seq":203,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:37.820163+0000","last_change":"2024-03-21T13:02:26.247826+0000","last_active":"2024-03-21T13:02:37.820163+0000","last_peered":"2024-03-21T13:02:37.820163+0000","last_clean":"2024-03-21T13:02:37.820163+0000","last_became_active":"2024-03-21T13:02:26.234494+0000","last_became_peered":"2024-03-21T13:02:26.234494+0000","last_unstale":"2024-03-21T13:02:37.820163+0000","last_undegraded":"2024-03-21T13:02:37.820163+0000","last_fullsized":"2024-03-21T13:02:37.820163+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:23.867564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:23.867564+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773701+0000","last_change":"2024-03-21T13:01:45.972893+0000","last_active":"2024-03-21T13:02:30.773701+0000","last_peered":"2024-03-21T13:02:30.773701+0000","last_clean":"2024-03-21T13:02:30.773701+0000","last_became_active":"2024-03-21T12:58:37.731988+0000","last_became_peered":"2024-03-21T12:58:37.731988+0000","last_unstale":"2024-03-21T13:02:30.773701+0000","last_undegraded":"2024-03-21T13:02:30.773701+0000","last_fullsized":"2024-03-21T13:02:30.773701+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:45.972809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:45.972809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:54.970355+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"71'10","reported_seq":172,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.789236+0000","last_change":"2024-03-21T13:02:14.389576+0000","last_active":"2024-03-21T13:02:30.789236+0000","last_peered":"2024-03-21T13:02:30.789236+0000","last_clean":"2024-03-21T13:02:30.789236+0000","last_became_active":"2024-03-21T13:02:14.353938+0000","last_became_peered":"2024-03-21T13:02:14.353938+0000","last_unstale":"2024-03-21T13:02:30.789236+0000","last_undegraded":"2024-03-21T13:02:30.789236+0000","last_fullsized":"2024-03-21T13:02:30.789236+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:00:06.197903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.197903+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":145,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.771837+0000","last_change":"2024-03-21T13:02:26.233510+0000","last_active":"2024-03-21T13:02:30.771837+0000","last_peered":"2024-03-21T13:02:30.771837+0000","last_clean":"2024-03-21T13:02:30.771837+0000","last_became_active":"2024-03-21T13:02:26.233049+0000","last_became_peered":"2024-03-21T13:02:26.233049+0000","last_unstale":"2024-03-21T13:02:30.771837+0000","last_undegraded":"2024-03-21T13:02:30.771837+0000","last_fullsized":"2024-03-21T13:02:30.771837+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:01:36.699469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:00.034611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":193,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:37.819332+0000","last_change":"2024-03-21T13:01:18.526684+0000","last_active":"2024-03-21T13:02:37.819332+0000","last_peered":"2024-03-21T13:02:37.819332+0000","last_clean":"2024-03-21T13:02:37.819332+0000","last_became_active":"2024-03-21T12:58:41.167826+0000","last_became_peered":"2024-03-21T12:58:41.167826+0000","last_unstale":"2024-03-21T13:02:37.819332+0000","last_undegraded":"2024-03-21T13:02:37.819332+0000","last_fullsized":"2024-03-21T13:02:37.819332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:18.526428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.526428+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":42,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773421+0000","last_change":"2024-03-21T13:01:48.212782+0000","last_active":"2024-03-21T13:02:30.773421+0000","last_peered":"2024-03-21T13:02:30.773421+0000","last_clean":"2024-03-21T13:02:30.773421+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:02:30.773421+0000","last_undegraded":"2024-03-21T13:02:30.773421+0000","last_fullsized":"2024-03-21T13:02:30.773421+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:35.801638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.801638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"71'9","reported_seq":61,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.774158+0000","last_change":"2024-03-21T13:02:26.266952+0000","last_active":"2024-03-21T13:02:30.774158+0000","last_peered":"2024-03-21T13:02:30.774158+0000","last_clean":"2024-03-21T13:02:30.774158+0000","last_became_active":"2024-03-21T13:02:26.239161+0000","last_became_peered":"2024-03-21T13:02:26.239161+0000","last_unstale":"2024-03-21T13:02:30.774158+0000","last_undegraded":"2024-03-21T13:02:30.774158+0000","last_fullsized":"2024-03-21T13:02:30.774158+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:00:19.686939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:19.686939+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":140,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.781682+0000","last_change":"2024-03-21T13:01:34.642737+0000","last_active":"2024-03-21T13:02:30.781682+0000","last_peered":"2024-03-21T13:02:30.781682+0000","last_clean":"2024-03-21T13:02:30.781682+0000","last_became_active":"2024-03-21T12:58:43.182126+0000","last_became_peered":"2024-03-21T12:58:43.182126+0000","last_unstale":"2024-03-21T13:02:30.781682+0000","last_undegraded":"2024-03-21T13:02:30.781682+0000","last_fullsized":"2024-03-21T13:02:30.781682+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:34.642613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:01:34.642613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:40.394869+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.788400+0000","last_change":"2024-03-21T13:02:14.354537+0000","last_active":"2024-03-21T13:02:30.788400+0000","last_peered":"2024-03-21T13:02:30.788400+0000","last_clean":"2024-03-21T13:02:30.788400+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:02:30.788400+0000","last_undegraded":"2024-03-21T13:02:30.788400+0000","last_fullsized":"2024-03-21T13:02:30.788400+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:12.121970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:12.121970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":155,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.774943+0000","last_change":"2024-03-21T13:01:27.075743+0000","last_active":"2024-03-21T13:02:30.774943+0000","last_peered":"2024-03-21T13:02:30.774943+0000","last_clean":"2024-03-21T13:02:30.774943+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:02:30.774943+0000","last_undegraded":"2024-03-21T13:02:30.774943+0000","last_fullsized":"2024-03-21T13:02:30.774943+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:01:27.074936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:27.074936+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:47.683462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'9","reported_seq":77,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.785504+0000","last_change":"2024-03-21T13:01:48.210983+0000","last_active":"2024-03-21T13:02:30.785504+0000","last_peered":"2024-03-21T13:02:30.785504+0000","last_clean":"2024-03-21T13:02:30.785504+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:02:30.785504+0000","last_undegraded":"2024-03-21T13:02:30.785504+0000","last_fullsized":"2024-03-21T13:02:30.785504+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:01:40.964615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.964615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:04.519841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773717+0000","last_change":"2024-03-21T13:00:08.197105+0000","last_active":"2024-03-21T13:02:30.773717+0000","last_peered":"2024-03-21T13:02:30.773717+0000","last_clean":"2024-03-21T13:02:30.773717+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:02:30.773717+0000","last_undegraded":"2024-03-21T13:02:30.773717+0000","last_fullsized":"2024-03-21T13:02:30.773717+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.196979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.196979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":132,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.785856+0000","last_change":"2024-03-21T13:02:26.243050+0000","last_active":"2024-03-21T13:02:30.785856+0000","last_peered":"2024-03-21T13:02:30.785856+0000","last_clean":"2024-03-21T13:02:30.785856+0000","last_became_active":"2024-03-21T13:02:26.242502+0000","last_became_peered":"2024-03-21T13:02:26.242502+0000","last_unstale":"2024-03-21T13:02:30.785856+0000","last_undegraded":"2024-03-21T13:02:30.785856+0000","last_fullsized":"2024-03-21T13:02:30.785856+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:32.639920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.639920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":157,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.776317+0000","last_change":"2024-03-21T13:02:26.252758+0000","last_active":"2024-03-21T13:02:30.776317+0000","last_peered":"2024-03-21T13:02:30.776317+0000","last_clean":"2024-03-21T13:02:30.776317+0000","last_became_active":"2024-03-21T13:02:26.252406+0000","last_became_peered":"2024-03-21T13:02:26.252406+0000","last_unstale":"2024-03-21T13:02:30.776317+0000","last_undegraded":"2024-03-21T13:02:30.776317+0000","last_fullsized":"2024-03-21T13:02:30.776317+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:41.621550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:41.621550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:46.508630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'6","reported_seq":162,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.788226+0000","last_change":"2024-03-21T13:02:14.384317+0000","last_active":"2024-03-21T13:02:30.788226+0000","last_peered":"2024-03-21T13:02:30.788226+0000","last_clean":"2024-03-21T13:02:30.788226+0000","last_became_active":"2024-03-21T13:02:14.352117+0000","last_became_peered":"2024-03-21T13:02:14.352117+0000","last_unstale":"2024-03-21T13:02:30.788226+0000","last_undegraded":"2024-03-21T13:02:30.788226+0000","last_fullsized":"2024-03-21T13:02:30.788226+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T12:59:46.176819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.176819+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"126'8472","reported_seq":14624,"reported_epoch":126,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:02:44.662226+0000","last_change":"2024-03-21T13:02:27.339033+0000","last_active":"2024-03-21T13:02:44.662226+0000","last_peered":"2024-03-21T13:02:44.662226+0000","last_clean":"2024-03-21T13:02:13.311845+0000","last_became_active":"2024-03-21T13:02:27.309968+0000","last_became_peered":"2024-03-21T13:02:27.309968+0000","last_unstale":"2024-03-21T13:02:44.662226+0000","last_undegraded":"2024-03-21T13:02:44.662226+0000","last_fullsized":"2024-03-21T13:02:44.662226+0000","mapping_epoch":124,"log_start":"106'4223","ondisk_log_start":"106'4223","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4249,"log_dups_size":0,"ondisk_log_size":4249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402784256,"num_objects":6146,"num_object_clones":0,"num_object_copies":18438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8572,"num_objects_unfound":0,"num_objects_dirty":6146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8472,"num_write_kb":467776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2020,"num_bytes_recovered":132382720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,5,1],"avail_no_missing":["6","1","5"],"object_location_counts":[{"shards":"1,5,6","objects":6146}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.776120+0000","last_change":"2024-03-21T13:02:26.253217+0000","last_active":"2024-03-21T13:02:30.776120+0000","last_peered":"2024-03-21T13:02:30.776120+0000","last_clean":"2024-03-21T13:02:30.776120+0000","last_became_active":"2024-03-21T13:02:26.252904+0000","last_became_peered":"2024-03-21T13:02:26.252904+0000","last_unstale":"2024-03-21T13:02:30.776120+0000","last_undegraded":"2024-03-21T13:02:30.776120+0000","last_fullsized":"2024-03-21T13:02:30.776120+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:01:43.075256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:59.639341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":161,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773889+0000","last_change":"2024-03-21T13:01:39.975587+0000","last_active":"2024-03-21T13:02:30.773889+0000","last_peered":"2024-03-21T13:02:30.773889+0000","last_clean":"2024-03-21T13:02:30.773889+0000","last_became_active":"2024-03-21T12:58:37.735037+0000","last_became_peered":"2024-03-21T12:58:37.735037+0000","last_unstale":"2024-03-21T13:02:30.773889+0000","last_undegraded":"2024-03-21T13:02:30.773889+0000","last_fullsized":"2024-03-21T13:02:30.773889+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:01:39.975456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.975456+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:53.447196+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'15","reported_seq":76,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773344+0000","last_change":"2024-03-21T13:02:26.260469+0000","last_active":"2024-03-21T13:02:30.773344+0000","last_peered":"2024-03-21T13:02:30.773344+0000","last_clean":"2024-03-21T13:02:30.773344+0000","last_became_active":"2024-03-21T13:02:26.237800+0000","last_became_peered":"2024-03-21T13:02:26.237800+0000","last_unstale":"2024-03-21T13:02:30.773344+0000","last_undegraded":"2024-03-21T13:02:30.773344+0000","last_fullsized":"2024-03-21T13:02:30.773344+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:00:27.640392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:27.640392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":192,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:37.819428+0000","last_change":"2024-03-21T13:00:20.176757+0000","last_active":"2024-03-21T13:02:37.819428+0000","last_peered":"2024-03-21T13:02:37.819428+0000","last_clean":"2024-03-21T13:02:37.819428+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:02:37.819428+0000","last_undegraded":"2024-03-21T13:02:37.819428+0000","last_fullsized":"2024-03-21T13:02:37.819428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.176653+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"126'8453","reported_seq":8538,"reported_epoch":126,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-21T13:02:44.882848+0000","last_change":"2024-03-21T13:02:27.317609+0000","last_active":"2024-03-21T13:02:44.882848+0000","last_peered":"2024-03-21T13:02:44.882848+0000","last_clean":"2024-03-21T13:02:25.722555+0000","last_became_active":"2024-03-21T13:02:27.294304+0000","last_became_peered":"2024-03-21T13:02:27.294304+0000","last_unstale":"2024-03-21T13:02:44.882848+0000","last_undegraded":"2024-03-21T13:02:44.882848+0000","last_fullsized":"2024-03-21T13:02:44.882848+0000","mapping_epoch":124,"log_start":"109'4829","ondisk_log_start":"109'4829","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3624,"log_dups_size":0,"ondisk_log_size":3624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398131200,"num_objects":6075,"num_object_clones":0,"num_object_copies":18225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6075,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8453,"num_write_kb":464896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,5],"avail_no_missing":["3","2","5"],"object_location_counts":[{"shards":"2,3,5","objects":6075}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.784245+0000","last_change":"2024-03-21T13:02:14.352120+0000","last_active":"2024-03-21T13:02:30.784245+0000","last_peered":"2024-03-21T13:02:30.784245+0000","last_clean":"2024-03-21T13:02:30.784245+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:02:30.784245+0000","last_undegraded":"2024-03-21T13:02:30.784245+0000","last_fullsized":"2024-03-21T13:02:30.784245+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:24.988792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:24.988792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":155,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.774429+0000","last_change":"2024-03-21T13:02:14.337089+0000","last_active":"2024-03-21T13:02:30.774429+0000","last_peered":"2024-03-21T13:02:30.774429+0000","last_clean":"2024-03-21T13:02:30.774429+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:02:30.774429+0000","last_undegraded":"2024-03-21T13:02:30.774429+0000","last_fullsized":"2024-03-21T13:02:30.774429+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.030821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.030821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:41.826123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"71'12","reported_seq":82,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.785050+0000","last_change":"2024-03-21T13:02:14.383361+0000","last_active":"2024-03-21T13:02:30.785050+0000","last_peered":"2024-03-21T13:02:30.785050+0000","last_clean":"2024-03-21T13:02:30.785050+0000","last_became_active":"2024-03-21T13:02:14.347102+0000","last_became_peered":"2024-03-21T13:02:14.347102+0000","last_unstale":"2024-03-21T13:02:30.785050+0000","last_undegraded":"2024-03-21T13:02:30.785050+0000","last_fullsized":"2024-03-21T13:02:30.785050+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:00:32.864608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.864608+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":156,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.771920+0000","last_change":"2024-03-21T13:02:26.235545+0000","last_active":"2024-03-21T13:02:30.771920+0000","last_peered":"2024-03-21T13:02:30.771920+0000","last_clean":"2024-03-21T13:02:30.771920+0000","last_became_active":"2024-03-21T13:02:26.235083+0000","last_became_peered":"2024-03-21T13:02:26.235083+0000","last_unstale":"2024-03-21T13:02:30.771920+0000","last_undegraded":"2024-03-21T13:02:30.771920+0000","last_fullsized":"2024-03-21T13:02:30.771920+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:40.658883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.658883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:46.000148+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"126'8624","reported_seq":17601,"reported_epoch":126,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:02:44.745714+0000","last_change":"2024-03-21T13:02:15.531917+0000","last_active":"2024-03-21T13:02:44.745714+0000","last_peered":"2024-03-21T13:02:44.745714+0000","last_clean":"2024-03-21T13:02:13.279925+0000","last_became_active":"2024-03-21T13:02:15.516624+0000","last_became_peered":"2024-03-21T13:02:15.516624+0000","last_unstale":"2024-03-21T13:02:44.745714+0000","last_undegraded":"2024-03-21T13:02:44.745714+0000","last_fullsized":"2024-03-21T13:02:44.745714+0000","mapping_epoch":118,"log_start":"107'4439","ondisk_log_start":"107'4439","created":92,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396099584,"num_objects":6044,"num_object_clones":0,"num_object_copies":18132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2094,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8624,"num_write_kb":469376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4447,"num_bytes_recovered":291438592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[0,4,1],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":6044}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.772953+0000","last_change":"2024-03-21T13:01:14.114540+0000","last_active":"2024-03-21T13:02:30.772953+0000","last_peered":"2024-03-21T13:02:30.772953+0000","last_clean":"2024-03-21T13:02:30.772953+0000","last_became_active":"2024-03-21T12:58:43.179007+0000","last_became_peered":"2024-03-21T12:58:43.179007+0000","last_unstale":"2024-03-21T13:02:30.772953+0000","last_undegraded":"2024-03-21T13:02:30.772953+0000","last_fullsized":"2024-03-21T13:02:30.772953+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:14.114420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:14.114420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:32.250957+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":150,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773938+0000","last_change":"2024-03-21T13:02:26.238463+0000","last_active":"2024-03-21T13:02:30.773938+0000","last_peered":"2024-03-21T13:02:30.773938+0000","last_clean":"2024-03-21T13:02:30.773938+0000","last_became_active":"2024-03-21T13:02:26.238116+0000","last_became_peered":"2024-03-21T13:02:26.238116+0000","last_unstale":"2024-03-21T13:02:30.773938+0000","last_undegraded":"2024-03-21T13:02:30.773938+0000","last_fullsized":"2024-03-21T13:02:30.773938+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:39.655215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.655215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:55.723130+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":155,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.774068+0000","last_change":"2024-03-21T13:02:26.237114+0000","last_active":"2024-03-21T13:02:30.774068+0000","last_peered":"2024-03-21T13:02:30.774068+0000","last_clean":"2024-03-21T13:02:30.774068+0000","last_became_active":"2024-03-21T13:02:26.236780+0000","last_became_peered":"2024-03-21T13:02:26.236780+0000","last_unstale":"2024-03-21T13:02:30.774068+0000","last_undegraded":"2024-03-21T13:02:30.774068+0000","last_fullsized":"2024-03-21T13:02:30.774068+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:18.012391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.012391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:47.239056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"71'19","reported_seq":230,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.776323+0000","last_change":"2024-03-21T13:02:26.275429+0000","last_active":"2024-03-21T13:02:30.776323+0000","last_peered":"2024-03-21T13:02:30.776323+0000","last_clean":"2024-03-21T13:02:30.776323+0000","last_became_active":"2024-03-21T13:02:26.240350+0000","last_became_peered":"2024-03-21T13:02:26.240350+0000","last_unstale":"2024-03-21T13:02:30.776323+0000","last_undegraded":"2024-03-21T13:02:30.776323+0000","last_fullsized":"2024-03-21T13:02:30.776323+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:01:18.623551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.623551+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:36.880940+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"126'8560","reported_seq":8718,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:44.705682+0000","last_change":"2024-03-21T13:02:26.586862+0000","last_active":"2024-03-21T13:02:44.705682+0000","last_peered":"2024-03-21T13:02:44.705682+0000","last_clean":"2024-03-21T13:02:44.705682+0000","last_became_active":"2024-03-21T13:00:42.708899+0000","last_became_peered":"2024-03-21T13:00:42.708899+0000","last_unstale":"2024-03-21T13:02:44.705682+0000","last_undegraded":"2024-03-21T13:02:44.705682+0000","last_fullsized":"2024-03-21T13:02:44.705682+0000","mapping_epoch":92,"log_start":"104'3841","ondisk_log_start":"104'3841","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7428","last_scrub_stamp":"2024-03-21T13:02:26.586741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:02:26.586741+0000","objects_scrubbed":7233,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:45.825198+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399507456,"num_objects":6096,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8560,"num_write_kb":468992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":42,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773343+0000","last_change":"2024-03-21T13:01:48.214335+0000","last_active":"2024-03-21T13:02:30.773343+0000","last_peered":"2024-03-21T13:02:30.773343+0000","last_clean":"2024-03-21T13:02:30.773343+0000","last_became_active":"2024-03-21T13:01:48.213995+0000","last_became_peered":"2024-03-21T13:01:48.213995+0000","last_unstale":"2024-03-21T13:02:30.773343+0000","last_undegraded":"2024-03-21T13:02:30.773343+0000","last_fullsized":"2024-03-21T13:02:30.773343+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:44.997946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:44.997946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:55.883675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":144,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.772478+0000","last_change":"2024-03-21T13:01:46.642867+0000","last_active":"2024-03-21T13:02:30.772478+0000","last_peered":"2024-03-21T13:02:30.772478+0000","last_clean":"2024-03-21T13:02:30.772478+0000","last_became_active":"2024-03-21T12:58:41.163843+0000","last_became_peered":"2024-03-21T12:58:41.163843+0000","last_unstale":"2024-03-21T13:02:30.772478+0000","last_undegraded":"2024-03-21T13:02:30.772478+0000","last_fullsized":"2024-03-21T13:02:30.772478+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:46.642749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.642749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:07.227477+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.789086+0000","last_change":"2024-03-21T13:02:26.244355+0000","last_active":"2024-03-21T13:02:30.789086+0000","last_peered":"2024-03-21T13:02:30.789086+0000","last_clean":"2024-03-21T13:02:30.789086+0000","last_became_active":"2024-03-21T13:02:26.243974+0000","last_became_peered":"2024-03-21T13:02:26.243974+0000","last_unstale":"2024-03-21T13:02:30.789086+0000","last_undegraded":"2024-03-21T13:02:30.789086+0000","last_fullsized":"2024-03-21T13:02:30.789086+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:33.998382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:33.998382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:00.799821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"86'13","reported_seq":106,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.771107+0000","last_change":"2024-03-21T13:02:14.369004+0000","last_active":"2024-03-21T13:02:30.771107+0000","last_peered":"2024-03-21T13:02:30.771107+0000","last_clean":"2024-03-21T13:02:30.771107+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:02:30.771107+0000","last_undegraded":"2024-03-21T13:02:30.771107+0000","last_fullsized":"2024-03-21T13:02:30.771107+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:01:26.174361+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:01:26.174361+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:44.518951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.7","version":"126'8734","reported_seq":8812,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:44.650812+0000","last_change":"2024-03-21T13:00:42.707087+0000","last_active":"2024-03-21T13:02:44.650812+0000","last_peered":"2024-03-21T13:02:44.650812+0000","last_clean":"2024-03-21T13:02:44.650812+0000","last_became_active":"2024-03-21T13:00:42.706774+0000","last_became_peered":"2024-03-21T13:00:42.706774+0000","last_unstale":"2024-03-21T13:02:44.650812+0000","last_undegraded":"2024-03-21T13:02:44.650812+0000","last_fullsized":"2024-03-21T13:02:44.650812+0000","mapping_epoch":92,"log_start":"106'4499","ondisk_log_start":"106'4499","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406061056,"num_objects":6196,"num_object_clones":0,"num_object_copies":18588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8734,"num_write_kb":477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":131,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.785943+0000","last_change":"2024-03-21T13:01:46.738822+0000","last_active":"2024-03-21T13:02:30.785943+0000","last_peered":"2024-03-21T13:02:30.785943+0000","last_clean":"2024-03-21T13:02:30.785943+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:02:30.785943+0000","last_undegraded":"2024-03-21T13:02:30.785943+0000","last_fullsized":"2024-03-21T13:02:30.785943+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.738659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:08.964050+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"67'2","reported_seq":46,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.776222+0000","last_change":"2024-03-21T13:02:26.266567+0000","last_active":"2024-03-21T13:02:30.776222+0000","last_peered":"2024-03-21T13:02:30.776222+0000","last_clean":"2024-03-21T13:02:30.776222+0000","last_became_active":"2024-03-21T13:02:26.243692+0000","last_became_peered":"2024-03-21T13:02:26.243692+0000","last_unstale":"2024-03-21T13:02:30.776222+0000","last_undegraded":"2024-03-21T13:02:30.776222+0000","last_fullsized":"2024-03-21T13:02:30.776222+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-21T13:01:41.017193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:01:41.017193+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:48.236093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":151,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.771224+0000","last_change":"2024-03-21T13:01:31.721299+0000","last_active":"2024-03-21T13:02:30.771224+0000","last_peered":"2024-03-21T13:02:30.771224+0000","last_clean":"2024-03-21T13:02:30.771224+0000","last_became_active":"2024-03-21T12:58:37.729454+0000","last_became_peered":"2024-03-21T12:58:37.729454+0000","last_unstale":"2024-03-21T13:02:30.771224+0000","last_undegraded":"2024-03-21T13:02:30.771224+0000","last_fullsized":"2024-03-21T13:02:30.771224+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:31.721222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.721222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:32.502719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"119'29","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:37.819382+0000","last_change":"2024-03-21T13:02:14.382134+0000","last_active":"2024-03-21T13:02:37.819382+0000","last_peered":"2024-03-21T13:02:37.819382+0000","last_clean":"2024-03-21T13:02:37.819382+0000","last_became_active":"2024-03-21T13:02:14.351114+0000","last_became_peered":"2024-03-21T13:02:14.351114+0000","last_unstale":"2024-03-21T13:02:37.819382+0000","last_undegraded":"2024-03-21T13:02:37.819382+0000","last_fullsized":"2024-03-21T13:02:37.819382+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'28","last_scrub_stamp":"2024-03-21T13:01:36.231420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:36.231420+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:47.255112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":30,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.771106+0000","last_change":"2024-03-21T13:02:14.336654+0000","last_active":"2024-03-21T13:02:30.771106+0000","last_peered":"2024-03-21T13:02:30.771106+0000","last_clean":"2024-03-21T13:02:30.771106+0000","last_became_active":"2024-03-21T13:02:14.336177+0000","last_became_peered":"2024-03-21T13:02:14.336177+0000","last_unstale":"2024-03-21T13:02:30.771106+0000","last_undegraded":"2024-03-21T13:02:30.771106+0000","last_fullsized":"2024-03-21T13:02:30.771106+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:36.760263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:36.760263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.4","version":"126'8524","reported_seq":15576,"reported_epoch":126,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-21T13:02:44.635089+0000","last_change":"2024-03-21T13:02:26.250493+0000","last_active":"2024-03-21T13:02:44.635089+0000","last_peered":"2024-03-21T13:02:44.635089+0000","last_clean":"2024-03-21T13:02:13.309754+0000","last_became_active":"2024-03-21T13:02:15.383776+0000","last_became_peered":"2024-03-21T13:02:15.383776+0000","last_unstale":"2024-03-21T13:02:44.635089+0000","last_undegraded":"2024-03-21T13:02:44.635089+0000","last_fullsized":"2024-03-21T13:02:44.635089+0000","mapping_epoch":118,"log_start":"107'4482","ondisk_log_start":"107'4482","created":92,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4042,"log_dups_size":0,"ondisk_log_size":4042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400556032,"num_objects":6112,"num_object_clones":0,"num_object_copies":18336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3096,"num_objects_unfound":0,"num_objects_dirty":6112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8524,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3478,"num_bytes_recovered":227934208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,1],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":6112}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.774929+0000","last_change":"2024-03-21T13:01:28.061976+0000","last_active":"2024-03-21T13:02:30.774929+0000","last_peered":"2024-03-21T13:02:30.774929+0000","last_clean":"2024-03-21T13:02:30.774929+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:02:30.774929+0000","last_undegraded":"2024-03-21T13:02:30.774929+0000","last_fullsized":"2024-03-21T13:02:30.774929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:28.061868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:28.061868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:39.070422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":36,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.772083+0000","last_change":"2024-03-21T13:02:21.275708+0000","last_active":"2024-03-21T13:02:30.772083+0000","last_peered":"2024-03-21T13:02:30.772083+0000","last_clean":"2024-03-21T13:02:30.772083+0000","last_became_active":"2024-03-21T13:02:14.328020+0000","last_became_peered":"2024-03-21T13:02:14.328020+0000","last_unstale":"2024-03-21T13:02:30.772083+0000","last_undegraded":"2024-03-21T13:02:30.772083+0000","last_fullsized":"2024-03-21T13:02:30.772083+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:02:21.275321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:02:21.275321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:47.786062+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":161,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.780967+0000","last_change":"2024-03-21T13:01:16.871013+0000","last_active":"2024-03-21T13:02:30.780967+0000","last_peered":"2024-03-21T13:02:30.780967+0000","last_clean":"2024-03-21T13:02:30.780967+0000","last_became_active":"2024-03-21T12:58:36.724067+0000","last_became_peered":"2024-03-21T12:58:36.724067+0000","last_unstale":"2024-03-21T13:02:30.780967+0000","last_undegraded":"2024-03-21T13:02:30.780967+0000","last_fullsized":"2024-03-21T13:02:30.780967+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:16.870806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:01:16.870806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:31.276770+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0.000743678,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'13","reported_seq":76,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773730+0000","last_change":"2024-03-21T13:02:26.260934+0000","last_active":"2024-03-21T13:02:30.773730+0000","last_peered":"2024-03-21T13:02:30.773730+0000","last_clean":"2024-03-21T13:02:30.773730+0000","last_became_active":"2024-03-21T13:02:26.234230+0000","last_became_peered":"2024-03-21T13:02:26.234230+0000","last_unstale":"2024-03-21T13:02:30.773730+0000","last_undegraded":"2024-03-21T13:02:30.773730+0000","last_fullsized":"2024-03-21T13:02:30.773730+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'13","last_scrub_stamp":"2024-03-21T13:01:31.190053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.190053+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:43.617567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":150,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.774657+0000","last_change":"2024-03-21T13:02:26.228478+0000","last_active":"2024-03-21T13:02:30.774657+0000","last_peered":"2024-03-21T13:02:30.774657+0000","last_clean":"2024-03-21T13:02:30.774657+0000","last_became_active":"2024-03-21T13:02:26.228128+0000","last_became_peered":"2024-03-21T13:02:26.228128+0000","last_unstale":"2024-03-21T13:02:30.774657+0000","last_undegraded":"2024-03-21T13:02:30.774657+0000","last_fullsized":"2024-03-21T13:02:30.774657+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:25.668960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:25.668960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:52.795418+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"126'8531","reported_seq":8613,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:44.642378+0000","last_change":"2024-03-21T13:01:48.219270+0000","last_active":"2024-03-21T13:02:44.642378+0000","last_peered":"2024-03-21T13:02:44.642378+0000","last_clean":"2024-03-21T13:02:44.642378+0000","last_became_active":"2024-03-21T13:01:48.213755+0000","last_became_peered":"2024-03-21T13:01:48.213755+0000","last_unstale":"2024-03-21T13:02:44.642378+0000","last_undegraded":"2024-03-21T13:02:44.642378+0000","last_fullsized":"2024-03-21T13:02:44.642378+0000","mapping_epoch":110,"log_start":"107'4462","ondisk_log_start":"107'4462","created":92,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4069,"log_dups_size":0,"ondisk_log_size":4069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399179776,"num_objects":6091,"num_object_clones":0,"num_object_copies":18273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8531,"num_write_kb":467904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.788313+0000","last_change":"2024-03-21T13:02:26.236072+0000","last_active":"2024-03-21T13:02:30.788313+0000","last_peered":"2024-03-21T13:02:30.788313+0000","last_clean":"2024-03-21T13:02:30.788313+0000","last_became_active":"2024-03-21T13:02:26.235640+0000","last_became_peered":"2024-03-21T13:02:26.235640+0000","last_unstale":"2024-03-21T13:02:30.788313+0000","last_undegraded":"2024-03-21T13:02:30.788313+0000","last_fullsized":"2024-03-21T13:02:30.788313+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:52.658377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.658377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":399,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773080+0000","last_change":"2024-03-21T13:00:16.180260+0000","last_active":"2024-03-21T13:02:30.773080+0000","last_peered":"2024-03-21T13:02:30.773080+0000","last_clean":"2024-03-21T13:02:30.773080+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:02:30.773080+0000","last_undegraded":"2024-03-21T13:02:30.773080+0000","last_fullsized":"2024-03-21T13:02:30.773080+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:00:16.180126+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:00:16.180126+0000","last_clean_scrub_stamp":"2024-03-21T13:00:16.180126+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.786309+0000","last_change":"2024-03-21T13:02:26.242969+0000","last_active":"2024-03-21T13:02:30.786309+0000","last_peered":"2024-03-21T13:02:30.786309+0000","last_clean":"2024-03-21T13:02:30.786309+0000","last_became_active":"2024-03-21T13:02:26.242456+0000","last_became_peered":"2024-03-21T13:02:26.242456+0000","last_unstale":"2024-03-21T13:02:30.786309+0000","last_undegraded":"2024-03-21T13:02:30.786309+0000","last_fullsized":"2024-03-21T13:02:30.786309+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:38.007486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.007486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":219,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:37.820290+0000","last_change":"2024-03-21T13:01:21.574857+0000","last_active":"2024-03-21T13:02:37.820290+0000","last_peered":"2024-03-21T13:02:37.820290+0000","last_clean":"2024-03-21T13:02:37.820290+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:02:37.820290+0000","last_undegraded":"2024-03-21T13:02:37.820290+0000","last_fullsized":"2024-03-21T13:02:37.820290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-21T13:01:21.574712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:21.574712+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:47.115069+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":147,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.774522+0000","last_change":"2024-03-21T13:02:26.235494+0000","last_active":"2024-03-21T13:02:30.774522+0000","last_peered":"2024-03-21T13:02:30.774522+0000","last_clean":"2024-03-21T13:02:30.774522+0000","last_became_active":"2024-03-21T13:02:26.235140+0000","last_became_peered":"2024-03-21T13:02:26.235140+0000","last_unstale":"2024-03-21T13:02:30.774522+0000","last_undegraded":"2024-03-21T13:02:30.774522+0000","last_fullsized":"2024-03-21T13:02:30.774522+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.968175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:45.853846+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"126'8571","reported_seq":8649,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:44.641264+0000","last_change":"2024-03-21T13:00:42.710083+0000","last_active":"2024-03-21T13:02:44.641264+0000","last_peered":"2024-03-21T13:02:44.641264+0000","last_clean":"2024-03-21T13:02:44.641264+0000","last_became_active":"2024-03-21T13:00:42.709753+0000","last_became_peered":"2024-03-21T13:00:42.709753+0000","last_unstale":"2024-03-21T13:02:44.641264+0000","last_undegraded":"2024-03-21T13:02:44.641264+0000","last_fullsized":"2024-03-21T13:02:44.641264+0000","mapping_epoch":92,"log_start":"106'4420","ondisk_log_start":"106'4420","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4151,"log_dups_size":0,"ondisk_log_size":4151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405995520,"num_objects":6195,"num_object_clones":0,"num_object_copies":18585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8571,"num_write_kb":472512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":18,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.788633+0000","last_change":"2024-03-21T13:02:26.243121+0000","last_active":"2024-03-21T13:02:30.788633+0000","last_peered":"2024-03-21T13:02:30.788633+0000","last_clean":"2024-03-21T13:02:30.788633+0000","last_became_active":"2024-03-21T13:02:26.242708+0000","last_became_peered":"2024-03-21T13:02:26.242708+0000","last_unstale":"2024-03-21T13:02:30.788633+0000","last_undegraded":"2024-03-21T13:02:30.788633+0000","last_fullsized":"2024-03-21T13:02:30.788633+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:20.854917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.854917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":42,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.772188+0000","last_change":"2024-03-21T13:01:48.212127+0000","last_active":"2024-03-21T13:02:30.772188+0000","last_peered":"2024-03-21T13:02:30.772188+0000","last_clean":"2024-03-21T13:02:30.772188+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:02:30.772188+0000","last_undegraded":"2024-03-21T13:02:30.772188+0000","last_fullsized":"2024-03-21T13:02:30.772188+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:01:34.066532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:50.405498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":402,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:43.376216+0000","last_change":"2024-03-21T13:02:14.358837+0000","last_active":"2024-03-21T13:02:43.376216+0000","last_peered":"2024-03-21T13:02:43.376216+0000","last_clean":"2024-03-21T13:02:43.376216+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:02:43.376216+0000","last_undegraded":"2024-03-21T13:02:43.376216+0000","last_fullsized":"2024-03-21T13:02:43.376216+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:00:15.164219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:00:15.164219+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":238,"num_read_kb":233,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":184,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.769482+0000","last_change":"2024-03-21T13:02:14.359827+0000","last_active":"2024-03-21T13:02:30.769482+0000","last_peered":"2024-03-21T13:02:30.769482+0000","last_clean":"2024-03-21T13:02:30.769482+0000","last_became_active":"2024-03-21T13:02:14.333917+0000","last_became_peered":"2024-03-21T13:02:14.333917+0000","last_unstale":"2024-03-21T13:02:30.769482+0000","last_undegraded":"2024-03-21T13:02:30.769482+0000","last_fullsized":"2024-03-21T13:02:30.769482+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:01:28.111201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:28.111201+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:40.512588+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773952+0000","last_change":"2024-03-21T13:02:26.232971+0000","last_active":"2024-03-21T13:02:30.773952+0000","last_peered":"2024-03-21T13:02:30.773952+0000","last_clean":"2024-03-21T13:02:30.773952+0000","last_became_active":"2024-03-21T13:02:26.232625+0000","last_became_peered":"2024-03-21T13:02:26.232625+0000","last_unstale":"2024-03-21T13:02:30.773952+0000","last_undegraded":"2024-03-21T13:02:30.773952+0000","last_fullsized":"2024-03-21T13:02:30.773952+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:34.982531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:34.982531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:42.653760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"126'8546","reported_seq":8636,"reported_epoch":126,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-21T13:02:44.825467+0000","last_change":"2024-03-21T13:02:27.381219+0000","last_active":"2024-03-21T13:02:44.825467+0000","last_peered":"2024-03-21T13:02:44.825467+0000","last_clean":"2024-03-21T13:02:13.315649+0000","last_became_active":"2024-03-21T13:02:27.316276+0000","last_became_peered":"2024-03-21T13:02:27.316276+0000","last_unstale":"2024-03-21T13:02:44.825467+0000","last_undegraded":"2024-03-21T13:02:44.825467+0000","last_fullsized":"2024-03-21T13:02:44.825467+0000","mapping_epoch":124,"log_start":"109'4958","ondisk_log_start":"109'4958","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3588,"log_dups_size":0,"ondisk_log_size":3588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410124288,"num_objects":6258,"num_object_clones":0,"num_object_copies":18774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12516,"num_objects_unfound":0,"num_objects_dirty":6258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":473728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,5,1],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":6258}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":28,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.777102+0000","last_change":"2024-03-21T13:02:14.336240+0000","last_active":"2024-03-21T13:02:30.777102+0000","last_peered":"2024-03-21T13:02:30.777102+0000","last_clean":"2024-03-21T13:02:30.777102+0000","last_became_active":"2024-03-21T13:02:14.335897+0000","last_became_peered":"2024-03-21T13:02:14.335897+0000","last_unstale":"2024-03-21T13:02:30.777102+0000","last_undegraded":"2024-03-21T13:02:30.777102+0000","last_fullsized":"2024-03-21T13:02:30.777102+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:21.040252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:21.040252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:40.380350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.781336+0000","last_change":"2024-03-21T13:02:26.238041+0000","last_active":"2024-03-21T13:02:30.781336+0000","last_peered":"2024-03-21T13:02:30.781336+0000","last_clean":"2024-03-21T13:02:30.781336+0000","last_became_active":"2024-03-21T13:02:26.237683+0000","last_became_peered":"2024-03-21T13:02:26.237683+0000","last_unstale":"2024-03-21T13:02:30.781336+0000","last_undegraded":"2024-03-21T13:02:30.781336+0000","last_fullsized":"2024-03-21T13:02:30.781336+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:24.072225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:24.072225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:43.387359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"71'19","reported_seq":89,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773991+0000","last_change":"2024-03-21T13:02:26.274805+0000","last_active":"2024-03-21T13:02:30.773991+0000","last_peered":"2024-03-21T13:02:30.773991+0000","last_clean":"2024-03-21T13:02:30.773991+0000","last_became_active":"2024-03-21T13:02:26.239140+0000","last_became_peered":"2024-03-21T13:02:26.239140+0000","last_unstale":"2024-03-21T13:02:30.773991+0000","last_undegraded":"2024-03-21T13:02:30.773991+0000","last_fullsized":"2024-03-21T13:02:30.773991+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:01:29.106715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:29.106715+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:48.135109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.776321+0000","last_change":"2024-03-21T13:02:26.252937+0000","last_active":"2024-03-21T13:02:30.776321+0000","last_peered":"2024-03-21T13:02:30.776321+0000","last_clean":"2024-03-21T13:02:30.776321+0000","last_became_active":"2024-03-21T13:02:26.252372+0000","last_became_peered":"2024-03-21T13:02:26.252372+0000","last_unstale":"2024-03-21T13:02:30.776321+0000","last_undegraded":"2024-03-21T13:02:30.776321+0000","last_fullsized":"2024-03-21T13:02:30.776321+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:22.601641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:22.601641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:34.500891+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"126'8559","reported_seq":8696,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:44.612039+0000","last_change":"2024-03-21T13:01:47.452788+0000","last_active":"2024-03-21T13:02:44.612039+0000","last_peered":"2024-03-21T13:02:44.612039+0000","last_clean":"2024-03-21T13:02:44.612039+0000","last_became_active":"2024-03-21T13:00:42.707521+0000","last_became_peered":"2024-03-21T13:00:42.707521+0000","last_unstale":"2024-03-21T13:02:44.612039+0000","last_undegraded":"2024-03-21T13:02:44.612039+0000","last_fullsized":"2024-03-21T13:02:44.612039+0000","mapping_epoch":92,"log_start":"108'4536","ondisk_log_start":"108'4536","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'5147","last_scrub_stamp":"2024-03-21T13:01:47.452664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:01:47.452664+0000","objects_scrubbed":5137,"log_size":4023,"log_dups_size":0,"ondisk_log_size":4023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:12.838546+0000","scrub_duration":887,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395640832,"num_objects":6037,"num_object_clones":0,"num_object_copies":18111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8559,"num_write_kb":467072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":42,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.782920+0000","last_change":"2024-03-21T13:01:48.209228+0000","last_active":"2024-03-21T13:02:30.782920+0000","last_peered":"2024-03-21T13:02:30.782920+0000","last_clean":"2024-03-21T13:02:30.782920+0000","last_became_active":"2024-03-21T13:01:48.208899+0000","last_became_peered":"2024-03-21T13:01:48.208899+0000","last_unstale":"2024-03-21T13:02:30.782920+0000","last_undegraded":"2024-03-21T13:02:30.782920+0000","last_fullsized":"2024-03-21T13:02:30.782920+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:31.121913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.121913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:35.731749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.785287+0000","last_change":"2024-03-21T13:02:26.244688+0000","last_active":"2024-03-21T13:02:30.785287+0000","last_peered":"2024-03-21T13:02:30.785287+0000","last_clean":"2024-03-21T13:02:30.785287+0000","last_became_active":"2024-03-21T13:02:26.243417+0000","last_became_peered":"2024-03-21T13:02:26.243417+0000","last_unstale":"2024-03-21T13:02:30.785287+0000","last_undegraded":"2024-03-21T13:02:30.785287+0000","last_fullsized":"2024-03-21T13:02:30.785287+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:20.666893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.666893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"86'15","reported_seq":74,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.770658+0000","last_change":"2024-03-21T13:02:26.263374+0000","last_active":"2024-03-21T13:02:30.770658+0000","last_peered":"2024-03-21T13:02:30.770658+0000","last_clean":"2024-03-21T13:02:30.770658+0000","last_became_active":"2024-03-21T13:02:26.232580+0000","last_became_peered":"2024-03-21T13:02:26.232580+0000","last_unstale":"2024-03-21T13:02:30.770658+0000","last_undegraded":"2024-03-21T13:02:30.770658+0000","last_fullsized":"2024-03-21T13:02:30.770658+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-03-21T13:01:27.655920+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:01:27.655920+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:31.613941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"125'12","reported_seq":91,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:37.819509+0000","last_change":"2024-03-21T13:02:26.269099+0000","last_active":"2024-03-21T13:02:37.819509+0000","last_peered":"2024-03-21T13:02:37.819509+0000","last_clean":"2024-03-21T13:02:37.819509+0000","last_became_active":"2024-03-21T13:02:26.253720+0000","last_became_peered":"2024-03-21T13:02:26.253720+0000","last_unstale":"2024-03-21T13:02:37.819509+0000","last_undegraded":"2024-03-21T13:02:37.819509+0000","last_fullsized":"2024-03-21T13:02:37.819509+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:25.175225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:25.175225+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"126'8652","reported_seq":8741,"reported_epoch":126,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-21T13:02:44.858623+0000","last_change":"2024-03-21T13:02:15.585871+0000","last_active":"2024-03-21T13:02:44.858623+0000","last_peered":"2024-03-21T13:02:44.858623+0000","last_clean":"2024-03-21T13:02:13.308104+0000","last_became_active":"2024-03-21T13:02:15.519968+0000","last_became_peered":"2024-03-21T13:02:15.519968+0000","last_unstale":"2024-03-21T13:02:44.858623+0000","last_undegraded":"2024-03-21T13:02:44.858623+0000","last_fullsized":"2024-03-21T13:02:44.858623+0000","mapping_epoch":118,"log_start":"109'5083","ondisk_log_start":"109'5083","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404488192,"num_objects":6172,"num_object_clones":0,"num_object_copies":18516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6172,"num_objects_unfound":0,"num_objects_dirty":6172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8652,"num_write_kb":474368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":6172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.774115+0000","last_change":"2024-03-21T13:01:50.086495+0000","last_active":"2024-03-21T13:02:30.774115+0000","last_peered":"2024-03-21T13:02:30.774115+0000","last_clean":"2024-03-21T13:02:30.774115+0000","last_became_active":"2024-03-21T12:58:43.353049+0000","last_became_peered":"2024-03-21T12:58:43.353049+0000","last_unstale":"2024-03-21T13:02:30.774115+0000","last_undegraded":"2024-03-21T13:02:30.774115+0000","last_fullsized":"2024-03-21T13:02:30.774115+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:50.086373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:50.086373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:58.927435+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":161,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.769588+0000","last_change":"2024-03-21T13:02:14.351423+0000","last_active":"2024-03-21T13:02:30.769588+0000","last_peered":"2024-03-21T13:02:30.769588+0000","last_clean":"2024-03-21T13:02:30.769588+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:02:30.769588+0000","last_undegraded":"2024-03-21T13:02:30.769588+0000","last_fullsized":"2024-03-21T13:02:30.769588+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:01:39.008715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.008715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:46.627750+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"87'21","reported_seq":108,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.776857+0000","last_change":"2024-03-21T13:02:26.263470+0000","last_active":"2024-03-21T13:02:30.776857+0000","last_peered":"2024-03-21T13:02:30.776857+0000","last_clean":"2024-03-21T13:02:30.776857+0000","last_became_active":"2024-03-21T13:02:26.227165+0000","last_became_peered":"2024-03-21T13:02:26.227165+0000","last_unstale":"2024-03-21T13:02:30.776857+0000","last_undegraded":"2024-03-21T13:02:30.776857+0000","last_fullsized":"2024-03-21T13:02:30.776857+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'21","last_scrub_stamp":"2024-03-21T13:01:29.911046+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:01:29.911046+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:33.593899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"119'9","reported_seq":205,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:42.822799+0000","last_change":"2024-03-21T13:02:14.375956+0000","last_active":"2024-03-21T13:02:42.822799+0000","last_peered":"2024-03-21T13:02:42.822799+0000","last_clean":"2024-03-21T13:02:42.822799+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:02:42.822799+0000","last_undegraded":"2024-03-21T13:02:42.822799+0000","last_fullsized":"2024-03-21T13:02:42.822799+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:25.700549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:25.700549+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"126'8530","reported_seq":8608,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:44.855882+0000","last_change":"2024-03-21T13:00:42.711111+0000","last_active":"2024-03-21T13:02:44.855882+0000","last_peered":"2024-03-21T13:02:44.855882+0000","last_clean":"2024-03-21T13:02:44.855882+0000","last_became_active":"2024-03-21T13:00:42.710829+0000","last_became_peered":"2024-03-21T13:00:42.710829+0000","last_unstale":"2024-03-21T13:02:44.855882+0000","last_undegraded":"2024-03-21T13:02:44.855882+0000","last_fullsized":"2024-03-21T13:02:44.855882+0000","mapping_epoch":92,"log_start":"109'4974","ondisk_log_start":"109'4974","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3556,"log_dups_size":0,"ondisk_log_size":3556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399507456,"num_objects":6096,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8530,"num_write_kb":468032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":56,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.788429+0000","last_change":"2024-03-21T13:01:27.382938+0000","last_active":"2024-03-21T13:02:30.788429+0000","last_peered":"2024-03-21T13:02:30.788429+0000","last_clean":"2024-03-21T13:02:30.788429+0000","last_became_active":"2024-03-21T13:01:27.382663+0000","last_became_peered":"2024-03-21T13:01:27.382663+0000","last_unstale":"2024-03-21T13:02:30.788429+0000","last_undegraded":"2024-03-21T13:02:30.788429+0000","last_fullsized":"2024-03-21T13:02:30.788429+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.823841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":143,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.788053+0000","last_change":"2024-03-21T12:59:43.240826+0000","last_active":"2024-03-21T13:02:30.788053+0000","last_peered":"2024-03-21T13:02:30.788053+0000","last_clean":"2024-03-21T13:02:30.788053+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:02:30.788053+0000","last_undegraded":"2024-03-21T13:02:30.788053+0000","last_fullsized":"2024-03-21T13:02:30.788053+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T12:59:43.240674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"71'18","reported_seq":172,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.784254+0000","last_change":"2024-03-21T12:59:48.179286+0000","last_active":"2024-03-21T13:02:30.784254+0000","last_peered":"2024-03-21T13:02:30.784254+0000","last_clean":"2024-03-21T13:02:30.784254+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:02:30.784254+0000","last_undegraded":"2024-03-21T13:02:30.784254+0000","last_fullsized":"2024-03-21T13:02:30.784254+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T12:59:48.179188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.179188+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":135,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.772015+0000","last_change":"2024-03-21T13:01:17.590965+0000","last_active":"2024-03-21T13:02:30.772015+0000","last_peered":"2024-03-21T13:02:30.772015+0000","last_clean":"2024-03-21T13:02:30.772015+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:02:30.772015+0000","last_undegraded":"2024-03-21T13:02:30.772015+0000","last_fullsized":"2024-03-21T13:02:30.772015+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:17.590831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:17.590831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:43.546310+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"126'8571","reported_seq":8649,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:44.667321+0000","last_change":"2024-03-21T13:00:42.705026+0000","last_active":"2024-03-21T13:02:44.667321+0000","last_peered":"2024-03-21T13:02:44.667321+0000","last_clean":"2024-03-21T13:02:44.667321+0000","last_became_active":"2024-03-21T13:00:42.704712+0000","last_became_peered":"2024-03-21T13:00:42.704712+0000","last_unstale":"2024-03-21T13:02:44.667321+0000","last_undegraded":"2024-03-21T13:02:44.667321+0000","last_fullsized":"2024-03-21T13:02:44.667321+0000","mapping_epoch":92,"log_start":"106'4368","ondisk_log_start":"106'4368","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4203,"log_dups_size":0,"ondisk_log_size":4203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399572992,"num_objects":6097,"num_object_clones":0,"num_object_copies":18291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8571,"num_write_kb":469376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":136,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773478+0000","last_change":"2024-03-21T13:02:26.236964+0000","last_active":"2024-03-21T13:02:30.773478+0000","last_peered":"2024-03-21T13:02:30.773478+0000","last_clean":"2024-03-21T13:02:30.773478+0000","last_became_active":"2024-03-21T13:02:26.236501+0000","last_became_peered":"2024-03-21T13:02:26.236501+0000","last_unstale":"2024-03-21T13:02:30.773478+0000","last_undegraded":"2024-03-21T13:02:30.773478+0000","last_fullsized":"2024-03-21T13:02:30.773478+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:24.875883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:24.875883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:42.208002+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"125'10","reported_seq":49,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:37.820841+0000","last_change":"2024-03-21T13:02:26.252598+0000","last_active":"2024-03-21T13:02:37.820841+0000","last_peered":"2024-03-21T13:02:37.820841+0000","last_clean":"2024-03-21T13:02:37.820841+0000","last_became_active":"2024-03-21T13:02:26.231372+0000","last_became_peered":"2024-03-21T13:02:26.231372+0000","last_unstale":"2024-03-21T13:02:37.820841+0000","last_undegraded":"2024-03-21T13:02:37.820841+0000","last_fullsized":"2024-03-21T13:02:37.820841+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:00:23.782010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:23.782010+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.783383+0000","last_change":"2024-03-21T13:02:14.352037+0000","last_active":"2024-03-21T13:02:30.783383+0000","last_peered":"2024-03-21T13:02:30.783383+0000","last_clean":"2024-03-21T13:02:30.783383+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:02:30.783383+0000","last_undegraded":"2024-03-21T13:02:30.783383+0000","last_fullsized":"2024-03-21T13:02:30.783383+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:41.231340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:41.231340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":88,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.788137+0000","last_change":"2024-03-21T13:02:14.377274+0000","last_active":"2024-03-21T13:02:30.788137+0000","last_peered":"2024-03-21T13:02:30.788137+0000","last_clean":"2024-03-21T13:02:30.788137+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:02:30.788137+0000","last_undegraded":"2024-03-21T13:02:30.788137+0000","last_fullsized":"2024-03-21T13:02:30.788137+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:01:46.167736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.167736+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:55.828764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"126'8448","reported_seq":11492,"reported_epoch":126,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-21T13:02:44.655530+0000","last_change":"2024-03-21T13:02:26.270079+0000","last_active":"2024-03-21T13:02:44.655530+0000","last_peered":"2024-03-21T13:02:44.655530+0000","last_clean":"2024-03-21T13:02:13.317064+0000","last_became_active":"2024-03-21T13:02:15.410023+0000","last_became_peered":"2024-03-21T13:02:15.410023+0000","last_unstale":"2024-03-21T13:02:44.655530+0000","last_undegraded":"2024-03-21T13:02:44.655530+0000","last_fullsized":"2024-03-21T13:02:44.655530+0000","mapping_epoch":118,"log_start":"106'4244","ondisk_log_start":"106'4244","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4204,"log_dups_size":0,"ondisk_log_size":4204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393412632,"num_objects":6004,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4733,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8448,"num_write_kb":462401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1478,"num_bytes_recovered":96862208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,1],"avail_no_missing":["6","0","1"],"object_location_counts":[{"shards":"0,1,6","objects":6004}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":142,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.788723+0000","last_change":"2024-03-21T13:02:26.249048+0000","last_active":"2024-03-21T13:02:30.788723+0000","last_peered":"2024-03-21T13:02:30.788723+0000","last_clean":"2024-03-21T13:02:30.788723+0000","last_became_active":"2024-03-21T13:02:26.243985+0000","last_became_peered":"2024-03-21T13:02:26.243985+0000","last_unstale":"2024-03-21T13:02:30.788723+0000","last_undegraded":"2024-03-21T13:02:30.788723+0000","last_fullsized":"2024-03-21T13:02:30.788723+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:00:27.958396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:27.958396+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.771742+0000","last_change":"2024-03-21T13:02:26.229605+0000","last_active":"2024-03-21T13:02:30.771742+0000","last_peered":"2024-03-21T13:02:30.771742+0000","last_clean":"2024-03-21T13:02:30.771742+0000","last_became_active":"2024-03-21T13:02:26.229296+0000","last_became_peered":"2024-03-21T13:02:26.229296+0000","last_unstale":"2024-03-21T13:02:30.771742+0000","last_undegraded":"2024-03-21T13:02:30.771742+0000","last_fullsized":"2024-03-21T13:02:30.771742+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:36.008632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:36.008632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:39.693665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":151,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.772044+0000","last_change":"2024-03-21T13:01:39.629358+0000","last_active":"2024-03-21T13:02:30.772044+0000","last_peered":"2024-03-21T13:02:30.772044+0000","last_clean":"2024-03-21T13:02:30.772044+0000","last_became_active":"2024-03-21T12:58:37.730858+0000","last_became_peered":"2024-03-21T12:58:37.730858+0000","last_unstale":"2024-03-21T13:02:30.772044+0000","last_undegraded":"2024-03-21T13:02:30.772044+0000","last_fullsized":"2024-03-21T13:02:30.772044+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:39.629248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.629248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:51.372076+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'11","reported_seq":176,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.770970+0000","last_change":"2024-03-21T13:02:14.381387+0000","last_active":"2024-03-21T13:02:30.770970+0000","last_peered":"2024-03-21T13:02:30.770970+0000","last_clean":"2024-03-21T13:02:30.770970+0000","last_became_active":"2024-03-21T13:02:14.348499+0000","last_became_peered":"2024-03-21T13:02:14.348499+0000","last_unstale":"2024-03-21T13:02:30.770970+0000","last_undegraded":"2024-03-21T13:02:30.770970+0000","last_fullsized":"2024-03-21T13:02:30.770970+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:00:38.769807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.769807+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"126'8600","reported_seq":11496,"reported_epoch":126,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:02:44.716984+0000","last_change":"2024-03-21T13:02:27.319180+0000","last_active":"2024-03-21T13:02:44.716984+0000","last_peered":"2024-03-21T13:02:44.716984+0000","last_clean":"2024-03-21T13:02:25.730230+0000","last_became_active":"2024-03-21T13:02:27.296672+0000","last_became_peered":"2024-03-21T13:02:27.296672+0000","last_unstale":"2024-03-21T13:02:44.716984+0000","last_undegraded":"2024-03-21T13:02:44.716984+0000","last_fullsized":"2024-03-21T13:02:44.716984+0000","mapping_epoch":124,"log_start":"104'3884","ondisk_log_start":"104'3884","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'7457","last_scrub_stamp":"2024-03-21T13:02:25.543659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:02:25.543659+0000","objects_scrubbed":7338,"log_size":4716,"log_dups_size":0,"ondisk_log_size":4716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"no scrub is scheduled","scrub_duration":3245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402915328,"num_objects":6148,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4888,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":471936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":89194496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,5],"avail_no_missing":["2","0","5"],"object_location_counts":[{"shards":"0,2,5","objects":6148}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":140,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.776804+0000","last_change":"2024-03-21T13:01:47.562081+0000","last_active":"2024-03-21T13:02:30.776804+0000","last_peered":"2024-03-21T13:02:30.776804+0000","last_clean":"2024-03-21T13:02:30.776804+0000","last_became_active":"2024-03-21T12:58:43.183380+0000","last_became_peered":"2024-03-21T12:58:43.183380+0000","last_unstale":"2024-03-21T13:02:30.776804+0000","last_undegraded":"2024-03-21T13:02:30.776804+0000","last_fullsized":"2024-03-21T13:02:30.776804+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:47.561991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:01:47.561991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:01.535862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":28,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.771900+0000","last_change":"2024-03-21T13:02:14.335457+0000","last_active":"2024-03-21T13:02:30.771900+0000","last_peered":"2024-03-21T13:02:30.771900+0000","last_clean":"2024-03-21T13:02:30.771900+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:02:30.771900+0000","last_undegraded":"2024-03-21T13:02:30.771900+0000","last_fullsized":"2024-03-21T13:02:30.771900+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:17.006200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:17.006200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":157,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.776439+0000","last_change":"2024-03-21T13:02:14.343797+0000","last_active":"2024-03-21T13:02:30.776439+0000","last_peered":"2024-03-21T13:02:30.776439+0000","last_clean":"2024-03-21T13:02:30.776439+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:02:30.776439+0000","last_undegraded":"2024-03-21T13:02:30.776439+0000","last_fullsized":"2024-03-21T13:02:30.776439+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:16.595403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:16.595403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":135,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.773642+0000","last_change":"2024-03-21T13:02:27.259863+0000","last_active":"2024-03-21T13:02:30.773642+0000","last_peered":"2024-03-21T13:02:30.773642+0000","last_clean":"2024-03-21T13:02:30.773642+0000","last_became_active":"2024-03-21T13:02:14.334882+0000","last_became_peered":"2024-03-21T13:02:14.334882+0000","last_unstale":"2024-03-21T13:02:30.773642+0000","last_undegraded":"2024-03-21T13:02:30.773642+0000","last_fullsized":"2024-03-21T13:02:30.773642+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-21T13:02:27.259731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:02:27.259731+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:35.457954+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"126'8638","reported_seq":8725,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:44.842141+0000","last_change":"2024-03-21T13:01:48.274014+0000","last_active":"2024-03-21T13:02:44.842141+0000","last_peered":"2024-03-21T13:02:44.842141+0000","last_clean":"2024-03-21T13:02:44.842141+0000","last_became_active":"2024-03-21T13:01:48.220955+0000","last_became_peered":"2024-03-21T13:01:48.220955+0000","last_unstale":"2024-03-21T13:02:44.842141+0000","last_undegraded":"2024-03-21T13:02:44.842141+0000","last_fullsized":"2024-03-21T13:02:44.842141+0000","mapping_epoch":110,"log_start":"109'5039","ondisk_log_start":"109'5039","created":92,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399638528,"num_objects":6098,"num_object_clones":0,"num_object_copies":18294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8638,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":36,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.774511+0000","last_change":"2024-03-21T13:02:26.228129+0000","last_active":"2024-03-21T13:02:30.774511+0000","last_peered":"2024-03-21T13:02:30.774511+0000","last_clean":"2024-03-21T13:02:30.774511+0000","last_became_active":"2024-03-21T13:02:26.227779+0000","last_became_peered":"2024-03-21T13:02:26.227779+0000","last_unstale":"2024-03-21T13:02:30.774511+0000","last_undegraded":"2024-03-21T13:02:30.774511+0000","last_fullsized":"2024-03-21T13:02:30.774511+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:32.225843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:32.225843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:58.973233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"125'10","reported_seq":213,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:37.819653+0000","last_change":"2024-03-21T13:02:26.263146+0000","last_active":"2024-03-21T13:02:37.819653+0000","last_peered":"2024-03-21T13:02:37.819653+0000","last_clean":"2024-03-21T13:02:37.819653+0000","last_became_active":"2024-03-21T13:02:26.235954+0000","last_became_peered":"2024-03-21T13:02:26.235954+0000","last_unstale":"2024-03-21T13:02:37.819653+0000","last_undegraded":"2024-03-21T13:02:37.819653+0000","last_fullsized":"2024-03-21T13:02:37.819653+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:41.660019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:41.660019+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:02:56.560186+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":143,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.783265+0000","last_change":"2024-03-21T13:00:25.960463+0000","last_active":"2024-03-21T13:02:30.783265+0000","last_peered":"2024-03-21T13:02:30.783265+0000","last_clean":"2024-03-21T13:02:30.783265+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:02:30.783265+0000","last_undegraded":"2024-03-21T13:02:30.783265+0000","last_fullsized":"2024-03-21T13:02:30.783265+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:25.960383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:25.960383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":90,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-21T13:02:30.788934+0000","last_change":"2024-03-21T13:01:27.363819+0000","last_active":"2024-03-21T13:02:30.788934+0000","last_peered":"2024-03-21T13:02:30.788934+0000","last_clean":"2024-03-21T13:02:30.788934+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:02:30.788934+0000","last_undegraded":"2024-03-21T13:02:30.788934+0000","last_fullsized":"2024-03-21T13:02:30.788934+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:00:10.785752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.785752+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6413615128,"num_objects":97865,"num_object_clones":0,"num_object_copies":293595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":48146,"num_objects_unfound":0,"num_objects_dirty":97865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137013,"num_write_kb":7516033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12786,"num_bytes_recovered":837943296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20110585856,"data_stored":20110573640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64732,"ondisk_log_size":64732,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":238,"num_read_kb":233,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":684,"num_read_kb":441,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":3548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":424,"ondisk_log_size":424,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168630,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2313964,"kb_used_data":2214692,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91431188,"statfs":{"total":95995035648,"available":93625536512,"internally_reserved":0,"allocated":2267844608,"data_stored":2267390727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462972,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2270392,"kb_used_data":2162864,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91474760,"statfs":{"total":95995035648,"available":93670154240,"internally_reserved":0,"allocated":2214772736,"data_stored":2214306553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574932,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3253676,"kb_used_data":3141796,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":90491476,"statfs":{"total":95995035648,"available":92663271424,"internally_reserved":0,"allocated":3217199104,"data_stored":3216744038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2586380,"kb_used_data":2482948,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":91158772,"statfs":{"total":95995035648,"available":93346582528,"internally_reserved":0,"allocated":2542538752,"data_stored":2542115734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3859916,"kb_used_data":3712068,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":89885236,"statfs":{"total":95995035648,"available":92042481664,"internally_reserved":0,"allocated":3801157632,"data_stored":3800670865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151321031},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018949,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2187292,"kb_used_data":2075604,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91557860,"statfs":{"total":95995035648,"available":93755248640,"internally_reserved":0,"allocated":2125418496,"data_stored":2124979619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724608,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3201372160,"data_stored":3201368088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2407731200,"data_stored":2407727128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2534211584,"data_stored":2534211584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3778019328,"data_stored":3778019328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2114387968,"data_stored":2114387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1615986688,"data_stored":1615986688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2194935808,"data_stored":2194931736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2263941120,"data_stored":2263941120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:02:54.321 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool set unique_pool_0 pgp_num 16 2024-03-21T13:02:54.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:54.406 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:54.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:54.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:54.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:54 smithi204 ceph-mon[33153]: pgmap v158: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 43344/278310 objects misplaced (15.574%); 21 MiB/s, 337 objects/s recovering 2024-03-21T13:02:54.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:54 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:02:54.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:54 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4140330268' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:02:54.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:54 smithi204 ceph-mon[33153]: osdmap e128: 8 total, 8 up, 6 in 2024-03-21T13:02:54.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:54.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:54 smithi163 ceph-mon[29621]: pgmap v158: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 43344/278310 objects misplaced (15.574%); 21 MiB/s, 337 objects/s recovering 2024-03-21T13:02:54.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:54 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:02:54.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:54 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4140330268' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:02:54.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:54 smithi163 ceph-mon[29621]: osdmap e128: 8 total, 8 up, 6 in 2024-03-21T13:02:54.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:54 smithi163 ceph-mon[40675]: pgmap v158: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 43344/278310 objects misplaced (15.574%); 21 MiB/s, 337 objects/s recovering 2024-03-21T13:02:54.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:54 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:02:54.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:54 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4140330268' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:02:54.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:54 smithi163 ceph-mon[40675]: osdmap e128: 8 total, 8 up, 6 in 2024-03-21T13:02:54.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:54.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:54.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:54.644 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:54.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:54.699 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:54.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:54.800 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:54.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:54.839 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:54.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:54.940 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:54.976 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:54.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:54.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:55.078 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:55.118 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:55.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:55.220 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:55.262 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:55.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:55.278 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:02:55.364 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:55.396 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:55.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:55.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:55.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:55.533 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:55.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:55.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:55.635 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:55.688 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:55.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:55.790 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:55.818 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:55.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:55.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:55.914 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:55.920 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:55.934 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:02:55.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:55.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:55.989 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:02:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:02:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:02:56.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:02:56.093 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:56.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:56.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:02:56.183 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:02:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:02:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:02:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.14820.0:149122 7.8 e128/92 7:1bd4d740:::benchmark_data_smithi163_80312_object31681:head v 128'9318, mlcod=128'9316)", 2024-03-21T13:02:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:56.181930+0000", 2024-03-21T13:02:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.00084474400000000003, 2024-03-21T13:02:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00087712700000000001, 2024-03-21T13:02:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:02:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:02:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:02:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:56.181930+0000", 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:56.181930+0000", 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:56.181931+0000", 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0279999999999999e-06 2024-03-21T13:02:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:56.181938+0000", 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0720000000000002e-06 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:56.181953+0000", 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5659999999999999e-05 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:56.181962+0000", 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3739999999999998e-06 2024-03-21T13:02:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:02:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:02:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:02:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:56.182033+0000", 2024-03-21T13:02:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1260000000000006e-05 2024-03-21T13:02:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:02:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:02:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:02:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:56.182053+0000", 2024-03-21T13:02:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9721999999999999e-05 2024-03-21T13:02:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:02:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:02:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:02:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:02:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-03-21T13:02:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 1 2024-03-21T13:02:56.187 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:02:56.194 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:56.234 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:56.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:56.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:56.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:56.383 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:56.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:56.484 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:56.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:56 smithi204 ceph-mon[33153]: pgmap v160: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 992 op/s; 41396/270381 objects misplaced (15.310%); 17 MiB/s, 277 objects/s recovering 2024-03-21T13:02:56.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:56 smithi163 ceph-mon[29621]: pgmap v160: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 992 op/s; 41396/270381 objects misplaced (15.310%); 17 MiB/s, 277 objects/s recovering 2024-03-21T13:02:56.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:56 smithi163 ceph-mon[40675]: pgmap v160: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 992 op/s; 41396/270381 objects misplaced (15.310%); 17 MiB/s, 277 objects/s recovering 2024-03-21T13:02:56.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:56.530 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:56.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:56.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:56.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:56.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:56.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:56.806 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:56.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:56.908 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:56.958 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:56.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:57.060 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:57.100 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:57.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:57.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:57.202 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:57.237 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:57.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:57.338 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:57.379 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:57.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:57.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:57.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:57.520 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:57.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:57.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:57.622 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:57.655 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:57.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:57.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:57.784 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:02:57.792 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:57.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:57.893 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:57.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:57.944 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:57.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:58.044 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:58.094 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:58.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:58.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:58.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:02:58.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:58.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:58.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:02:58.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-21T13:02:58.335 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:58.346 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:02:58 smithi163 ceph-mon[29621]: pgmap v161: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 38741/262329 objects misplaced (14.768%); 25 MiB/s, 406 objects/s recovering 2024-03-21T13:02:58.346 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:02:58 smithi163 ceph-mon[40675]: pgmap v161: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 38741/262329 objects misplaced (14.768%); 25 MiB/s, 406 objects/s recovering 2024-03-21T13:02:58.376 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:58.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:58.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:58.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:02:58 smithi204 ceph-mon[33153]: pgmap v161: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 38741/262329 objects misplaced (14.768%); 25 MiB/s, 406 objects/s recovering 2024-03-21T13:02:58.521 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:58.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:58.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:58.623 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:58.665 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:02:58.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:58.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:58.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:58.775 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:58.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:58.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:58.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:58.905 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:02:58.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:58.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:02:59.039 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:59.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:59.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:59.164 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:02:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:59.204 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:59.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:02:59.305 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:59.345 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:59.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:59.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:59.446 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:02:59.485 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:59.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:59.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:02:59.587 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:02:59.610 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:59.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:02:59.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:02:59.742 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:02:59.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:59.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:02:59.844 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:02:59.883 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:02:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:02:59.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:02:59.985 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:00.011 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:00.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:00.112 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:00.154 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:00.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:00.256 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:00.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:00.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:00.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:00.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:00.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:00 smithi163 ceph-mon[29621]: pgmap v162: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 666 op/s; 38741/262329 objects misplaced (14.768%); 13 MiB/s, 202 objects/s recovering 2024-03-21T13:03:00.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:00 smithi163 ceph-mon[40675]: pgmap v162: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 666 op/s; 38741/262329 objects misplaced (14.768%); 13 MiB/s, 202 objects/s recovering 2024-03-21T13:03:00.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:00.559 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:00.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:00.570 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:00 smithi204 ceph-mon[33153]: pgmap v162: 148 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 140 active+clean; 5.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 666 op/s; 38741/262329 objects misplaced (14.768%); 13 MiB/s, 202 objects/s recovering 2024-03-21T13:03:00.661 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:00.690 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:00.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:00.792 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:00.815 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:00.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:00.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:00.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:00.941 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:00.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:01.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:01.071 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:01.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:01.173 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:01.197 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:01.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:01.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:01.333 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:01.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:01.434 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:01.469 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:01.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:01.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:01.571 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:01.600 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:01 smithi204 ceph-mon[33153]: osdmap e129: 8 total, 8 up, 6 in 2024-03-21T13:03:01.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:03:01 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:03:01.610 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:01.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:01.712 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:01.745 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:01.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:01.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:01 smithi163 ceph-mon[29621]: osdmap e129: 8 total, 8 up, 6 in 2024-03-21T13:03:01.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:01 smithi163 ceph-mon[40675]: osdmap e129: 8 total, 8 up, 6 in 2024-03-21T13:03:01.845 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:01.889 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:01.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:01.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:01.991 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:02.022 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:02.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:02.124 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:02.148 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:02.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:02.250 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:02.288 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:02.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:02.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:02.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:02.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:02.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:02.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:02.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:02.556 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:02.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:02.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:02.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:02.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:02.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:02.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:02 smithi204 ceph-mon[33153]: pgmap v164: 148 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 140 active+clean; 5.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 33863/247056 objects misplaced (13.707%); 30 MiB/s, 477 objects/s recovering 2024-03-21T13:03:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:02 smithi204 ceph-mon[33153]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:03:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:02 smithi204 ceph-mon[33153]: osdmap e130: 8 total, 8 up, 6 in 2024-03-21T13:03:02.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:02 smithi163 ceph-mon[29621]: pgmap v164: 148 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 140 active+clean; 5.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 33863/247056 objects misplaced (13.707%); 30 MiB/s, 477 objects/s recovering 2024-03-21T13:03:02.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:02 smithi163 ceph-mon[29621]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:03:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:02 smithi163 ceph-mon[29621]: osdmap e130: 8 total, 8 up, 6 in 2024-03-21T13:03:02.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:02 smithi163 ceph-mon[40675]: pgmap v164: 148 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 140 active+clean; 5.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 33863/247056 objects misplaced (13.707%); 30 MiB/s, 477 objects/s recovering 2024-03-21T13:03:02.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:02 smithi163 ceph-mon[40675]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:03:02.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:02 smithi163 ceph-mon[40675]: osdmap e130: 8 total, 8 up, 6 in 2024-03-21T13:03:02.795 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:02.832 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:02.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:02.933 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:02.971 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:02.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:02.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:03.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:03.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:03.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:03.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:03.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:03.234 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:03.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:03.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:03.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:03.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:03.468 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:03.514 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:03.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:03.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:03 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:03:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:03 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:03:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:03 smithi163 ceph-mon[40675]: osdmap e131: 8 total, 8 up, 6 in 2024-03-21T13:03:03.555 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:03.555 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:03 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:03:03.555 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:03 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:03:03.555 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:03 smithi163 ceph-mon[29621]: osdmap e131: 8 total, 8 up, 6 in 2024-03-21T13:03:03.617 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:03.664 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:03.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:03.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:03.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:03 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:03:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:03 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:03:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:03 smithi204 ceph-mon[33153]: osdmap e131: 8 total, 8 up, 6 in 2024-03-21T13:03:03.766 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:03.835 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:03.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:03.936 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:03.993 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:03.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:03.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:04.095 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:04.160 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:04.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:04.261 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:04.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:04.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:04.355 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:03:04.355 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-03-21T13:03:04.355 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-03-21T13:03:04.355 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-03-21T13:03:04.355 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:03:04.400 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:04.450 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:04.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:04.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:04.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:04.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:04.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:04.686 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:04.724 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:04.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:04.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:04 smithi204 ceph-mon[33153]: pgmap v167: 148 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 140 active+clean; 5.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 849 op/s; 33863/247056 objects misplaced (13.707%); 23 MiB/s, 366 objects/s recovering 2024-03-21T13:03:04.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:04 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:04.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:04 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:04.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:04 smithi163 ceph-mon[40675]: pgmap v167: 148 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 140 active+clean; 5.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 849 op/s; 33863/247056 objects misplaced (13.707%); 23 MiB/s, 366 objects/s recovering 2024-03-21T13:03:04.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:04 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:04.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:04 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:04 smithi163 ceph-mon[29621]: pgmap v167: 148 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 140 active+clean; 5.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 849 op/s; 33863/247056 objects misplaced (13.707%); 23 MiB/s, 366 objects/s recovering 2024-03-21T13:03:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:04 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:04 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:04.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:04.857 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:04.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:04.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:04.959 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:05.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:05.013 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:05.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:05.079 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:05.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:05.146 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:05.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:05.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:05.185 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:03:05.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-21T13:03:05.231 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:03:05.248 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:05.268 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:03:05.314 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:05.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:05.324 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:03:05.368 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:03:05.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:05.426 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:03:05.447 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:05.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:05.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:05.477 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:03:05.519 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:03:05.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:03:05.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:03:05.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:03:05.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:03:05.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:03:05.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:03:05.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:03:05.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:03:05.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:05.549 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:03:05.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:05.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:05.684 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:05.744 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:05.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:05.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:05.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:03:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:03:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:03:05.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:05.875 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:05.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:05.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:05.959 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:05 smithi204 ceph-mon[33153]: pgmap v168: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.9 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.22k op/s; 31285/240438 objects misplaced (13.012%); 38 MiB/s, 602 objects/s recovering 2024-03-21T13:03:05.959 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:05.959 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:05.959 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:05 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/763024467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:03:05.977 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:06.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:06.013 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:06.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:06.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:06.151 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:06.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:06.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:06.219 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:05 smithi163 ceph-mon[29621]: pgmap v168: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.9 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.22k op/s; 31285/240438 objects misplaced (13.012%); 38 MiB/s, 602 objects/s recovering 2024-03-21T13:03:06.219 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:06.219 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:06.219 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:05 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/763024467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:03:06.220 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:05 smithi163 ceph-mon[40675]: pgmap v168: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.9 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.22k op/s; 31285/240438 objects misplaced (13.012%); 38 MiB/s, 602 objects/s recovering 2024-03-21T13:03:06.220 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:06.220 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:06.220 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:05 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/763024467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:03:06.252 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:06.288 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:06.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:06.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:06.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:06.421 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:06.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:06.523 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:06.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:06.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:06.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:06.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:06.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:06.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:06.793 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:06.831 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:06.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:06.932 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:06.966 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:06 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2513765245' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-21T13:03:06.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:06.972 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:06.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:06.994 INFO:teuthology.orchestra.run.smithi163.stderr:set pool 7 pgp_num to 16 2024-03-21T13:03:07.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:07.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:07.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:07.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:07.123 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:06 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2513765245' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-21T13:03:07.123 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:06 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2513765245' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-21T13:03:07.212 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:07.251 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:07.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:07.353 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:07.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:07.389 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:07.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:07.487 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:03:07.487 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:03:07.490 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:07.547 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:07.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:07.648 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:07.692 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:07.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:07.793 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:07.837 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:07.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:07.938 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:07.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:07.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:07.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:08.078 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:08.115 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:08.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:08.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:08.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:07 smithi163 ceph-mon[29621]: pgmap v169: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 797 B/s rd, 0 B/s wr, 617 op/s; 28804/235182 objects misplaced (12.248%); 28 MiB/s, 440 objects/s recovering 2024-03-21T13:03:08.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:07 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2513765245' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-03-21T13:03:08.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:07 smithi163 ceph-mon[29621]: osdmap e132: 8 total, 8 up, 6 in 2024-03-21T13:03:08.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:07 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:08.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:07 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:03:08.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:07 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:08.180 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:07 smithi163 ceph-mon[40675]: pgmap v169: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 797 B/s rd, 0 B/s wr, 617 op/s; 28804/235182 objects misplaced (12.248%); 28 MiB/s, 440 objects/s recovering 2024-03-21T13:03:08.180 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:07 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2513765245' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-03-21T13:03:08.180 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:07 smithi163 ceph-mon[40675]: osdmap e132: 8 total, 8 up, 6 in 2024-03-21T13:03:08.181 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:07 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:08.181 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:07 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:03:08.181 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:07 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:08.191 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:07 smithi204 ceph-mon[33153]: pgmap v169: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 797 B/s rd, 0 B/s wr, 617 op/s; 28804/235182 objects misplaced (12.248%); 28 MiB/s, 440 objects/s recovering 2024-03-21T13:03:08.191 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:07 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2513765245' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-03-21T13:03:08.191 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:07 smithi204 ceph-mon[33153]: osdmap e132: 8 total, 8 up, 6 in 2024-03-21T13:03:08.191 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:07 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:08.192 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:07 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:03:08.192 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:07 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:08.217 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:08.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-21T13:03:08.254 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:08.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:08.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:08.356 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:08.402 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:08.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:08.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:08.541 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:08.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:08.643 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:08.650 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:03:08.684 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:08.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:08.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:08.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:08.821 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:08.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:08.910 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:03:08.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:08.957 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:08.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:09.059 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:09.105 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:09.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:09.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:09.252 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:09.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:09.354 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:09.399 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:09.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:09.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:09.501 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:09.526 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:09.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:09.628 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:09.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:09.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:09.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:09.793 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:09.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:09.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:09.894 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:09.917 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:09.924 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:09.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:09.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:10.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:10.060 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:10.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:10.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:10.162 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:10.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:10.197 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:10.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:10.239 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:09 smithi163 ceph-mon[29621]: pgmap v171: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 690 B/s rd, 0 B/s wr, 534 op/s; 28804/235182 objects misplaced (12.248%); 24 MiB/s, 381 objects/s recovering 2024-03-21T13:03:10.239 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:09 smithi163 ceph-mon[40675]: pgmap v171: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 690 B/s rd, 0 B/s wr, 534 op/s; 28804/235182 objects misplaced (12.248%); 24 MiB/s, 381 objects/s recovering 2024-03-21T13:03:10.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:09 smithi204 ceph-mon[33153]: pgmap v171: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 690 B/s rd, 0 B/s wr, 534 op/s; 28804/235182 objects misplaced (12.248%); 24 MiB/s, 381 objects/s recovering 2024-03-21T13:03:10.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:10.323 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:10.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:10.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:10.458 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:10.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:10.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:10.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:10.586 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:10.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:10.688 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:10.734 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:10.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:10.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:10.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:10.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:10.900 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:03:10.973 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:11.015 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:11.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:11.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:11.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:11.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:11.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:11.153 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:10 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2783204942' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:11.162 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:10 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2783204942' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:11.162 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:10 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2783204942' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:11.251 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:11.289 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:11.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:11.391 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:11.430 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:11.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:11.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:11.505 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:03:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:03:11.532 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:11.573 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:11.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:11.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:11.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:11.710 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:11.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:11.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:11.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:11.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:11.848 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:11.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:11.949 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:11.986 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:11.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:12.087 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:12.110 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:03:12.110 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:03:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:12.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:12.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:12.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:12.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:11 smithi204 ceph-mon[33153]: pgmap v172: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 24684/220365 objects misplaced (11.201%); 35 MiB/s, 567 objects/s recovering 2024-03-21T13:03:12.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:11 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:03:12.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:11 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2783204942' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:03:12.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:11 smithi204 ceph-mon[33153]: osdmap e133: 8 total, 8 up, 6 in 2024-03-21T13:03:12.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:11 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:03:12.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:11 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:03:12.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:11 smithi163 ceph-mon[29621]: pgmap v172: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 24684/220365 objects misplaced (11.201%); 35 MiB/s, 567 objects/s recovering 2024-03-21T13:03:12.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:11 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:03:12.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:11 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2783204942' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:03:12.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:11 smithi163 ceph-mon[29621]: osdmap e133: 8 total, 8 up, 6 in 2024-03-21T13:03:12.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:11 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:03:12.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:11 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:03:12.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:11 smithi163 ceph-mon[40675]: pgmap v172: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 24684/220365 objects misplaced (11.201%); 35 MiB/s, 567 objects/s recovering 2024-03-21T13:03:12.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:11 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:03:12.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:11 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2783204942' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:03:12.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:11 smithi163 ceph-mon[40675]: osdmap e133: 8 total, 8 up, 6 in 2024-03-21T13:03:12.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:11 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:03:12.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:11 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:03:12.265 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:12.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:12.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:12.367 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:12.403 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:12.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:12.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:12.505 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:12.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:12.534 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:12.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:12.635 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:12.676 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:12.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:12.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:12.778 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:12.819 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:12.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:12.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-21T13:03:12.909 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:03:12.910 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:03:12.910 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-21T13:03:12.910 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-21T13:03:12.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:12.963 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:12.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:13.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:13.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:12 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:13.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:12 smithi204 ceph-mon[33153]: osdmap e134: 8 total, 8 up, 6 in 2024-03-21T13:03:13.092 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:13.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:13.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:13.194 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:13.206 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:03:13.246 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:13.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:13.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:13.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:12 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:13.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:12 smithi163 ceph-mon[29621]: osdmap e134: 8 total, 8 up, 6 in 2024-03-21T13:03:13.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:12 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:13.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:12 smithi163 ceph-mon[40675]: osdmap e134: 8 total, 8 up, 6 in 2024-03-21T13:03:13.349 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:13.382 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:13.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:13.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:13.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:13.518 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:13.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:13.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:13.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:13.653 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:13.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:13.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:13.779 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:13.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:13.880 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:13.917 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:13.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:14.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:14.055 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:14.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:14.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:14.191 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:14.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:14.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:13 smithi204 ceph-mon[33153]: pgmap v175: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 823 op/s; 24684/220365 objects misplaced (11.201%); 20 MiB/s, 319 objects/s recovering 2024-03-21T13:03:14.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:13 smithi163 ceph-mon[29621]: pgmap v175: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 823 op/s; 24684/220365 objects misplaced (11.201%); 20 MiB/s, 319 objects/s recovering 2024-03-21T13:03:14.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:13 smithi163 ceph-mon[40675]: pgmap v175: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 823 op/s; 24684/220365 objects misplaced (11.201%); 20 MiB/s, 319 objects/s recovering 2024-03-21T13:03:14.293 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:14.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:14.334 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:14.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:14.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:14.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:14.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:14.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:14.572 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:14.620 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:14.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:14.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:14.722 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:14.757 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:14.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:14.859 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:14.890 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:14.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:14.991 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:15.026 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:15.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:15.128 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:15.164 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:15.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:15.266 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:15.308 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:15.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:15.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:15.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:15.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:15.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:15.544 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:15.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:15.587 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:15.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:15.666 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:03:15.667 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:03:15.688 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:15.724 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:15.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:15.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:15.844 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:03:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:03:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:03:15.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:15.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:15.959 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:15.997 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:15.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:16.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:16.103 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:15 smithi163 ceph-mon[29621]: pgmap v176: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.00k op/s; 22720/211737 objects misplaced (10.730%); 24 MiB/s, 382 objects/s recovering 2024-03-21T13:03:16.103 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:15 smithi163 ceph-mon[40675]: pgmap v176: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.00k op/s; 22720/211737 objects misplaced (10.730%); 24 MiB/s, 382 objects/s recovering 2024-03-21T13:03:16.125 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:16.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:16.148 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:03:16.173 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:03:16.207 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:03:16.226 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:16.248 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:03:16.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:15 smithi204 ceph-mon[33153]: pgmap v176: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.00k op/s; 22720/211737 objects misplaced (10.730%); 24 MiB/s, 382 objects/s recovering 2024-03-21T13:03:16.262 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:16.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:16.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:16.288 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:03:16.295 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:16.323 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:03:16.355 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:03:16.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:16.380 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:03:16.381 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:03:16.382 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:03:16.382 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:03:16.382 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:03:16.382 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:03:16.382 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:03:16.382 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:03:16.382 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:03:16.391 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:16.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:16.493 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:16.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:16.524 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:16.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:16.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:16.652 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:16.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:16.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:16.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:16.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:16.784 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:16.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:16.885 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:16.913 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:16.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:17.015 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:17.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:17.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:17.152 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:17.181 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:17.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:17.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:17.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:16 smithi204 ceph-mon[33153]: osdmap e135: 8 total, 8 up, 6 in 2024-03-21T13:03:17.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:16 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/624674533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:03:17.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:16 smithi163 ceph-mon[29621]: osdmap e135: 8 total, 8 up, 6 in 2024-03-21T13:03:17.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:16 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/624674533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:03:17.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:16 smithi163 ceph-mon[40675]: osdmap e135: 8 total, 8 up, 6 in 2024-03-21T13:03:17.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:16 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/624674533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:03:17.283 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:17.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:17.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:17.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:17.417 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:17.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:17.461 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:17.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:17.562 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:17.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-21T13:03:17.607 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:17.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:17.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:17.709 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:17.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:17.753 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:17.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:17.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:17.887 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:03:17.894 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:17.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:17.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:17.961 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:03:17.996 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:18.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:18.048 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:18.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:18.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:18.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:18.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:18.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:18.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:17 smithi204 ceph-mon[33153]: pgmap v178: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 843 op/s; 20575/205182 objects misplaced (10.028%); 21 MiB/s, 328 objects/s recovering 2024-03-21T13:03:18.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:17 smithi204 ceph-mon[33153]: osdmap e136: 8 total, 8 up, 6 in 2024-03-21T13:03:18.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:17 smithi163 ceph-mon[29621]: pgmap v178: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 843 op/s; 20575/205182 objects misplaced (10.028%); 21 MiB/s, 328 objects/s recovering 2024-03-21T13:03:18.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:17 smithi163 ceph-mon[29621]: osdmap e136: 8 total, 8 up, 6 in 2024-03-21T13:03:18.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:17 smithi163 ceph-mon[40675]: pgmap v178: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 843 op/s; 20575/205182 objects misplaced (10.028%); 21 MiB/s, 328 objects/s recovering 2024-03-21T13:03:18.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:17 smithi163 ceph-mon[40675]: osdmap e136: 8 total, 8 up, 6 in 2024-03-21T13:03:18.281 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:18.324 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:18.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:18.425 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:18.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:18.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:18.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:18.588 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:18.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:18.690 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:18.736 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:18.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:18.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:18.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:18.871 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:18.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:18.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:18.919 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:18.973 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:19.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:19.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:19.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:19.141 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:19.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:19.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:19.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:19.273 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:19.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:19.375 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:19.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:19.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:19.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:19.508 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:19.542 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:19.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:19.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:19.644 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:19.663 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:03:19.665 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:03:19.665 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 136, 2024-03-21T13:03:19.665 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "d4bf79c0-e781-11ee-95cd-87774f69a715", 2024-03-21T13:03:19.666 INFO:teuthology.orchestra.run.smithi163.stdout: "created": "2024-03-21T12:52:56.031419+0000", 2024-03-21T13:03:19.666 INFO:teuthology.orchestra.run.smithi163.stdout: "modified": "2024-03-21T13:03:16.950940+0000", 2024-03-21T13:03:19.666 INFO:teuthology.orchestra.run.smithi163.stdout: "last_up_change": "2024-03-21T12:58:23.775302+0000", 2024-03-21T13:03:19.666 INFO:teuthology.orchestra.run.smithi163.stdout: "last_in_change": "2024-03-21T13:02:25.729226+0000", 2024-03-21T13:03:19.666 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-21T13:03:19.666 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_num": 5806080, 2024-03-21T13:03:19.666 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_set": [ 2024-03-21T13:03:19.666 INFO:teuthology.orchestra.run.smithi163.stdout: "nodeep-scrub", 2024-03-21T13:03:19.667 INFO:teuthology.orchestra.run.smithi163.stdout: "noscrub", 2024-03-21T13:03:19.667 INFO:teuthology.orchestra.run.smithi163.stdout: "pglog_hardlimit", 2024-03-21T13:03:19.667 INFO:teuthology.orchestra.run.smithi163.stdout: "purged_snapdirs", 2024-03-21T13:03:19.667 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_deletes", 2024-03-21T13:03:19.667 INFO:teuthology.orchestra.run.smithi163.stdout: "sortbitwise" 2024-03-21T13:03:19.667 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:03:19.667 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_version": 18, 2024-03-21T13:03:19.667 INFO:teuthology.orchestra.run.smithi163.stdout: "full_ratio": 0.94999998807907104, 2024-03-21T13:03:19.667 INFO:teuthology.orchestra.run.smithi163.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-21T13:03:19.668 INFO:teuthology.orchestra.run.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-21T13:03:19.668 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_snapshot": "", 2024-03-21T13:03:19.668 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_max": 7, 2024-03-21T13:03:19.668 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd": 8, 2024-03-21T13:03:19.668 INFO:teuthology.orchestra.run.smithi163.stdout: "require_min_compat_client": "luminous", 2024-03-21T13:03:19.668 INFO:teuthology.orchestra.run.smithi163.stdout: "min_compat_client": "jewel", 2024-03-21T13:03:19.668 INFO:teuthology.orchestra.run.smithi163.stdout: "require_osd_release": "squid", 2024-03-21T13:03:19.668 INFO:teuthology.orchestra.run.smithi163.stdout: "allow_crimson": false, 2024-03-21T13:03:19.668 INFO:teuthology.orchestra.run.smithi163.stdout: "pools": [ 2024-03-21T13:03:19.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.669 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 1, 2024-03-21T13:03:19.669 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".mgr", 2024-03-21T13:03:19.669 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:56:20.557490+0000", 2024-03-21T13:03:19.669 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:03:19.669 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:03:19.669 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:03:19.669 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:03:19.669 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:03:19.669 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:03:19.669 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:03:19.669 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:03:19.670 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:03:19.670 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:03:19.670 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:03:19.670 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:03:19.670 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 1, 2024-03-21T13:03:19.670 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 1, 2024-03-21T13:03:19.670 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 1, 2024-03-21T13:03:19.670 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 1, 2024-03-21T13:03:19.670 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 1, 2024-03-21T13:03:19.671 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:03:19.671 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:03:19.671 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:03:19.671 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:03:19.671 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:03:19.671 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:03:19.671 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:03:19.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.671 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "21", 2024-03-21T13:03:19.671 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:03:19.671 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:03:19.671 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:03:19.671 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:03:19.672 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:03:19.672 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:03:19.672 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:03:19.672 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:03:19.672 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:03:19.672 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:03:19.672 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:03:19.672 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:03:19.672 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:03:19.672 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:03:19.672 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:03:19.672 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:03:19.672 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:03:19.673 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:03:19.673 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:03:19.673 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:03:19.673 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:03:19.673 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:03:19.673 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:03:19.673 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:03:19.673 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:03:19.673 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:03:19.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.673 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:03:19.673 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:03:19.673 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:03:19.674 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:03:19.674 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:03:19.674 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:03:19.674 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:03:19.674 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:03:19.674 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:03:19.674 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:03:19.674 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:03:19.674 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-03-21T13:03:19.674 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_max": 32, 2024-03-21T13:03:19.674 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_min": 1 2024-03-21T13:03:19.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.674 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:03:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": {} 2024-03-21T13:03:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:03:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:03:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 7.8899998664855957, 2024-03-21T13:03:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 7.8899998664855957, 2024-03-21T13:03:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.37999999523162842, 2024-03-21T13:03:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 3, 2024-03-21T13:03:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 3, 2024-03-21T13:03:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.84565353393554688, 2024-03-21T13:03:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.85000002384185791, 2024-03-21T13:03:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-03-21T13:03:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 2, 2024-03-21T13:03:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "datapool", 2024-03-21T13:03:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:35.713383+0000", 2024-03-21T13:03:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-03-21T13:03:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-21T13:03:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:03:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:03:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:03:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:03:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:03:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 3, 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 3, 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 3, 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 3, 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 3, 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:03:19.677 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:03:19.678 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:03:19.678 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:03:19.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.678 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "67", 2024-03-21T13:03:19.678 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:03:19.678 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:03:19.678 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:03:19.678 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:03:19.678 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:03:19.678 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 2, 2024-03-21T13:03:19.678 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 67, 2024-03-21T13:03:19.678 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:03:19.678 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:03:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:03:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd": {} 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.6500000953674316, 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.6500000953674316, 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.62999999523162842, 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.6699999570846558, 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.6699999570846558, 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.84565353393554688, 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.85000002384185791, 2024-03-21T13:03:19.681 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-03-21T13:03:19.682 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.682 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 3, 2024-03-21T13:03:19.682 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".rgw.root", 2024-03-21T13:03:19.682 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:36.436384+0000", 2024-03-21T13:03:19.682 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:03:19.682 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:03:19.682 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:03:19.682 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:03:19.682 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:03:19.682 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:03:19.683 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:03:19.684 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:03:19.684 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:03:19.684 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:03:19.684 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:03:19.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.684 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "65", 2024-03-21T13:03:19.684 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:03:19.684 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:03:19.684 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:03:19.684 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:03:19.684 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:03:19.684 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:03:19.684 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:03:19.685 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:03:19.686 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:03:19.686 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:03:19.686 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:03:19.686 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:03:19.686 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:03:19.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.686 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:03:19.686 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:03:19.686 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:03:19.686 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:03:19.686 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:03:19.686 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:03:19.686 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.5099999904632568, 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.5099999904632568, 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.75, 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-21T13:03:19.687 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.84565353393554688, 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.85000002384185791, 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 4, 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.log", 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:38.049304+0000", 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:03:19.688 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:03:19.689 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:03:19.690 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:03:19.690 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:03:19.690 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:03:19.690 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:03:19.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.690 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "67", 2024-03-21T13:03:19.690 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:03:19.690 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:03:19.690 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:03:19.690 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:03:19.690 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:03:19.690 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:03:19.691 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:03:19.692 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:03:19.692 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:03:19.692 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:03:19.692 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:03:19.692 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:03:19.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.692 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:03:19.692 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:03:19.692 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:03:19.692 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:03:19.692 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:03:19.692 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:03:19.692 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:03:19.693 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:03:19.693 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:03:19.693 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:03:19.693 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:03:19.693 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:03:19.693 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:03:19.693 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:03:19.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.693 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:03:19.693 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:03:19.693 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 3.25, 2024-03-21T13:03:19.693 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 3.25, 2024-03-21T13:03:19.693 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.75, 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 2.440000057220459, 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 2.440000057220459, 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.84565353393554688, 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.85000002384185791, 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 5, 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.control", 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:39.990512+0000", 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:03:19.694 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:03:19.695 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:03:19.695 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:03:19.695 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:03:19.695 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:03:19.695 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:03:19.695 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:03:19.695 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:03:19.695 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:03:19.695 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.696 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "69", 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:03:19.697 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:03:19.698 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:03:19.698 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:03:19.698 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:03:19.698 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:03:19.698 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:03:19.698 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:03:19.698 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:03:19.698 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:03:19.698 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:03:19.698 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:03:19.698 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:03:19.698 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:03:19.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:03:19.699 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:03:19.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.700 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:03:19.700 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:03:19.700 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2, 2024-03-21T13:03:19.700 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2, 2024-03-21T13:03:19.700 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.75, 2024-03-21T13:03:19.700 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.5, 2024-03-21T13:03:19.700 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.5, 2024-03-21T13:03:19.701 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.84565353393554688, 2024-03-21T13:03:19.701 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.85000002384185791, 2024-03-21T13:03:19.701 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-03-21T13:03:19.701 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.701 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 6, 2024-03-21T13:03:19.701 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.meta", 2024-03-21T13:03:19.701 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:41.463686+0000", 2024-03-21T13:03:19.701 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:03:19.701 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:03:19.701 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:03:19.701 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:03:19.702 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "71", 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:03:19.703 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:03:19.704 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:03:19.705 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_bias": 4 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 3, 2024-03-21T13:03:19.706 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 3, 2024-03-21T13:03:19.707 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.75, 2024-03-21T13:03:19.707 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 2.25, 2024-03-21T13:03:19.707 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 2.25, 2024-03-21T13:03:19.707 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.84565353393554688, 2024-03-21T13:03:19.707 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.85000002384185791, 2024-03-21T13:03:19.707 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-03-21T13:03:19.707 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.707 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 7, 2024-03-21T13:03:19.707 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "unique_pool_0", 2024-03-21T13:03:19.707 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T13:00:41.658316+0000", 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 16, 2024-03-21T13:03:19.708 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 16, 2024-03-21T13:03:19.709 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 16, 2024-03-21T13:03:19.709 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 16, 2024-03-21T13:03:19.709 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 16, 2024-03-21T13:03:19.709 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:03:19.709 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:03:19.709 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:03:19.709 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:03:19.709 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:03:19.709 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:03:19.709 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:03:19.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.709 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "132", 2024-03-21T13:03:19.709 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:03:19.710 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:03:19.711 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:03:19.712 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.5099999904632568, 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.5099999904632568, 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.75, 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.84565353393554688, 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.85000002384185791, 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.713 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f", 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0.0852813720703125, 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 8, 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 128, 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6801", 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6802", 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.715 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6804", 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.716 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.717 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6803", 2024-03-21T13:03:19.717 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:03:19.717 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.717 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.717 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6801/3086410981", 2024-03-21T13:03:19.717 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6802/3086410981", 2024-03-21T13:03:19.717 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6804/3086410981", 2024-03-21T13:03:19.718 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:19.718 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6803/3086410981", 2024-03-21T13:03:19.718 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:03:19.718 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:03:19.718 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:03:19.718 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.719 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-03-21T13:03:19.719 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "36e05f89-c238-41f7-be65-1553b529e8ce", 2024-03-21T13:03:19.719 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:03:19.719 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 0, 2024-03-21T13:03:19.719 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 0, 2024-03-21T13:03:19.719 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:03:19.719 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:03:19.719 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:03:19.719 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 13, 2024-03-21T13:03:19.719 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 110, 2024-03-21T13:03:19.719 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:03:19.719 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:03:19.720 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:03:19.720 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.720 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.720 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6805", 2024-03-21T13:03:19.720 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:03:19.720 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.720 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.720 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:03:19.720 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.720 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6806", 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6808", 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.722 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:03:19.722 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.722 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.722 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6807", 2024-03-21T13:03:19.722 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:03:19.722 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.722 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.722 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6805/3666062784", 2024-03-21T13:03:19.722 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6806/3666062784", 2024-03-21T13:03:19.722 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6808/3666062784", 2024-03-21T13:03:19.722 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6807/3666062784", 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f", 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:03:19.723 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:03:19.724 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 18, 2024-03-21T13:03:19.724 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 124, 2024-03-21T13:03:19.724 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:03:19.724 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:03:19.724 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:03:19.724 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.724 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.724 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6809", 2024-03-21T13:03:19.724 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:03:19.724 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.724 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.725 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:03:19.725 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.725 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.725 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6810", 2024-03-21T13:03:19.725 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:03:19.725 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.725 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.725 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:03:19.725 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.725 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.726 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6812", 2024-03-21T13:03:19.726 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:03:19.726 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.726 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.726 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:03:19.726 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.726 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.726 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6811", 2024-03-21T13:03:19.726 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:03:19.726 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6809/4009213507", 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6810/4009213507", 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6812/4009213507", 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6811/4009213507", 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-03-21T13:03:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d", 2024-03-21T13:03:19.728 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:03:19.728 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:03:19.728 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:03:19.728 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:03:19.728 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:03:19.728 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:03:19.728 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 25, 2024-03-21T13:03:19.728 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 124, 2024-03-21T13:03:19.728 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:03:19.728 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:03:19.728 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:03:19.728 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.729 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.729 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6813", 2024-03-21T13:03:19.729 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:03:19.729 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.729 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.729 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:03:19.729 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.729 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.729 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6814", 2024-03-21T13:03:19.729 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:03:19.730 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.730 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.730 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:03:19.730 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.730 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.730 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6816", 2024-03-21T13:03:19.730 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:03:19.730 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.730 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.731 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:03:19.731 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.731 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.731 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6815", 2024-03-21T13:03:19.731 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:03:19.731 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.731 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.731 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6813/3981878189", 2024-03-21T13:03:19.731 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6814/3981878189", 2024-03-21T13:03:19.732 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6816/3981878189", 2024-03-21T13:03:19.732 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6815/3981878189", 2024-03-21T13:03:19.732 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:03:19.732 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:03:19.732 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:03:19.732 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.732 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-03-21T13:03:19.732 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471", 2024-03-21T13:03:19.732 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:03:19.732 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:03:19.733 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:03:19.733 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0.6799468994140625, 2024-03-21T13:03:19.733 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:03:19.733 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:03:19.733 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 30, 2024-03-21T13:03:19.733 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 123, 2024-03-21T13:03:19.733 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:03:19.733 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:03:19.733 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:03:19.733 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.733 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.733 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6800", 2024-03-21T13:03:19.734 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:03:19.734 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.734 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.734 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:03:19.734 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.734 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.734 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6801", 2024-03-21T13:03:19.734 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:03:19.734 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.734 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.735 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:03:19.735 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.735 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.735 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6803", 2024-03-21T13:03:19.735 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:03:19.735 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.735 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.735 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:03:19.735 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.735 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6802", 2024-03-21T13:03:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:03:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6800/719276054", 2024-03-21T13:03:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6801/719276054", 2024-03-21T13:03:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6803/719276054", 2024-03-21T13:03:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6802/719276054", 2024-03-21T13:03:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:03:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:03:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:03:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1", 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 0, 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 0, 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 39, 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 123, 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:03:19.737 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:03:19.738 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:03:19.738 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.738 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.738 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6804", 2024-03-21T13:03:19.738 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:03:19.738 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.738 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.738 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:03:19.738 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:19.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:19.740 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.740 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6805", 2024-03-21T13:03:19.740 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:03:19.740 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.740 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.740 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:03:19.740 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6807", 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6806", 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.741 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.742 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6804/1346947668", 2024-03-21T13:03:19.742 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6805/1346947668", 2024-03-21T13:03:19.742 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6807/1346947668", 2024-03-21T13:03:19.742 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6806/1346947668", 2024-03-21T13:03:19.742 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:03:19.742 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:03:19.742 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:03:19.742 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.742 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-03-21T13:03:19.742 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b", 2024-03-21T13:03:19.743 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:03:19.743 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:03:19.743 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:03:19.743 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:03:19.743 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:03:19.743 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:03:19.743 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 47, 2024-03-21T13:03:19.743 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 135, 2024-03-21T13:03:19.743 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:03:19.743 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:03:19.743 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:03:19.743 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.744 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.744 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6808", 2024-03-21T13:03:19.744 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:03:19.744 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.744 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.744 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:03:19.744 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.744 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.744 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6809", 2024-03-21T13:03:19.744 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:03:19.744 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6811", 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6810", 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6808/200077988", 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6809/200077988", 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6811/200077988", 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6810/200077988", 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:03:19.746 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49", 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 56, 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 129, 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:03:19.747 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6812", 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6813", 2024-03-21T13:03:19.748 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6815", 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:03:19.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6814", 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6812/315664888", 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6813/315664888", 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6815/315664888", 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6814/315664888", 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:03:19.750 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_xinfo": [ 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:02:19.697874+0000", 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 65536, 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:02:18.653335+0000", 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:03:19.752 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:02:26.419877+0000", 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:03:11.631115+0000", 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-03-21T13:03:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:03:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:03:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:03:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:03:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:03:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:02:08.890428+0000", 2024-03-21T13:03:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:03:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-03-21T13:03:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:03:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 65536, 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:02:51.764974+0000", 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:03:01.908480+0000", 2024-03-21T13:03:19.755 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:02:11.917673+0000", 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap": [], 2024-03-21T13:03:19.756 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_items": [], 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_primaries": [], 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_temp": [ 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "7.0", 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: 5, 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: 1 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "7.2", 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:03:19.757 INFO:teuthology.orchestra.run.smithi163.stdout: 2, 2024-03-21T13:03:19.758 INFO:teuthology.orchestra.run.smithi163.stdout: 1 2024-03-21T13:03:19.758 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.758 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.758 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "7.4", 2024-03-21T13:03:19.758 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:03:19.758 INFO:teuthology.orchestra.run.smithi163.stdout: 4, 2024-03-21T13:03:19.758 INFO:teuthology.orchestra.run.smithi163.stdout: 0, 2024-03-21T13:03:19.758 INFO:teuthology.orchestra.run.smithi163.stdout: 1 2024-03-21T13:03:19.758 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.758 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.758 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "7.a", 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: 2, 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: 5 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "7.d", 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: 6, 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: 0, 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: 1 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "7.e", 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: 2, 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: 0, 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: 5 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_temp": [], 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: "blocklist": { 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3453855427": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/2571244897": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/3073378854": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:03:19.760 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1334591766": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:03:19.761 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/116934309": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:03:19.761 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3890775846": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:03:19.761 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1783921524": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:03:19.761 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2553440211": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:03:19.761 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1812694077": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:03:19.761 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1070540283": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:03:19.761 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/2354157786": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:03:19.761 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1591817392": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:03:19.761 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/4152198951": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:03:19.761 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/1798115633": "2024-03-22T12:54:00.224135+0000" 2024-03-21T13:03:19.761 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.761 INFO:teuthology.orchestra.run.smithi163.stdout: "range_blocklist": {}, 2024-03-21T13:03:19.761 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profiles": { 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: "default": { 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van" 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps_queue": [], 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: "new_removed_snaps": [], 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: "new_purged_snaps": [], 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_node_flags": {}, 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: "device_class_flags": {}, 2024-03-21T13:03:19.762 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode": { 2024-03-21T13:03:19.763 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_enabled": false, 2024-03-21T13:03:19.763 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_bucket_count": 0, 2024-03-21T13:03:19.763 INFO:teuthology.orchestra.run.smithi163.stdout: "degraded_stretch_mode": 0, 2024-03-21T13:03:19.763 INFO:teuthology.orchestra.run.smithi163.stdout: "recovering_stretch_mode": 0, 2024-03-21T13:03:19.763 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_bucket": 0 2024-03-21T13:03:19.763 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:19.763 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:03:19.841 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:19.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:19.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:19.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:19.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:19.979 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:03:20.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:20.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:20.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:20.135 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:20.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:20.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:20.237 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:20.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:19 smithi204 ceph-mon[33153]: pgmap v180: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 755 B/s rd, 0 B/s wr, 747 op/s; 20575/205182 objects misplaced (10.028%); 18 MiB/s, 291 objects/s recovering 2024-03-21T13:03:20.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:20.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:19 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3207186785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:03:20.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:19 smithi163 ceph-mon[29621]: pgmap v180: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 755 B/s rd, 0 B/s wr, 747 op/s; 20575/205182 objects misplaced (10.028%); 18 MiB/s, 291 objects/s recovering 2024-03-21T13:03:20.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:20.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:19 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3207186785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:03:20.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:19 smithi163 ceph-mon[40675]: pgmap v180: 148 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 141 active+clean; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 755 B/s rd, 0 B/s wr, 747 op/s; 20575/205182 objects misplaced (10.028%); 18 MiB/s, 291 objects/s recovering 2024-03-21T13:03:20.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:20.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:19 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3207186785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:03:20.284 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:20.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:20.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:20.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:20.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:20.430 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:20.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:20.531 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:20.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:20.571 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:20.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:20.672 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:20.697 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:20.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:20.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:20.799 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:20.832 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:20.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:20.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:20.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:20.964 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:20.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:20.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:21.066 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:21.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:21.104 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:21.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:21.206 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:21.246 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:21.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:21.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:21.277 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:21 smithi204 ceph-mon[33153]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:03:21.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:21.378 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:21.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:21.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:21 smithi163 ceph-mon[29621]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:03:21.452 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:21 smithi163 ceph-mon[40675]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:03:21.480 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:21.503 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:21.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:21.580 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:03:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:03:21.605 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:21.646 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:21.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:21.747 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:21.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:21.781 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:21.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:21.784 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:03:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-03-21T13:03:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-03-21T13:03:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:03:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:5937 7.0 7:0cfef427:::benchmark_data_smithi163_80312_object5936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:03:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:00:56.236353+0000", 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 145.54279122099999, 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.080287529999999996, 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5937 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236353+0000", 2024-03-21T13:03:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236353+0000", 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236354+0000", 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3859999999999999e-06 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236405+0000", 2024-03-21T13:03:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1201e-05 2024-03-21T13:03:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236424+0000", 2024-03-21T13:03:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8842e-05 2024-03-21T13:03:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236432+0000", 2024-03-21T13:03:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9990000000000008e-06 2024-03-21T13:03:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236490+0000", 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7497999999999997e-05 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236613+0000", 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012285099999999999 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236761+0000", 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014846799999999999 2024-03-21T13:03:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.237277+0000", 2024-03-21T13:03:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000516195 2024-03-21T13:03:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.238004+0000", 2024-03-21T13:03:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00072655799999999998 2024-03-21T13:03:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.316548+0000", 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.078544055000000002 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.316590+0000", 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2398000000000003e-05 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.316640+0000", 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0078999999999999e-05 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:5965 7.0 7:03da52ce:::benchmark_data_smithi163_80312_object5964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:00:56.326347+0000", 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 145.45279760400001, 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.10961245, 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5965 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326347+0000", 2024-03-21T13:03:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326347+0000", 2024-03-21T13:03:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326348+0000", 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.451e-06 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326386+0000", 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8075999999999999e-05 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326398+0000", 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2306999999999999e-05 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326408+0000", 2024-03-21T13:03:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4299999999999995e-06 2024-03-21T13:03:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326474+0000", 2024-03-21T13:03:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6591e-05 2024-03-21T13:03:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326610+0000", 2024-03-21T13:03:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013582700000000001 2024-03-21T13:03:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326782+0000", 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017180799999999999 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.327361+0000", 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00057854599999999999 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.328867+0000", 2024-03-21T13:03:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001506082 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.435844+0000", 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.10697730799999999 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.435914+0000", 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9698999999999998e-05 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.435959+0000", 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5324999999999997e-05 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:5979 7.0 7:060be99b:::benchmark_data_smithi163_80312_object5978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:00:56.330543+0000", 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 145.44860151099999, 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.10700628399999999, 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5979 2024-03-21T13:03:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330543+0000", 2024-03-21T13:03:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330543+0000", 2024-03-21T13:03:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330544+0000", 2024-03-21T13:03:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5209999999999999e-06 2024-03-21T13:03:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330628+0000", 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3535000000000006e-05 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330637+0000", 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7720000000000006e-06 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330644+0000", 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6730000000000003e-06 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330706+0000", 2024-03-21T13:03:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2160000000000001e-05 2024-03-21T13:03:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330804+0000", 2024-03-21T13:03:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7522999999999994e-05 2024-03-21T13:03:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:03:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330931+0000", 2024-03-21T13:03:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000127584 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.331447+0000", 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00051556400000000004 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.331956+0000", 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000509302 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.437417+0000", 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.105460939 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.437501+0000", 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4078000000000004e-05 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.437549+0000", 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7633000000000003e-05 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:66554 7.0 7:00032620:::benchmark_data_smithi163_80312_object66553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:01:32.675052+0000", 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 109.104092568, 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.094370187999999994, 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 66554 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675052+0000", 2024-03-21T13:03:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675052+0000", 2024-03-21T13:03:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675053+0000", 2024-03-21T13:03:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.178e-06 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675115+0000", 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2138999999999996e-05 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675125+0000", 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0362e-05 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675131+0000", 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2519999999999996e-06 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675182+0000", 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0618000000000002e-05 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675272+0000", 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0174999999999996e-05 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675363+0000", 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0754999999999994e-05 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.676991+0000", 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016278849999999999 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.677307+0000", 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031613000000000002 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.769314+0000", 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.092007163000000003 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.769377+0000", 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2644000000000005e-05 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.769422+0000", 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4886999999999999e-05 2024-03-21T13:03:21.809 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.810 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.810 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.810 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:94688 7.0 7:0e49193b:::benchmark_data_smithi163_80312_object94687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:03:21.810 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:01:59.258336+0000", 2024-03-21T13:03:21.810 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 82.520808493000004, 2024-03-21T13:03:21.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.099394761999999998, 2024-03-21T13:03:21.810 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.810 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.810 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.810 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 94688 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258336+0000", 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258336+0000", 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258337+0000", 2024-03-21T13:03:21.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2440000000000001e-06 2024-03-21T13:03:21.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.813 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258393+0000", 2024-03-21T13:03:21.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6415000000000001e-05 2024-03-21T13:03:21.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.813 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258405+0000", 2024-03-21T13:03:21.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.135e-05 2024-03-21T13:03:21.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258411+0000", 2024-03-21T13:03:21.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.993e-06 2024-03-21T13:03:21.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258461+0000", 2024-03-21T13:03:21.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9886000000000003e-05 2024-03-21T13:03:21.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258555+0000", 2024-03-21T13:03:21.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4494999999999996e-05 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258650+0000", 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4661999999999996e-05 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.264002+0000", 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053522109999999999 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.264051+0000", 2024-03-21T13:03:21.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8980999999999997e-05 2024-03-21T13:03:21.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.357625+0000", 2024-03-21T13:03:21.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.093574288000000005 2024-03-21T13:03:21.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.357687+0000", 2024-03-21T13:03:21.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1737999999999997e-05 2024-03-21T13:03:21.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.357730+0000", 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3498999999999998e-05 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:94691 7.0 7:0868e98a:::benchmark_data_smithi163_80312_object94690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:01:59.260462+0000", 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 82.518682018000007, 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.097889528000000003, 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.817 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.818 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.818 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.818 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.818 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 94691 2024-03-21T13:03:21.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.818 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260462+0000", 2024-03-21T13:03:21.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260462+0000", 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260463+0000", 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6899999999999996e-07 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260528+0000", 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5376000000000006e-05 2024-03-21T13:03:21.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260538+0000", 2024-03-21T13:03:21.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6600000000000007e-06 2024-03-21T13:03:21.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260555+0000", 2024-03-21T13:03:21.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7503000000000002e-05 2024-03-21T13:03:21.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260943+0000", 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038779500000000002 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.261055+0000", 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112186 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.261202+0000", 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014699900000000001 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.264043+0000", 2024-03-21T13:03:21.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028405769999999999 2024-03-21T13:03:21.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.264118+0000", 2024-03-21T13:03:21.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4924000000000003e-05 2024-03-21T13:03:21.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.358274+0000", 2024-03-21T13:03:21.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.09415635 2024-03-21T13:03:21.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.358311+0000", 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7169000000000003e-05 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.358352+0000", 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0219999999999998e-05 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.823 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:95064 7.0 7:0c9f881e:::benchmark_data_smithi163_80312_object95063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:01:59.727240+0000", 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 82.051904261000004, 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.080850353999999999, 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 95064 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727240+0000", 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727240+0000", 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727241+0000", 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.167e-06 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727287+0000", 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5596999999999997e-05 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727298+0000", 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1144e-05 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727304+0000", 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3470000000000004e-06 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727352+0000", 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8359000000000002e-05 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727438+0000", 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5470000000000007e-05 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727545+0000", 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107428 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.730267+0000", 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027215310000000001 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.731653+0000", 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001386034 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.807949+0000", 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.076295899 2024-03-21T13:03:21.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.808042+0000", 2024-03-21T13:03:21.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3126000000000003e-05 2024-03-21T13:03:21.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.808090+0000", 2024-03-21T13:03:21.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8251999999999998e-05 2024-03-21T13:03:21.829 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.829 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.829 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116056 7.b 7:df017f68:::benchmark_data_smithi163_80312_object116055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.467833+0000", 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 59.311311469000003, 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.085221341000000006, 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116056 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467833+0000", 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467833+0000", 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467844+0000", 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1499e-05 2024-03-21T13:03:21.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467906+0000", 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1583999999999999e-05 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467918+0000", 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2690999999999999e-05 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467926+0000", 2024-03-21T13:03:21.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0190000000000006e-06 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467984+0000", 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8004000000000002e-05 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.468133+0000", 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014849500000000001 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,6", 2024-03-21T13:03:21.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.468321+0000", 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018788600000000001 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.471683+0000", 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003362659 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.471770+0000", 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6512000000000001e-05 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.475037+0000", 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032672109999999999 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.552843+0000", 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.077806288000000001 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.552926+0000", 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2328000000000002e-05 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.553054+0000", 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012816499999999999 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116068 7.0 7:09402460:::benchmark_data_smithi163_80312_object116067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.483157+0000", 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 59.295986861999999, 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.081778124999999993, 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.836 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116068 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483157+0000", 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483157+0000", 2024-03-21T13:03:21.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483159+0000", 2024-03-21T13:03:21.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.407e-06 2024-03-21T13:03:21.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483192+0000", 2024-03-21T13:03:21.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3175e-05 2024-03-21T13:03:21.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483203+0000", 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1457999999999999e-05 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483211+0000", 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6620000000000005e-06 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483281+0000", 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9734999999999996e-05 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483422+0000", 2024-03-21T13:03:21.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014167800000000001 2024-03-21T13:03:21.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:03:21.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483585+0000", 2024-03-21T13:03:21.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016310599999999999 2024-03-21T13:03:21.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.486348+0000", 2024-03-21T13:03:21.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027625190000000002 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487663+0000", 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001315309 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.563548+0000", 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.075884518999999998 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.564765+0000", 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001216983 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.564819+0000", 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4404999999999998e-05 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.564935+0000", 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011616900000000001 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.842 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116075 7.0 7:08926736:::benchmark_data_smithi163_80312_object116074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.487709+0000", 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 59.291435407999998, 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.078736329999999993, 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116075 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487709+0000", 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487709+0000", 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487709+0000", 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6599999999999995e-07 2024-03-21T13:03:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487766+0000", 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.681e-05 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487775+0000", 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5820000000000002e-06 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487782+0000", 2024-03-21T13:03:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2169999999999997e-06 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.490331+0000", 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025494060000000002 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.490445+0000", 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000113776 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:03:21.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.490581+0000", 2024-03-21T13:03:21.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000136224 2024-03-21T13:03:21.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.492567+0000", 2024-03-21T13:03:21.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001985503 2024-03-21T13:03:21.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.493189+0000", 2024-03-21T13:03:21.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00062200299999999999 2024-03-21T13:03:21.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.565262+0000", 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072073034999999994 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.566264+0000", 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010021769999999999 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.566331+0000", 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6910999999999997e-05 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.566445+0000", 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011391999999999999 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116240 7.b 7:d015c35c:::benchmark_data_smithi163_80312_object116239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.720013+0000", 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 59.059131272999998, 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.130214526, 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.849 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116240 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720013+0000", 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720013+0000", 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720014+0000", 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4440000000000001e-06 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720050+0000", 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5490000000000001e-05 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720064+0000", 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3773e-05 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720072+0000", 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2290000000000003e-06 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720132+0000", 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9831999999999999e-05 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720279+0000", 2024-03-21T13:03:21.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014782799999999999 2024-03-21T13:03:21.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,6", 2024-03-21T13:03:21.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720455+0000", 2024-03-21T13:03:21.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017585499999999999 2024-03-21T13:03:21.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.723242+0000", 2024-03-21T13:03:21.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027871419999999998 2024-03-21T13:03:21.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.723669+0000", 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000426933 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.723963+0000", 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029338200000000002 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.850012+0000", 2024-03-21T13:03:21.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.126049095 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.850083+0000", 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1653e-05 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.850227+0000", 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014386999999999999 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116247 7.0 7:0a4d72c9:::benchmark_data_smithi163_80312_object116246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.726834+0000", 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 59.052310493, 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.130439006, 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116247 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726834+0000", 2024-03-21T13:03:21.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726834+0000", 2024-03-21T13:03:21.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726835+0000", 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4959999999999999e-06 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726865+0000", 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0080999999999999e-05 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726886+0000", 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1182000000000001e-05 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726895+0000", 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4719999999999995e-06 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726971+0000", 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5972999999999999e-05 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.727099+0000", 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000128162 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.727267+0000", 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016778600000000001 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730152+0000", 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028849560000000002 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.732900+0000", 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027485600000000002 2024-03-21T13:03:21.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.848690+0000", 2024-03-21T13:03:21.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.11578986600000001 2024-03-21T13:03:21.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.857079+0000", 2024-03-21T13:03:21.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0083888729999999998 2024-03-21T13:03:21.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.857149+0000", 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.991e-05 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.857273+0000", 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012368899999999999 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116251 7.0 7:017df485:::benchmark_data_smithi163_80312_object116250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.730529+0000", 2024-03-21T13:03:21.863 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 59.048614704999999, 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12804560300000001, 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116251 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730529+0000", 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730529+0000", 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730530+0000", 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0809999999999999e-06 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730582+0000", 2024-03-21T13:03:21.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1319000000000001e-05 2024-03-21T13:03:21.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730592+0000", 2024-03-21T13:03:21.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0332e-05 2024-03-21T13:03:21.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730598+0000", 2024-03-21T13:03:21.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3289999999999999e-06 2024-03-21T13:03:21.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730645+0000", 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6916000000000003e-05 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730750+0000", 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104949 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730879+0000", 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012888 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.732868+0000", 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001989237 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.733022+0000", 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000153108 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.858130+0000", 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.125108673 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.858342+0000", 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000211428 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.858408+0000", 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5899e-05 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.858575+0000", 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016745200000000001 2024-03-21T13:03:21.869 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.870 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.870 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.870 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116535 7.b 7:db9c5de4:::benchmark_data_smithi163_80312_object116534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:03:21.870 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:23.124212+0000", 2024-03-21T13:03:21.870 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 58.654932023000001, 2024-03-21T13:03:21.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.092375593000000006, 2024-03-21T13:03:21.870 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.870 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.870 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.870 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.870 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.871 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116535 2024-03-21T13:03:21.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.871 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124212+0000", 2024-03-21T13:03:21.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124212+0000", 2024-03-21T13:03:21.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124213+0000", 2024-03-21T13:03:21.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3700000000000001e-07 2024-03-21T13:03:21.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124242+0000", 2024-03-21T13:03:21.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9159e-05 2024-03-21T13:03:21.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124253+0000", 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0869e-05 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124259+0000", 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9410000000000001e-06 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.127981+0000", 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003721673 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.130206+0000", 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022252589999999998 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,6", 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.130421+0000", 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000214828 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.136255+0000", 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058337229999999999 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.137633+0000", 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013781570000000001 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.138730+0000", 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010970929999999999 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.216445+0000", 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.077715203999999996 2024-03-21T13:03:21.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.216500+0000", 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4321999999999999e-05 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.216588+0000", 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7928000000000002e-05 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.876 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116566 7.b 7:ddb81152:::benchmark_data_smithi163_80312_object116565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:23.152336+0000", 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 58.626807774, 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.089229614999999998, 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116566 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152336+0000", 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152336+0000", 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152338+0000", 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.57e-06 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152379+0000", 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1118000000000002e-05 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152399+0000", 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0415e-05 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152423+0000", 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3938999999999999e-05 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152487+0000", 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3590000000000006e-05 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.153143+0000", 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00065597699999999997 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,6", 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.153343+0000", 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020033 2024-03-21T13:03:21.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.156773+0000", 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034299539999999998 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.157158+0000", 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000385153 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.157320+0000", 2024-03-21T13:03:21.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016143900000000001 2024-03-21T13:03:21.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.241425+0000", 2024-03-21T13:03:21.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.084105503999999998 2024-03-21T13:03:21.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:03:21.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.241470+0000", 2024-03-21T13:03:21.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5158999999999999e-05 2024-03-21T13:03:21.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.241566+0000", 2024-03-21T13:03:21.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5466999999999999e-05 2024-03-21T13:03:21.883 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.883 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.883 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.883 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118902 7.0 7:007d3f07:::benchmark_data_smithi163_80312_object1461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:03:21.883 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.702328+0000", 2024-03-21T13:03:21.883 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 56.076815772000003, 2024-03-21T13:03:21.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.511295948, 2024-03-21T13:03:21.883 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118902 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702328+0000", 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702328+0000", 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702329+0000", 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1510000000000001e-06 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702334+0000", 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0300000000000004e-06 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702343+0000", 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2029999999999998e-06 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702350+0000", 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.182e-06 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702410+0000", 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.05e-05 2024-03-21T13:03:21.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:21.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702498+0000", 2024-03-21T13:03:21.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7113000000000004e-05 2024-03-21T13:03:21.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:03:21.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702603+0000", 2024-03-21T13:03:21.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105567 2024-03-21T13:03:21.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:21.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.705654+0000", 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030509489999999998 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.709103+0000", 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003448914 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.712195+0000", 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030919419999999999 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.750528+0000", 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.038333000999999998 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.750541+0000", 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3454e-05 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213611+0000", 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46306928400000003 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213624+0000", 2024-03-21T13:03:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3658e-05 2024-03-21T13:03:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118942 7.0 7.1f908400 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:03:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.753006+0000", 2024-03-21T13:03:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 56.026137628000001, 2024-03-21T13:03:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46065019600000001, 2024-03-21T13:03:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118942 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753006+0000", 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753006+0000", 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753008+0000", 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3999999999999999e-06 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753011+0000", 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5899999999999999e-06 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753020+0000", 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.242e-06 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753027+0000", 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3220000000000004e-06 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.754026+0000", 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00099910600000000004 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.754036+0000", 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7110000000000007e-06 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213649+0000", 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45961280399999999 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213657+0000", 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0209999999999999e-06 2024-03-21T13:03:21.894 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118952 7.0 7.3905ab10 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.767903+0000", 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 56.011240753999999, 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44580361899999998, 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.895 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118952 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767903+0000", 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767903+0000", 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767905+0000", 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5370000000000001e-06 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767914+0000", 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0880000000000001e-06 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767925+0000", 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1231e-05 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767934+0000", 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4889999999999995e-06 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.768019+0000", 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4860999999999999e-05 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.768029+0000", 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0558e-05 2024-03-21T13:03:21.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213698+0000", 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44566897700000002 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213707+0000", 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8780000000000004e-06 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.900 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118953 7.0 7.83d475e0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:03:21.900 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.767956+0000", 2024-03-21T13:03:21.900 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 56.011188400000002, 2024-03-21T13:03:21.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44578293800000002, 2024-03-21T13:03:21.900 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.900 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:03:21.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.900 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118953 2024-03-21T13:03:21.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.900 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767956+0000", 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767956+0000", 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767956+0000", 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7199999999999999e-07 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767959+0000", 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9289999999999998e-06 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767966+0000", 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3280000000000002e-06 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767971+0000", 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6180000000000003e-06 2024-03-21T13:03:21.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.768055+0000", 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3486999999999995e-05 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.768062+0000", 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9040000000000003e-06 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213730+0000", 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44566886999999999 2024-03-21T13:03:21.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213739+0000", 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2300000000000008e-06 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118970 7.b 7.e8b90beb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.796717+0000", 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 55.982427139999999, 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42020922900000002, 2024-03-21T13:03:21.904 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118970 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796717+0000", 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796717+0000", 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796719+0000", 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0040000000000002e-06 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796728+0000", 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9570000000000008e-06 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796748+0000", 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9741e-05 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796758+0000", 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9569999999999997e-06 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796848+0000", 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0754000000000006e-05 2024-03-21T13:03:21.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796860+0000", 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1265e-05 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.216916+0000", 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42005610100000002 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.216926+0000", 2024-03-21T13:03:21.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.045e-05 2024-03-21T13:03:21.909 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.909 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.909 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.909 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:21.909 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:21.909 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:03:21.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:21.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:21.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:22.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:22.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:22.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:22.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:22.220 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:22.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:22.323 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:22.363 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:22.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:22.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:22.465 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:22.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:22 smithi204 ceph-mon[33153]: pgmap v181: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 16221/191805 objects misplaced (8.457%); 39 MiB/s, 625 objects/s recovering 2024-03-21T13:03:22.503 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:22.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:22.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:22.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:22 smithi163 ceph-mon[29621]: pgmap v181: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 16221/191805 objects misplaced (8.457%); 39 MiB/s, 625 objects/s recovering 2024-03-21T13:03:22.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:22 smithi163 ceph-mon[40675]: pgmap v181: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 16221/191805 objects misplaced (8.457%); 39 MiB/s, 625 objects/s recovering 2024-03-21T13:03:22.605 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:22.639 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:22.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:22.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:22.741 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:22.781 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:22.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:22.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:22.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:22.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:22.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:23.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:23.050 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:23.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:23.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:23.097 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-21T13:03:23.152 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:23.192 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:23.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:23.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:23.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:23.339 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:23.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:23.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:23.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:23.484 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:23.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:23.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:23 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:23.506 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:03:23.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:23 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:23.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:23 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:23.586 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:23.594 INFO:tasks.thrashosds.thrasher:j is {'epoch': 136, 'fsid': 'd4bf79c0-e781-11ee-95cd-87774f69a715', 'created': '2024-03-21T12:52:56.031419+0000', 'modified': '2024-03-21T13:03:16.950940+0000', 'last_up_change': '2024-03-21T12:58:23.775302+0000', 'last_in_change': '2024-03-21T13:02:25.729226+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T12:56:20.557490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8456535339355469, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.8500000238418579}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-21T12:58:35.713383+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.8456535339355469, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.8500000238418579}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-21T12:58:36.436384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8456535339355469, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.8500000238418579}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-21T12:58:38.049304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.25, 'score_stable': 3.25, 'optimal_score': 0.75, 'raw_score_acting': 2.440000057220459, 'raw_score_stable': 2.440000057220459, 'primary_affinity_weighted': 0.8456535339355469, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.8500000238418579}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-21T12:58:39.990512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8456535339355469, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.8500000238418579}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-21T12:58:41.463686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8456535339355469, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.8500000238418579}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-21T13:00:41.658316+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8456535339355469, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.8500000238418579}}], 'osds': [{'osd': 0, 'uuid': 'c34a3b25-9e02-46e6-8f9a-d0af2615258f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.0852813720703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 128, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6801', 'nonce': 3086410981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6802', 'nonce': 3086410981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6804', 'nonce': 3086410981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6803', 'nonce': 3086410981}]}, 'public_addr': '172.21.15.163:6801/3086410981', 'cluster_addr': '172.21.15.163:6802/3086410981', 'heartbeat_back_addr': '172.21.15.163:6804/3086410981', 'heartbeat_front_addr': '172.21.15.163:6803/3086410981', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '36e05f89-c238-41f7-be65-1553b529e8ce', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6805', 'nonce': 3666062784}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6806', 'nonce': 3666062784}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6808', 'nonce': 3666062784}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6807', 'nonce': 3666062784}]}, 'public_addr': '172.21.15.163:6805/3666062784', 'cluster_addr': '172.21.15.163:6806/3666062784', 'heartbeat_back_addr': '172.21.15.163:6808/3666062784', 'heartbeat_front_addr': '172.21.15.163:6807/3666062784', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '968ec78d-25b8-4905-b328-d490c38ca85f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 124, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6809', 'nonce': 4009213507}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6810', 'nonce': 4009213507}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6812', 'nonce': 4009213507}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6811', 'nonce': 4009213507}]}, 'public_addr': '172.21.15.163:6809/4009213507', 'cluster_addr': '172.21.15.163:6810/4009213507', 'heartbeat_back_addr': '172.21.15.163:6812/4009213507', 'heartbeat_front_addr': '172.21.15.163:6811/4009213507', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '67dfa4f6-1709-48f6-971d-c6287233492d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 124, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6813', 'nonce': 3981878189}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6814', 'nonce': 3981878189}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6816', 'nonce': 3981878189}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6815', 'nonce': 3981878189}]}, 'public_addr': '172.21.15.163:6813/3981878189', 'cluster_addr': '172.21.15.163:6814/3981878189', 'heartbeat_back_addr': '172.21.15.163:6816/3981878189', 'heartbeat_front_addr': '172.21.15.163:6815/3981878189', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7fd311a9-f3c2-456d-8ab1-341cf5b33471', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6799468994140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 123, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 719276054}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6801', 'nonce': 719276054}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6803', 'nonce': 719276054}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 719276054}]}, 'public_addr': '172.21.15.204:6800/719276054', 'cluster_addr': '172.21.15.204:6801/719276054', 'heartbeat_back_addr': '172.21.15.204:6803/719276054', 'heartbeat_front_addr': '172.21.15.204:6802/719276054', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a87759c8-251c-415d-9387-a49f7d2f36d1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 123, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 1346947668}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6805', 'nonce': 1346947668}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6807', 'nonce': 1346947668}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 1346947668}]}, 'public_addr': '172.21.15.204:6804/1346947668', 'cluster_addr': '172.21.15.204:6805/1346947668', 'heartbeat_back_addr': '172.21.15.204:6807/1346947668', 'heartbeat_front_addr': '172.21.15.204:6806/1346947668', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5eb135cd-f677-4d0c-b23a-738f31615b6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 200077988}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6809', 'nonce': 200077988}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6811', 'nonce': 200077988}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 200077988}]}, 'public_addr': '172.21.15.204:6808/200077988', 'cluster_addr': '172.21.15.204:6809/200077988', 'heartbeat_back_addr': '172.21.15.204:6811/200077988', 'heartbeat_front_addr': '172.21.15.204:6810/200077988', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4f528e10-2c93-4bc2-a54c-06ca4132ed49', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 315664888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6813', 'nonce': 315664888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6815', 'nonce': 315664888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 315664888}]}, 'public_addr': '172.21.15.204:6812/315664888', 'cluster_addr': '172.21.15.204:6813/315664888', 'heartbeat_back_addr': '172.21.15.204:6815/315664888', 'heartbeat_front_addr': '172.21.15.204:6814/315664888', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:02:19.697874+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-21T13:02:18.653335+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:02:26.419877+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:03:11.631115+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:02:08.890428+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-21T13:02:51.764974+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:03:01.908480+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:02:11.917673+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.0', 'osds': [3, 5, 1]}, {'pgid': '7.2', 'osds': [3, 2, 1]}, {'pgid': '7.4', 'osds': [4, 0, 1]}, {'pgid': '7.a', 'osds': [3, 2, 5]}, {'pgid': '7.d', 'osds': [6, 0, 1]}, {'pgid': '7.e', 'osds': [2, 0, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.163:0/3453855427': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/2571244897': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:6800/3073378854': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/1334591766': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/116934309': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:0/3890775846': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/1783921524': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:6800/2553440211': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1812694077': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1070540283': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/2354157786': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1591817392': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/4152198951': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:6800/1798115633': '2024-03-22T12:54:00.224135+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-21T13:03:23.595 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:03:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:23.634 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:23.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:23.735 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:23.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:23.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:23.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:23.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:23.933 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:23.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:24.034 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:24.084 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:24.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:24.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:24.186 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:24.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:24.265 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:24.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:24.284 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:24.366 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:24.396 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:24.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:24.460 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:24 smithi163 ceph-mon[40675]: pgmap v182: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 830 op/s; 16221/191805 objects misplaced (8.457%); 31 MiB/s, 492 objects/s recovering 2024-03-21T13:03:24.460 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:24 smithi163 ceph-mon[29621]: pgmap v182: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 830 op/s; 16221/191805 objects misplaced (8.457%); 31 MiB/s, 492 objects/s recovering 2024-03-21T13:03:24.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:24.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:24 smithi204 ceph-mon[33153]: pgmap v182: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 830 op/s; 16221/191805 objects misplaced (8.457%); 31 MiB/s, 492 objects/s recovering 2024-03-21T13:03:24.524 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:24.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:24.584 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:24.625 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:24.652 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:24.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:24.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:24.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:24.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:24.780 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:24.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:24.881 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:24.907 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:24.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:24.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:25.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:25.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:25.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:25.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:25.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:25.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:25.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:25.263 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:25.273 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:03:25.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:25.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:25.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:25.434 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:25.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:25.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:25.468 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:25 smithi163 ceph-mon[29621]: osdmap e137: 8 total, 8 up, 6 in 2024-03-21T13:03:25.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:25 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1620896917' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:03:25.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:25 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:03:25.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:25 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:03:25.469 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:25 smithi163 ceph-mon[40675]: osdmap e137: 8 total, 8 up, 6 in 2024-03-21T13:03:25.469 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:25 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1620896917' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:03:25.469 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:25 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:03:25.469 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:25 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:03:25.494 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:25 smithi204 ceph-mon[33153]: osdmap e137: 8 total, 8 up, 6 in 2024-03-21T13:03:25.494 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:25 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1620896917' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:03:25.495 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:25 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:03:25.495 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:25 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:03:25.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:25.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:25.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:25.668 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:25.709 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:25.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:25.762 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:03:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:03:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:03:25.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:25.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:25.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:25.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:25.948 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:25.987 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:25.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:26.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:26.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:26.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:26.223 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:26.258 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:26.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:26.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:26.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:26.398 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:26.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:26.437 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:26 smithi163 ceph-mon[29621]: pgmap v184: 148 pgs: 1 peering, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 893 op/s; 15526/183765 objects misplaced (8.449%); 28 MiB/s, 443 objects/s recovering 2024-03-21T13:03:26.438 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:26 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:03:26.438 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:26 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1620896917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:03:26.438 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:26 smithi163 ceph-mon[29621]: osdmap e138: 8 total, 8 up, 6 in 2024-03-21T13:03:26.438 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:26 smithi163 ceph-mon[29621]: 7.d scrub starts 2024-03-21T13:03:26.438 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:26 smithi163 ceph-mon[40675]: pgmap v184: 148 pgs: 1 peering, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 893 op/s; 15526/183765 objects misplaced (8.449%); 28 MiB/s, 443 objects/s recovering 2024-03-21T13:03:26.438 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:26 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:03:26.438 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:26 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1620896917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:03:26.438 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:26 smithi163 ceph-mon[40675]: osdmap e138: 8 total, 8 up, 6 in 2024-03-21T13:03:26.438 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:26 smithi163 ceph-mon[40675]: 7.d scrub starts 2024-03-21T13:03:26.498 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:26.541 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:26.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:26.552 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:26 smithi204 ceph-mon[33153]: pgmap v184: 148 pgs: 1 peering, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 893 op/s; 15526/183765 objects misplaced (8.449%); 28 MiB/s, 443 objects/s recovering 2024-03-21T13:03:26.552 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:26 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:03:26.552 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:26 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1620896917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:03:26.552 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:26 smithi204 ceph-mon[33153]: osdmap e138: 8 total, 8 up, 6 in 2024-03-21T13:03:26.552 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:26 smithi204 ceph-mon[33153]: 7.d scrub starts 2024-03-21T13:03:26.642 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:26.646 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:03:26.647 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:03:26.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.647 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.14820.0:177513 7.0 e138/124 7:0ead96e9:::benchmark_data_smithi163_80312_object60072:head v 138'11097, mlcod=138'11095)", 2024-03-21T13:03:26.647 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:26.641007+0000", 2024-03-21T13:03:26.647 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.003227677, 2024-03-21T13:03:26.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032570619999999998, 2024-03-21T13:03:26.647 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:26.647 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:03:26.647 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:26.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:26.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641007+0000", 2024-03-21T13:03:26.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:26.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641007+0000", 2024-03-21T13:03:26.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:26.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641007+0000", 2024-03-21T13:03:26.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.74e-07 2024-03-21T13:03:26.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641017+0000", 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0122999999999999e-05 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641022+0000", 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0520000000000001e-06 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641026+0000", 2024-03-21T13:03:26.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4780000000000002e-06 2024-03-21T13:03:26.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:26.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641080+0000", 2024-03-21T13:03:26.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3510000000000001e-05 2024-03-21T13:03:26.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:26.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641103+0000", 2024-03-21T13:03:26.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3390000000000001e-05 2024-03-21T13:03:26.650 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.650 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:26.650 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.651 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.14820.0:177516 7.0 e138/124 7:0b413d94:::benchmark_data_smithi163_80312_object60075:head v 138'11098, mlcod=138'11095)", 2024-03-21T13:03:26.651 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:26.641093+0000", 2024-03-21T13:03:26.651 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0031414979999999999, 2024-03-21T13:03:26.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032159290000000002, 2024-03-21T13:03:26.651 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:26.651 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:03:26.651 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:26.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:26.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641093+0000", 2024-03-21T13:03:26.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641093+0000", 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641093+0000", 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9299999999999999e-07 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641096+0000", 2024-03-21T13:03:26.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7539999999999998e-06 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641100+0000", 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3959999999999999e-06 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641104+0000", 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9650000000000002e-06 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:26.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.643145+0000", 2024-03-21T13:03:26.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002040923 2024-03-21T13:03:26.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:26.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.643164+0000", 2024-03-21T13:03:26.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9335e-05 2024-03-21T13:03:26.654 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.654 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:26.654 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.654 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:177518 7.c 7:358ca7b4:::benchmark_data_smithi163_80312_object60077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-21T13:03:26.654 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:26.641414+0000", 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0028205309999999998, 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029356719999999998, 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 177518 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641414+0000", 2024-03-21T13:03:26.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641414+0000", 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641415+0000", 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.186e-06 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:26.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641419+0000", 2024-03-21T13:03:26.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.092e-06 2024-03-21T13:03:26.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:26.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641428+0000", 2024-03-21T13:03:26.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2280000000000004e-06 2024-03-21T13:03:26.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:26.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641435+0000", 2024-03-21T13:03:26.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.443e-06 2024-03-21T13:03:26.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641490+0000", 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5322000000000003e-05 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641613+0000", 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012303499999999999 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641748+0000", 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000134771 2024-03-21T13:03:26.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.643607+0000", 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0018588319999999999 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.643964+0000", 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00035687099999999998 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:177501 7.d 7:b0abbb2d:::benchmark_data_smithi163_80312_object60060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:26.627129+0000", 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.017105532, 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017289891000000002, 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 177501 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:26.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.627129+0000", 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.627129+0000", 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.627130+0000", 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0699999999999996e-07 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:26.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.627134+0000", 2024-03-21T13:03:26.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4309999999999996e-06 2024-03-21T13:03:26.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:26.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.627142+0000", 2024-03-21T13:03:26.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9589999999999995e-06 2024-03-21T13:03:26.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:26.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.627152+0000", 2024-03-21T13:03:26.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8819999999999996e-06 2024-03-21T13:03:26.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.630170+0000", 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030179109999999999 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.630282+0000", 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112002 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.630402+0000", 2024-03-21T13:03:26.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000120029 2024-03-21T13:03:26.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:26.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.633387+0000", 2024-03-21T13:03:26.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029847929999999999 2024-03-21T13:03:26.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:26.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641123+0000", 2024-03-21T13:03:26.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0077368979999999999 2024-03-21T13:03:26.664 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:177519 7.7 7:e887129a:::benchmark_data_smithi163_80312_object60078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:26.644167+0000", 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 6.6885999999999996e-05, 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00034338199999999999, 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 177519 2024-03-21T13:03:26.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.666 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:26.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:26.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644167+0000", 2024-03-21T13:03:26.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:26.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644167+0000", 2024-03-21T13:03:26.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644169+0000", 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1519999999999999e-06 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644177+0000", 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6859999999999995e-06 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644190+0000", 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3141e-05 2024-03-21T13:03:26.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644200+0000", 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7999999999999993e-06 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644255+0000", 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4729999999999999e-05 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644335+0000", 2024-03-21T13:03:26.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9658000000000002e-05 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644408+0000", 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3410000000000004e-05 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.14820.0:177511 7.7 e138/92)", 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:26.644366+0000", 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4294967295.9998689, 2024-03-21T13:03:26.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018990000000000001, 2024-03-21T13:03:26.670 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:26.670 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "queued for pg", 2024-03-21T13:03:26.670 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:26.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:26.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644366+0000", 2024-03-21T13:03:26.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:26.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644366+0000", 2024-03-21T13:03:26.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:26.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644366+0000", 2024-03-21T13:03:26.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7e-07 2024-03-21T13:03:26.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:26.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644369+0000", 2024-03-21T13:03:26.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7350000000000001e-06 2024-03-21T13:03:26.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:26.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644374+0000", 2024-03-21T13:03:26.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4790000000000004e-06 2024-03-21T13:03:26.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:26.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644378+0000", 2024-03-21T13:03:26.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9269999999999998e-06 2024-03-21T13:03:26.672 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.672 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:26.672 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.672 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.14820.0:177512 7.c e138/92 ondisk, result = 0)", 2024-03-21T13:03:26.672 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:26.644424+0000", 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4294967295.9998102, 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016022200000000001, 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644424+0000", 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644424+0000", 2024-03-21T13:03:26.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644425+0000", 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.96e-07 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644427+0000", 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4399999999999999e-06 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644431+0000", 2024-03-21T13:03:26.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2440000000000004e-06 2024-03-21T13:03:26.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:26.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644435+0000", 2024-03-21T13:03:26.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4489999999999999e-06 2024-03-21T13:03:26.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:26.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644498+0000", 2024-03-21T13:03:26.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3471000000000003e-05 2024-03-21T13:03:26.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:26.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644503+0000", 2024-03-21T13:03:26.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9239999999999998e-06 2024-03-21T13:03:26.676 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.676 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:26.676 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:177509 7.d 7:b7ade8b8:::benchmark_data_smithi163_80312_object60068:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:26.634101+0000", 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.010133144, 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010531633, 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 177509 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:26.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:26.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.634101+0000", 2024-03-21T13:03:26.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:26.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.634101+0000", 2024-03-21T13:03:26.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:26.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.634102+0000", 2024-03-21T13:03:26.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7900000000000007e-07 2024-03-21T13:03:26.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:26.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.634106+0000", 2024-03-21T13:03:26.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.07e-06 2024-03-21T13:03:26.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:26.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.634115+0000", 2024-03-21T13:03:26.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3350000000000007e-06 2024-03-21T13:03:26.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.634120+0000", 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.823e-06 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.634159+0000", 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8379e-05 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.634237+0000", 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8140999999999996e-05 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.634309+0000", 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2547999999999997e-05 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.636188+0000", 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001878722 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.641144+0000", 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0049560180000000004 2024-03-21T13:03:26.681 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.682 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:26.682 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.682 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:177511 7.7 7:ecfbe6ea:::benchmark_data_smithi163_80312_object60070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-21T13:03:26.682 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:26.636363+0000", 2024-03-21T13:03:26.682 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0078710970000000005, 2024-03-21T13:03:26.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0083289499999999999, 2024-03-21T13:03:26.682 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:26.682 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:03:26.682 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:26.682 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:26.682 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 177511 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.636363+0000", 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.636363+0000", 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.636364+0000", 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9999999999999997e-07 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.636367+0000", 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4599999999999999e-06 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.636376+0000", 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1419999999999993e-06 2024-03-21T13:03:26.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.636381+0000", 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5910000000000003e-06 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.636441+0000", 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9777e-05 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.636685+0000", 2024-03-21T13:03:26.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000244415 2024-03-21T13:03:26.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:03:26.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.636804+0000", 2024-03-21T13:03:26.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118788 2024-03-21T13:03:26.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:26.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.639514+0000", 2024-03-21T13:03:26.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027102049999999998 2024-03-21T13:03:26.686 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.686 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:26.686 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:177515 7.d 7:b42563a0:::benchmark_data_smithi163_80312_object60074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:26.637285+0000", 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0069495889999999999, 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074630219999999997, 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 177515 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.687 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.637285+0000", 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.637285+0000", 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.637285+0000", 2024-03-21T13:03:26.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3399999999999998e-07 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.637289+0000", 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9210000000000002e-06 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.637299+0000", 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2250000000000006e-06 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.637305+0000", 2024-03-21T13:03:26.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7619999999999998e-06 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.637358+0000", 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2707999999999997e-05 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.637482+0000", 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000123855 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:03:26.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.637620+0000", 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013829900000000001 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.639522+0000", 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001901759 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:26.644064+0000", 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0045420030000000002 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:03:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.692 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:03:26.692 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-03-21T13:03:26.692 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 10 2024-03-21T13:03:26.692 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:03:26.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:26.693 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:26.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:26.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:26.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:26.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:26.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:26.929 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:26.969 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:26.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:27.071 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:27.111 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:27.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:27.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:27.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:27.247 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:27.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:27.348 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:27.396 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:27.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:27.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:27.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:27.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:27 smithi163 ceph-mon[29621]: 7.d scrub ok 2024-03-21T13:03:27.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:27 smithi163 ceph-mon[40675]: 7.d scrub ok 2024-03-21T13:03:27.522 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-21T13:03:27.542 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:27.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:27.575 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:27 smithi204 ceph-mon[33153]: 7.d scrub ok 2024-03-21T13:03:27.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:27.682 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:27.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:27.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:27.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:27.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:27.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:27.969 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:27.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:27.976 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:03:28.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:28.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:28.113 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:28.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:28.214 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:28.261 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:28.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:28.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:28.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:28.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:28.420 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:28 smithi163 ceph-mon[29621]: pgmap v186: 148 pgs: 2 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 12724/177732 objects misplaced (7.159%); 42 MiB/s, 673 objects/s recovering 2024-03-21T13:03:28.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:28 smithi163 ceph-mon[40675]: pgmap v186: 148 pgs: 2 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 12724/177732 objects misplaced (7.159%); 42 MiB/s, 673 objects/s recovering 2024-03-21T13:03:28.483 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:03:28.504 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:28.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:28.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:28.725 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:28.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:28.747 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:03:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:28 smithi204 ceph-mon[33153]: pgmap v186: 148 pgs: 2 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 12724/177732 objects misplaced (7.159%); 42 MiB/s, 673 objects/s recovering 2024-03-21T13:03:28.793 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:03:28.827 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:28.838 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:03:28.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:28.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:28.894 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:03:28.941 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:03:28.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:28.989 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:28.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:28.996 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:03:29.003 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:29.045 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:03:29.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:29.096 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:03:29.110 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:03:29.110 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:03:29.111 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:03:29.111 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:03:29.111 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:03:29.111 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:03:29.111 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:03:29.111 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:03:29.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:29.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:29.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:29.224 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:29.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:29.266 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:29.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:29.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:03:29.401 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:29 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/996105538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:03:29.401 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:29 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/996105538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:03:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:29.402 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:29.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:29.405 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:03:29.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:29.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:29.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:29.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:29.642 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:29 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/996105538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:03:29.644 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:29.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:29.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:29.787 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:29.825 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:29.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:29.926 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:29.963 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:29.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:29.982 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:03:29.982 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:03:30.064 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:30.100 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:30.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:30.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:30.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:30.238 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:30.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:30.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:30.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:30.370 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:30.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:30.472 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:30.487 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-03-21T13:03:28.697939+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3870169536,"num_objects":59244,"num_object_clones":0,"num_object_copies":177732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12724,"num_objects_unfound":0,"num_objects_dirty":59244,"num_whiteouts":0,"num_read":1199,"num_read_kb":1045,"num_write":176447,"num_write_kb":7518494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30522,"num_bytes_recovered":1990072263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65097,"ondisk_log_size":65097,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":11342160,"kb_used_data":10498088,"kb_used_omap":9,"kb_used_meta":843638,"kb_avail":551128752,"statfs":{"total":575970213888,"available":564355842048,"internally_reserved":0,"allocated":10750042112,"data_stored":10747578590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":863885990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,3,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-307429376,"num_objects":-4691,"num_object_clones":0,"num_object_copies":-14073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3497,"num_objects_unfound":0,"num_objects_dirty":-4691,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4691,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2356,"num_bytes_recovered":154402816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002324"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":173,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266045+0000","last_change":"2024-03-21T13:02:26.223957+0000","last_active":"2024-03-21T13:03:24.266045+0000","last_peered":"2024-03-21T13:03:24.266045+0000","last_clean":"2024-03-21T13:03:24.266045+0000","last_became_active":"2024-03-21T13:02:26.223292+0000","last_became_peered":"2024-03-21T13:02:26.223292+0000","last_unstale":"2024-03-21T13:03:24.266045+0000","last_undegraded":"2024-03-21T13:03:24.266045+0000","last_fullsized":"2024-03-21T13:03:24.266045+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:01:35.694549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":40,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279452+0000","last_change":"2024-03-21T13:02:26.234070+0000","last_active":"2024-03-21T13:03:24.279452+0000","last_peered":"2024-03-21T13:03:24.279452+0000","last_clean":"2024-03-21T13:03:24.279452+0000","last_became_active":"2024-03-21T13:02:26.233709+0000","last_became_peered":"2024-03-21T13:02:26.233709+0000","last_unstale":"2024-03-21T13:03:24.279452+0000","last_undegraded":"2024-03-21T13:03:24.279452+0000","last_fullsized":"2024-03-21T13:03:24.279452+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:25.947092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:25.947092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":40,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.268531+0000","last_change":"2024-03-21T13:02:26.237862+0000","last_active":"2024-03-21T13:03:24.268531+0000","last_peered":"2024-03-21T13:03:24.268531+0000","last_clean":"2024-03-21T13:03:24.268531+0000","last_became_active":"2024-03-21T13:02:26.237501+0000","last_became_peered":"2024-03-21T13:02:26.237501+0000","last_unstale":"2024-03-21T13:03:24.268531+0000","last_undegraded":"2024-03-21T13:03:24.268531+0000","last_fullsized":"2024-03-21T13:03:24.268531+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:24.141445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:24.141445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"89'13","reported_seq":221,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.265845+0000","last_change":"2024-03-21T13:02:14.370743+0000","last_active":"2024-03-21T13:03:24.265845+0000","last_peered":"2024-03-21T13:03:24.265845+0000","last_clean":"2024-03-21T13:03:24.265845+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:03:24.265845+0000","last_undegraded":"2024-03-21T13:03:24.265845+0000","last_fullsized":"2024-03-21T13:03:24.265845+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:01:37.692932+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":153,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.280603+0000","last_change":"2024-03-21T13:00:05.173374+0000","last_active":"2024-03-21T13:03:24.280603+0000","last_peered":"2024-03-21T13:03:24.280603+0000","last_clean":"2024-03-21T13:03:24.280603+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:03:24.280603+0000","last_undegraded":"2024-03-21T13:03:24.280603+0000","last_fullsized":"2024-03-21T13:03:24.280603+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:00:05.173276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":84,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.275285+0000","last_change":"2024-03-21T13:02:14.330937+0000","last_active":"2024-03-21T13:03:24.275285+0000","last_peered":"2024-03-21T13:03:24.275285+0000","last_clean":"2024-03-21T13:03:24.275285+0000","last_became_active":"2024-03-21T13:02:14.330467+0000","last_became_peered":"2024-03-21T13:02:14.330467+0000","last_unstale":"2024-03-21T13:03:24.275285+0000","last_undegraded":"2024-03-21T13:03:24.275285+0000","last_fullsized":"2024-03-21T13:03:24.275285+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:00:16.731434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":173,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.286055+0000","last_change":"2024-03-21T13:01:38.766164+0000","last_active":"2024-03-21T13:03:24.286055+0000","last_peered":"2024-03-21T13:03:24.286055+0000","last_clean":"2024-03-21T13:03:24.286055+0000","last_became_active":"2024-03-21T12:58:37.728383+0000","last_became_peered":"2024-03-21T12:58:37.728383+0000","last_unstale":"2024-03-21T13:03:24.286055+0000","last_undegraded":"2024-03-21T13:03:24.286055+0000","last_fullsized":"2024-03-21T13:03:24.286055+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_clean_scrub_stamp":"2024-03-21T13:01:38.765896+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"71'15","reported_seq":186,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279907+0000","last_change":"2024-03-21T13:00:31.008873+0000","last_active":"2024-03-21T13:03:24.279907+0000","last_peered":"2024-03-21T13:03:24.279907+0000","last_clean":"2024-03-21T13:03:24.279907+0000","last_became_active":"2024-03-21T12:58:39.750396+0000","last_became_peered":"2024-03-21T12:58:39.750396+0000","last_unstale":"2024-03-21T13:03:24.279907+0000","last_undegraded":"2024-03-21T13:03:24.279907+0000","last_fullsized":"2024-03-21T13:03:24.279907+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:00:31.008733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:31.008733+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":52,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.278940+0000","last_change":"2024-03-21T13:02:14.336859+0000","last_active":"2024-03-21T13:03:24.278940+0000","last_peered":"2024-03-21T13:03:24.278940+0000","last_clean":"2024-03-21T13:03:24.278940+0000","last_became_active":"2024-03-21T13:02:14.334221+0000","last_became_peered":"2024-03-21T13:02:14.334221+0000","last_unstale":"2024-03-21T13:03:24.278940+0000","last_undegraded":"2024-03-21T13:03:24.278940+0000","last_fullsized":"2024-03-21T13:03:24.278940+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:27.137810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:27.137810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":38,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.283351+0000","last_change":"2024-03-21T13:02:26.237088+0000","last_active":"2024-03-21T13:03:24.283351+0000","last_peered":"2024-03-21T13:03:24.283351+0000","last_clean":"2024-03-21T13:03:24.283351+0000","last_became_active":"2024-03-21T13:02:26.236707+0000","last_became_peered":"2024-03-21T13:02:26.236707+0000","last_unstale":"2024-03-21T13:03:24.283351+0000","last_undegraded":"2024-03-21T13:03:24.283351+0000","last_fullsized":"2024-03-21T13:03:24.283351+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:01:42.076564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:28.336438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":38,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.275277+0000","last_change":"2024-03-21T13:02:26.245694+0000","last_active":"2024-03-21T13:03:24.275277+0000","last_peered":"2024-03-21T13:03:24.275277+0000","last_clean":"2024-03-21T13:03:24.275277+0000","last_became_active":"2024-03-21T13:02:26.245402+0000","last_became_peered":"2024-03-21T13:02:26.245402+0000","last_unstale":"2024-03-21T13:03:24.275277+0000","last_undegraded":"2024-03-21T13:03:24.275277+0000","last_fullsized":"2024-03-21T13:03:24.275277+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:22.892585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:22.892585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":155,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.288260+0000","last_change":"2024-03-21T13:01:27.370969+0000","last_active":"2024-03-21T13:03:24.288260+0000","last_peered":"2024-03-21T13:03:24.288260+0000","last_clean":"2024-03-21T13:03:24.288260+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:03:24.288260+0000","last_undegraded":"2024-03-21T13:03:24.288260+0000","last_fullsized":"2024-03-21T13:03:24.288260+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.880161+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279986+0000","last_change":"2024-03-21T13:02:26.229621+0000","last_active":"2024-03-21T13:03:24.279986+0000","last_peered":"2024-03-21T13:03:24.279986+0000","last_clean":"2024-03-21T13:03:24.279986+0000","last_became_active":"2024-03-21T13:02:26.229397+0000","last_became_peered":"2024-03-21T13:02:26.229397+0000","last_unstale":"2024-03-21T13:03:24.279986+0000","last_undegraded":"2024-03-21T13:03:24.279986+0000","last_fullsized":"2024-03-21T13:03:24.279986+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:32.971665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:32.971665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":169,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279373+0000","last_change":"2024-03-21T13:02:14.336087+0000","last_active":"2024-03-21T13:03:24.279373+0000","last_peered":"2024-03-21T13:03:24.279373+0000","last_clean":"2024-03-21T13:03:24.279373+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:03:24.279373+0000","last_undegraded":"2024-03-21T13:03:24.279373+0000","last_fullsized":"2024-03-21T13:03:24.279373+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:17.015065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:17.015065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266278+0000","last_change":"2024-03-21T13:02:26.230571+0000","last_active":"2024-03-21T13:03:24.266278+0000","last_peered":"2024-03-21T13:03:24.266278+0000","last_clean":"2024-03-21T13:03:24.266278+0000","last_became_active":"2024-03-21T13:02:26.230057+0000","last_became_peered":"2024-03-21T13:02:26.230057+0000","last_unstale":"2024-03-21T13:03:24.266278+0000","last_undegraded":"2024-03-21T13:03:24.266278+0000","last_fullsized":"2024-03-21T13:03:24.266278+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:14.869934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:14.869934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":157,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.280346+0000","last_change":"2024-03-21T13:02:14.383093+0000","last_active":"2024-03-21T13:03:24.280346+0000","last_peered":"2024-03-21T13:03:24.280346+0000","last_clean":"2024-03-21T13:03:24.280346+0000","last_became_active":"2024-03-21T13:02:14.349072+0000","last_became_peered":"2024-03-21T13:02:14.349072+0000","last_unstale":"2024-03-21T13:03:24.280346+0000","last_undegraded":"2024-03-21T13:03:24.280346+0000","last_fullsized":"2024-03-21T13:03:24.280346+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T12:59:53.952401+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":168,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266683+0000","last_change":"2024-03-21T13:02:26.227096+0000","last_active":"2024-03-21T13:03:24.266683+0000","last_peered":"2024-03-21T13:03:24.266683+0000","last_clean":"2024-03-21T13:03:24.266683+0000","last_became_active":"2024-03-21T13:02:26.226675+0000","last_became_peered":"2024-03-21T13:02:26.226675+0000","last_unstale":"2024-03-21T13:03:24.266683+0000","last_undegraded":"2024-03-21T13:03:24.266683+0000","last_fullsized":"2024-03-21T13:03:24.266683+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:24.662938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:24.662938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":56,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266918+0000","last_change":"2024-03-21T13:02:26.232199+0000","last_active":"2024-03-21T13:03:24.266918+0000","last_peered":"2024-03-21T13:03:24.266918+0000","last_clean":"2024-03-21T13:03:24.266918+0000","last_became_active":"2024-03-21T13:02:26.231879+0000","last_became_peered":"2024-03-21T13:02:26.231879+0000","last_unstale":"2024-03-21T13:03:24.266918+0000","last_undegraded":"2024-03-21T13:03:24.266918+0000","last_fullsized":"2024-03-21T13:03:24.266918+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:18.990434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.990434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":64,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279652+0000","last_change":"2024-03-21T13:01:48.210250+0000","last_active":"2024-03-21T13:03:24.279652+0000","last_peered":"2024-03-21T13:03:24.279652+0000","last_clean":"2024-03-21T13:03:24.279652+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:03:24.279652+0000","last_undegraded":"2024-03-21T13:03:24.279652+0000","last_fullsized":"2024-03-21T13:03:24.279652+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:44.005379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:44.005379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":217,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.269163+0000","last_change":"2024-03-21T13:02:14.385705+0000","last_active":"2024-03-21T13:03:24.269163+0000","last_peered":"2024-03-21T13:03:24.269163+0000","last_clean":"2024-03-21T13:03:24.269163+0000","last_became_active":"2024-03-21T13:02:14.350563+0000","last_became_peered":"2024-03-21T13:02:14.350563+0000","last_unstale":"2024-03-21T13:03:24.269163+0000","last_undegraded":"2024-03-21T13:03:24.269163+0000","last_fullsized":"2024-03-21T13:03:24.269163+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:01:36.603903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:36.603903+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":160,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.283728+0000","last_change":"2024-03-21T13:02:26.243438+0000","last_active":"2024-03-21T13:03:24.283728+0000","last_peered":"2024-03-21T13:03:24.283728+0000","last_clean":"2024-03-21T13:03:24.283728+0000","last_became_active":"2024-03-21T13:02:26.242482+0000","last_became_peered":"2024-03-21T13:02:26.242482+0000","last_unstale":"2024-03-21T13:03:24.283728+0000","last_undegraded":"2024-03-21T13:03:24.283728+0000","last_fullsized":"2024-03-21T13:03:24.283728+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:35.036335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.036335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":78,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279604+0000","last_change":"2024-03-21T13:01:27.367176+0000","last_active":"2024-03-21T13:03:24.279604+0000","last_peered":"2024-03-21T13:03:24.279604+0000","last_clean":"2024-03-21T13:03:24.279604+0000","last_became_active":"2024-03-21T13:01:27.366846+0000","last_became_peered":"2024-03-21T13:01:27.366846+0000","last_unstale":"2024-03-21T13:03:24.279604+0000","last_undegraded":"2024-03-21T13:03:24.279604+0000","last_fullsized":"2024-03-21T13:03:24.279604+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.904419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.904419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":179,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266284+0000","last_change":"2024-03-21T13:02:14.329782+0000","last_active":"2024-03-21T13:03:24.266284+0000","last_peered":"2024-03-21T13:03:24.266284+0000","last_clean":"2024-03-21T13:03:24.266284+0000","last_became_active":"2024-03-21T13:02:14.329156+0000","last_became_peered":"2024-03-21T13:02:14.329156+0000","last_unstale":"2024-03-21T13:03:24.266284+0000","last_undegraded":"2024-03-21T13:03:24.266284+0000","last_fullsized":"2024-03-21T13:03:24.266284+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:27.704194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_clean_scrub_stamp":"2024-03-21T13:01:27.704194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'12","reported_seq":220,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.284273+0000","last_change":"2024-03-21T13:02:14.362313+0000","last_active":"2024-03-21T13:03:24.284273+0000","last_peered":"2024-03-21T13:03:24.284273+0000","last_clean":"2024-03-21T13:03:24.284273+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:03:24.284273+0000","last_undegraded":"2024-03-21T13:03:24.284273+0000","last_fullsized":"2024-03-21T13:03:24.284273+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:01:39.755345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.755345+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279711+0000","last_change":"2024-03-21T13:02:26.254057+0000","last_active":"2024-03-21T13:03:24.279711+0000","last_peered":"2024-03-21T13:03:24.279711+0000","last_clean":"2024-03-21T13:03:24.279711+0000","last_became_active":"2024-03-21T13:02:26.237656+0000","last_became_peered":"2024-03-21T13:02:26.237656+0000","last_unstale":"2024-03-21T13:03:24.279711+0000","last_undegraded":"2024-03-21T13:03:24.279711+0000","last_fullsized":"2024-03-21T13:03:24.279711+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:01:22.076398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:22.076398+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":235,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.275211+0000","last_change":"2024-03-21T13:01:30.587787+0000","last_active":"2024-03-21T13:03:24.275211+0000","last_peered":"2024-03-21T13:03:24.275211+0000","last_clean":"2024-03-21T13:03:24.275211+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:03:24.275211+0000","last_undegraded":"2024-03-21T13:03:24.275211+0000","last_fullsized":"2024-03-21T13:03:24.275211+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:30.587660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:30.587660+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":62,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279519+0000","last_change":"2024-03-21T13:02:14.372498+0000","last_active":"2024-03-21T13:03:24.279519+0000","last_peered":"2024-03-21T13:03:24.279519+0000","last_clean":"2024-03-21T13:03:24.279519+0000","last_became_active":"2024-03-21T13:02:14.348666+0000","last_became_peered":"2024-03-21T13:02:14.348666+0000","last_unstale":"2024-03-21T13:03:24.279519+0000","last_undegraded":"2024-03-21T13:03:24.279519+0000","last_fullsized":"2024-03-21T13:03:24.279519+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:01:37.244557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:37.244557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"71'10","reported_seq":98,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.280028+0000","last_change":"2024-03-21T13:01:48.218829+0000","last_active":"2024-03-21T13:03:24.280028+0000","last_peered":"2024-03-21T13:03:24.280028+0000","last_clean":"2024-03-21T13:03:24.280028+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:03:24.280028+0000","last_undegraded":"2024-03-21T13:03:24.280028+0000","last_fullsized":"2024-03-21T13:03:24.280028+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:01:17.088732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:17.088732+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":40,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266343+0000","last_change":"2024-03-21T13:02:26.230240+0000","last_active":"2024-03-21T13:03:24.266343+0000","last_peered":"2024-03-21T13:03:24.266343+0000","last_clean":"2024-03-21T13:03:24.266343+0000","last_became_active":"2024-03-21T13:02:26.229188+0000","last_became_peered":"2024-03-21T13:02:26.229188+0000","last_unstale":"2024-03-21T13:03:24.266343+0000","last_undegraded":"2024-03-21T13:03:24.266343+0000","last_fullsized":"2024-03-21T13:03:24.266343+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:33.251265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:33.251265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":64,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279863+0000","last_change":"2024-03-21T13:01:48.218869+0000","last_active":"2024-03-21T13:03:24.279863+0000","last_peered":"2024-03-21T13:03:24.279863+0000","last_clean":"2024-03-21T13:03:24.279863+0000","last_became_active":"2024-03-21T13:01:48.215160+0000","last_became_peered":"2024-03-21T13:01:48.215160+0000","last_unstale":"2024-03-21T13:03:24.279863+0000","last_undegraded":"2024-03-21T13:03:24.279863+0000","last_fullsized":"2024-03-21T13:03:24.279863+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:39.968654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.968654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":171,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.287632+0000","last_change":"2024-03-21T13:02:14.344996+0000","last_active":"2024-03-21T13:03:24.287632+0000","last_peered":"2024-03-21T13:03:24.287632+0000","last_clean":"2024-03-21T13:03:24.287632+0000","last_became_active":"2024-03-21T13:02:14.341286+0000","last_became_peered":"2024-03-21T13:02:14.341286+0000","last_unstale":"2024-03-21T13:03:24.287632+0000","last_undegraded":"2024-03-21T13:03:24.287632+0000","last_fullsized":"2024-03-21T13:03:24.287632+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:49.178785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:49.178785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'11","reported_seq":225,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.275261+0000","last_change":"2024-03-21T13:02:26.277298+0000","last_active":"2024-03-21T13:03:24.275261+0000","last_peered":"2024-03-21T13:03:24.275261+0000","last_clean":"2024-03-21T13:03:24.275261+0000","last_became_active":"2024-03-21T13:02:26.254436+0000","last_became_peered":"2024-03-21T13:02:26.254436+0000","last_unstale":"2024-03-21T13:03:24.275261+0000","last_undegraded":"2024-03-21T13:03:24.275261+0000","last_fullsized":"2024-03-21T13:03:24.275261+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:01:48.562468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:48.562468+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":160,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279508+0000","last_change":"2024-03-21T13:01:35.005181+0000","last_active":"2024-03-21T13:03:24.279508+0000","last_peered":"2024-03-21T13:03:24.279508+0000","last_clean":"2024-03-21T13:03:24.279508+0000","last_became_active":"2024-03-21T12:58:43.183475+0000","last_became_peered":"2024-03-21T12:58:43.183475+0000","last_unstale":"2024-03-21T13:03:24.279508+0000","last_undegraded":"2024-03-21T13:03:24.279508+0000","last_fullsized":"2024-03-21T13:03:24.279508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:01:35.005026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.274707+0000","last_change":"2024-03-21T13:01:20.591831+0000","last_active":"2024-03-21T13:03:24.274707+0000","last_peered":"2024-03-21T13:03:24.274707+0000","last_clean":"2024-03-21T13:03:24.274707+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:03:24.274707+0000","last_undegraded":"2024-03-21T13:03:24.274707+0000","last_fullsized":"2024-03-21T13:03:24.274707+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.591636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.591636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:29.607861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":40,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.283597+0000","last_change":"2024-03-21T13:02:26.243710+0000","last_active":"2024-03-21T13:03:24.283597+0000","last_peered":"2024-03-21T13:03:24.283597+0000","last_clean":"2024-03-21T13:03:24.283597+0000","last_became_active":"2024-03-21T13:02:26.242973+0000","last_became_peered":"2024-03-21T13:02:26.242973+0000","last_unstale":"2024-03-21T13:03:24.283597+0000","last_undegraded":"2024-03-21T13:03:24.283597+0000","last_fullsized":"2024-03-21T13:03:24.283597+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:32.789993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.789993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":180,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.287717+0000","last_change":"2024-03-21T13:02:14.369668+0000","last_active":"2024-03-21T13:03:24.287717+0000","last_peered":"2024-03-21T13:03:24.287717+0000","last_clean":"2024-03-21T13:03:24.287717+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:03:24.287717+0000","last_undegraded":"2024-03-21T13:03:24.287717+0000","last_fullsized":"2024-03-21T13:03:24.287717+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-21T13:01:30.807979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:30.807979+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":153,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.280878+0000","last_change":"2024-03-21T13:00:10.110413+0000","last_active":"2024-03-21T13:03:24.280878+0000","last_peered":"2024-03-21T13:03:24.280878+0000","last_clean":"2024-03-21T13:03:24.280878+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:03:24.280878+0000","last_undegraded":"2024-03-21T13:03:24.280878+0000","last_fullsized":"2024-03-21T13:03:24.280878+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:10.110335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.110335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":156,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279446+0000","last_change":"2024-03-21T13:00:18.163092+0000","last_active":"2024-03-21T13:03:24.279446+0000","last_peered":"2024-03-21T13:03:24.279446+0000","last_clean":"2024-03-21T13:03:24.279446+0000","last_became_active":"2024-03-21T12:58:41.164784+0000","last_became_peered":"2024-03-21T12:58:41.164784+0000","last_unstale":"2024-03-21T13:03:24.279446+0000","last_undegraded":"2024-03-21T13:03:24.279446+0000","last_fullsized":"2024-03-21T13:03:24.279446+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:18.162988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:18.162988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279769+0000","last_change":"2024-03-21T13:02:26.236549+0000","last_active":"2024-03-21T13:03:24.279769+0000","last_peered":"2024-03-21T13:03:24.279769+0000","last_clean":"2024-03-21T13:03:24.279769+0000","last_became_active":"2024-03-21T13:02:26.236235+0000","last_became_peered":"2024-03-21T13:02:26.236235+0000","last_unstale":"2024-03-21T13:03:24.279769+0000","last_undegraded":"2024-03-21T13:03:24.279769+0000","last_fullsized":"2024-03-21T13:03:24.279769+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:02:21.087308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:02:21.087308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:24.336763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"71'11","reported_seq":181,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279820+0000","last_change":"2024-03-21T12:59:52.181828+0000","last_active":"2024-03-21T13:03:24.279820+0000","last_peered":"2024-03-21T13:03:24.279820+0000","last_clean":"2024-03-21T13:03:24.279820+0000","last_became_active":"2024-03-21T12:58:39.753014+0000","last_became_peered":"2024-03-21T12:58:39.753014+0000","last_unstale":"2024-03-21T13:03:24.279820+0000","last_undegraded":"2024-03-21T13:03:24.279820+0000","last_fullsized":"2024-03-21T13:03:24.279820+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T12:59:52.181697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.181697+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":61,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.274782+0000","last_change":"2024-03-21T13:02:26.253063+0000","last_active":"2024-03-21T13:03:24.274782+0000","last_peered":"2024-03-21T13:03:24.274782+0000","last_clean":"2024-03-21T13:03:24.274782+0000","last_became_active":"2024-03-21T13:02:26.252488+0000","last_became_peered":"2024-03-21T13:02:26.252488+0000","last_unstale":"2024-03-21T13:03:24.274782+0000","last_undegraded":"2024-03-21T13:03:24.274782+0000","last_fullsized":"2024-03-21T13:03:24.274782+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:22.083890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:22.083890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.280559+0000","last_change":"2024-03-21T13:02:26.233084+0000","last_active":"2024-03-21T13:03:24.280559+0000","last_peered":"2024-03-21T13:03:24.280559+0000","last_clean":"2024-03-21T13:03:24.280559+0000","last_became_active":"2024-03-21T13:02:26.232682+0000","last_became_peered":"2024-03-21T13:02:26.232682+0000","last_unstale":"2024-03-21T13:03:24.280559+0000","last_undegraded":"2024-03-21T13:03:24.280559+0000","last_fullsized":"2024-03-21T13:03:24.280559+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:38.995215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.995215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":171,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.283133+0000","last_change":"2024-03-21T13:01:43.982673+0000","last_active":"2024-03-21T13:03:24.283133+0000","last_peered":"2024-03-21T13:03:24.283133+0000","last_clean":"2024-03-21T13:03:24.283133+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:03:24.283133+0000","last_undegraded":"2024-03-21T13:03:24.283133+0000","last_fullsized":"2024-03-21T13:03:24.283133+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:43.982553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:43.982553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"86'13","reported_seq":129,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.275055+0000","last_change":"2024-03-21T13:02:14.351413+0000","last_active":"2024-03-21T13:03:24.275055+0000","last_peered":"2024-03-21T13:03:24.275055+0000","last_clean":"2024-03-21T13:03:24.275055+0000","last_became_active":"2024-03-21T13:02:14.330196+0000","last_became_peered":"2024-03-21T13:02:14.330196+0000","last_unstale":"2024-03-21T13:03:24.275055+0000","last_undegraded":"2024-03-21T13:03:24.275055+0000","last_fullsized":"2024-03-21T13:03:24.275055+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:01:14.027717+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:01:14.027717+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":160,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279407+0000","last_change":"2024-03-21T13:02:26.229754+0000","last_active":"2024-03-21T13:03:24.279407+0000","last_peered":"2024-03-21T13:03:24.279407+0000","last_clean":"2024-03-21T13:03:24.279407+0000","last_became_active":"2024-03-21T13:02:26.229425+0000","last_became_peered":"2024-03-21T13:02:26.229425+0000","last_unstale":"2024-03-21T13:03:24.279407+0000","last_undegraded":"2024-03-21T13:03:24.279407+0000","last_fullsized":"2024-03-21T13:03:24.279407+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:50.729022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:50.729022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":40,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.288355+0000","last_change":"2024-03-21T13:02:26.242867+0000","last_active":"2024-03-21T13:03:24.288355+0000","last_peered":"2024-03-21T13:03:24.288355+0000","last_clean":"2024-03-21T13:03:24.288355+0000","last_became_active":"2024-03-21T13:02:26.242491+0000","last_became_peered":"2024-03-21T13:02:26.242491+0000","last_unstale":"2024-03-21T13:03:24.288355+0000","last_undegraded":"2024-03-21T13:03:24.288355+0000","last_fullsized":"2024-03-21T13:03:24.288355+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.029629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.029629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":174,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266366+0000","last_change":"2024-03-21T13:01:20.648033+0000","last_active":"2024-03-21T13:03:24.266366+0000","last_peered":"2024-03-21T13:03:24.266366+0000","last_clean":"2024-03-21T13:03:24.266366+0000","last_became_active":"2024-03-21T12:58:37.731659+0000","last_became_peered":"2024-03-21T12:58:37.731659+0000","last_unstale":"2024-03-21T13:03:24.266366+0000","last_undegraded":"2024-03-21T13:03:24.266366+0000","last_fullsized":"2024-03-21T13:03:24.266366+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.647931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.647931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":217,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.267041+0000","last_change":"2024-03-21T13:02:26.258852+0000","last_active":"2024-03-21T13:03:24.267041+0000","last_peered":"2024-03-21T13:03:24.267041+0000","last_clean":"2024-03-21T13:03:24.267041+0000","last_became_active":"2024-03-21T13:02:26.225627+0000","last_became_peered":"2024-03-21T13:02:26.225627+0000","last_unstale":"2024-03-21T13:03:24.267041+0000","last_undegraded":"2024-03-21T13:03:24.267041+0000","last_fullsized":"2024-03-21T13:03:24.267041+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:01:19.642156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:19.642156+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":64,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279711+0000","last_change":"2024-03-21T13:01:48.215368+0000","last_active":"2024-03-21T13:03:24.279711+0000","last_peered":"2024-03-21T13:03:24.279711+0000","last_clean":"2024-03-21T13:03:24.279711+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:03:24.279711+0000","last_undegraded":"2024-03-21T13:03:24.279711+0000","last_fullsized":"2024-03-21T13:03:24.279711+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.062842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.062842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"125'10","reported_seq":233,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.283909+0000","last_change":"2024-03-21T13:02:26.247826+0000","last_active":"2024-03-21T13:03:24.283909+0000","last_peered":"2024-03-21T13:03:24.283909+0000","last_clean":"2024-03-21T13:03:24.283909+0000","last_became_active":"2024-03-21T13:02:26.234494+0000","last_became_peered":"2024-03-21T13:02:26.234494+0000","last_unstale":"2024-03-21T13:03:24.283909+0000","last_undegraded":"2024-03-21T13:03:24.283909+0000","last_fullsized":"2024-03-21T13:03:24.283909+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:23.867564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:23.867564+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":172,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279314+0000","last_change":"2024-03-21T13:01:45.972893+0000","last_active":"2024-03-21T13:03:24.279314+0000","last_peered":"2024-03-21T13:03:24.279314+0000","last_clean":"2024-03-21T13:03:24.279314+0000","last_became_active":"2024-03-21T12:58:37.731988+0000","last_became_peered":"2024-03-21T12:58:37.731988+0000","last_unstale":"2024-03-21T13:03:24.279314+0000","last_undegraded":"2024-03-21T13:03:24.279314+0000","last_fullsized":"2024-03-21T13:03:24.279314+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:45.972809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:45.972809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"71'10","reported_seq":195,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.280175+0000","last_change":"2024-03-21T13:02:14.389576+0000","last_active":"2024-03-21T13:03:24.280175+0000","last_peered":"2024-03-21T13:03:24.280175+0000","last_clean":"2024-03-21T13:03:24.280175+0000","last_became_active":"2024-03-21T13:02:14.353938+0000","last_became_peered":"2024-03-21T13:02:14.353938+0000","last_unstale":"2024-03-21T13:03:24.280175+0000","last_undegraded":"2024-03-21T13:03:24.280175+0000","last_fullsized":"2024-03-21T13:03:24.280175+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:00:06.197903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.197903+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":167,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266983+0000","last_change":"2024-03-21T13:02:26.233510+0000","last_active":"2024-03-21T13:03:24.266983+0000","last_peered":"2024-03-21T13:03:24.266983+0000","last_clean":"2024-03-21T13:03:24.266983+0000","last_became_active":"2024-03-21T13:02:26.233049+0000","last_became_peered":"2024-03-21T13:02:26.233049+0000","last_unstale":"2024-03-21T13:03:24.266983+0000","last_undegraded":"2024-03-21T13:03:24.266983+0000","last_fullsized":"2024-03-21T13:03:24.266983+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:01:36.699469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":224,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.280258+0000","last_change":"2024-03-21T13:01:18.526684+0000","last_active":"2024-03-21T13:03:24.280258+0000","last_peered":"2024-03-21T13:03:24.280258+0000","last_clean":"2024-03-21T13:03:24.280258+0000","last_became_active":"2024-03-21T12:58:41.167826+0000","last_became_peered":"2024-03-21T12:58:41.167826+0000","last_unstale":"2024-03-21T13:03:24.280258+0000","last_undegraded":"2024-03-21T13:03:24.280258+0000","last_fullsized":"2024-03-21T13:03:24.280258+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:18.526428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.526428+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":64,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279261+0000","last_change":"2024-03-21T13:01:48.212782+0000","last_active":"2024-03-21T13:03:24.279261+0000","last_peered":"2024-03-21T13:03:24.279261+0000","last_clean":"2024-03-21T13:03:24.279261+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:03:24.279261+0000","last_undegraded":"2024-03-21T13:03:24.279261+0000","last_fullsized":"2024-03-21T13:03:24.279261+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:35.801638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.801638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"71'9","reported_seq":83,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.283233+0000","last_change":"2024-03-21T13:02:26.266952+0000","last_active":"2024-03-21T13:03:24.283233+0000","last_peered":"2024-03-21T13:03:24.283233+0000","last_clean":"2024-03-21T13:03:24.283233+0000","last_became_active":"2024-03-21T13:02:26.239161+0000","last_became_peered":"2024-03-21T13:02:26.239161+0000","last_unstale":"2024-03-21T13:03:24.283233+0000","last_undegraded":"2024-03-21T13:03:24.283233+0000","last_fullsized":"2024-03-21T13:03:24.283233+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:00:19.686939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:19.686939+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.269569+0000","last_change":"2024-03-21T13:01:34.642737+0000","last_active":"2024-03-21T13:03:24.269569+0000","last_peered":"2024-03-21T13:03:24.269569+0000","last_clean":"2024-03-21T13:03:24.269569+0000","last_became_active":"2024-03-21T12:58:43.182126+0000","last_became_peered":"2024-03-21T12:58:43.182126+0000","last_unstale":"2024-03-21T13:03:24.269569+0000","last_undegraded":"2024-03-21T13:03:24.269569+0000","last_fullsized":"2024-03-21T13:03:24.269569+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:34.642613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:01:34.642613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":163,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.284236+0000","last_change":"2024-03-21T13:02:14.354537+0000","last_active":"2024-03-21T13:03:24.284236+0000","last_peered":"2024-03-21T13:03:24.284236+0000","last_clean":"2024-03-21T13:03:24.284236+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:03:24.284236+0000","last_undegraded":"2024-03-21T13:03:24.284236+0000","last_fullsized":"2024-03-21T13:03:24.284236+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:12.121970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:12.121970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":178,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279185+0000","last_change":"2024-03-21T13:01:27.075743+0000","last_active":"2024-03-21T13:03:24.279185+0000","last_peered":"2024-03-21T13:03:24.279185+0000","last_clean":"2024-03-21T13:03:24.279185+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:03:24.279185+0000","last_undegraded":"2024-03-21T13:03:24.279185+0000","last_fullsized":"2024-03-21T13:03:24.279185+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:01:27.074936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:27.074936+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'9","reported_seq":99,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.288240+0000","last_change":"2024-03-21T13:01:48.210983+0000","last_active":"2024-03-21T13:03:24.288240+0000","last_peered":"2024-03-21T13:03:24.288240+0000","last_clean":"2024-03-21T13:03:24.288240+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:03:24.288240+0000","last_undegraded":"2024-03-21T13:03:24.288240+0000","last_fullsized":"2024-03-21T13:03:24.288240+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:01:40.964615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.964615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":151,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.282994+0000","last_change":"2024-03-21T13:00:08.197105+0000","last_active":"2024-03-21T13:03:24.282994+0000","last_peered":"2024-03-21T13:03:24.282994+0000","last_clean":"2024-03-21T13:03:24.282994+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:03:24.282994+0000","last_undegraded":"2024-03-21T13:03:24.282994+0000","last_fullsized":"2024-03-21T13:03:24.282994+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.196979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.196979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.286811+0000","last_change":"2024-03-21T13:02:26.243050+0000","last_active":"2024-03-21T13:03:24.286811+0000","last_peered":"2024-03-21T13:03:24.286811+0000","last_clean":"2024-03-21T13:03:24.286811+0000","last_became_active":"2024-03-21T13:02:26.242502+0000","last_became_peered":"2024-03-21T13:02:26.242502+0000","last_unstale":"2024-03-21T13:03:24.286811+0000","last_undegraded":"2024-03-21T13:03:24.286811+0000","last_fullsized":"2024-03-21T13:03:24.286811+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:32.639920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.639920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":180,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.275101+0000","last_change":"2024-03-21T13:02:26.252758+0000","last_active":"2024-03-21T13:03:24.275101+0000","last_peered":"2024-03-21T13:03:24.275101+0000","last_clean":"2024-03-21T13:03:24.275101+0000","last_became_active":"2024-03-21T13:02:26.252406+0000","last_became_peered":"2024-03-21T13:02:26.252406+0000","last_unstale":"2024-03-21T13:03:24.275101+0000","last_undegraded":"2024-03-21T13:03:24.275101+0000","last_fullsized":"2024-03-21T13:03:24.275101+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:41.621550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:41.621550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'6","reported_seq":184,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.284055+0000","last_change":"2024-03-21T13:02:14.384317+0000","last_active":"2024-03-21T13:03:24.284055+0000","last_peered":"2024-03-21T13:03:24.284055+0000","last_clean":"2024-03-21T13:03:24.284055+0000","last_became_active":"2024-03-21T13:02:14.352117+0000","last_became_peered":"2024-03-21T13:02:14.352117+0000","last_unstale":"2024-03-21T13:03:24.284055+0000","last_undegraded":"2024-03-21T13:03:24.284055+0000","last_fullsized":"2024-03-21T13:03:24.284055+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T12:59:46.176819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.176819+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"137'10907","reported_seq":27845,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.325777+0000","last_change":"2024-03-21T13:03:16.986397+0000","last_active":"2024-03-21T13:03:24.325777+0000","last_peered":"2024-03-21T13:03:24.325777+0000","last_clean":"2024-03-21T13:03:24.325777+0000","last_became_active":"2024-03-21T13:03:16.964637+0000","last_became_peered":"2024-03-21T13:03:16.964637+0000","last_unstale":"2024-03-21T13:03:24.325777+0000","last_undegraded":"2024-03-21T13:03:24.325777+0000","last_fullsized":"2024-03-21T13:03:24.325777+0000","mapping_epoch":135,"log_start":"119'6723","ondisk_log_start":"119'6723","created":92,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4184,"log_dups_size":0,"ondisk_log_size":4184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243204096,"num_objects":3711,"num_object_clones":0,"num_object_copies":11133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10907,"num_write_kb":467776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5606,"num_bytes_recovered":367394816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":38,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.274507+0000","last_change":"2024-03-21T13:02:26.253217+0000","last_active":"2024-03-21T13:03:24.274507+0000","last_peered":"2024-03-21T13:03:24.274507+0000","last_clean":"2024-03-21T13:03:24.274507+0000","last_became_active":"2024-03-21T13:02:26.252904+0000","last_became_peered":"2024-03-21T13:02:26.252904+0000","last_unstale":"2024-03-21T13:03:24.274507+0000","last_undegraded":"2024-03-21T13:03:24.274507+0000","last_fullsized":"2024-03-21T13:03:24.274507+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:01:43.075256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":184,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.280325+0000","last_change":"2024-03-21T13:01:39.975587+0000","last_active":"2024-03-21T13:03:24.280325+0000","last_peered":"2024-03-21T13:03:24.280325+0000","last_clean":"2024-03-21T13:03:24.280325+0000","last_became_active":"2024-03-21T12:58:37.735037+0000","last_became_peered":"2024-03-21T12:58:37.735037+0000","last_unstale":"2024-03-21T13:03:24.280325+0000","last_undegraded":"2024-03-21T13:03:24.280325+0000","last_fullsized":"2024-03-21T13:03:24.280325+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:01:39.975456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.975456+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'15","reported_seq":98,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279218+0000","last_change":"2024-03-21T13:02:26.260469+0000","last_active":"2024-03-21T13:03:24.279218+0000","last_peered":"2024-03-21T13:03:24.279218+0000","last_clean":"2024-03-21T13:03:24.279218+0000","last_became_active":"2024-03-21T13:02:26.237800+0000","last_became_peered":"2024-03-21T13:02:26.237800+0000","last_unstale":"2024-03-21T13:03:24.279218+0000","last_undegraded":"2024-03-21T13:03:24.279218+0000","last_fullsized":"2024-03-21T13:03:24.279218+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:00:27.640392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:27.640392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":223,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279543+0000","last_change":"2024-03-21T13:00:20.176757+0000","last_active":"2024-03-21T13:03:24.279543+0000","last_peered":"2024-03-21T13:03:24.279543+0000","last_clean":"2024-03-21T13:03:24.279543+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:03:24.279543+0000","last_undegraded":"2024-03-21T13:03:24.279543+0000","last_fullsized":"2024-03-21T13:03:24.279543+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.176653+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"137'10839","reported_seq":17525,"reported_epoch":137,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:03:24.986642+0000","last_change":"2024-03-21T13:02:57.325811+0000","last_active":"2024-03-21T13:03:24.986642+0000","last_peered":"2024-03-21T13:03:24.986642+0000","last_clean":"2024-03-21T13:02:25.722555+0000","last_became_active":"2024-03-21T13:02:27.294304+0000","last_became_peered":"2024-03-21T13:02:27.294304+0000","last_unstale":"2024-03-21T13:03:24.986642+0000","last_undegraded":"2024-03-21T13:03:24.986642+0000","last_fullsized":"2024-03-21T13:03:24.986642+0000","mapping_epoch":124,"log_start":"122'7229","ondisk_log_start":"122'7229","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3610,"log_dups_size":0,"ondisk_log_size":3610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241762304,"num_objects":3689,"num_object_clones":0,"num_object_copies":11067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":902,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10839,"num_write_kb":464896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3287,"num_bytes_recovered":215416832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,5],"avail_no_missing":["3","2","5"],"object_location_counts":[{"shards":"2,3,5","objects":3689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":160,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.280691+0000","last_change":"2024-03-21T13:02:14.352120+0000","last_active":"2024-03-21T13:03:24.280691+0000","last_peered":"2024-03-21T13:03:24.280691+0000","last_clean":"2024-03-21T13:03:24.280691+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:03:24.280691+0000","last_undegraded":"2024-03-21T13:03:24.280691+0000","last_fullsized":"2024-03-21T13:03:24.280691+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:24.988792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:24.988792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":178,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279845+0000","last_change":"2024-03-21T13:02:14.337089+0000","last_active":"2024-03-21T13:03:24.279845+0000","last_peered":"2024-03-21T13:03:24.279845+0000","last_clean":"2024-03-21T13:03:24.279845+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:03:24.279845+0000","last_undegraded":"2024-03-21T13:03:24.279845+0000","last_fullsized":"2024-03-21T13:03:24.279845+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.030821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.030821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"71'12","reported_seq":104,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279678+0000","last_change":"2024-03-21T13:02:14.383361+0000","last_active":"2024-03-21T13:03:24.279678+0000","last_peered":"2024-03-21T13:03:24.279678+0000","last_clean":"2024-03-21T13:03:24.279678+0000","last_became_active":"2024-03-21T13:02:14.347102+0000","last_became_peered":"2024-03-21T13:02:14.347102+0000","last_unstale":"2024-03-21T13:03:24.279678+0000","last_undegraded":"2024-03-21T13:03:24.279678+0000","last_fullsized":"2024-03-21T13:03:24.279678+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:00:32.864608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.864608+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":179,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.269059+0000","last_change":"2024-03-21T13:02:26.235545+0000","last_active":"2024-03-21T13:03:24.269059+0000","last_peered":"2024-03-21T13:03:24.269059+0000","last_clean":"2024-03-21T13:03:24.269059+0000","last_became_active":"2024-03-21T13:02:26.235083+0000","last_became_peered":"2024-03-21T13:02:26.235083+0000","last_unstale":"2024-03-21T13:03:24.269059+0000","last_undegraded":"2024-03-21T13:03:24.269059+0000","last_fullsized":"2024-03-21T13:03:24.269059+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:40.658883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.658883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"137'10996","reported_seq":23856,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.347709+0000","last_change":"2024-03-21T13:03:01.305905+0000","last_active":"2024-03-21T13:03:24.347709+0000","last_peered":"2024-03-21T13:03:24.347709+0000","last_clean":"2024-03-21T13:03:24.347709+0000","last_became_active":"2024-03-21T13:03:01.285305+0000","last_became_peered":"2024-03-21T13:03:01.285305+0000","last_unstale":"2024-03-21T13:03:24.347709+0000","last_undegraded":"2024-03-21T13:03:24.347709+0000","last_fullsized":"2024-03-21T13:03:24.347709+0000","mapping_epoch":129,"log_start":"119'6939","ondisk_log_start":"119'6939","created":92,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4057,"log_dups_size":0,"ondisk_log_size":4057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240648192,"num_objects":3672,"num_object_clones":0,"num_object_copies":11016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10996,"num_write_kb":469376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6372,"num_bytes_recovered":417595392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":160,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.278970+0000","last_change":"2024-03-21T13:01:14.114540+0000","last_active":"2024-03-21T13:03:24.278970+0000","last_peered":"2024-03-21T13:03:24.278970+0000","last_clean":"2024-03-21T13:03:24.278970+0000","last_became_active":"2024-03-21T12:58:43.179007+0000","last_became_peered":"2024-03-21T12:58:43.179007+0000","last_unstale":"2024-03-21T13:03:24.278970+0000","last_undegraded":"2024-03-21T13:03:24.278970+0000","last_fullsized":"2024-03-21T13:03:24.278970+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:14.114420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:14.114420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":173,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.277904+0000","last_change":"2024-03-21T13:02:26.238463+0000","last_active":"2024-03-21T13:03:24.277904+0000","last_peered":"2024-03-21T13:03:24.277904+0000","last_clean":"2024-03-21T13:03:24.277904+0000","last_became_active":"2024-03-21T13:02:26.238116+0000","last_became_peered":"2024-03-21T13:02:26.238116+0000","last_unstale":"2024-03-21T13:03:24.277904+0000","last_undegraded":"2024-03-21T13:03:24.277904+0000","last_fullsized":"2024-03-21T13:03:24.277904+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:39.655215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.655215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":178,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.283224+0000","last_change":"2024-03-21T13:02:26.237114+0000","last_active":"2024-03-21T13:03:24.283224+0000","last_peered":"2024-03-21T13:03:24.283224+0000","last_clean":"2024-03-21T13:03:24.283224+0000","last_became_active":"2024-03-21T13:02:26.236780+0000","last_became_peered":"2024-03-21T13:02:26.236780+0000","last_unstale":"2024-03-21T13:03:24.283224+0000","last_undegraded":"2024-03-21T13:03:24.283224+0000","last_fullsized":"2024-03-21T13:03:24.283224+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:18.012391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.012391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"71'19","reported_seq":253,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.274691+0000","last_change":"2024-03-21T13:02:26.275429+0000","last_active":"2024-03-21T13:03:24.274691+0000","last_peered":"2024-03-21T13:03:24.274691+0000","last_clean":"2024-03-21T13:03:24.274691+0000","last_became_active":"2024-03-21T13:02:26.240350+0000","last_became_peered":"2024-03-21T13:02:26.240350+0000","last_unstale":"2024-03-21T13:03:24.274691+0000","last_undegraded":"2024-03-21T13:03:24.274691+0000","last_fullsized":"2024-03-21T13:03:24.274691+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:01:18.623551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.623551+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"137'10976","reported_seq":11158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.308869+0000","last_change":"2024-03-21T13:02:26.586862+0000","last_active":"2024-03-21T13:03:24.308869+0000","last_peered":"2024-03-21T13:03:24.308869+0000","last_clean":"2024-03-21T13:03:24.308869+0000","last_became_active":"2024-03-21T13:00:42.708899+0000","last_became_peered":"2024-03-21T13:00:42.708899+0000","last_unstale":"2024-03-21T13:03:24.308869+0000","last_undegraded":"2024-03-21T13:03:24.308869+0000","last_fullsized":"2024-03-21T13:03:24.308869+0000","mapping_epoch":92,"log_start":"114'6241","ondisk_log_start":"114'6241","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7428","last_scrub_stamp":"2024-03-21T13:02:26.586741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:02:26.586741+0000","objects_scrubbed":7233,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:31.661033+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241172480,"num_objects":3680,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10976,"num_write_kb":468992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":64,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279070+0000","last_change":"2024-03-21T13:01:48.214335+0000","last_active":"2024-03-21T13:03:24.279070+0000","last_peered":"2024-03-21T13:03:24.279070+0000","last_clean":"2024-03-21T13:03:24.279070+0000","last_became_active":"2024-03-21T13:01:48.213995+0000","last_became_peered":"2024-03-21T13:01:48.213995+0000","last_unstale":"2024-03-21T13:03:24.279070+0000","last_undegraded":"2024-03-21T13:03:24.279070+0000","last_fullsized":"2024-03-21T13:03:24.279070+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:44.997946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:44.997946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":167,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.268568+0000","last_change":"2024-03-21T13:01:46.642867+0000","last_active":"2024-03-21T13:03:24.268568+0000","last_peered":"2024-03-21T13:03:24.268568+0000","last_clean":"2024-03-21T13:03:24.268568+0000","last_became_active":"2024-03-21T12:58:41.163843+0000","last_became_peered":"2024-03-21T12:58:41.163843+0000","last_unstale":"2024-03-21T13:03:24.268568+0000","last_undegraded":"2024-03-21T13:03:24.268568+0000","last_fullsized":"2024-03-21T13:03:24.268568+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:46.642749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.642749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":38,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279992+0000","last_change":"2024-03-21T13:02:26.244355+0000","last_active":"2024-03-21T13:03:24.279992+0000","last_peered":"2024-03-21T13:03:24.279992+0000","last_clean":"2024-03-21T13:03:24.279992+0000","last_became_active":"2024-03-21T13:02:26.243974+0000","last_became_peered":"2024-03-21T13:02:26.243974+0000","last_unstale":"2024-03-21T13:03:24.279992+0000","last_undegraded":"2024-03-21T13:03:24.279992+0000","last_fullsized":"2024-03-21T13:03:24.279992+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:33.998382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:33.998382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"86'13","reported_seq":128,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266665+0000","last_change":"2024-03-21T13:02:14.369004+0000","last_active":"2024-03-21T13:03:24.266665+0000","last_peered":"2024-03-21T13:03:24.266665+0000","last_clean":"2024-03-21T13:03:24.266665+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:03:24.266665+0000","last_undegraded":"2024-03-21T13:03:24.266665+0000","last_fullsized":"2024-03-21T13:03:24.266665+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:01:26.174361+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:01:26.174361+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.7","version":"137'11202","reported_seq":11304,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.326632+0000","last_change":"2024-03-21T13:00:42.707087+0000","last_active":"2024-03-21T13:03:24.326632+0000","last_peered":"2024-03-21T13:03:24.326632+0000","last_clean":"2024-03-21T13:03:24.326632+0000","last_became_active":"2024-03-21T13:00:42.706774+0000","last_became_peered":"2024-03-21T13:00:42.706774+0000","last_unstale":"2024-03-21T13:03:24.326632+0000","last_undegraded":"2024-03-21T13:03:24.326632+0000","last_fullsized":"2024-03-21T13:03:24.326632+0000","mapping_epoch":92,"log_start":"119'6999","ondisk_log_start":"119'6999","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4203,"log_dups_size":0,"ondisk_log_size":4203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244318208,"num_objects":3728,"num_object_clones":0,"num_object_copies":11184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11202,"num_write_kb":477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.286894+0000","last_change":"2024-03-21T13:01:46.738822+0000","last_active":"2024-03-21T13:03:24.286894+0000","last_peered":"2024-03-21T13:03:24.286894+0000","last_clean":"2024-03-21T13:03:24.286894+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:03:24.286894+0000","last_undegraded":"2024-03-21T13:03:24.286894+0000","last_fullsized":"2024-03-21T13:03:24.286894+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.738659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"67'2","reported_seq":68,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.274589+0000","last_change":"2024-03-21T13:02:26.266567+0000","last_active":"2024-03-21T13:03:24.274589+0000","last_peered":"2024-03-21T13:03:24.274589+0000","last_clean":"2024-03-21T13:03:24.274589+0000","last_became_active":"2024-03-21T13:02:26.243692+0000","last_became_peered":"2024-03-21T13:02:26.243692+0000","last_unstale":"2024-03-21T13:03:24.274589+0000","last_undegraded":"2024-03-21T13:03:24.274589+0000","last_fullsized":"2024-03-21T13:03:24.274589+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-21T13:01:41.017193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:01:41.017193+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":174,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266763+0000","last_change":"2024-03-21T13:01:31.721299+0000","last_active":"2024-03-21T13:03:24.266763+0000","last_peered":"2024-03-21T13:03:24.266763+0000","last_clean":"2024-03-21T13:03:24.266763+0000","last_became_active":"2024-03-21T12:58:37.729454+0000","last_became_peered":"2024-03-21T12:58:37.729454+0000","last_unstale":"2024-03-21T13:03:24.266763+0000","last_undegraded":"2024-03-21T13:03:24.266763+0000","last_fullsized":"2024-03-21T13:03:24.266763+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:31.721222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.721222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"119'29","reported_seq":156,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.269419+0000","last_change":"2024-03-21T13:02:14.382134+0000","last_active":"2024-03-21T13:03:24.269419+0000","last_peered":"2024-03-21T13:03:24.269419+0000","last_clean":"2024-03-21T13:03:24.269419+0000","last_became_active":"2024-03-21T13:02:14.351114+0000","last_became_peered":"2024-03-21T13:02:14.351114+0000","last_unstale":"2024-03-21T13:03:24.269419+0000","last_undegraded":"2024-03-21T13:03:24.269419+0000","last_fullsized":"2024-03-21T13:03:24.269419+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'28","last_scrub_stamp":"2024-03-21T13:01:36.231420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:36.231420+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":52,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266499+0000","last_change":"2024-03-21T13:02:14.336654+0000","last_active":"2024-03-21T13:03:24.266499+0000","last_peered":"2024-03-21T13:03:24.266499+0000","last_clean":"2024-03-21T13:03:24.266499+0000","last_became_active":"2024-03-21T13:02:14.336177+0000","last_became_peered":"2024-03-21T13:02:14.336177+0000","last_unstale":"2024-03-21T13:03:24.266499+0000","last_undegraded":"2024-03-21T13:03:24.266499+0000","last_fullsized":"2024-03-21T13:03:24.266499+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:36.760263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:36.760263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.4","version":"137'10992","reported_seq":18071,"reported_epoch":137,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-21T13:03:24.302739+0000","last_change":"2024-03-21T13:02:56.254633+0000","last_active":"2024-03-21T13:03:24.302739+0000","last_peered":"2024-03-21T13:03:24.302739+0000","last_clean":"2024-03-21T13:02:13.309754+0000","last_became_active":"2024-03-21T13:02:15.383776+0000","last_became_peered":"2024-03-21T13:02:15.383776+0000","last_unstale":"2024-03-21T13:03:24.302739+0000","last_undegraded":"2024-03-21T13:03:24.302739+0000","last_fullsized":"2024-03-21T13:03:24.302739+0000","mapping_epoch":118,"log_start":"119'6982","ondisk_log_start":"119'6982","created":92,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4010,"log_dups_size":0,"ondisk_log_size":4010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238813184,"num_objects":3644,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1856,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10992,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3478,"num_bytes_recovered":227934208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,1],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":3644}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":160,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.283275+0000","last_change":"2024-03-21T13:01:28.061976+0000","last_active":"2024-03-21T13:03:24.283275+0000","last_peered":"2024-03-21T13:03:24.283275+0000","last_clean":"2024-03-21T13:03:24.283275+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:03:24.283275+0000","last_undegraded":"2024-03-21T13:03:24.283275+0000","last_fullsized":"2024-03-21T13:03:24.283275+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:28.061868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:28.061868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":58,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.269197+0000","last_change":"2024-03-21T13:02:21.275708+0000","last_active":"2024-03-21T13:03:24.269197+0000","last_peered":"2024-03-21T13:03:24.269197+0000","last_clean":"2024-03-21T13:03:24.269197+0000","last_became_active":"2024-03-21T13:02:14.328020+0000","last_became_peered":"2024-03-21T13:02:14.328020+0000","last_unstale":"2024-03-21T13:03:24.269197+0000","last_undegraded":"2024-03-21T13:03:24.269197+0000","last_fullsized":"2024-03-21T13:03:24.269197+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:02:21.275321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:02:21.275321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:47.786062+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":184,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.268833+0000","last_change":"2024-03-21T13:01:16.871013+0000","last_active":"2024-03-21T13:03:24.268833+0000","last_peered":"2024-03-21T13:03:24.268833+0000","last_clean":"2024-03-21T13:03:24.268833+0000","last_became_active":"2024-03-21T12:58:36.724067+0000","last_became_peered":"2024-03-21T12:58:36.724067+0000","last_unstale":"2024-03-21T13:03:24.268833+0000","last_undegraded":"2024-03-21T13:03:24.268833+0000","last_fullsized":"2024-03-21T13:03:24.268833+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:16.870806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:01:16.870806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0.000743678,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'13","reported_seq":98,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.280108+0000","last_change":"2024-03-21T13:02:26.260934+0000","last_active":"2024-03-21T13:03:24.280108+0000","last_peered":"2024-03-21T13:03:24.280108+0000","last_clean":"2024-03-21T13:03:24.280108+0000","last_became_active":"2024-03-21T13:02:26.234230+0000","last_became_peered":"2024-03-21T13:02:26.234230+0000","last_unstale":"2024-03-21T13:03:24.280108+0000","last_undegraded":"2024-03-21T13:03:24.280108+0000","last_fullsized":"2024-03-21T13:03:24.280108+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'13","last_scrub_stamp":"2024-03-21T13:01:31.190053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.190053+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":173,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.272882+0000","last_change":"2024-03-21T13:02:26.228478+0000","last_active":"2024-03-21T13:03:24.272882+0000","last_peered":"2024-03-21T13:03:24.272882+0000","last_clean":"2024-03-21T13:03:24.272882+0000","last_became_active":"2024-03-21T13:02:26.228128+0000","last_became_peered":"2024-03-21T13:02:26.228128+0000","last_unstale":"2024-03-21T13:03:24.272882+0000","last_undegraded":"2024-03-21T13:03:24.272882+0000","last_fullsized":"2024-03-21T13:03:24.272882+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:25.668960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:25.668960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"137'10961","reported_seq":11066,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.342367+0000","last_change":"2024-03-21T13:01:48.219270+0000","last_active":"2024-03-21T13:03:24.342367+0000","last_peered":"2024-03-21T13:03:24.342367+0000","last_clean":"2024-03-21T13:03:24.342367+0000","last_became_active":"2024-03-21T13:01:48.213755+0000","last_became_peered":"2024-03-21T13:01:48.213755+0000","last_unstale":"2024-03-21T13:03:24.342367+0000","last_undegraded":"2024-03-21T13:03:24.342367+0000","last_fullsized":"2024-03-21T13:03:24.342367+0000","mapping_epoch":110,"log_start":"119'6862","ondisk_log_start":"119'6862","created":92,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239927296,"num_objects":3661,"num_object_clones":0,"num_object_copies":10983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10961,"num_write_kb":467904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":38,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.284149+0000","last_change":"2024-03-21T13:02:26.236072+0000","last_active":"2024-03-21T13:03:24.284149+0000","last_peered":"2024-03-21T13:03:24.284149+0000","last_clean":"2024-03-21T13:03:24.284149+0000","last_became_active":"2024-03-21T13:02:26.235640+0000","last_became_peered":"2024-03-21T13:02:26.235640+0000","last_unstale":"2024-03-21T13:03:24.284149+0000","last_undegraded":"2024-03-21T13:03:24.284149+0000","last_fullsized":"2024-03-21T13:03:24.284149+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:52.658377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.658377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":421,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279084+0000","last_change":"2024-03-21T13:00:16.180260+0000","last_active":"2024-03-21T13:03:24.279084+0000","last_peered":"2024-03-21T13:03:24.279084+0000","last_clean":"2024-03-21T13:03:24.279084+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:03:24.279084+0000","last_undegraded":"2024-03-21T13:03:24.279084+0000","last_fullsized":"2024-03-21T13:03:24.279084+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:00:16.180126+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:00:16.180126+0000","last_clean_scrub_stamp":"2024-03-21T13:00:16.180126+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":172,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.287822+0000","last_change":"2024-03-21T13:02:26.242969+0000","last_active":"2024-03-21T13:03:24.287822+0000","last_peered":"2024-03-21T13:03:24.287822+0000","last_clean":"2024-03-21T13:03:24.287822+0000","last_became_active":"2024-03-21T13:02:26.242456+0000","last_became_peered":"2024-03-21T13:02:26.242456+0000","last_unstale":"2024-03-21T13:03:24.287822+0000","last_undegraded":"2024-03-21T13:03:24.287822+0000","last_fullsized":"2024-03-21T13:03:24.287822+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:38.007486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.007486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":250,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.275378+0000","last_change":"2024-03-21T13:01:21.574857+0000","last_active":"2024-03-21T13:03:24.275378+0000","last_peered":"2024-03-21T13:03:24.275378+0000","last_clean":"2024-03-21T13:03:24.275378+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:03:24.275378+0000","last_undegraded":"2024-03-21T13:03:24.275378+0000","last_fullsized":"2024-03-21T13:03:24.275378+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-21T13:01:21.574712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:21.574712+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279733+0000","last_change":"2024-03-21T13:02:26.235494+0000","last_active":"2024-03-21T13:03:24.279733+0000","last_peered":"2024-03-21T13:03:24.279733+0000","last_clean":"2024-03-21T13:03:24.279733+0000","last_became_active":"2024-03-21T13:02:26.235140+0000","last_became_peered":"2024-03-21T13:02:26.235140+0000","last_unstale":"2024-03-21T13:03:24.279733+0000","last_undegraded":"2024-03-21T13:03:24.279733+0000","last_fullsized":"2024-03-21T13:03:24.279733+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.968175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"137'11037","reported_seq":11139,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.297082+0000","last_change":"2024-03-21T13:00:42.710083+0000","last_active":"2024-03-21T13:03:24.297082+0000","last_peered":"2024-03-21T13:03:24.297082+0000","last_clean":"2024-03-21T13:03:24.297082+0000","last_became_active":"2024-03-21T13:00:42.709753+0000","last_became_peered":"2024-03-21T13:00:42.709753+0000","last_unstale":"2024-03-21T13:03:24.297082+0000","last_undegraded":"2024-03-21T13:03:24.297082+0000","last_fullsized":"2024-03-21T13:03:24.297082+0000","mapping_epoch":92,"log_start":"119'6920","ondisk_log_start":"119'6920","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4117,"log_dups_size":0,"ondisk_log_size":4117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244383744,"num_objects":3729,"num_object_clones":0,"num_object_copies":11187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11037,"num_write_kb":472512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":40,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279420+0000","last_change":"2024-03-21T13:02:26.243121+0000","last_active":"2024-03-21T13:03:24.279420+0000","last_peered":"2024-03-21T13:03:24.279420+0000","last_clean":"2024-03-21T13:03:24.279420+0000","last_became_active":"2024-03-21T13:02:26.242708+0000","last_became_peered":"2024-03-21T13:02:26.242708+0000","last_unstale":"2024-03-21T13:03:24.279420+0000","last_undegraded":"2024-03-21T13:03:24.279420+0000","last_fullsized":"2024-03-21T13:03:24.279420+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:20.854917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.854917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":64,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266155+0000","last_change":"2024-03-21T13:01:48.212127+0000","last_active":"2024-03-21T13:03:24.266155+0000","last_peered":"2024-03-21T13:03:24.266155+0000","last_clean":"2024-03-21T13:03:24.266155+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:03:24.266155+0000","last_undegraded":"2024-03-21T13:03:24.266155+0000","last_fullsized":"2024-03-21T13:03:24.266155+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:01:34.066532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":464,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279128+0000","last_change":"2024-03-21T13:02:14.358837+0000","last_active":"2024-03-21T13:03:24.279128+0000","last_peered":"2024-03-21T13:03:24.279128+0000","last_clean":"2024-03-21T13:03:24.279128+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:03:24.279128+0000","last_undegraded":"2024-03-21T13:03:24.279128+0000","last_fullsized":"2024-03-21T13:03:24.279128+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:00:15.164219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:00:15.164219+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":278,"num_read_kb":273,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":207,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.284209+0000","last_change":"2024-03-21T13:02:14.359827+0000","last_active":"2024-03-21T13:03:24.284209+0000","last_peered":"2024-03-21T13:03:24.284209+0000","last_clean":"2024-03-21T13:03:24.284209+0000","last_became_active":"2024-03-21T13:02:14.333917+0000","last_became_peered":"2024-03-21T13:02:14.333917+0000","last_unstale":"2024-03-21T13:03:24.284209+0000","last_undegraded":"2024-03-21T13:03:24.284209+0000","last_fullsized":"2024-03-21T13:03:24.284209+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:01:28.111201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:28.111201+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:28.336018+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":37,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.280338+0000","last_change":"2024-03-21T13:02:26.232971+0000","last_active":"2024-03-21T13:03:24.280338+0000","last_peered":"2024-03-21T13:03:24.280338+0000","last_clean":"2024-03-21T13:03:24.280338+0000","last_became_active":"2024-03-21T13:02:26.232625+0000","last_became_peered":"2024-03-21T13:02:26.232625+0000","last_unstale":"2024-03-21T13:03:24.280338+0000","last_undegraded":"2024-03-21T13:03:24.280338+0000","last_fullsized":"2024-03-21T13:03:24.280338+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:34.982531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:34.982531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"137'10999","reported_seq":11116,"reported_epoch":137,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-21T13:03:24.305684+0000","last_change":"2024-03-21T13:02:57.394983+0000","last_active":"2024-03-21T13:03:24.305684+0000","last_peered":"2024-03-21T13:03:24.305684+0000","last_clean":"2024-03-21T13:02:13.315649+0000","last_became_active":"2024-03-21T13:02:27.316276+0000","last_became_peered":"2024-03-21T13:02:27.316276+0000","last_unstale":"2024-03-21T13:03:24.305684+0000","last_undegraded":"2024-03-21T13:03:24.305684+0000","last_fullsized":"2024-03-21T13:03:24.305684+0000","mapping_epoch":124,"log_start":"122'7458","ondisk_log_start":"122'7458","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3541,"log_dups_size":0,"ondisk_log_size":3541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249364480,"num_objects":3805,"num_object_clones":0,"num_object_copies":11415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7610,"num_objects_unfound":0,"num_objects_dirty":3805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10999,"num_write_kb":473728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,5,1],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":3805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":50,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.274660+0000","last_change":"2024-03-21T13:02:14.336240+0000","last_active":"2024-03-21T13:03:24.274660+0000","last_peered":"2024-03-21T13:03:24.274660+0000","last_clean":"2024-03-21T13:03:24.274660+0000","last_became_active":"2024-03-21T13:02:14.335897+0000","last_became_peered":"2024-03-21T13:02:14.335897+0000","last_unstale":"2024-03-21T13:03:24.274660+0000","last_undegraded":"2024-03-21T13:03:24.274660+0000","last_fullsized":"2024-03-21T13:03:24.274660+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:21.040252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:21.040252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":56,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.269265+0000","last_change":"2024-03-21T13:02:26.238041+0000","last_active":"2024-03-21T13:03:24.269265+0000","last_peered":"2024-03-21T13:03:24.269265+0000","last_clean":"2024-03-21T13:03:24.269265+0000","last_became_active":"2024-03-21T13:02:26.237683+0000","last_became_peered":"2024-03-21T13:02:26.237683+0000","last_unstale":"2024-03-21T13:03:24.269265+0000","last_undegraded":"2024-03-21T13:03:24.269265+0000","last_fullsized":"2024-03-21T13:03:24.269265+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:24.072225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:24.072225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"71'19","reported_seq":111,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279741+0000","last_change":"2024-03-21T13:02:26.274805+0000","last_active":"2024-03-21T13:03:24.279741+0000","last_peered":"2024-03-21T13:03:24.279741+0000","last_clean":"2024-03-21T13:03:24.279741+0000","last_became_active":"2024-03-21T13:02:26.239140+0000","last_became_peered":"2024-03-21T13:02:26.239140+0000","last_unstale":"2024-03-21T13:03:24.279741+0000","last_undegraded":"2024-03-21T13:03:24.279741+0000","last_fullsized":"2024-03-21T13:03:24.279741+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:01:29.106715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:29.106715+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":172,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.274802+0000","last_change":"2024-03-21T13:02:26.252937+0000","last_active":"2024-03-21T13:03:24.274802+0000","last_peered":"2024-03-21T13:03:24.274802+0000","last_clean":"2024-03-21T13:03:24.274802+0000","last_became_active":"2024-03-21T13:02:26.252372+0000","last_became_peered":"2024-03-21T13:02:26.252372+0000","last_unstale":"2024-03-21T13:03:24.274802+0000","last_undegraded":"2024-03-21T13:03:24.274802+0000","last_fullsized":"2024-03-21T13:03:24.274802+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:22.601641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:22.601641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"137'10953","reported_seq":11114,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.331255+0000","last_change":"2024-03-21T13:01:47.452788+0000","last_active":"2024-03-21T13:03:24.331255+0000","last_peered":"2024-03-21T13:03:24.331255+0000","last_clean":"2024-03-21T13:03:24.331255+0000","last_became_active":"2024-03-21T13:00:42.707521+0000","last_became_peered":"2024-03-21T13:00:42.707521+0000","last_unstale":"2024-03-21T13:03:24.331255+0000","last_undegraded":"2024-03-21T13:03:24.331255+0000","last_fullsized":"2024-03-21T13:03:24.331255+0000","mapping_epoch":92,"log_start":"119'6936","ondisk_log_start":"119'6936","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'5147","last_scrub_stamp":"2024-03-21T13:01:47.452664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:01:47.452664+0000","objects_scrubbed":5137,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":887,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238747648,"num_objects":3643,"num_object_clones":0,"num_object_copies":10929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10953,"num_write_kb":467072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":64,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.287538+0000","last_change":"2024-03-21T13:01:48.209228+0000","last_active":"2024-03-21T13:03:24.287538+0000","last_peered":"2024-03-21T13:03:24.287538+0000","last_clean":"2024-03-21T13:03:24.287538+0000","last_became_active":"2024-03-21T13:01:48.208899+0000","last_became_peered":"2024-03-21T13:01:48.208899+0000","last_unstale":"2024-03-21T13:03:24.287538+0000","last_undegraded":"2024-03-21T13:03:24.287538+0000","last_fullsized":"2024-03-21T13:03:24.287538+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:31.121913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.121913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":38,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.284078+0000","last_change":"2024-03-21T13:02:26.244688+0000","last_active":"2024-03-21T13:03:24.284078+0000","last_peered":"2024-03-21T13:03:24.284078+0000","last_clean":"2024-03-21T13:03:24.284078+0000","last_became_active":"2024-03-21T13:02:26.243417+0000","last_became_peered":"2024-03-21T13:02:26.243417+0000","last_unstale":"2024-03-21T13:03:24.284078+0000","last_undegraded":"2024-03-21T13:03:24.284078+0000","last_fullsized":"2024-03-21T13:03:24.284078+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:20.666893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.666893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"86'15","reported_seq":96,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266342+0000","last_change":"2024-03-21T13:02:26.263374+0000","last_active":"2024-03-21T13:03:24.266342+0000","last_peered":"2024-03-21T13:03:24.266342+0000","last_clean":"2024-03-21T13:03:24.266342+0000","last_became_active":"2024-03-21T13:02:26.232580+0000","last_became_peered":"2024-03-21T13:02:26.232580+0000","last_unstale":"2024-03-21T13:03:24.266342+0000","last_undegraded":"2024-03-21T13:03:24.266342+0000","last_fullsized":"2024-03-21T13:03:24.266342+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-03-21T13:01:27.655920+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:01:27.655920+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"125'12","reported_seq":121,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.274951+0000","last_change":"2024-03-21T13:02:26.269099+0000","last_active":"2024-03-21T13:03:24.274951+0000","last_peered":"2024-03-21T13:03:24.274951+0000","last_clean":"2024-03-21T13:03:24.274951+0000","last_became_active":"2024-03-21T13:02:26.253720+0000","last_became_peered":"2024-03-21T13:02:26.253720+0000","last_unstale":"2024-03-21T13:03:24.274951+0000","last_undegraded":"2024-03-21T13:03:24.274951+0000","last_fullsized":"2024-03-21T13:03:24.274951+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:25.175225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:25.175225+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"137'11085","reported_seq":14129,"reported_epoch":137,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:03:24.987671+0000","last_change":"2024-03-21T13:03:15.603630+0000","last_active":"2024-03-21T13:03:24.987671+0000","last_peered":"2024-03-21T13:03:24.987671+0000","last_clean":"2024-03-21T13:02:13.308104+0000","last_became_active":"2024-03-21T13:02:15.519968+0000","last_became_peered":"2024-03-21T13:02:15.519968+0000","last_unstale":"2024-03-21T13:03:24.987671+0000","last_undegraded":"2024-03-21T13:03:24.987671+0000","last_fullsized":"2024-03-21T13:03:24.987671+0000","mapping_epoch":118,"log_start":"122'7483","ondisk_log_start":"122'7483","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3602,"log_dups_size":0,"ondisk_log_size":3602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245039104,"num_objects":3739,"num_object_clones":0,"num_object_copies":11217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2356,"num_objects_unfound":0,"num_objects_dirty":3739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":474368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1463,"num_bytes_recovered":95879168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":3739}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":160,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279845+0000","last_change":"2024-03-21T13:01:50.086495+0000","last_active":"2024-03-21T13:03:24.279845+0000","last_peered":"2024-03-21T13:03:24.279845+0000","last_clean":"2024-03-21T13:03:24.279845+0000","last_became_active":"2024-03-21T12:58:43.353049+0000","last_became_peered":"2024-03-21T12:58:43.353049+0000","last_unstale":"2024-03-21T13:03:24.279845+0000","last_undegraded":"2024-03-21T13:03:24.279845+0000","last_fullsized":"2024-03-21T13:03:24.279845+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:50.086373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:50.086373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":184,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.284314+0000","last_change":"2024-03-21T13:02:14.351423+0000","last_active":"2024-03-21T13:03:24.284314+0000","last_peered":"2024-03-21T13:03:24.284314+0000","last_clean":"2024-03-21T13:03:24.284314+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:03:24.284314+0000","last_undegraded":"2024-03-21T13:03:24.284314+0000","last_fullsized":"2024-03-21T13:03:24.284314+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:01:39.008715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.008715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:28.336401+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"87'21","reported_seq":130,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.274477+0000","last_change":"2024-03-21T13:02:26.263470+0000","last_active":"2024-03-21T13:03:24.274477+0000","last_peered":"2024-03-21T13:03:24.274477+0000","last_clean":"2024-03-21T13:03:24.274477+0000","last_became_active":"2024-03-21T13:02:26.227165+0000","last_became_peered":"2024-03-21T13:02:26.227165+0000","last_unstale":"2024-03-21T13:03:24.274477+0000","last_undegraded":"2024-03-21T13:03:24.274477+0000","last_fullsized":"2024-03-21T13:03:24.274477+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'21","last_scrub_stamp":"2024-03-21T13:01:29.911046+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:01:29.911046+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"119'9","reported_seq":235,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.288105+0000","last_change":"2024-03-21T13:02:14.375956+0000","last_active":"2024-03-21T13:03:24.288105+0000","last_peered":"2024-03-21T13:03:24.288105+0000","last_clean":"2024-03-21T13:03:24.288105+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:03:24.288105+0000","last_undegraded":"2024-03-21T13:03:24.288105+0000","last_fullsized":"2024-03-21T13:03:24.288105+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:25.700549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:25.700549+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"137'10969","reported_seq":11071,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.351090+0000","last_change":"2024-03-21T13:00:42.711111+0000","last_active":"2024-03-21T13:03:24.351090+0000","last_peered":"2024-03-21T13:03:24.351090+0000","last_clean":"2024-03-21T13:03:24.351090+0000","last_became_active":"2024-03-21T13:00:42.710829+0000","last_became_peered":"2024-03-21T13:00:42.710829+0000","last_unstale":"2024-03-21T13:03:24.351090+0000","last_undegraded":"2024-03-21T13:03:24.351090+0000","last_fullsized":"2024-03-21T13:03:24.351090+0000","mapping_epoch":92,"log_start":"122'7374","ondisk_log_start":"122'7374","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3595,"log_dups_size":0,"ondisk_log_size":3595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239665152,"num_objects":3657,"num_object_clones":0,"num_object_copies":10971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10969,"num_write_kb":468032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":78,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.288443+0000","last_change":"2024-03-21T13:01:27.382938+0000","last_active":"2024-03-21T13:03:24.288443+0000","last_peered":"2024-03-21T13:03:24.288443+0000","last_clean":"2024-03-21T13:03:24.288443+0000","last_became_active":"2024-03-21T13:01:27.382663+0000","last_became_peered":"2024-03-21T13:01:27.382663+0000","last_unstale":"2024-03-21T13:03:24.288443+0000","last_undegraded":"2024-03-21T13:03:24.288443+0000","last_fullsized":"2024-03-21T13:03:24.288443+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.823841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.283866+0000","last_change":"2024-03-21T12:59:43.240826+0000","last_active":"2024-03-21T13:03:24.283866+0000","last_peered":"2024-03-21T13:03:24.283866+0000","last_clean":"2024-03-21T13:03:24.283866+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:03:24.283866+0000","last_undegraded":"2024-03-21T13:03:24.283866+0000","last_fullsized":"2024-03-21T13:03:24.283866+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T12:59:43.240674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"71'18","reported_seq":194,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.287944+0000","last_change":"2024-03-21T12:59:48.179286+0000","last_active":"2024-03-21T13:03:24.287944+0000","last_peered":"2024-03-21T13:03:24.287944+0000","last_clean":"2024-03-21T13:03:24.287944+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:03:24.287944+0000","last_undegraded":"2024-03-21T13:03:24.287944+0000","last_fullsized":"2024-03-21T13:03:24.287944+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T12:59:48.179188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.179188+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.265849+0000","last_change":"2024-03-21T13:01:17.590965+0000","last_active":"2024-03-21T13:03:24.265849+0000","last_peered":"2024-03-21T13:03:24.265849+0000","last_clean":"2024-03-21T13:03:24.265849+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:03:24.265849+0000","last_undegraded":"2024-03-21T13:03:24.265849+0000","last_fullsized":"2024-03-21T13:03:24.265849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:17.590831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:17.590831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"137'11057","reported_seq":11159,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.338617+0000","last_change":"2024-03-21T13:00:42.705026+0000","last_active":"2024-03-21T13:03:24.338617+0000","last_peered":"2024-03-21T13:03:24.338617+0000","last_clean":"2024-03-21T13:03:24.338617+0000","last_became_active":"2024-03-21T13:00:42.704712+0000","last_became_peered":"2024-03-21T13:00:42.704712+0000","last_unstale":"2024-03-21T13:03:24.338617+0000","last_undegraded":"2024-03-21T13:03:24.338617+0000","last_fullsized":"2024-03-21T13:03:24.338617+0000","mapping_epoch":92,"log_start":"119'6868","ondisk_log_start":"119'6868","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4189,"log_dups_size":0,"ondisk_log_size":4189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236650496,"num_objects":3611,"num_object_clones":0,"num_object_copies":10833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11057,"num_write_kb":469376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.274397+0000","last_change":"2024-03-21T13:02:26.236964+0000","last_active":"2024-03-21T13:03:24.274397+0000","last_peered":"2024-03-21T13:03:24.274397+0000","last_clean":"2024-03-21T13:03:24.274397+0000","last_became_active":"2024-03-21T13:02:26.236501+0000","last_became_peered":"2024-03-21T13:02:26.236501+0000","last_unstale":"2024-03-21T13:03:24.274397+0000","last_undegraded":"2024-03-21T13:03:24.274397+0000","last_fullsized":"2024-03-21T13:03:24.274397+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:24.875883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:24.875883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"125'10","reported_seq":79,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.285236+0000","last_change":"2024-03-21T13:02:26.252598+0000","last_active":"2024-03-21T13:03:24.285236+0000","last_peered":"2024-03-21T13:03:24.285236+0000","last_clean":"2024-03-21T13:03:24.285236+0000","last_became_active":"2024-03-21T13:02:26.231372+0000","last_became_peered":"2024-03-21T13:02:26.231372+0000","last_unstale":"2024-03-21T13:03:24.285236+0000","last_undegraded":"2024-03-21T13:03:24.285236+0000","last_fullsized":"2024-03-21T13:03:24.285236+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:00:23.782010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:23.782010+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:28.335982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":172,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.287864+0000","last_change":"2024-03-21T13:02:14.352037+0000","last_active":"2024-03-21T13:03:24.287864+0000","last_peered":"2024-03-21T13:03:24.287864+0000","last_clean":"2024-03-21T13:03:24.287864+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:03:24.287864+0000","last_undegraded":"2024-03-21T13:03:24.287864+0000","last_fullsized":"2024-03-21T13:03:24.287864+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:41.231340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:41.231340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":110,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.283952+0000","last_change":"2024-03-21T13:02:14.377274+0000","last_active":"2024-03-21T13:03:24.283952+0000","last_peered":"2024-03-21T13:03:24.283952+0000","last_clean":"2024-03-21T13:03:24.283952+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:03:24.283952+0000","last_undegraded":"2024-03-21T13:03:24.283952+0000","last_fullsized":"2024-03-21T13:03:24.283952+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:01:46.167736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.167736+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"136'10829","reported_seq":20853,"reported_epoch":136,"state":"peering","last_fresh":"2024-03-21T13:03:24.241724+0000","last_change":"2024-03-21T13:03:24.278003+0000","last_active":"2024-03-21T13:03:24.278003+0000","last_peered":"2024-03-21T13:03:24.241724+0000","last_clean":"2024-03-21T13:03:24.241724+0000","last_became_active":"2024-03-21T13:02:15.410023+0000","last_became_peered":"2024-03-21T13:02:15.410023+0000","last_unstale":"2024-03-21T13:03:24.241724+0000","last_undegraded":"2024-03-21T13:03:24.241724+0000","last_fullsized":"2024-03-21T13:03:24.241724+0000","mapping_epoch":137,"log_start":"116'6644","ondisk_log_start":"116'6644","created":92,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237371416,"num_objects":3623,"num_object_clones":0,"num_object_copies":10869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10829,"num_write_kb":462401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4954,"num_bytes_recovered":324599832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":165,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279510+0000","last_change":"2024-03-21T13:02:26.249048+0000","last_active":"2024-03-21T13:03:24.279510+0000","last_peered":"2024-03-21T13:03:24.279510+0000","last_clean":"2024-03-21T13:03:24.279510+0000","last_became_active":"2024-03-21T13:02:26.243985+0000","last_became_peered":"2024-03-21T13:02:26.243985+0000","last_unstale":"2024-03-21T13:03:24.279510+0000","last_undegraded":"2024-03-21T13:03:24.279510+0000","last_fullsized":"2024-03-21T13:03:24.279510+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:00:27.958396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:27.958396+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":38,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266936+0000","last_change":"2024-03-21T13:02:26.229605+0000","last_active":"2024-03-21T13:03:24.266936+0000","last_peered":"2024-03-21T13:03:24.266936+0000","last_clean":"2024-03-21T13:03:24.266936+0000","last_became_active":"2024-03-21T13:02:26.229296+0000","last_became_peered":"2024-03-21T13:02:26.229296+0000","last_unstale":"2024-03-21T13:03:24.266936+0000","last_undegraded":"2024-03-21T13:03:24.266936+0000","last_fullsized":"2024-03-21T13:03:24.266936+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:36.008632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:36.008632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":174,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.265926+0000","last_change":"2024-03-21T13:01:39.629358+0000","last_active":"2024-03-21T13:03:24.265926+0000","last_peered":"2024-03-21T13:03:24.265926+0000","last_clean":"2024-03-21T13:03:24.265926+0000","last_became_active":"2024-03-21T12:58:37.730858+0000","last_became_peered":"2024-03-21T12:58:37.730858+0000","last_unstale":"2024-03-21T13:03:24.265926+0000","last_undegraded":"2024-03-21T13:03:24.265926+0000","last_fullsized":"2024-03-21T13:03:24.265926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:39.629248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.629248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'11","reported_seq":199,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266570+0000","last_change":"2024-03-21T13:02:14.381387+0000","last_active":"2024-03-21T13:03:24.266570+0000","last_peered":"2024-03-21T13:03:24.266570+0000","last_clean":"2024-03-21T13:03:24.266570+0000","last_became_active":"2024-03-21T13:02:14.348499+0000","last_became_peered":"2024-03-21T13:02:14.348499+0000","last_unstale":"2024-03-21T13:03:24.266570+0000","last_undegraded":"2024-03-21T13:03:24.266570+0000","last_fullsized":"2024-03-21T13:03:24.266570+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:00:38.769807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.769807+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"136'11031","reported_seq":21584,"reported_epoch":136,"state":"peering","last_fresh":"2024-03-21T13:03:24.225452+0000","last_change":"2024-03-21T13:03:24.281784+0000","last_active":"2024-03-21T13:03:24.281784+0000","last_peered":"2024-03-21T13:03:24.225452+0000","last_clean":"2024-03-21T13:03:24.225452+0000","last_became_active":"2024-03-21T13:02:27.296672+0000","last_became_peered":"2024-03-21T13:02:27.296672+0000","last_unstale":"2024-03-21T13:03:24.225452+0000","last_undegraded":"2024-03-21T13:03:24.225452+0000","last_fullsized":"2024-03-21T13:03:24.225452+0000","mapping_epoch":137,"log_start":"114'6284","ondisk_log_start":"114'6284","created":92,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'7457","last_scrub_stamp":"2024-03-21T13:02:25.543659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:02:25.543659+0000","objects_scrubbed":7338,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"no scrub is scheduled","scrub_duration":3245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243597312,"num_objects":3717,"num_object_clones":0,"num_object_copies":11151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11031,"num_write_kb":471936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5179,"num_bytes_recovered":339410944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":163,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.268631+0000","last_change":"2024-03-21T13:01:47.562081+0000","last_active":"2024-03-21T13:03:24.268631+0000","last_peered":"2024-03-21T13:03:24.268631+0000","last_clean":"2024-03-21T13:03:24.268631+0000","last_became_active":"2024-03-21T12:58:43.183380+0000","last_became_peered":"2024-03-21T12:58:43.183380+0000","last_unstale":"2024-03-21T13:03:24.268631+0000","last_undegraded":"2024-03-21T13:03:24.268631+0000","last_fullsized":"2024-03-21T13:03:24.268631+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:47.561991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:01:47.561991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":50,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.266451+0000","last_change":"2024-03-21T13:02:14.335457+0000","last_active":"2024-03-21T13:03:24.266451+0000","last_peered":"2024-03-21T13:03:24.266451+0000","last_clean":"2024-03-21T13:03:24.266451+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:03:24.266451+0000","last_undegraded":"2024-03-21T13:03:24.266451+0000","last_fullsized":"2024-03-21T13:03:24.266451+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:17.006200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:17.006200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":180,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.274893+0000","last_change":"2024-03-21T13:02:14.343797+0000","last_active":"2024-03-21T13:03:24.274893+0000","last_peered":"2024-03-21T13:03:24.274893+0000","last_clean":"2024-03-21T13:03:24.274893+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:03:24.274893+0000","last_undegraded":"2024-03-21T13:03:24.274893+0000","last_fullsized":"2024-03-21T13:03:24.274893+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:16.595403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:16.595403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":157,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.274616+0000","last_change":"2024-03-21T13:02:27.259863+0000","last_active":"2024-03-21T13:03:24.274616+0000","last_peered":"2024-03-21T13:03:24.274616+0000","last_clean":"2024-03-21T13:03:24.274616+0000","last_became_active":"2024-03-21T13:02:14.334882+0000","last_became_peered":"2024-03-21T13:02:14.334882+0000","last_unstale":"2024-03-21T13:03:24.274616+0000","last_undegraded":"2024-03-21T13:03:24.274616+0000","last_fullsized":"2024-03-21T13:03:24.274616+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-21T13:02:27.259731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:02:27.259731+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:03:35.457954+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"137'10999","reported_seq":11110,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.305950+0000","last_change":"2024-03-21T13:01:48.274014+0000","last_active":"2024-03-21T13:03:24.305950+0000","last_peered":"2024-03-21T13:03:24.305950+0000","last_clean":"2024-03-21T13:03:24.305950+0000","last_became_active":"2024-03-21T13:01:48.220955+0000","last_became_peered":"2024-03-21T13:01:48.220955+0000","last_unstale":"2024-03-21T13:03:24.305950+0000","last_undegraded":"2024-03-21T13:03:24.305950+0000","last_fullsized":"2024-03-21T13:03:24.305950+0000","mapping_epoch":110,"log_start":"122'7439","ondisk_log_start":"122'7439","created":92,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3560,"log_dups_size":0,"ondisk_log_size":3560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244908032,"num_objects":3737,"num_object_clones":0,"num_object_copies":11211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10999,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":58,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.272783+0000","last_change":"2024-03-21T13:02:26.228129+0000","last_active":"2024-03-21T13:03:24.272783+0000","last_peered":"2024-03-21T13:03:24.272783+0000","last_clean":"2024-03-21T13:03:24.272783+0000","last_became_active":"2024-03-21T13:02:26.227779+0000","last_became_peered":"2024-03-21T13:02:26.227779+0000","last_unstale":"2024-03-21T13:03:24.272783+0000","last_undegraded":"2024-03-21T13:03:24.272783+0000","last_fullsized":"2024-03-21T13:03:24.272783+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:32.225843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:32.225843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"125'10","reported_seq":244,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.274901+0000","last_change":"2024-03-21T13:02:26.263146+0000","last_active":"2024-03-21T13:03:24.274901+0000","last_peered":"2024-03-21T13:03:24.274901+0000","last_clean":"2024-03-21T13:03:24.274901+0000","last_became_active":"2024-03-21T13:02:26.235954+0000","last_became_peered":"2024-03-21T13:02:26.235954+0000","last_unstale":"2024-03-21T13:03:24.274901+0000","last_undegraded":"2024-03-21T13:03:24.274901+0000","last_fullsized":"2024-03-21T13:03:24.274901+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:41.660019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:41.660019+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.280443+0000","last_change":"2024-03-21T13:00:25.960463+0000","last_active":"2024-03-21T13:03:24.280443+0000","last_peered":"2024-03-21T13:03:24.280443+0000","last_clean":"2024-03-21T13:03:24.280443+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:03:24.280443+0000","last_undegraded":"2024-03-21T13:03:24.280443+0000","last_fullsized":"2024-03-21T13:03:24.280443+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:25.960383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:25.960383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":112,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-21T13:03:24.279904+0000","last_change":"2024-03-21T13:01:27.363819+0000","last_active":"2024-03-21T13:03:24.279904+0000","last_peered":"2024-03-21T13:03:24.279904+0000","last_clean":"2024-03-21T13:03:24.279904+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:03:24.279904+0000","last_undegraded":"2024-03-21T13:03:24.279904+0000","last_fullsized":"2024-03-21T13:03:24.279904+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:00:10.785752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.785752+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3869573144,"num_objects":59046,"num_object_clones":0,"num_object_copies":177138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12724,"num_objects_unfound":0,"num_objects_dirty":59046,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175832,"num_write_kb":7516033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30340,"num_bytes_recovered":1988296728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12454084608,"data_stored":12454068320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64451,"ondisk_log_size":64451,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":278,"num_read_kb":273,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":684,"num_read_kb":441,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":3548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":424,"ondisk_log_size":424,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168638,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1618440,"kb_used_data":1478976,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92126712,"statfs":{"total":95995035648,"available":94337753088,"internally_reserved":0,"allocated":1514471424,"data_stored":1514043849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142801354},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462980,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1413668,"kb_used_data":1287968,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92331484,"statfs":{"total":95995035648,"available":94547439616,"internally_reserved":0,"allocated":1318879232,"data_stored":1318492608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128580038},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574940,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2019864,"kb_used_data":1890004,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":91725288,"statfs":{"total":95995035648,"available":93926694912,"internally_reserved":0,"allocated":1935364096,"data_stored":1934988589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1811880,"kb_used_data":1664032,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":91933272,"statfs":{"total":95995035648,"available":94139670528,"internally_reserved":0,"allocated":1703968768,"data_stored":1703572056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2780524,"kb_used_data":2604964,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":90964628,"statfs":{"total":95995035648,"available":93147779072,"internally_reserved":0,"allocated":2667483136,"data_stored":2667018603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":179763655},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018957,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1697784,"kb_used_data":1572144,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92047368,"statfs":{"total":95995035648,"available":94256504832,"internally_reserved":0,"allocated":1609875456,"data_stored":1609462885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724616,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1933971456,"data_stored":1933967384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":970592256,"data_stored":970588184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1698824192,"data_stored":1698824192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2661355520,"data_stored":2661351448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1612447744,"data_stored":1612447744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":734724096,"data_stored":734724096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1317539840,"data_stored":1317535768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1524629504,"data_stored":1524629504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:03:30.495 INFO:tasks.thrashosds.thrasher:Setting 6.11 to [7, 3, 4] 2024-03-21T13:03:30.496 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.11', '7', '3', '4'] 2024-03-21T13:03:30.496 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pg-upmap 6.11 7 3 4 2024-03-21T13:03:30.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:30.520 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:30.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:30.547 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:30 smithi163 ceph-mon[29621]: pgmap v187: 148 pgs: 2 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 586 op/s; 12724/177732 objects misplaced (7.159%); 18 MiB/s, 294 objects/s recovering 2024-03-21T13:03:30.547 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:30 smithi163 ceph-mon[29621]: 5.e scrub starts 2024-03-21T13:03:30.547 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:30 smithi163 ceph-mon[29621]: 5.e scrub ok 2024-03-21T13:03:30.547 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:30 smithi163 ceph-mon[40675]: pgmap v187: 148 pgs: 2 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 586 op/s; 12724/177732 objects misplaced (7.159%); 18 MiB/s, 294 objects/s recovering 2024-03-21T13:03:30.548 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:30 smithi163 ceph-mon[40675]: 5.e scrub starts 2024-03-21T13:03:30.548 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:30 smithi163 ceph-mon[40675]: 5.e scrub ok 2024-03-21T13:03:30.621 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:30.665 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:30.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:30.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:30.682 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:30 smithi204 ceph-mon[33153]: pgmap v187: 148 pgs: 2 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 586 op/s; 12724/177732 objects misplaced (7.159%); 18 MiB/s, 294 objects/s recovering 2024-03-21T13:03:30.682 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:30 smithi204 ceph-mon[33153]: 5.e scrub starts 2024-03-21T13:03:30.682 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:30 smithi204 ceph-mon[33153]: 5.e scrub ok 2024-03-21T13:03:30.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-21T13:03:30.767 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:30.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:30.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:30.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:30.905 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:30.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:30.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:31.045 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:31.082 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:03:31.087 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:31.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:31.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:31.189 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:31.231 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:31.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:31.333 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:31.374 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:31.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:31.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:31.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:31.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:31 smithi204 ceph-mon[33153]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:03:31.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:03:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:03:31.508 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:31.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:31.549 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:31 smithi163 ceph-mon[29621]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:03:31.549 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:31 smithi163 ceph-mon[40675]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:03:31.610 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:31.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:31.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:31.744 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:31.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:31.881 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:31.922 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:31.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:31.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:32.024 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:32.068 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:32.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:32.170 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:32.219 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:32.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:32.297 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:32.320 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:32.376 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:32.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:32.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:32 smithi163 ceph-mon[29621]: pgmap v188: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 3.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 9991/161598 objects misplaced (6.183%); 30 MiB/s, 477 objects/s recovering 2024-03-21T13:03:32.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:32 smithi163 ceph-mon[29621]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:03:32.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:32 smithi163 ceph-mon[29621]: 4.6 scrub starts 2024-03-21T13:03:32.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:32 smithi163 ceph-mon[29621]: 4.6 scrub ok 2024-03-21T13:03:32.435 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:32 smithi163 ceph-mon[40675]: pgmap v188: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 3.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 9991/161598 objects misplaced (6.183%); 30 MiB/s, 477 objects/s recovering 2024-03-21T13:03:32.435 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:32 smithi163 ceph-mon[40675]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:03:32.435 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:32 smithi163 ceph-mon[40675]: 4.6 scrub starts 2024-03-21T13:03:32.435 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:32 smithi163 ceph-mon[40675]: 4.6 scrub ok 2024-03-21T13:03:32.477 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:32.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:32.610 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:32.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:32.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:32.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:32.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:32 smithi204 ceph-mon[33153]: pgmap v188: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 3.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 9991/161598 objects misplaced (6.183%); 30 MiB/s, 477 objects/s recovering 2024-03-21T13:03:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:32 smithi204 ceph-mon[33153]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:03:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:32 smithi204 ceph-mon[33153]: 4.6 scrub starts 2024-03-21T13:03:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:32 smithi204 ceph-mon[33153]: 4.6 scrub ok 2024-03-21T13:03:32.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:32.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:32.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:32.864 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:32.892 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:32.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:32.994 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:33.023 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:33.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:33.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:33.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:33.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:33.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:33.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:33.280 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:33.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:33.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:33.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:33.420 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:33.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:33.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:33.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:33 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:33.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:33 smithi204 ceph-mon[33153]: osdmap e139: 8 total, 8 up, 6 in 2024-03-21T13:03:33.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:33 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:33.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:33 smithi163 ceph-mon[29621]: osdmap e139: 8 total, 8 up, 6 in 2024-03-21T13:03:33.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:33 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:33.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:33 smithi163 ceph-mon[40675]: osdmap e139: 8 total, 8 up, 6 in 2024-03-21T13:03:33.522 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:33.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:33.559 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:33.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:33.626 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:03:33.660 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:33.722 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:33.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:33.823 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:33.836 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:03:33.836 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:03:33.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:33.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:33.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:33.957 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:34.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:34.004 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:34.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:34.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:34.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:34.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:34.238 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:34.291 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:34.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:34.360 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:03:34.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:34.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:34.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:34.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:34 smithi204 ceph-mon[33153]: pgmap v190: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 3.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 923 op/s; 9991/161598 objects misplaced (6.183%); 26 MiB/s, 413 objects/s recovering 2024-03-21T13:03:34.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:34 smithi204 ceph-mon[33153]: osdmap e140: 8 total, 8 up, 6 in 2024-03-21T13:03:34.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:34 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2690668971' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:34.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:34 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:34.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:34 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:34.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:34 smithi163 ceph-mon[29621]: pgmap v190: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 3.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 923 op/s; 9991/161598 objects misplaced (6.183%); 26 MiB/s, 413 objects/s recovering 2024-03-21T13:03:34.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:34 smithi163 ceph-mon[29621]: osdmap e140: 8 total, 8 up, 6 in 2024-03-21T13:03:34.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:34 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2690668971' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:34.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:34 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:34.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:34 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:34.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:34 smithi163 ceph-mon[40675]: pgmap v190: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 3.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 923 op/s; 9991/161598 objects misplaced (6.183%); 26 MiB/s, 413 objects/s recovering 2024-03-21T13:03:34.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:34 smithi163 ceph-mon[40675]: osdmap e140: 8 total, 8 up, 6 in 2024-03-21T13:03:34.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:34 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2690668971' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:34.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:34 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:34.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:34 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:34.527 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:34.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:34.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:34.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:34.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:34.701 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:34.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:34.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-21T13:03:34.803 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:34.839 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:34.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:34.940 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:34.986 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:34.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:34.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:35.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:35.122 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:03:35.126 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:35.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:35.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:35.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:35.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:35.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:35.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:35.391 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:35.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:35.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:35.493 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:35.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:35 smithi204 ceph-mon[33153]: 4.e scrub starts 2024-03-21T13:03:35.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:35 smithi204 ceph-mon[33153]: 4.e scrub ok 2024-03-21T13:03:35.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:35 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:03:35.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:35 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:03:35.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:35 smithi204 ceph-mon[33153]: osdmap e141: 8 total, 8 up, 6 in 2024-03-21T13:03:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:35.536 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:35.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:35.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:35 smithi163 ceph-mon[29621]: 4.e scrub starts 2024-03-21T13:03:35.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:35 smithi163 ceph-mon[29621]: 4.e scrub ok 2024-03-21T13:03:35.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:35 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:03:35.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:35 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:03:35.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:35 smithi163 ceph-mon[29621]: osdmap e141: 8 total, 8 up, 6 in 2024-03-21T13:03:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:35 smithi163 ceph-mon[40675]: 4.e scrub starts 2024-03-21T13:03:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:35 smithi163 ceph-mon[40675]: 4.e scrub ok 2024-03-21T13:03:35.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:35 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:03:35.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:35 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:03:35.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:35 smithi163 ceph-mon[40675]: osdmap e141: 8 total, 8 up, 6 in 2024-03-21T13:03:35.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:35.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:35.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:35.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:35.804 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:35.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:35.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:35.891 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:03:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:03:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:03:35.904 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:35.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:35.935 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:35.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:36.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:36.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:36.069 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:36.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:36.170 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:36.211 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:36.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:36.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:36.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:36.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:36.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:36.372 INFO:teuthology.orchestra.run.smithi163.stderr:set 6.11 pg_upmap mapping to [7,3,4] 2024-03-21T13:03:36.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:36.467 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:36.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:36.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:36 smithi163 ceph-mon[29621]: pgmap v193: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 3.1 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.37k op/s; 9823/153063 objects misplaced (6.418%); 15 MiB/s, 243 objects/s recovering 2024-03-21T13:03:36.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:36 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/854668487' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.11", "id": [7, 3, 4]}]: dispatch 2024-03-21T13:03:36.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:36 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.11", "id": [7, 3, 4]}]: dispatch 2024-03-21T13:03:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:36 smithi163 ceph-mon[40675]: pgmap v193: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 3.1 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.37k op/s; 9823/153063 objects misplaced (6.418%); 15 MiB/s, 243 objects/s recovering 2024-03-21T13:03:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:36 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/854668487' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.11", "id": [7, 3, 4]}]: dispatch 2024-03-21T13:03:36.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:36 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.11", "id": [7, 3, 4]}]: dispatch 2024-03-21T13:03:36.569 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:36.607 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:36.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:36.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:36.728 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:36 smithi204 ceph-mon[33153]: pgmap v193: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 3.1 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.37k op/s; 9823/153063 objects misplaced (6.418%); 15 MiB/s, 243 objects/s recovering 2024-03-21T13:03:36.728 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:36 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/854668487' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.11", "id": [7, 3, 4]}]: dispatch 2024-03-21T13:03:36.729 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:36 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.11", "id": [7, 3, 4]}]: dispatch 2024-03-21T13:03:36.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:36.743 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:36.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:36.845 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:36.885 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:36.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:36.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:36.987 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:37.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:37.021 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:37.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:37.122 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:37.169 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:37.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:37.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:37.272 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:37.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:37.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:37.406 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:37.436 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:37.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:37.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:37.475 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:37 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:03:37.475 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:37 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:03:37.476 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:37 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.11", "id": [7, 3, 4]}]': finished 2024-03-21T13:03:37.476 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:37 smithi204 ceph-mon[33153]: osdmap e142: 8 total, 8 up, 6 in 2024-03-21T13:03:37.476 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:37 smithi204 ceph-mon[33153]: pgmap v195: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.0 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 6640/146427 objects misplaced (4.535%); 26 MiB/s, 418 objects/s recovering 2024-03-21T13:03:37.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:37.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:37.576 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:37.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:37.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:37.705 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:37.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:37.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:37 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:03:37.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:37 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:03:37.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:37 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.11", "id": [7, 3, 4]}]': finished 2024-03-21T13:03:37.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:37 smithi163 ceph-mon[29621]: osdmap e142: 8 total, 8 up, 6 in 2024-03-21T13:03:37.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:37 smithi163 ceph-mon[29621]: pgmap v195: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.0 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 6640/146427 objects misplaced (4.535%); 26 MiB/s, 418 objects/s recovering 2024-03-21T13:03:37.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:37 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:03:37.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:37 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:03:37.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:37 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.11", "id": [7, 3, 4]}]': finished 2024-03-21T13:03:37.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:37 smithi163 ceph-mon[40675]: osdmap e142: 8 total, 8 up, 6 in 2024-03-21T13:03:37.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:37 smithi163 ceph-mon[40675]: pgmap v195: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.0 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 6640/146427 objects misplaced (4.535%); 26 MiB/s, 418 objects/s recovering 2024-03-21T13:03:37.807 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:37.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:37.857 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:37.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:37.948 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:03:37.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:37.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:37.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:38.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:38.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:38.125 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:38.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:38.222 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:03:38.222 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:03:38.226 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:38.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:38.271 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:38.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:38.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:38.414 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:38.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:38.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[29621]: 5.5 scrub starts 2024-03-21T13:03:38.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[29621]: 5.5 scrub ok 2024-03-21T13:03:38.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[29621]: 5.1b scrub starts 2024-03-21T13:03:38.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[29621]: 5.1b scrub ok 2024-03-21T13:03:38.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[29621]: osdmap e143: 8 total, 8 up, 6 in 2024-03-21T13:03:38.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:03:38.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:03:38.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[40675]: 5.5 scrub starts 2024-03-21T13:03:38.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[40675]: 5.5 scrub ok 2024-03-21T13:03:38.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[40675]: 5.1b scrub starts 2024-03-21T13:03:38.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[40675]: 5.1b scrub ok 2024-03-21T13:03:38.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[40675]: osdmap e143: 8 total, 8 up, 6 in 2024-03-21T13:03:38.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:03:38.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:38 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:03:38.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:38.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:38.552 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:38.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:38.643 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:38 smithi204 ceph-mon[33153]: 5.5 scrub starts 2024-03-21T13:03:38.643 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:38 smithi204 ceph-mon[33153]: 5.5 scrub ok 2024-03-21T13:03:38.643 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:38 smithi204 ceph-mon[33153]: 5.1b scrub starts 2024-03-21T13:03:38.643 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:38 smithi204 ceph-mon[33153]: 5.1b scrub ok 2024-03-21T13:03:38.643 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:38 smithi204 ceph-mon[33153]: osdmap e143: 8 total, 8 up, 6 in 2024-03-21T13:03:38.643 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:38 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:03:38.644 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:38 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:03:38.653 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:38.676 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:38.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:38.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:38.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:38.820 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:38.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:38.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:38.945 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:38.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:38.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:39.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:39.073 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:39.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:39.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:39.174 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:39.211 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:39.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:39.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:39.313 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:39.353 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:39.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:39.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:39.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:39.482 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:39.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:39.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:39.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:39 smithi204 ceph-mon[33153]: osdmap e144: 8 total, 8 up, 6 in 2024-03-21T13:03:39.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:39 smithi204 ceph-mon[33153]: pgmap v198: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.0 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 6640/146427 objects misplaced (4.535%); 26 MiB/s, 419 objects/s recovering 2024-03-21T13:03:39.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:39.617 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:39.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:39.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:39.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:39.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:39.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:39.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:39 smithi163 ceph-mon[29621]: osdmap e144: 8 total, 8 up, 6 in 2024-03-21T13:03:39.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:39 smithi163 ceph-mon[29621]: pgmap v198: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.0 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 6640/146427 objects misplaced (4.535%); 26 MiB/s, 419 objects/s recovering 2024-03-21T13:03:39.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:39 smithi163 ceph-mon[40675]: osdmap e144: 8 total, 8 up, 6 in 2024-03-21T13:03:39.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:39 smithi163 ceph-mon[40675]: pgmap v198: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.0 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 6640/146427 objects misplaced (4.535%); 26 MiB/s, 419 objects/s recovering 2024-03-21T13:03:39.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:39.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:39.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:39.978 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:40.013 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:40.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:40.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:40.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:40.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:40.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:40.242 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:40.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:40.277 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:40.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:40.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:40.412 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:40.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:40.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:40.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:40.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:40.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:40.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:40.640 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:40.679 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:40.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:40 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:03:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:40 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:03:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:40 smithi204 ceph-mon[33153]: osdmap e145: 8 total, 8 up, 6 in 2024-03-21T13:03:40.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:40 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:03:40.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:40 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:03:40.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:40 smithi163 ceph-mon[40675]: osdmap e145: 8 total, 8 up, 6 in 2024-03-21T13:03:40.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:40 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:03:40.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:40 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:03:40.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:40 smithi163 ceph-mon[29621]: osdmap e145: 8 total, 8 up, 6 in 2024-03-21T13:03:40.780 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:40.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:40.807 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:40.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:40.908 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:40.950 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:40.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:41.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-21T13:03:41.052 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:41.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:41.079 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:41.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:41.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:41.220 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:41.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:41.322 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:41.343 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:03:41.345 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:41.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:41.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:41.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:41.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:41.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:41.505 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:03:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:03:41.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:41 smithi204 ceph-mon[33153]: pgmap v200: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 2.7 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 4914/131184 objects misplaced (3.746%); 11 MiB/s, 178 objects/s recovering 2024-03-21T13:03:41.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:41 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:03:41.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:41 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:03:41.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:41 smithi204 ceph-mon[33153]: 5.19 scrub starts 2024-03-21T13:03:41.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:41 smithi204 ceph-mon[33153]: 5.19 scrub ok 2024-03-21T13:03:41.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:41 smithi204 ceph-mon[33153]: osdmap e146: 8 total, 8 up, 6 in 2024-03-21T13:03:41.578 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:41.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:41.603 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:41.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:41.704 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:41.729 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:41.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:41.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:41 smithi163 ceph-mon[29621]: pgmap v200: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 2.7 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 4914/131184 objects misplaced (3.746%); 11 MiB/s, 178 objects/s recovering 2024-03-21T13:03:41.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:41 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:03:41.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:41 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:03:41.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:41 smithi163 ceph-mon[29621]: 5.19 scrub starts 2024-03-21T13:03:41.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:41 smithi163 ceph-mon[29621]: 5.19 scrub ok 2024-03-21T13:03:41.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:41 smithi163 ceph-mon[29621]: osdmap e146: 8 total, 8 up, 6 in 2024-03-21T13:03:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:41 smithi163 ceph-mon[40675]: pgmap v200: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 2.7 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 4914/131184 objects misplaced (3.746%); 11 MiB/s, 178 objects/s recovering 2024-03-21T13:03:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:41 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:03:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:41 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:03:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:41 smithi163 ceph-mon[40675]: 5.19 scrub starts 2024-03-21T13:03:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:41 smithi163 ceph-mon[40675]: 5.19 scrub ok 2024-03-21T13:03:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:41 smithi163 ceph-mon[40675]: osdmap e146: 8 total, 8 up, 6 in 2024-03-21T13:03:41.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:41.856 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:41.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:41.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:41.958 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:41.984 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:41.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:42.085 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:42.110 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:42.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:42.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:42.212 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:42.237 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:42.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:42.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:42.363 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:42.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:42.465 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:42.492 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:42.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:42.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:42.594 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:42.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:42.634 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:42.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:42.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:42 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:42.664 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:42 smithi204 ceph-mon[33153]: 6.15 scrub starts 2024-03-21T13:03:42.664 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:42 smithi204 ceph-mon[33153]: 6.15 scrub ok 2024-03-21T13:03:42.664 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:42 smithi204 ceph-mon[33153]: osdmap e147: 8 total, 8 up, 6 in 2024-03-21T13:03:42.734 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:42.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:42 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:42.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:42 smithi163 ceph-mon[29621]: 6.15 scrub starts 2024-03-21T13:03:42.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:42 smithi163 ceph-mon[29621]: 6.15 scrub ok 2024-03-21T13:03:42.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:42 smithi163 ceph-mon[29621]: osdmap e147: 8 total, 8 up, 6 in 2024-03-21T13:03:42.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:42 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:42.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:42 smithi163 ceph-mon[40675]: 6.15 scrub starts 2024-03-21T13:03:42.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:42 smithi163 ceph-mon[40675]: 6.15 scrub ok 2024-03-21T13:03:42.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:42 smithi163 ceph-mon[40675]: osdmap e147: 8 total, 8 up, 6 in 2024-03-21T13:03:42.775 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:42.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:42.877 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:42.917 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:42.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:43.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:43.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:43.057 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:43.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:43.158 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:43.186 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:43.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:43.287 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:43.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:43.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:43.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:43.342 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:03:43.343 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:03:43.417 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:43.455 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:43.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:43.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:43.557 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:43.599 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:43.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:43.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:43.673 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:43 smithi204 ceph-mon[33153]: pgmap v203: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 2.7 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 4914/131184 objects misplaced (3.746%); 11 MiB/s, 179 objects/s recovering 2024-03-21T13:03:43.673 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:43 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:03:43.674 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:43 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:03:43.701 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:43.738 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:43.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:43.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:43 smithi163 ceph-mon[29621]: pgmap v203: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 2.7 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 4914/131184 objects misplaced (3.746%); 11 MiB/s, 179 objects/s recovering 2024-03-21T13:03:43.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:43 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:03:43.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:43 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:03:43.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:43 smithi163 ceph-mon[40675]: pgmap v203: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 2.7 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 4914/131184 objects misplaced (3.746%); 11 MiB/s, 179 objects/s recovering 2024-03-21T13:03:43.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:43 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:03:43.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:43 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:03:43.841 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:43.878 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:43.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:43.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:43.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:44.013 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:44.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:44.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:44.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:44.148 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:44.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:44.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:44.172 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-21T13:03:44.250 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:44.287 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:44.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:44.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:44.389 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:44.425 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:44.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:44.527 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:44.550 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:44.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:44.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:44.552 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:03:44.652 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:44.675 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:44 smithi204 ceph-mon[33153]: osdmap e148: 8 total, 8 up, 6 in 2024-03-21T13:03:44.675 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:44 smithi204 ceph-mon[33153]: 5.10 scrub starts 2024-03-21T13:03:44.675 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:44 smithi204 ceph-mon[33153]: 5.10 scrub ok 2024-03-21T13:03:44.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:44.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:44.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:44 smithi163 ceph-mon[40675]: osdmap e148: 8 total, 8 up, 6 in 2024-03-21T13:03:44.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:44 smithi163 ceph-mon[40675]: 5.10 scrub starts 2024-03-21T13:03:44.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:44 smithi163 ceph-mon[40675]: 5.10 scrub ok 2024-03-21T13:03:44.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:44 smithi163 ceph-mon[29621]: osdmap e148: 8 total, 8 up, 6 in 2024-03-21T13:03:44.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:44 smithi163 ceph-mon[29621]: 5.10 scrub starts 2024-03-21T13:03:44.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:44 smithi163 ceph-mon[29621]: 5.10 scrub ok 2024-03-21T13:03:44.792 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:44.820 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:44.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:44.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:44.922 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:44.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:45.062 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:45.100 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:45.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:45.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:45.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:45.239 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:45.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:45.340 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:45.363 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:45.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:45.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:45.465 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:45.504 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:45.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:45.605 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:45.644 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:45.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:45.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:45.722 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:45 smithi204 ceph-mon[33153]: 5.c scrub starts 2024-03-21T13:03:45.722 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:45 smithi204 ceph-mon[33153]: 5.c scrub ok 2024-03-21T13:03:45.722 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:45 smithi204 ceph-mon[33153]: pgmap v205: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 2.5 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 1.39k op/s; 4769/124296 objects misplaced (3.837%); 9.1 MiB/s, 145 objects/s recovering 2024-03-21T13:03:45.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:45 smithi163 ceph-mon[40675]: 5.c scrub starts 2024-03-21T13:03:45.726 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:45 smithi163 ceph-mon[40675]: 5.c scrub ok 2024-03-21T13:03:45.726 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:45 smithi163 ceph-mon[40675]: pgmap v205: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 2.5 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 1.39k op/s; 4769/124296 objects misplaced (3.837%); 9.1 MiB/s, 145 objects/s recovering 2024-03-21T13:03:45.726 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:03:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:03:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:03:45.726 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:45 smithi163 ceph-mon[29621]: 5.c scrub starts 2024-03-21T13:03:45.726 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:45 smithi163 ceph-mon[29621]: 5.c scrub ok 2024-03-21T13:03:45.726 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:45 smithi163 ceph-mon[29621]: pgmap v205: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 2.5 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 1.39k op/s; 4769/124296 objects misplaced (3.837%); 9.1 MiB/s, 145 objects/s recovering 2024-03-21T13:03:45.746 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:45.790 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:45.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:45.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:45.892 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:45.932 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:45.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:46.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:46.064 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:46.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:46.165 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:46.189 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:46.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:46.290 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:46.335 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:46.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:46.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:46.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:46.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:46.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:46.564 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:03:46.564 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:03:46.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:46.603 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:46.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:46.705 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:46.739 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:46.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:46.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:46 smithi204 ceph-mon[33153]: 2.0 scrub starts 2024-03-21T13:03:46.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:46 smithi204 ceph-mon[33153]: 2.0 scrub ok 2024-03-21T13:03:46.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:46 smithi163 ceph-mon[29621]: 2.0 scrub starts 2024-03-21T13:03:46.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:46 smithi163 ceph-mon[29621]: 2.0 scrub ok 2024-03-21T13:03:46.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:46 smithi163 ceph-mon[40675]: 2.0 scrub starts 2024-03-21T13:03:46.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:46 smithi163 ceph-mon[40675]: 2.0 scrub ok 2024-03-21T13:03:46.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:46.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:46.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:46.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:46.977 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:47.005 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:47.061 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:47.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:47.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:47.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:47.196 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:47.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:47.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:47.298 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:47.352 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:47.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:47.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:47.454 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:47.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:47.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:47.536 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:47 smithi163 ceph-mon[29621]: pgmap v206: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.4 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 702 op/s; 3187/118545 objects misplaced (2.688%); 7.2 MiB/s, 114 objects/s recovering 2024-03-21T13:03:47.536 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:47 smithi163 ceph-mon[40675]: pgmap v206: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.4 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 702 op/s; 3187/118545 objects misplaced (2.688%); 7.2 MiB/s, 114 objects/s recovering 2024-03-21T13:03:47.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:47.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:47.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:47.681 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:47 smithi204 ceph-mon[33153]: pgmap v206: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.4 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 702 op/s; 3187/118545 objects misplaced (2.688%); 7.2 MiB/s, 114 objects/s recovering 2024-03-21T13:03:47.721 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:47.725 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-21T13:03:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:47.763 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:47.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:47.864 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:47.911 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:47.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:48.012 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:48.062 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:48.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:48.163 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:48.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:48.215 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:48.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:48.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:48.349 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:48.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:48.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:48.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:48.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:48.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:48.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:48.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:48.615 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:48.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:48.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:48.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:48.750 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:48.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:48.851 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:48.902 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:48.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:49.004 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:49.031 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:49.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:49.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:49.122 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:03:49.134 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:49.161 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:03:49.177 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:49.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:49.198 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:03:49.241 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:03:49.279 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:49.289 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:03:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:49.335 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:49.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:49.344 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:03:49.385 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:03:49.423 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:03:49.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:03:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:03:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:03:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:03:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:03:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:03:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:03:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:03:49.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:49.467 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:49.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:49.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:49.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:49.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:49.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:49.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:49.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:49.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:49.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:49.780 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:49.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:49.860 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:49.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:49.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:49.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:49 smithi163 ceph-mon[29621]: 4.13 scrub starts 2024-03-21T13:03:49.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:49 smithi163 ceph-mon[29621]: 4.13 scrub ok 2024-03-21T13:03:49.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:49 smithi163 ceph-mon[29621]: pgmap v207: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.4 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 684 B/s rd, 0 B/s wr, 563 op/s; 3187/118545 objects misplaced (2.688%); 5.7 MiB/s, 92 objects/s recovering 2024-03-21T13:03:49.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:49 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:49.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:49 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:49.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:49 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/536456019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:03:49.930 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:49 smithi163 ceph-mon[40675]: 4.13 scrub starts 2024-03-21T13:03:49.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:49 smithi163 ceph-mon[40675]: 4.13 scrub ok 2024-03-21T13:03:49.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:49 smithi163 ceph-mon[40675]: pgmap v207: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.4 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 684 B/s rd, 0 B/s wr, 563 op/s; 3187/118545 objects misplaced (2.688%); 5.7 MiB/s, 92 objects/s recovering 2024-03-21T13:03:49.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:49 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:49.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:49 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:49.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:49 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/536456019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:03:49.962 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:50.009 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:50.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:50.111 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:50.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:50.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:50.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:50.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:49 smithi204 ceph-mon[33153]: 4.13 scrub starts 2024-03-21T13:03:50.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:49 smithi204 ceph-mon[33153]: 4.13 scrub ok 2024-03-21T13:03:50.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:49 smithi204 ceph-mon[33153]: pgmap v207: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.4 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 684 B/s rd, 0 B/s wr, 563 op/s; 3187/118545 objects misplaced (2.688%); 5.7 MiB/s, 92 objects/s recovering 2024-03-21T13:03:50.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:49 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:03:50.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:49 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:50.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:49 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/536456019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:03:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:50.294 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:50.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:50.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:50.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:50.435 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:50.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:50.536 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:50.577 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:50.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:50.678 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:50.715 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:50.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:50.798 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:03:50.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:50.859 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:50.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:50.960 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:51.004 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:51.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:51.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:50 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2422063363' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:03:51.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:50 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:03:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:50 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2422063363' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:03:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:50 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:03:51.106 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:51.163 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:51.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:51.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:50 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2422063363' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:03:51.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:50 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:03:51.265 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:51.300 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:51.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:51.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:51.402 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:51.439 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:51.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:51.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:51.571 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:51.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:51.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:51.674 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:51.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:51.712 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:51.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:51.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:03:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:03:51.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:51.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:51.847 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:51.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:51.948 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:51.962 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:03:51.962 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:03:51.962 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-03-21T13:03:51.962 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-21T13:03:51.963 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-21T13:03:51.963 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-03-21T13:03:52.005 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:52.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:51 smithi163 ceph-mon[29621]: pgmap v208: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.0 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 2088/100620 objects misplaced (2.075%); 7.5 MiB/s, 119 objects/s recovering 2024-03-21T13:03:52.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:51 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:03:52.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:51 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:03:52.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:51 smithi163 ceph-mon[29621]: osdmap e149: 8 total, 8 up, 6 in 2024-03-21T13:03:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:52.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:52.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:51 smithi163 ceph-mon[40675]: pgmap v208: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.0 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 2088/100620 objects misplaced (2.075%); 7.5 MiB/s, 119 objects/s recovering 2024-03-21T13:03:52.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:51 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:03:52.008 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:51 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:03:52.008 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:51 smithi163 ceph-mon[40675]: osdmap e149: 8 total, 8 up, 6 in 2024-03-21T13:03:52.060 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:51 smithi204 ceph-mon[33153]: pgmap v208: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.0 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 2088/100620 objects misplaced (2.075%); 7.5 MiB/s, 119 objects/s recovering 2024-03-21T13:03:52.060 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:51 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:03:52.061 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:51 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:03:52.061 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:51 smithi204 ceph-mon[33153]: osdmap e149: 8 total, 8 up, 6 in 2024-03-21T13:03:52.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:52.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:52.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:52.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:52.238 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:52.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:52.266 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:52.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:52.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:52.398 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:52.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:52.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:52.500 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:52.530 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:52.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:52.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:52.632 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:52.689 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:52.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:52.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:52.844 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:52.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:52.889 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:52 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:52.889 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:52 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:52.946 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:52.954 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:52.996 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:52.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:52.997 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:03:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:03:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:205235 7.f e149/110 7:fec47def:::benchmark_data_smithi163_80312_object87794:head v 149'12898, mlcod=149'12896)", 2024-03-21T13:03:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:52.992326+0000", 2024-03-21T13:03:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.0023734509999999999, 2024-03-21T13:03:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024110799999999999, 2024-03-21T13:03:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:03:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:03:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:03:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:03:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992326+0000", 2024-03-21T13:03:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:03:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:03:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992326+0000", 2024-03-21T13:03:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:03:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:03:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992328+0000", 2024-03-21T13:03:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5230000000000001e-06 2024-03-21T13:03:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:03:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992336+0000", 2024-03-21T13:03:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3410000000000003e-06 2024-03-21T13:03:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:03:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992352+0000", 2024-03-21T13:03:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6011999999999998e-05 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992361+0000", 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3380000000000005e-06 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992569+0000", 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020862699999999999 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:03:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992645+0000", 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5945000000000006e-05 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:205241 7.f e149/110)", 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:52.992384+0000", 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.0023161599999999998, 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00241398, 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "queued for pg", 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:03:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992384+0000", 2024-03-21T13:03:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:03:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:03:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992384+0000", 2024-03-21T13:03:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:03:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:03:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992384+0000", 2024-03-21T13:03:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2899999999999999e-07 2024-03-21T13:03:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992388+0000", 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2250000000000002e-06 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992394+0000", 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6790000000000002e-06 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992398+0000", 2024-03-21T13:03:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6280000000000004e-06 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:205245 7.d e149/137 7:b982f89f:::benchmark_data_smithi163_80312_object87804:head v 149'12633, mlcod=149'12629)", 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:52.992435+0000", 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.002265282, 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024175400000000001, 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:03:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992435+0000", 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992435+0000", 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992435+0000", 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0200000000000002e-07 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:03:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992441+0000", 2024-03-21T13:03:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9880000000000001e-06 2024-03-21T13:03:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:03:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992449+0000", 2024-03-21T13:03:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7020000000000001e-06 2024-03-21T13:03:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:03:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992455+0000", 2024-03-21T13:03:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3670000000000002e-06 2024-03-21T13:03:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992811+0000", 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035587699999999999 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:52.992847+0000", 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6226000000000002e-05 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:03:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:03:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 3 2024-03-21T13:03:53.008 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:03:53.008 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:52 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:53.097 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:53.141 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:53.141 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:03:53.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:53.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:53.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:53.272 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:53.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:53.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:53.374 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:53.421 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:53.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:53.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:53.531 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:53.567 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:53.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:53.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:53.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:53.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:53.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:53.725 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-21T13:03:53.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:53.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:53.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:53.937 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:53.983 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:53.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:54.068 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:53 smithi163 ceph-mon[29621]: pgmap v210: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.0 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s; 2088/100620 objects misplaced (2.075%); 6.8 MiB/s, 109 objects/s recovering 2024-03-21T13:03:54.068 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:53 smithi163 ceph-mon[40675]: pgmap v210: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.0 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s; 2088/100620 objects misplaced (2.075%); 6.8 MiB/s, 109 objects/s recovering 2024-03-21T13:03:54.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:54.137 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:54.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:54.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:54.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:54.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:53 smithi204 ceph-mon[33153]: pgmap v210: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.0 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s; 2088/100620 objects misplaced (2.075%); 6.8 MiB/s, 109 objects/s recovering 2024-03-21T13:03:54.272 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:54.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:54.364 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:03:54.374 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:54.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:54.463 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:54.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:54.564 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:54.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:54.619 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:54.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:54.720 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:54.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:54.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:54.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:54.853 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:54.891 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:54.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:54.993 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:55.033 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:55.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:55.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:55.134 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:55.159 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:55.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:55.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:55.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:55.217 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:55.264 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:55.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:55.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:55.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:55.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:55.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:55.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:55.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:55.585 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:55.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:55.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:55.687 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:55.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:55.737 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:55.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:55.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:55.841 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:55 smithi163 ceph-mon[29621]: pgmap v211: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 1.8 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 1905/90660 objects misplaced (2.101%); 6.3 MiB/s, 101 objects/s recovering 2024-03-21T13:03:55.841 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:55 smithi163 ceph-mon[40675]: pgmap v211: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 1.8 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 1905/90660 objects misplaced (2.101%); 6.3 MiB/s, 101 objects/s recovering 2024-03-21T13:03:55.841 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:03:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:03:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:03:55.866 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:55.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:55.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:55.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:55.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:55.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:56.098 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:56.136 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:56.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:56.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:56.237 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:56.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:55 smithi204 ceph-mon[33153]: pgmap v211: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 1.8 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 1905/90660 objects misplaced (2.101%); 6.3 MiB/s, 101 objects/s recovering 2024-03-21T13:03:56.282 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:56.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:56.384 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:56.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:56.422 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:56.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:56.523 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:56.558 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:56.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:56.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:56.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:56.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:56.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:56.793 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:56.822 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:56.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:56.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:56.924 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:56.966 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:56.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:56.984 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:03:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-21T13:03:56.984 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:03:57.067 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:57.104 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:57.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:57.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:57.236 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:57.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:57.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:57.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:57.370 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:57.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:57.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:57.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:57.501 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:57.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:57.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:57.602 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:57.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:57.742 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:57.781 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:57.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:57.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:03:57.911 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:57.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:58.013 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:58.036 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:58.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:58.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:03:58.163 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:58.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:58.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:03:58.165 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:57 smithi163 ceph-mon[29621]: pgmap v212: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 146 active+clean; 1.7 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s; 1105/82533 objects misplaced (1.339%); 4.2 MiB/s, 66 objects/s recovering 2024-03-21T13:03:58.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:57 smithi163 ceph-mon[40675]: pgmap v212: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 146 active+clean; 1.7 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s; 1105/82533 objects misplaced (1.339%); 4.2 MiB/s, 66 objects/s recovering 2024-03-21T13:03:58.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:57 smithi204 ceph-mon[33153]: pgmap v212: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 146 active+clean; 1.7 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s; 1105/82533 objects misplaced (1.339%); 4.2 MiB/s, 66 objects/s recovering 2024-03-21T13:03:58.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:58.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:58.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:58.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:58.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:58.418 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:58.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:58.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:58.519 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:58.546 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:58.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:58.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:58.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:58.676 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:58.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:58.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:58.778 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:03:58.806 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:58.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:58.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:03:58.824 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:03:58.908 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:03:58.937 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:58.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:03:59.039 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:59.073 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:59.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:59.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:59.175 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:03:59.213 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:59.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:59.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:03:59.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:58 smithi163 ceph-mon[29621]: osdmap e150: 8 total, 8 up, 6 in 2024-03-21T13:03:59.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:58 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1734611102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:59.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:58 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:59.239 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:58 smithi163 ceph-mon[40675]: osdmap e150: 8 total, 8 up, 6 in 2024-03-21T13:03:59.239 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:58 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1734611102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:59.239 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:58 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:59.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:58 smithi204 ceph-mon[33153]: osdmap e150: 8 total, 8 up, 6 in 2024-03-21T13:03:59.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:58 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1734611102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:59.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:58 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:03:59.315 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:03:59.345 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:59.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:03:59.447 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:03:59.484 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:03:59.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:59.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:03:59.586 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:03:59.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:59.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:03:59.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:59.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:59.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:59.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:59.850 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:03:59.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:03:59.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:03:59.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:03:59.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:00.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:59 smithi163 ceph-mon[29621]: pgmap v214: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 146 active+clean; 1.7 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 753 op/s; 1105/82533 objects misplaced (1.339%); 2.6 MiB/s, 42 objects/s recovering 2024-03-21T13:04:00.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:59 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:04:00.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:59 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:04:00.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:03:59 smithi163 ceph-mon[29621]: osdmap e151: 8 total, 8 up, 6 in 2024-03-21T13:04:00.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:59 smithi163 ceph-mon[40675]: pgmap v214: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 146 active+clean; 1.7 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 753 op/s; 1105/82533 objects misplaced (1.339%); 2.6 MiB/s, 42 objects/s recovering 2024-03-21T13:04:00.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:59 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:04:00.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:59 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:04:00.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:03:59 smithi163 ceph-mon[40675]: osdmap e151: 8 total, 8 up, 6 in 2024-03-21T13:04:00.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:00.015 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:00.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:00.116 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:00.150 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:00.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:00.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:00.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:00.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:59 smithi204 ceph-mon[33153]: pgmap v214: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 146 active+clean; 1.7 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 753 op/s; 1105/82533 objects misplaced (1.339%); 2.6 MiB/s, 42 objects/s recovering 2024-03-21T13:04:00.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:59 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:04:00.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:59 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:04:00.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:03:59 smithi204 ceph-mon[33153]: osdmap e151: 8 total, 8 up, 6 in 2024-03-21T13:04:00.294 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:00.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:00.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:00.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:00.428 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:00.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:00.530 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:00.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:00.568 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:00.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:00.669 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:00.710 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:00.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:00.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:00.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:00.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:00.845 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:00.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:00.946 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:00.987 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:00.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:01.089 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:01.137 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:01.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:01.239 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:01.279 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:01.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:01.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:01.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:01.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:01.515 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:01.557 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:01.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:01.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:01.658 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:01.709 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:01.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:01.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:01.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:04:01 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:04:01.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:01.857 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:01.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:01.959 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:02.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:02.005 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:02.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:02.106 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:02.166 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:02.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:02.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:02.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:02.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:02.401 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:02.438 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:02.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:02.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:02 smithi163 ceph-mon[29621]: pgmap v216: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.3 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 687/65955 objects misplaced (1.042%); 3.7 MiB/s, 58 objects/s recovering 2024-03-21T13:04:02.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:02 smithi163 ceph-mon[29621]: osdmap e152: 8 total, 8 up, 6 in 2024-03-21T13:04:02.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:02 smithi163 ceph-mon[40675]: pgmap v216: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.3 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 687/65955 objects misplaced (1.042%); 3.7 MiB/s, 58 objects/s recovering 2024-03-21T13:04:02.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:02 smithi163 ceph-mon[40675]: osdmap e152: 8 total, 8 up, 6 in 2024-03-21T13:04:02.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:02 smithi204 ceph-mon[33153]: pgmap v216: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.3 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 687/65955 objects misplaced (1.042%); 3.7 MiB/s, 58 objects/s recovering 2024-03-21T13:04:02.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:02 smithi204 ceph-mon[33153]: osdmap e152: 8 total, 8 up, 6 in 2024-03-21T13:04:02.522 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:04:02.540 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:02.604 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:02.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:02.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:02.721 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:04:02.733 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:02.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:02.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:02.762 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:04:02.807 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:04:02.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:02.861 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:04:02.864 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:02.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:02.903 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:04:02.925 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:04:02.925 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:04:02.926 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-21T13:04:02.926 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-21T13:04:02.926 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:04:02.949 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:04:02.966 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:03.000 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:04:03.009 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:04:03.009 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:04:03.030 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:03.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:03.046 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:04:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:04:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:04:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:04:03.067 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:04:03.067 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:04:03.067 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:04:03.067 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:04:03.067 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:04:03.131 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:03.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:03.178 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:03.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:03.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:03.312 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:03.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:03.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:03.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:03.444 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:03.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:03.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:03.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:03.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:03 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1756403508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:04:03.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:03.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:03 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1756403508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:04:03.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:03.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:03 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1756403508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:04:03.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:03.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:03.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:03.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:03.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:03.713 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:03.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:03.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:03.845 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:03.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:03.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:03.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:03.977 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:03.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:03.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:04.078 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:04.120 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:04.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:04.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:04.222 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:04.258 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:04.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:04.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:04.359 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:04.397 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:04.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:04.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:04.414 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:04 smithi163 ceph-mon[29621]: pgmap v218: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.3 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 921 op/s; 687/65955 objects misplaced (1.042%); 1.4 MiB/s, 21 objects/s recovering 2024-03-21T13:04:04.414 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:04 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:04.414 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:04 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:04.415 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:04 smithi163 ceph-mon[40675]: pgmap v218: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.3 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 921 op/s; 687/65955 objects misplaced (1.042%); 1.4 MiB/s, 21 objects/s recovering 2024-03-21T13:04:04.415 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:04 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:04.415 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:04 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:04.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:04.532 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:04.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:04.633 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:04.672 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:04.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:04.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:04.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:04 smithi204 ceph-mon[33153]: pgmap v218: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.3 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 921 op/s; 687/65955 objects misplaced (1.042%); 1.4 MiB/s, 21 objects/s recovering 2024-03-21T13:04:04.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:04 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:04.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:04 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:04.775 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:04.806 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:04.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:04.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:04.908 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:04.949 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:04.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:04.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:05.051 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:05.083 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:05.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:05.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:05.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-21T13:04:05.173 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:05.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:05.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:05.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:05.272 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:05.315 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:05.358 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:05.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:05.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:05.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:05.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:05.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:05.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:05.625 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:05.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:05.727 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:05.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:05.767 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:05.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:05.841 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:04:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:04:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:04:05.869 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:05.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:05.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:06.011 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:06.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:06.151 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:06.191 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:06.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:06.293 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:06.335 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:06.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:06.436 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:06.462 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:04:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:06.473 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:06.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:06.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:06 smithi163 ceph-mon[29621]: pgmap v219: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.1 GiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 466/54213 objects misplaced (0.860%); 1.2 MiB/s, 19 objects/s recovering 2024-03-21T13:04:06.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:06 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:04:06.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:06 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:04:06.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:06 smithi163 ceph-mon[29621]: osdmap e153: 8 total, 8 up, 6 in 2024-03-21T13:04:06.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:06 smithi163 ceph-mon[29621]: osdmap e154: 8 total, 8 up, 6 in 2024-03-21T13:04:06.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:06 smithi163 ceph-mon[40675]: pgmap v219: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.1 GiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 466/54213 objects misplaced (0.860%); 1.2 MiB/s, 19 objects/s recovering 2024-03-21T13:04:06.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:06 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:04:06.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:06 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:04:06.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:06 smithi163 ceph-mon[40675]: osdmap e153: 8 total, 8 up, 6 in 2024-03-21T13:04:06.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:06 smithi163 ceph-mon[40675]: osdmap e154: 8 total, 8 up, 6 in 2024-03-21T13:04:06.575 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:06.601 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:06.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:06.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:06.703 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:06.729 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:06.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:06.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:06 smithi204 ceph-mon[33153]: pgmap v219: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.1 GiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 466/54213 objects misplaced (0.860%); 1.2 MiB/s, 19 objects/s recovering 2024-03-21T13:04:06.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:06 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:04:06.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:06 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:04:06.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:06 smithi204 ceph-mon[33153]: osdmap e153: 8 total, 8 up, 6 in 2024-03-21T13:04:06.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:06 smithi204 ceph-mon[33153]: osdmap e154: 8 total, 8 up, 6 in 2024-03-21T13:04:06.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:06.860 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:06.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:06.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:06.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:06.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:06.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:07.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:07.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:07.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:07.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:07.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:07.253 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:07.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:07.355 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:07.393 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:07.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:07.495 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:07.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:07.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:07.628 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:07.672 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:07.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:07.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:07.805 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:07.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:07.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:07.907 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:07.952 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:07.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:07.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:08.053 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:08.100 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:08.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:08.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:08.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:08.236 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:08.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:08.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:08.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:08.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:08.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:08.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:08.509 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:08.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:08.611 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:08.657 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:08.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:08 smithi204 ceph-mon[33153]: pgmap v222: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 931 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 466/45264 objects misplaced (1.030%) 2024-03-21T13:04:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:08.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:08 smithi163 ceph-mon[29621]: pgmap v222: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 931 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 466/45264 objects misplaced (1.030%) 2024-03-21T13:04:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:08 smithi163 ceph-mon[40675]: pgmap v222: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 931 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 466/45264 objects misplaced (1.030%) 2024-03-21T13:04:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:08.795 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:08.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:08.897 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:08.939 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:08.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:09.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:09.085 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:09.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:09.186 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:09.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:09.230 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:09.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:09.331 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:09.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:09.387 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:09.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:09.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:09.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:09.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:09.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:09.627 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:09.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:09.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:09.822 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:09.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:09.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:09.924 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:09.982 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:09.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:10.075 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:10.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:10.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:10.155 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:10.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:10.256 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:10.285 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:10.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:10.387 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:10.434 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:10.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:10.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:10 smithi163 ceph-mon[29621]: pgmap v223: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 931 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 684 B/s rd, 0 B/s wr, 922 op/s; 466/45264 objects misplaced (1.030%) 2024-03-21T13:04:10.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:10 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:10.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:10 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:10.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:10 smithi163 ceph-mon[40675]: pgmap v223: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 931 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 684 B/s rd, 0 B/s wr, 922 op/s; 466/45264 objects misplaced (1.030%) 2024-03-21T13:04:10.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:10 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:10.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:10 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:10.536 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:10.593 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:10.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:10.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:10.658 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:10.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:10.722 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:10.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:10.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:10.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:10 smithi204 ceph-mon[33153]: pgmap v223: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 931 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 684 B/s rd, 0 B/s wr, 922 op/s; 466/45264 objects misplaced (1.030%) 2024-03-21T13:04:10.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:10 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:10.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:10 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:10.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:10.854 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:10.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:10.956 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:10.995 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:10.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:11.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:11.126 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:04:11.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:11.129 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:11.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:11.138 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:04:11.230 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:11.253 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:04:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:11.272 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:11.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:11.373 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:11.407 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:11.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:11.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:11.509 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:11.546 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:11.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:11.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:11.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:11.653 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:11 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3765885807' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:04:11.653 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:11 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:04:11.653 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:11 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:04:11.654 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:11 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:04:11.654 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:11 smithi204 ceph-mon[33153]: osdmap e155: 8 total, 8 up, 6 in 2024-03-21T13:04:11.654 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:04:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:04:11.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:11.681 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:11.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:11 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3765885807' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:04:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:11 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:04:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:11 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:04:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:11 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:04:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:11 smithi163 ceph-mon[29621]: osdmap e155: 8 total, 8 up, 6 in 2024-03-21T13:04:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:11 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3765885807' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:04:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:11 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:04:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:11 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:04:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:11 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:04:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:11 smithi163 ceph-mon[40675]: osdmap e155: 8 total, 8 up, 6 in 2024-03-21T13:04:11.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:11.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:11.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:11.916 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:11.959 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:11.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:11.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:12.061 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:12.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:12.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:12.202 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:12.240 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:12.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:12.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:12.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:12.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:12.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:12.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:12.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:12.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:12 smithi204 ceph-mon[33153]: pgmap v224: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 532 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.66k op/s; 264/26106 objects misplaced (1.011%) 2024-03-21T13:04:12.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:12 smithi204 ceph-mon[33153]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:04:12.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:12 smithi204 ceph-mon[33153]: 1.0 scrub starts 2024-03-21T13:04:12.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:12 smithi204 ceph-mon[33153]: 1.0 scrub ok 2024-03-21T13:04:12.521 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:12.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:12.623 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:12.662 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:12.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:12.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:12 smithi163 ceph-mon[29621]: pgmap v224: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 532 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.66k op/s; 264/26106 objects misplaced (1.011%) 2024-03-21T13:04:12.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:12 smithi163 ceph-mon[29621]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:04:12.670 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:12 smithi163 ceph-mon[29621]: 1.0 scrub starts 2024-03-21T13:04:12.670 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:12 smithi163 ceph-mon[29621]: 1.0 scrub ok 2024-03-21T13:04:12.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:12 smithi163 ceph-mon[40675]: pgmap v224: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 532 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.66k op/s; 264/26106 objects misplaced (1.011%) 2024-03-21T13:04:12.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:12 smithi163 ceph-mon[40675]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:04:12.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:12 smithi163 ceph-mon[40675]: 1.0 scrub starts 2024-03-21T13:04:12.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:12 smithi163 ceph-mon[40675]: 1.0 scrub ok 2024-03-21T13:04:12.717 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-03-21T13:04:10.721063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":557849024,"num_objects":8702,"num_object_clones":0,"num_object_copies":26106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":264,"num_objects_unfound":0,"num_objects_dirty":8702,"num_whiteouts":0,"num_read":1244,"num_read_kb":1090,"num_write":226989,"num_write_kb":7518494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36079,"num_bytes_recovered":2354190279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65017,"ondisk_log_size":65017,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":2841584,"kb_used_data":1807424,"kb_used_omap":9,"kb_used_meta":1033846,"kb_avail":559629328,"statfs":{"total":575970213888,"available":573060431872,"internally_reserved":0,"allocated":1850802176,"data_stored":1848001634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":1058658982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-870514688,"num_objects":-13283,"num_object_clones":0,"num_object_copies":-39849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-423,"num_objects_unfound":0,"num_objects_dirty":-13283,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13283,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003873"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.237484+0000","last_change":"2024-03-21T13:02:26.223957+0000","last_active":"2024-03-21T13:04:06.237484+0000","last_peered":"2024-03-21T13:04:06.237484+0000","last_clean":"2024-03-21T13:04:06.237484+0000","last_became_active":"2024-03-21T13:02:26.223292+0000","last_became_peered":"2024-03-21T13:02:26.223292+0000","last_unstale":"2024-03-21T13:04:06.237484+0000","last_undegraded":"2024-03-21T13:04:06.237484+0000","last_fullsized":"2024-03-21T13:04:06.237484+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:01:35.694549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.085658+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":82,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243948+0000","last_change":"2024-03-21T13:03:40.962728+0000","last_active":"2024-03-21T13:04:06.243948+0000","last_peered":"2024-03-21T13:04:06.243948+0000","last_clean":"2024-03-21T13:04:06.243948+0000","last_became_active":"2024-03-21T13:02:26.233709+0000","last_became_peered":"2024-03-21T13:02:26.233709+0000","last_unstale":"2024-03-21T13:04:06.243948+0000","last_undegraded":"2024-03-21T13:04:06.243948+0000","last_fullsized":"2024-03-21T13:04:06.243948+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:40.962617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:03:40.962617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:54.846879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":74,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.252010+0000","last_change":"2024-03-21T13:02:26.237862+0000","last_active":"2024-03-21T13:04:06.252010+0000","last_peered":"2024-03-21T13:04:06.252010+0000","last_clean":"2024-03-21T13:04:06.252010+0000","last_became_active":"2024-03-21T13:02:26.237501+0000","last_became_peered":"2024-03-21T13:02:26.237501+0000","last_unstale":"2024-03-21T13:04:06.252010+0000","last_undegraded":"2024-03-21T13:04:06.252010+0000","last_fullsized":"2024-03-21T13:04:06.252010+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:24.141445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:24.141445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"89'13","reported_seq":256,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.237381+0000","last_change":"2024-03-21T13:02:14.370743+0000","last_active":"2024-03-21T13:04:06.237381+0000","last_peered":"2024-03-21T13:04:06.237381+0000","last_clean":"2024-03-21T13:04:06.237381+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:04:06.237381+0000","last_undegraded":"2024-03-21T13:04:06.237381+0000","last_fullsized":"2024-03-21T13:04:06.237381+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:01:37.692932+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.085629+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":188,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.248865+0000","last_change":"2024-03-21T13:00:05.173374+0000","last_active":"2024-03-21T13:04:06.248865+0000","last_peered":"2024-03-21T13:04:06.248865+0000","last_clean":"2024-03-21T13:04:06.248865+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:04:06.248865+0000","last_undegraded":"2024-03-21T13:04:06.248865+0000","last_fullsized":"2024-03-21T13:04:06.248865+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:00:05.173276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367723+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":119,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244835+0000","last_change":"2024-03-21T13:02:14.330937+0000","last_active":"2024-03-21T13:04:06.244835+0000","last_peered":"2024-03-21T13:04:06.244835+0000","last_clean":"2024-03-21T13:04:06.244835+0000","last_became_active":"2024-03-21T13:02:14.330467+0000","last_became_peered":"2024-03-21T13:02:14.330467+0000","last_unstale":"2024-03-21T13:04:06.244835+0000","last_undegraded":"2024-03-21T13:04:06.244835+0000","last_fullsized":"2024-03-21T13:04:06.244835+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:00:16.731434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.685989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":207,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.256487+0000","last_change":"2024-03-21T13:01:38.766164+0000","last_active":"2024-03-21T13:04:06.256487+0000","last_peered":"2024-03-21T13:04:06.256487+0000","last_clean":"2024-03-21T13:04:06.256487+0000","last_became_active":"2024-03-21T12:58:37.728383+0000","last_became_peered":"2024-03-21T12:58:37.728383+0000","last_unstale":"2024-03-21T13:04:06.256487+0000","last_undegraded":"2024-03-21T13:04:06.256487+0000","last_fullsized":"2024-03-21T13:04:06.256487+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_clean_scrub_stamp":"2024-03-21T13:01:38.765896+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.392181+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"71'15","reported_seq":221,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.246400+0000","last_change":"2024-03-21T13:00:31.008873+0000","last_active":"2024-03-21T13:04:06.246400+0000","last_peered":"2024-03-21T13:04:06.246400+0000","last_clean":"2024-03-21T13:04:06.246400+0000","last_became_active":"2024-03-21T12:58:39.750396+0000","last_became_peered":"2024-03-21T12:58:39.750396+0000","last_unstale":"2024-03-21T13:04:06.246400+0000","last_undegraded":"2024-03-21T13:04:06.246400+0000","last_fullsized":"2024-03-21T13:04:06.246400+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:00:31.008733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:31.008733+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.368100+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":86,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243362+0000","last_change":"2024-03-21T13:02:14.336859+0000","last_active":"2024-03-21T13:04:06.243362+0000","last_peered":"2024-03-21T13:04:06.243362+0000","last_clean":"2024-03-21T13:04:06.243362+0000","last_became_active":"2024-03-21T13:02:14.334221+0000","last_became_peered":"2024-03-21T13:02:14.334221+0000","last_unstale":"2024-03-21T13:04:06.243362+0000","last_undegraded":"2024-03-21T13:04:06.243362+0000","last_fullsized":"2024-03-21T13:04:06.243362+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:27.137810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:27.137810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":80,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243931+0000","last_change":"2024-03-21T13:03:37.292304+0000","last_active":"2024-03-21T13:04:06.243931+0000","last_peered":"2024-03-21T13:04:06.243931+0000","last_clean":"2024-03-21T13:04:06.243931+0000","last_became_active":"2024-03-21T13:02:26.236707+0000","last_became_peered":"2024-03-21T13:02:26.236707+0000","last_unstale":"2024-03-21T13:04:06.243931+0000","last_undegraded":"2024-03-21T13:04:06.243931+0000","last_fullsized":"2024-03-21T13:04:06.243931+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:37.292196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:03:37.292196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:56.507676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":72,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.246065+0000","last_change":"2024-03-21T13:02:26.245694+0000","last_active":"2024-03-21T13:04:06.246065+0000","last_peered":"2024-03-21T13:04:06.246065+0000","last_clean":"2024-03-21T13:04:06.246065+0000","last_became_active":"2024-03-21T13:02:26.245402+0000","last_became_peered":"2024-03-21T13:02:26.245402+0000","last_unstale":"2024-03-21T13:04:06.246065+0000","last_undegraded":"2024-03-21T13:04:06.246065+0000","last_fullsized":"2024-03-21T13:04:06.246065+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:22.892585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:22.892585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":190,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245825+0000","last_change":"2024-03-21T13:01:27.370969+0000","last_active":"2024-03-21T13:04:06.245825+0000","last_peered":"2024-03-21T13:04:06.245825+0000","last_clean":"2024-03-21T13:04:06.245825+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:04:06.245825+0000","last_undegraded":"2024-03-21T13:04:06.245825+0000","last_fullsized":"2024-03-21T13:04:06.245825+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.880161+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.382186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":192,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245538+0000","last_change":"2024-03-21T13:02:26.229621+0000","last_active":"2024-03-21T13:04:06.245538+0000","last_peered":"2024-03-21T13:04:06.245538+0000","last_clean":"2024-03-21T13:04:06.245538+0000","last_became_active":"2024-03-21T13:02:26.229397+0000","last_became_peered":"2024-03-21T13:02:26.229397+0000","last_unstale":"2024-03-21T13:04:06.245538+0000","last_undegraded":"2024-03-21T13:04:06.245538+0000","last_fullsized":"2024-03-21T13:04:06.245538+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:32.971665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:32.971665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.392071+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":204,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243692+0000","last_change":"2024-03-21T13:02:14.336087+0000","last_active":"2024-03-21T13:04:06.243692+0000","last_peered":"2024-03-21T13:04:06.243692+0000","last_clean":"2024-03-21T13:04:06.243692+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:04:06.243692+0000","last_undegraded":"2024-03-21T13:04:06.243692+0000","last_fullsized":"2024-03-21T13:04:06.243692+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:17.015065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:17.015065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":204,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238082+0000","last_change":"2024-03-21T13:02:26.230571+0000","last_active":"2024-03-21T13:04:06.238082+0000","last_peered":"2024-03-21T13:04:06.238082+0000","last_clean":"2024-03-21T13:04:06.238082+0000","last_became_active":"2024-03-21T13:02:26.230057+0000","last_became_peered":"2024-03-21T13:02:26.230057+0000","last_unstale":"2024-03-21T13:04:06.238082+0000","last_undegraded":"2024-03-21T13:04:06.238082+0000","last_fullsized":"2024-03-21T13:04:06.238082+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:14.869934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:14.869934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.084961+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":192,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.248583+0000","last_change":"2024-03-21T13:02:14.383093+0000","last_active":"2024-03-21T13:04:06.248583+0000","last_peered":"2024-03-21T13:04:06.248583+0000","last_clean":"2024-03-21T13:04:06.248583+0000","last_became_active":"2024-03-21T13:02:14.349072+0000","last_became_peered":"2024-03-21T13:02:14.349072+0000","last_unstale":"2024-03-21T13:04:06.248583+0000","last_undegraded":"2024-03-21T13:04:06.248583+0000","last_fullsized":"2024-03-21T13:04:06.248583+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T12:59:53.952401+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":202,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238234+0000","last_change":"2024-03-21T13:02:26.227096+0000","last_active":"2024-03-21T13:04:06.238234+0000","last_peered":"2024-03-21T13:04:06.238234+0000","last_clean":"2024-03-21T13:04:06.238234+0000","last_became_active":"2024-03-21T13:02:26.226675+0000","last_became_peered":"2024-03-21T13:02:26.226675+0000","last_unstale":"2024-03-21T13:04:06.238234+0000","last_undegraded":"2024-03-21T13:04:06.238234+0000","last_fullsized":"2024-03-21T13:04:06.238234+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:24.662938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:24.662938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.085042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":90,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238993+0000","last_change":"2024-03-21T13:02:26.232199+0000","last_active":"2024-03-21T13:04:06.238993+0000","last_peered":"2024-03-21T13:04:06.238993+0000","last_clean":"2024-03-21T13:04:06.238993+0000","last_became_active":"2024-03-21T13:02:26.231879+0000","last_became_peered":"2024-03-21T13:02:26.231879+0000","last_unstale":"2024-03-21T13:04:06.238993+0000","last_undegraded":"2024-03-21T13:04:06.238993+0000","last_fullsized":"2024-03-21T13:04:06.238993+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:18.990434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.990434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.084934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":98,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243944+0000","last_change":"2024-03-21T13:01:48.210250+0000","last_active":"2024-03-21T13:04:06.243944+0000","last_peered":"2024-03-21T13:04:06.243944+0000","last_clean":"2024-03-21T13:04:06.243944+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:04:06.243944+0000","last_undegraded":"2024-03-21T13:04:06.243944+0000","last_fullsized":"2024-03-21T13:04:06.243944+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:44.005379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:44.005379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":252,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.251130+0000","last_change":"2024-03-21T13:02:14.385705+0000","last_active":"2024-03-21T13:04:06.251130+0000","last_peered":"2024-03-21T13:04:06.251130+0000","last_clean":"2024-03-21T13:04:06.251130+0000","last_became_active":"2024-03-21T13:02:14.350563+0000","last_became_peered":"2024-03-21T13:02:14.350563+0000","last_unstale":"2024-03-21T13:04:06.251130+0000","last_undegraded":"2024-03-21T13:04:06.251130+0000","last_fullsized":"2024-03-21T13:04:06.251130+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:01:36.603903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:36.603903+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100344+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":194,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.250257+0000","last_change":"2024-03-21T13:02:26.243438+0000","last_active":"2024-03-21T13:04:06.250257+0000","last_peered":"2024-03-21T13:04:06.250257+0000","last_clean":"2024-03-21T13:04:06.250257+0000","last_became_active":"2024-03-21T13:02:26.242482+0000","last_became_peered":"2024-03-21T13:02:26.242482+0000","last_unstale":"2024-03-21T13:04:06.250257+0000","last_undegraded":"2024-03-21T13:04:06.250257+0000","last_fullsized":"2024-03-21T13:04:06.250257+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:35.036335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.036335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.391772+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":113,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.246839+0000","last_change":"2024-03-21T13:01:27.367176+0000","last_active":"2024-03-21T13:04:06.246839+0000","last_peered":"2024-03-21T13:04:06.246839+0000","last_clean":"2024-03-21T13:04:06.246839+0000","last_became_active":"2024-03-21T13:01:27.366846+0000","last_became_peered":"2024-03-21T13:01:27.366846+0000","last_unstale":"2024-03-21T13:04:06.246839+0000","last_undegraded":"2024-03-21T13:04:06.246839+0000","last_fullsized":"2024-03-21T13:04:06.246839+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:50.904419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T12:59:50.904419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":214,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238441+0000","last_change":"2024-03-21T13:02:14.329782+0000","last_active":"2024-03-21T13:04:06.238441+0000","last_peered":"2024-03-21T13:04:06.238441+0000","last_clean":"2024-03-21T13:04:06.238441+0000","last_became_active":"2024-03-21T13:02:14.329156+0000","last_became_peered":"2024-03-21T13:02:14.329156+0000","last_unstale":"2024-03-21T13:04:06.238441+0000","last_undegraded":"2024-03-21T13:04:06.238441+0000","last_fullsized":"2024-03-21T13:04:06.238441+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:27.704194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_clean_scrub_stamp":"2024-03-21T13:01:27.704194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.084907+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'12","reported_seq":255,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.246398+0000","last_change":"2024-03-21T13:02:14.362313+0000","last_active":"2024-03-21T13:04:06.246398+0000","last_peered":"2024-03-21T13:04:06.246398+0000","last_clean":"2024-03-21T13:04:06.246398+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:04:06.246398+0000","last_undegraded":"2024-03-21T13:04:06.246398+0000","last_fullsized":"2024-03-21T13:04:06.246398+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:01:39.755345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.755345+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.392237+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":205,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244188+0000","last_change":"2024-03-21T13:02:26.254057+0000","last_active":"2024-03-21T13:04:06.244188+0000","last_peered":"2024-03-21T13:04:06.244188+0000","last_clean":"2024-03-21T13:04:06.244188+0000","last_became_active":"2024-03-21T13:02:26.237656+0000","last_became_peered":"2024-03-21T13:02:26.237656+0000","last_unstale":"2024-03-21T13:04:06.244188+0000","last_undegraded":"2024-03-21T13:04:06.244188+0000","last_fullsized":"2024-03-21T13:04:06.244188+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:01:22.076398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:22.076398+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":278,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.248022+0000","last_change":"2024-03-21T13:01:30.587787+0000","last_active":"2024-03-21T13:04:06.248022+0000","last_peered":"2024-03-21T13:04:06.248022+0000","last_clean":"2024-03-21T13:04:06.248022+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:04:06.248022+0000","last_undegraded":"2024-03-21T13:04:06.248022+0000","last_fullsized":"2024-03-21T13:04:06.248022+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:30.587660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:30.587660+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":96,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245855+0000","last_change":"2024-03-21T13:02:14.372498+0000","last_active":"2024-03-21T13:04:06.245855+0000","last_peered":"2024-03-21T13:04:06.245855+0000","last_clean":"2024-03-21T13:04:06.245855+0000","last_became_active":"2024-03-21T13:02:14.348666+0000","last_became_peered":"2024-03-21T13:02:14.348666+0000","last_unstale":"2024-03-21T13:04:06.245855+0000","last_undegraded":"2024-03-21T13:04:06.245855+0000","last_fullsized":"2024-03-21T13:04:06.245855+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:01:37.244557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:37.244557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.392042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"71'10","reported_seq":133,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245007+0000","last_change":"2024-03-21T13:01:48.218829+0000","last_active":"2024-03-21T13:04:06.245007+0000","last_peered":"2024-03-21T13:04:06.245007+0000","last_clean":"2024-03-21T13:04:06.245007+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:04:06.245007+0000","last_undegraded":"2024-03-21T13:04:06.245007+0000","last_fullsized":"2024-03-21T13:04:06.245007+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:01:17.088732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:17.088732+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":74,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238639+0000","last_change":"2024-03-21T13:02:26.230240+0000","last_active":"2024-03-21T13:04:06.238639+0000","last_peered":"2024-03-21T13:04:06.238639+0000","last_clean":"2024-03-21T13:04:06.238639+0000","last_became_active":"2024-03-21T13:02:26.229188+0000","last_became_peered":"2024-03-21T13:02:26.229188+0000","last_unstale":"2024-03-21T13:04:06.238639+0000","last_undegraded":"2024-03-21T13:04:06.238639+0000","last_fullsized":"2024-03-21T13:04:06.238639+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:33.251265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:33.251265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.085016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":99,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244813+0000","last_change":"2024-03-21T13:01:48.218869+0000","last_active":"2024-03-21T13:04:06.244813+0000","last_peered":"2024-03-21T13:04:06.244813+0000","last_clean":"2024-03-21T13:04:06.244813+0000","last_became_active":"2024-03-21T13:01:48.215160+0000","last_became_peered":"2024-03-21T13:01:48.215160+0000","last_unstale":"2024-03-21T13:04:06.244813+0000","last_undegraded":"2024-03-21T13:04:06.244813+0000","last_fullsized":"2024-03-21T13:04:06.244813+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:39.968654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.968654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":206,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.248702+0000","last_change":"2024-03-21T13:02:14.344996+0000","last_active":"2024-03-21T13:04:06.248702+0000","last_peered":"2024-03-21T13:04:06.248702+0000","last_clean":"2024-03-21T13:04:06.248702+0000","last_became_active":"2024-03-21T13:02:14.341286+0000","last_became_peered":"2024-03-21T13:02:14.341286+0000","last_unstale":"2024-03-21T13:04:06.248702+0000","last_undegraded":"2024-03-21T13:04:06.248702+0000","last_fullsized":"2024-03-21T13:04:06.248702+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:49.178785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:49.178785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'11","reported_seq":259,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.248080+0000","last_change":"2024-03-21T13:02:26.277298+0000","last_active":"2024-03-21T13:04:06.248080+0000","last_peered":"2024-03-21T13:04:06.248080+0000","last_clean":"2024-03-21T13:04:06.248080+0000","last_became_active":"2024-03-21T13:02:26.254436+0000","last_became_peered":"2024-03-21T13:02:26.254436+0000","last_unstale":"2024-03-21T13:04:06.248080+0000","last_undegraded":"2024-03-21T13:04:06.248080+0000","last_fullsized":"2024-03-21T13:04:06.248080+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:01:48.562468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:48.562468+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686849+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":194,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243742+0000","last_change":"2024-03-21T13:01:35.005181+0000","last_active":"2024-03-21T13:04:06.243742+0000","last_peered":"2024-03-21T13:04:06.243742+0000","last_clean":"2024-03-21T13:04:06.243742+0000","last_became_active":"2024-03-21T12:58:43.183475+0000","last_became_peered":"2024-03-21T12:58:43.183475+0000","last_unstale":"2024-03-21T13:04:06.243742+0000","last_undegraded":"2024-03-21T13:04:06.243742+0000","last_fullsized":"2024-03-21T13:04:06.243742+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:01:35.005026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":200,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245392+0000","last_change":"2024-03-21T13:01:20.591831+0000","last_active":"2024-03-21T13:04:06.245392+0000","last_peered":"2024-03-21T13:04:06.245392+0000","last_clean":"2024-03-21T13:04:06.245392+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:04:06.245392+0000","last_undegraded":"2024-03-21T13:04:06.245392+0000","last_fullsized":"2024-03-21T13:04:06.245392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.591636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.591636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":74,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.250484+0000","last_change":"2024-03-21T13:02:26.243710+0000","last_active":"2024-03-21T13:04:06.250484+0000","last_peered":"2024-03-21T13:04:06.250484+0000","last_clean":"2024-03-21T13:04:06.250484+0000","last_became_active":"2024-03-21T13:02:26.242973+0000","last_became_peered":"2024-03-21T13:02:26.242973+0000","last_unstale":"2024-03-21T13:04:06.250484+0000","last_undegraded":"2024-03-21T13:04:06.250484+0000","last_fullsized":"2024-03-21T13:04:06.250484+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:32.789993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.789993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.368152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":214,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.248790+0000","last_change":"2024-03-21T13:02:14.369668+0000","last_active":"2024-03-21T13:04:06.248790+0000","last_peered":"2024-03-21T13:04:06.248790+0000","last_clean":"2024-03-21T13:04:06.248790+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:04:06.248790+0000","last_undegraded":"2024-03-21T13:04:06.248790+0000","last_fullsized":"2024-03-21T13:04:06.248790+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-21T13:01:30.807979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:30.807979+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.392103+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":188,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.249215+0000","last_change":"2024-03-21T13:00:10.110413+0000","last_active":"2024-03-21T13:04:06.249215+0000","last_peered":"2024-03-21T13:04:06.249215+0000","last_clean":"2024-03-21T13:04:06.249215+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:04:06.249215+0000","last_undegraded":"2024-03-21T13:04:06.249215+0000","last_fullsized":"2024-03-21T13:04:06.249215+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:10.110335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.110335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":198,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243965+0000","last_change":"2024-03-21T13:03:43.905627+0000","last_active":"2024-03-21T13:04:06.243965+0000","last_peered":"2024-03-21T13:04:06.243965+0000","last_clean":"2024-03-21T13:04:06.243965+0000","last_became_active":"2024-03-21T12:58:41.164784+0000","last_became_peered":"2024-03-21T12:58:41.164784+0000","last_unstale":"2024-03-21T13:04:06.243965+0000","last_undegraded":"2024-03-21T13:04:06.243965+0000","last_fullsized":"2024-03-21T13:04:06.243965+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:43.905509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:03:43.905509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:05:04.582225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":196,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244694+0000","last_change":"2024-03-21T13:02:26.236549+0000","last_active":"2024-03-21T13:04:06.244694+0000","last_peered":"2024-03-21T13:04:06.244694+0000","last_clean":"2024-03-21T13:04:06.244694+0000","last_became_active":"2024-03-21T13:02:26.236235+0000","last_became_peered":"2024-03-21T13:02:26.236235+0000","last_unstale":"2024-03-21T13:04:06.244694+0000","last_undegraded":"2024-03-21T13:04:06.244694+0000","last_fullsized":"2024-03-21T13:04:06.244694+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:02:21.087308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:02:21.087308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"71'11","reported_seq":215,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.246291+0000","last_change":"2024-03-21T12:59:52.181828+0000","last_active":"2024-03-21T13:04:06.246291+0000","last_peered":"2024-03-21T13:04:06.246291+0000","last_clean":"2024-03-21T13:04:06.246291+0000","last_became_active":"2024-03-21T12:58:39.753014+0000","last_became_peered":"2024-03-21T12:58:39.753014+0000","last_unstale":"2024-03-21T13:04:06.246291+0000","last_undegraded":"2024-03-21T13:04:06.246291+0000","last_fullsized":"2024-03-21T13:04:06.246291+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T12:59:52.181697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.181697+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":95,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244716+0000","last_change":"2024-03-21T13:02:26.253063+0000","last_active":"2024-03-21T13:04:06.244716+0000","last_peered":"2024-03-21T13:04:06.244716+0000","last_clean":"2024-03-21T13:04:06.244716+0000","last_became_active":"2024-03-21T13:02:26.252488+0000","last_became_peered":"2024-03-21T13:02:26.252488+0000","last_unstale":"2024-03-21T13:04:06.244716+0000","last_undegraded":"2024-03-21T13:04:06.244716+0000","last_fullsized":"2024-03-21T13:04:06.244716+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:22.083890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:22.083890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":204,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.248840+0000","last_change":"2024-03-21T13:02:26.233084+0000","last_active":"2024-03-21T13:04:06.248840+0000","last_peered":"2024-03-21T13:04:06.248840+0000","last_clean":"2024-03-21T13:04:06.248840+0000","last_became_active":"2024-03-21T13:02:26.232682+0000","last_became_peered":"2024-03-21T13:02:26.232682+0000","last_unstale":"2024-03-21T13:04:06.248840+0000","last_undegraded":"2024-03-21T13:04:06.248840+0000","last_fullsized":"2024-03-21T13:04:06.248840+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:38.995215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.995215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.381933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":206,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244215+0000","last_change":"2024-03-21T13:01:43.982673+0000","last_active":"2024-03-21T13:04:06.244215+0000","last_peered":"2024-03-21T13:04:06.244215+0000","last_clean":"2024-03-21T13:04:06.244215+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:04:06.244215+0000","last_undegraded":"2024-03-21T13:04:06.244215+0000","last_fullsized":"2024-03-21T13:04:06.244215+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:43.982553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:43.982553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"86'13","reported_seq":163,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245743+0000","last_change":"2024-03-21T13:02:14.351413+0000","last_active":"2024-03-21T13:04:06.245743+0000","last_peered":"2024-03-21T13:04:06.245743+0000","last_clean":"2024-03-21T13:04:06.245743+0000","last_became_active":"2024-03-21T13:02:14.330196+0000","last_became_peered":"2024-03-21T13:02:14.330196+0000","last_unstale":"2024-03-21T13:04:06.245743+0000","last_undegraded":"2024-03-21T13:04:06.245743+0000","last_fullsized":"2024-03-21T13:04:06.245743+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:01:14.027717+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:01:14.027717+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":33,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243357+0000","last_change":"2024-03-21T13:03:37.387568+0000","last_active":"2024-03-21T13:04:06.243357+0000","last_peered":"2024-03-21T13:04:06.243357+0000","last_clean":"2024-03-21T13:04:06.243357+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:04:06.243357+0000","last_undegraded":"2024-03-21T13:04:06.243357+0000","last_fullsized":"2024-03-21T13:04:06.243357+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:50.729022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:50.729022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":74,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245914+0000","last_change":"2024-03-21T13:02:26.242867+0000","last_active":"2024-03-21T13:04:06.245914+0000","last_peered":"2024-03-21T13:04:06.245914+0000","last_clean":"2024-03-21T13:04:06.245914+0000","last_became_active":"2024-03-21T13:02:26.242491+0000","last_became_peered":"2024-03-21T13:02:26.242491+0000","last_unstale":"2024-03-21T13:04:06.245914+0000","last_undegraded":"2024-03-21T13:04:06.245914+0000","last_fullsized":"2024-03-21T13:04:06.245914+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.029629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.029629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.391932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238174+0000","last_change":"2024-03-21T13:01:20.648033+0000","last_active":"2024-03-21T13:04:06.238174+0000","last_peered":"2024-03-21T13:04:06.238174+0000","last_clean":"2024-03-21T13:04:06.238174+0000","last_became_active":"2024-03-21T12:58:37.731659+0000","last_became_peered":"2024-03-21T12:58:37.731659+0000","last_unstale":"2024-03-21T13:04:06.238174+0000","last_undegraded":"2024-03-21T13:04:06.238174+0000","last_fullsized":"2024-03-21T13:04:06.238174+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.647931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.647931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.085068+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":259,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238039+0000","last_change":"2024-03-21T13:03:48.009391+0000","last_active":"2024-03-21T13:04:06.238039+0000","last_peered":"2024-03-21T13:04:06.238039+0000","last_clean":"2024-03-21T13:04:06.238039+0000","last_became_active":"2024-03-21T13:02:26.225627+0000","last_became_peered":"2024-03-21T13:02:26.225627+0000","last_unstale":"2024-03-21T13:04:06.238039+0000","last_undegraded":"2024-03-21T13:04:06.238039+0000","last_fullsized":"2024-03-21T13:04:06.238039+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:03:48.009227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:03:48.009227+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:55.474153+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":98,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244692+0000","last_change":"2024-03-21T13:01:48.215368+0000","last_active":"2024-03-21T13:04:06.244692+0000","last_peered":"2024-03-21T13:04:06.244692+0000","last_clean":"2024-03-21T13:04:06.244692+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:04:06.244692+0000","last_undegraded":"2024-03-21T13:04:06.244692+0000","last_fullsized":"2024-03-21T13:04:06.244692+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.062842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.062842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"125'10","reported_seq":276,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245861+0000","last_change":"2024-03-21T13:02:26.247826+0000","last_active":"2024-03-21T13:04:06.245861+0000","last_peered":"2024-03-21T13:04:06.245861+0000","last_clean":"2024-03-21T13:04:06.245861+0000","last_became_active":"2024-03-21T13:02:26.234494+0000","last_became_peered":"2024-03-21T13:02:26.234494+0000","last_unstale":"2024-03-21T13:04:06.245861+0000","last_undegraded":"2024-03-21T13:04:06.245861+0000","last_fullsized":"2024-03-21T13:04:06.245861+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:23.867564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:23.867564+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":206,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243840+0000","last_change":"2024-03-21T13:01:45.972893+0000","last_active":"2024-03-21T13:04:06.243840+0000","last_peered":"2024-03-21T13:04:06.243840+0000","last_clean":"2024-03-21T13:04:06.243840+0000","last_became_active":"2024-03-21T12:58:37.731988+0000","last_became_peered":"2024-03-21T12:58:37.731988+0000","last_unstale":"2024-03-21T13:04:06.243840+0000","last_undegraded":"2024-03-21T13:04:06.243840+0000","last_fullsized":"2024-03-21T13:04:06.243840+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:45.972809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:45.972809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"71'10","reported_seq":229,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.248482+0000","last_change":"2024-03-21T13:02:14.389576+0000","last_active":"2024-03-21T13:04:06.248482+0000","last_peered":"2024-03-21T13:04:06.248482+0000","last_clean":"2024-03-21T13:04:06.248482+0000","last_became_active":"2024-03-21T13:02:14.353938+0000","last_became_peered":"2024-03-21T13:02:14.353938+0000","last_unstale":"2024-03-21T13:04:06.248482+0000","last_undegraded":"2024-03-21T13:04:06.248482+0000","last_fullsized":"2024-03-21T13:04:06.248482+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:00:06.197903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:06.197903+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.368033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":202,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238836+0000","last_change":"2024-03-21T13:02:26.233510+0000","last_active":"2024-03-21T13:04:06.238836+0000","last_peered":"2024-03-21T13:04:06.238836+0000","last_clean":"2024-03-21T13:04:06.238836+0000","last_became_active":"2024-03-21T13:02:26.233049+0000","last_became_peered":"2024-03-21T13:02:26.233049+0000","last_unstale":"2024-03-21T13:04:06.238836+0000","last_undegraded":"2024-03-21T13:04:06.238836+0000","last_fullsized":"2024-03-21T13:04:06.238836+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:01:36.699469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.085696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":267,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.248632+0000","last_change":"2024-03-21T13:01:18.526684+0000","last_active":"2024-03-21T13:04:06.248632+0000","last_peered":"2024-03-21T13:04:06.248632+0000","last_clean":"2024-03-21T13:04:06.248632+0000","last_became_active":"2024-03-21T12:58:41.167826+0000","last_became_peered":"2024-03-21T12:58:41.167826+0000","last_unstale":"2024-03-21T13:04:06.248632+0000","last_undegraded":"2024-03-21T13:04:06.248632+0000","last_fullsized":"2024-03-21T13:04:06.248632+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:18.526428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.526428+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":99,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243622+0000","last_change":"2024-03-21T13:01:48.212782+0000","last_active":"2024-03-21T13:04:06.243622+0000","last_peered":"2024-03-21T13:04:06.243622+0000","last_clean":"2024-03-21T13:04:06.243622+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:04:06.243622+0000","last_undegraded":"2024-03-21T13:04:06.243622+0000","last_fullsized":"2024-03-21T13:04:06.243622+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:35.801638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.801638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.747293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"71'9","reported_seq":117,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244488+0000","last_change":"2024-03-21T13:02:26.266952+0000","last_active":"2024-03-21T13:04:06.244488+0000","last_peered":"2024-03-21T13:04:06.244488+0000","last_clean":"2024-03-21T13:04:06.244488+0000","last_became_active":"2024-03-21T13:02:26.239161+0000","last_became_peered":"2024-03-21T13:02:26.239161+0000","last_unstale":"2024-03-21T13:04:06.244488+0000","last_undegraded":"2024-03-21T13:04:06.244488+0000","last_fullsized":"2024-03-21T13:04:06.244488+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:00:19.686939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:19.686939+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.251528+0000","last_change":"2024-03-21T13:01:34.642737+0000","last_active":"2024-03-21T13:04:06.251528+0000","last_peered":"2024-03-21T13:04:06.251528+0000","last_clean":"2024-03-21T13:04:06.251528+0000","last_became_active":"2024-03-21T12:58:43.182126+0000","last_became_peered":"2024-03-21T12:58:43.182126+0000","last_unstale":"2024-03-21T13:04:06.251528+0000","last_undegraded":"2024-03-21T13:04:06.251528+0000","last_fullsized":"2024-03-21T13:04:06.251528+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:34.642613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:01:34.642613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100264+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":198,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.250995+0000","last_change":"2024-03-21T13:02:14.354537+0000","last_active":"2024-03-21T13:04:06.250995+0000","last_peered":"2024-03-21T13:04:06.250995+0000","last_clean":"2024-03-21T13:04:06.250995+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:04:06.250995+0000","last_undegraded":"2024-03-21T13:04:06.250995+0000","last_fullsized":"2024-03-21T13:04:06.250995+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:12.121970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:12.121970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":212,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243700+0000","last_change":"2024-03-21T13:01:27.075743+0000","last_active":"2024-03-21T13:04:06.243700+0000","last_peered":"2024-03-21T13:04:06.243700+0000","last_clean":"2024-03-21T13:04:06.243700+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:04:06.243700+0000","last_undegraded":"2024-03-21T13:04:06.243700+0000","last_fullsized":"2024-03-21T13:04:06.243700+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:01:27.074936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:27.074936+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'9","reported_seq":133,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.256390+0000","last_change":"2024-03-21T13:01:48.210983+0000","last_active":"2024-03-21T13:04:06.256390+0000","last_peered":"2024-03-21T13:04:06.256390+0000","last_clean":"2024-03-21T13:04:06.256390+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:04:06.256390+0000","last_undegraded":"2024-03-21T13:04:06.256390+0000","last_fullsized":"2024-03-21T13:04:06.256390+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:01:40.964615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.964615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.392266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":194,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244196+0000","last_change":"2024-03-21T13:03:41.952266+0000","last_active":"2024-03-21T13:04:06.244196+0000","last_peered":"2024-03-21T13:04:06.244196+0000","last_clean":"2024-03-21T13:04:06.244196+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:04:06.244196+0000","last_undegraded":"2024-03-21T13:04:06.244196+0000","last_fullsized":"2024-03-21T13:04:06.244196+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:41.952189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:03:41.952189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:44.103950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":188,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.256674+0000","last_change":"2024-03-21T13:02:26.243050+0000","last_active":"2024-03-21T13:04:06.256674+0000","last_peered":"2024-03-21T13:04:06.256674+0000","last_clean":"2024-03-21T13:04:06.256674+0000","last_became_active":"2024-03-21T13:02:26.242502+0000","last_became_peered":"2024-03-21T13:02:26.242502+0000","last_unstale":"2024-03-21T13:04:06.256674+0000","last_undegraded":"2024-03-21T13:04:06.256674+0000","last_fullsized":"2024-03-21T13:04:06.256674+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:32.639920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.639920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.368065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":214,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244776+0000","last_change":"2024-03-21T13:02:26.252758+0000","last_active":"2024-03-21T13:04:06.244776+0000","last_peered":"2024-03-21T13:04:06.244776+0000","last_clean":"2024-03-21T13:04:06.244776+0000","last_became_active":"2024-03-21T13:02:26.252406+0000","last_became_peered":"2024-03-21T13:02:26.252406+0000","last_unstale":"2024-03-21T13:04:06.244776+0000","last_undegraded":"2024-03-21T13:04:06.244776+0000","last_fullsized":"2024-03-21T13:04:06.244776+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:41.621550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:41.621550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'6","reported_seq":219,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.250807+0000","last_change":"2024-03-21T13:02:14.384317+0000","last_active":"2024-03-21T13:04:06.250807+0000","last_peered":"2024-03-21T13:04:06.250807+0000","last_clean":"2024-03-21T13:04:06.250807+0000","last_became_active":"2024-03-21T13:02:14.352117+0000","last_became_peered":"2024-03-21T13:02:14.352117+0000","last_unstale":"2024-03-21T13:04:06.250807+0000","last_undegraded":"2024-03-21T13:04:06.250807+0000","last_fullsized":"2024-03-21T13:04:06.250807+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T12:59:46.176819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:46.176819+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"154'14011","reported_seq":30983,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:09.625161+0000","last_change":"2024-03-21T13:03:16.986397+0000","last_active":"2024-03-21T13:04:09.625161+0000","last_peered":"2024-03-21T13:04:09.625161+0000","last_clean":"2024-03-21T13:04:09.625161+0000","last_became_active":"2024-03-21T13:03:16.964637+0000","last_became_peered":"2024-03-21T13:03:16.964637+0000","last_unstale":"2024-03-21T13:04:09.625161+0000","last_undegraded":"2024-03-21T13:04:09.625161+0000","last_fullsized":"2024-03-21T13:04:09.625161+0000","mapping_epoch":135,"log_start":"131'9823","ondisk_log_start":"131'9823","created":92,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4188,"log_dups_size":0,"ondisk_log_size":4188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39780352,"num_objects":607,"num_object_clones":0,"num_object_copies":1821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14011,"num_write_kb":467776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5606,"num_bytes_recovered":367394816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":72,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245318+0000","last_change":"2024-03-21T13:02:26.253217+0000","last_active":"2024-03-21T13:04:06.245318+0000","last_peered":"2024-03-21T13:04:06.245318+0000","last_clean":"2024-03-21T13:04:06.245318+0000","last_became_active":"2024-03-21T13:02:26.252904+0000","last_became_peered":"2024-03-21T13:02:26.252904+0000","last_unstale":"2024-03-21T13:04:06.245318+0000","last_undegraded":"2024-03-21T13:04:06.245318+0000","last_fullsized":"2024-03-21T13:04:06.245318+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:01:43.075256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":218,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244700+0000","last_change":"2024-03-21T13:01:39.975587+0000","last_active":"2024-03-21T13:04:06.244700+0000","last_peered":"2024-03-21T13:04:06.244700+0000","last_clean":"2024-03-21T13:04:06.244700+0000","last_became_active":"2024-03-21T12:58:37.735037+0000","last_became_peered":"2024-03-21T12:58:37.735037+0000","last_unstale":"2024-03-21T13:04:06.244700+0000","last_undegraded":"2024-03-21T13:04:06.244700+0000","last_fullsized":"2024-03-21T13:04:06.244700+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:01:39.975456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.975456+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'15","reported_seq":132,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243792+0000","last_change":"2024-03-21T13:02:26.260469+0000","last_active":"2024-03-21T13:04:06.243792+0000","last_peered":"2024-03-21T13:04:06.243792+0000","last_clean":"2024-03-21T13:04:06.243792+0000","last_became_active":"2024-03-21T13:02:26.237800+0000","last_became_peered":"2024-03-21T13:02:26.237800+0000","last_unstale":"2024-03-21T13:04:06.243792+0000","last_undegraded":"2024-03-21T13:04:06.243792+0000","last_fullsized":"2024-03-21T13:04:06.243792+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:00:27.640392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:27.640392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":266,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244108+0000","last_change":"2024-03-21T13:00:20.176757+0000","last_active":"2024-03-21T13:04:06.244108+0000","last_peered":"2024-03-21T13:04:06.244108+0000","last_clean":"2024-03-21T13:04:06.244108+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:04:06.244108+0000","last_undegraded":"2024-03-21T13:04:06.244108+0000","last_fullsized":"2024-03-21T13:04:06.244108+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.176653+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.747368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"154'14028","reported_seq":22462,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:10.079897+0000","last_change":"2024-03-21T13:03:33.368858+0000","last_active":"2024-03-21T13:04:10.079897+0000","last_peered":"2024-03-21T13:04:10.079897+0000","last_clean":"2024-03-21T13:04:10.079897+0000","last_became_active":"2024-03-21T13:03:33.338786+0000","last_became_peered":"2024-03-21T13:03:33.338786+0000","last_unstale":"2024-03-21T13:04:10.079897+0000","last_undegraded":"2024-03-21T13:04:10.079897+0000","last_fullsized":"2024-03-21T13:04:10.079897+0000","mapping_epoch":139,"log_start":"136'10429","ondisk_log_start":"136'10429","created":92,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:13.404523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":500,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14028,"num_write_kb":464896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4140,"num_bytes_recovered":271319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":194,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.248947+0000","last_change":"2024-03-21T13:02:14.352120+0000","last_active":"2024-03-21T13:04:06.248947+0000","last_peered":"2024-03-21T13:04:06.248947+0000","last_clean":"2024-03-21T13:04:06.248947+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:04:06.248947+0000","last_undegraded":"2024-03-21T13:04:06.248947+0000","last_fullsized":"2024-03-21T13:04:06.248947+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:24.988792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:24.988792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.382018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":212,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244881+0000","last_change":"2024-03-21T13:02:14.337089+0000","last_active":"2024-03-21T13:04:06.244881+0000","last_peered":"2024-03-21T13:04:06.244881+0000","last_clean":"2024-03-21T13:04:06.244881+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:04:06.244881+0000","last_undegraded":"2024-03-21T13:04:06.244881+0000","last_fullsized":"2024-03-21T13:04:06.244881+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.030821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.030821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"71'12","reported_seq":138,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.246038+0000","last_change":"2024-03-21T13:02:14.383361+0000","last_active":"2024-03-21T13:04:06.246038+0000","last_peered":"2024-03-21T13:04:06.246038+0000","last_clean":"2024-03-21T13:04:06.246038+0000","last_became_active":"2024-03-21T13:02:14.347102+0000","last_became_peered":"2024-03-21T13:02:14.347102+0000","last_unstale":"2024-03-21T13:04:06.246038+0000","last_undegraded":"2024-03-21T13:04:06.246038+0000","last_fullsized":"2024-03-21T13:04:06.246038+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:00:32.864608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.864608+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.382218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":213,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.254145+0000","last_change":"2024-03-21T13:02:26.235545+0000","last_active":"2024-03-21T13:04:06.254145+0000","last_peered":"2024-03-21T13:04:06.254145+0000","last_clean":"2024-03-21T13:04:06.254145+0000","last_became_active":"2024-03-21T13:02:26.235083+0000","last_became_peered":"2024-03-21T13:02:26.235083+0000","last_unstale":"2024-03-21T13:04:06.254145+0000","last_undegraded":"2024-03-21T13:04:06.254145+0000","last_fullsized":"2024-03-21T13:04:06.254145+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:40.658883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.658883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100369+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"154'14092","reported_seq":26986,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:09.671864+0000","last_change":"2024-03-21T13:03:01.305905+0000","last_active":"2024-03-21T13:04:09.671864+0000","last_peered":"2024-03-21T13:04:09.671864+0000","last_clean":"2024-03-21T13:04:09.671864+0000","last_became_active":"2024-03-21T13:03:01.285305+0000","last_became_peered":"2024-03-21T13:03:01.285305+0000","last_unstale":"2024-03-21T13:04:09.671864+0000","last_undegraded":"2024-03-21T13:04:09.671864+0000","last_fullsized":"2024-03-21T13:04:09.671864+0000","mapping_epoch":129,"log_start":"132'10039","ondisk_log_start":"132'10039","created":92,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4053,"log_dups_size":0,"ondisk_log_size":4053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.747442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":576,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14092,"num_write_kb":469376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6372,"num_bytes_recovered":417595392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":194,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243418+0000","last_change":"2024-03-21T13:01:14.114540+0000","last_active":"2024-03-21T13:04:06.243418+0000","last_peered":"2024-03-21T13:04:06.243418+0000","last_clean":"2024-03-21T13:04:06.243418+0000","last_became_active":"2024-03-21T12:58:43.179007+0000","last_became_peered":"2024-03-21T12:58:43.179007+0000","last_unstale":"2024-03-21T13:04:06.243418+0000","last_undegraded":"2024-03-21T13:04:06.243418+0000","last_fullsized":"2024-03-21T13:04:06.243418+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:14.114420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:14.114420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":207,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.252007+0000","last_change":"2024-03-21T13:02:26.238463+0000","last_active":"2024-03-21T13:04:06.252007+0000","last_peered":"2024-03-21T13:04:06.252007+0000","last_clean":"2024-03-21T13:04:06.252007+0000","last_became_active":"2024-03-21T13:02:26.238116+0000","last_became_peered":"2024-03-21T13:02:26.238116+0000","last_unstale":"2024-03-21T13:04:06.252007+0000","last_undegraded":"2024-03-21T13:04:06.252007+0000","last_fullsized":"2024-03-21T13:04:06.252007+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:39.655215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.655215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100445+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":212,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244388+0000","last_change":"2024-03-21T13:02:26.237114+0000","last_active":"2024-03-21T13:04:06.244388+0000","last_peered":"2024-03-21T13:04:06.244388+0000","last_clean":"2024-03-21T13:04:06.244388+0000","last_became_active":"2024-03-21T13:02:26.236780+0000","last_became_peered":"2024-03-21T13:02:26.236780+0000","last_unstale":"2024-03-21T13:04:06.244388+0000","last_undegraded":"2024-03-21T13:04:06.244388+0000","last_fullsized":"2024-03-21T13:04:06.244388+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:18.012391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.012391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"71'19","reported_seq":287,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245501+0000","last_change":"2024-03-21T13:02:26.275429+0000","last_active":"2024-03-21T13:04:06.245501+0000","last_peered":"2024-03-21T13:04:06.245501+0000","last_clean":"2024-03-21T13:04:06.245501+0000","last_became_active":"2024-03-21T13:02:26.240350+0000","last_became_peered":"2024-03-21T13:02:26.240350+0000","last_unstale":"2024-03-21T13:04:06.245501+0000","last_undegraded":"2024-03-21T13:04:06.245501+0000","last_fullsized":"2024-03-21T13:04:06.245501+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:01:18.623551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.623551+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686531+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"154'14139","reported_seq":14355,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:09.817390+0000","last_change":"2024-03-21T13:02:26.586862+0000","last_active":"2024-03-21T13:04:09.817390+0000","last_peered":"2024-03-21T13:04:09.817390+0000","last_clean":"2024-03-21T13:04:09.817390+0000","last_became_active":"2024-03-21T13:00:42.708899+0000","last_became_peered":"2024-03-21T13:00:42.708899+0000","last_unstale":"2024-03-21T13:04:09.817390+0000","last_undegraded":"2024-03-21T13:04:09.817390+0000","last_fullsized":"2024-03-21T13:04:09.817390+0000","mapping_epoch":92,"log_start":"128'9341","ondisk_log_start":"128'9341","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7428","last_scrub_stamp":"2024-03-21T13:02:26.586741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:02:26.586741+0000","objects_scrubbed":7233,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33882112,"num_objects":517,"num_object_clones":0,"num_object_copies":1551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14139,"num_write_kb":468992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":99,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243502+0000","last_change":"2024-03-21T13:01:48.214335+0000","last_active":"2024-03-21T13:04:06.243502+0000","last_peered":"2024-03-21T13:04:06.243502+0000","last_clean":"2024-03-21T13:04:06.243502+0000","last_became_active":"2024-03-21T13:01:48.213995+0000","last_became_peered":"2024-03-21T13:01:48.213995+0000","last_unstale":"2024-03-21T13:04:06.243502+0000","last_undegraded":"2024-03-21T13:04:06.243502+0000","last_fullsized":"2024-03-21T13:04:06.243502+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:44.997946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:44.997946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":201,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.257799+0000","last_change":"2024-03-21T13:01:46.642867+0000","last_active":"2024-03-21T13:04:06.257799+0000","last_peered":"2024-03-21T13:04:06.257799+0000","last_clean":"2024-03-21T13:04:06.257799+0000","last_became_active":"2024-03-21T12:58:41.163843+0000","last_became_peered":"2024-03-21T12:58:41.163843+0000","last_unstale":"2024-03-21T13:04:06.257799+0000","last_undegraded":"2024-03-21T13:04:06.257799+0000","last_fullsized":"2024-03-21T13:04:06.257799+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:46.642749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.642749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100545+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":72,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.246918+0000","last_change":"2024-03-21T13:02:26.244355+0000","last_active":"2024-03-21T13:04:06.246918+0000","last_peered":"2024-03-21T13:04:06.246918+0000","last_clean":"2024-03-21T13:04:06.246918+0000","last_became_active":"2024-03-21T13:02:26.243974+0000","last_became_peered":"2024-03-21T13:02:26.243974+0000","last_unstale":"2024-03-21T13:04:06.246918+0000","last_undegraded":"2024-03-21T13:04:06.246918+0000","last_fullsized":"2024-03-21T13:04:06.246918+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:33.998382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:33.998382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.392207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"86'13","reported_seq":162,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238031+0000","last_change":"2024-03-21T13:02:14.369004+0000","last_active":"2024-03-21T13:04:06.238031+0000","last_peered":"2024-03-21T13:04:06.238031+0000","last_clean":"2024-03-21T13:04:06.238031+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:04:06.238031+0000","last_undegraded":"2024-03-21T13:04:06.238031+0000","last_fullsized":"2024-03-21T13:04:06.238031+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:01:26.174361+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:01:26.174361+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.085126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.7","version":"154'14402","reported_seq":14538,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:09.674222+0000","last_change":"2024-03-21T13:00:42.707087+0000","last_active":"2024-03-21T13:04:09.674222+0000","last_peered":"2024-03-21T13:04:09.674222+0000","last_clean":"2024-03-21T13:04:09.674222+0000","last_became_active":"2024-03-21T13:00:42.706774+0000","last_became_peered":"2024-03-21T13:00:42.706774+0000","last_unstale":"2024-03-21T13:04:09.674222+0000","last_undegraded":"2024-03-21T13:04:09.674222+0000","last_fullsized":"2024-03-21T13:04:09.674222+0000","mapping_epoch":92,"log_start":"132'10199","ondisk_log_start":"132'10199","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4203,"log_dups_size":0,"ondisk_log_size":4203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34603008,"num_objects":528,"num_object_clones":0,"num_object_copies":1584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14402,"num_write_kb":477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":188,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.256750+0000","last_change":"2024-03-21T13:01:46.738822+0000","last_active":"2024-03-21T13:04:06.256750+0000","last_peered":"2024-03-21T13:04:06.256750+0000","last_clean":"2024-03-21T13:04:06.256750+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:04:06.256750+0000","last_undegraded":"2024-03-21T13:04:06.256750+0000","last_fullsized":"2024-03-21T13:04:06.256750+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.738659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.392291+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"67'2","reported_seq":102,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245386+0000","last_change":"2024-03-21T13:02:26.266567+0000","last_active":"2024-03-21T13:04:06.245386+0000","last_peered":"2024-03-21T13:04:06.245386+0000","last_clean":"2024-03-21T13:04:06.245386+0000","last_became_active":"2024-03-21T13:02:26.243692+0000","last_became_peered":"2024-03-21T13:02:26.243692+0000","last_unstale":"2024-03-21T13:04:06.245386+0000","last_undegraded":"2024-03-21T13:04:06.245386+0000","last_fullsized":"2024-03-21T13:04:06.245386+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-21T13:01:41.017193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:01:41.017193+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.237817+0000","last_change":"2024-03-21T13:01:31.721299+0000","last_active":"2024-03-21T13:04:06.237817+0000","last_peered":"2024-03-21T13:04:06.237817+0000","last_clean":"2024-03-21T13:04:06.237817+0000","last_became_active":"2024-03-21T12:58:37.729454+0000","last_became_peered":"2024-03-21T12:58:37.729454+0000","last_unstale":"2024-03-21T13:04:06.237817+0000","last_undegraded":"2024-03-21T13:04:06.237817+0000","last_fullsized":"2024-03-21T13:04:06.237817+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:31.721222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.721222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.084878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"119'29","reported_seq":199,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.251429+0000","last_change":"2024-03-21T13:02:14.382134+0000","last_active":"2024-03-21T13:04:06.251429+0000","last_peered":"2024-03-21T13:04:06.251429+0000","last_clean":"2024-03-21T13:04:06.251429+0000","last_became_active":"2024-03-21T13:02:14.351114+0000","last_became_peered":"2024-03-21T13:02:14.351114+0000","last_unstale":"2024-03-21T13:04:06.251429+0000","last_undegraded":"2024-03-21T13:04:06.251429+0000","last_fullsized":"2024-03-21T13:04:06.251429+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'28","last_scrub_stamp":"2024-03-21T13:01:36.231420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:36.231420+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":94,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238744+0000","last_change":"2024-03-21T13:03:36.904209+0000","last_active":"2024-03-21T13:04:06.238744+0000","last_peered":"2024-03-21T13:04:06.238744+0000","last_clean":"2024-03-21T13:04:06.238744+0000","last_became_active":"2024-03-21T13:02:14.336177+0000","last_became_peered":"2024-03-21T13:02:14.336177+0000","last_unstale":"2024-03-21T13:04:06.238744+0000","last_undegraded":"2024-03-21T13:04:06.238744+0000","last_fullsized":"2024-03-21T13:04:06.238744+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:36.904112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:03:36.904112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:58.936647+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.4","version":"154'14105","reported_seq":21227,"reported_epoch":154,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-21T13:04:09.657850+0000","last_change":"2024-03-21T13:03:56.256633+0000","last_active":"2024-03-21T13:04:09.657850+0000","last_peered":"2024-03-21T13:04:09.657850+0000","last_clean":"2024-03-21T13:02:13.309754+0000","last_became_active":"2024-03-21T13:02:15.383776+0000","last_became_peered":"2024-03-21T13:02:15.383776+0000","last_unstale":"2024-03-21T13:04:09.657850+0000","last_undegraded":"2024-03-21T13:04:09.657850+0000","last_fullsized":"2024-03-21T13:04:09.657850+0000","mapping_epoch":118,"log_start":"133'10182","ondisk_log_start":"133'10182","created":92,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3923,"log_dups_size":0,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34799616,"num_objects":531,"num_object_clones":0,"num_object_copies":1593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":264,"num_objects_unfound":0,"num_objects_dirty":531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14105,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3478,"num_bytes_recovered":227934208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,1],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":531}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":194,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244595+0000","last_change":"2024-03-21T13:01:28.061976+0000","last_active":"2024-03-21T13:04:06.244595+0000","last_peered":"2024-03-21T13:04:06.244595+0000","last_clean":"2024-03-21T13:04:06.244595+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:04:06.244595+0000","last_undegraded":"2024-03-21T13:04:06.244595+0000","last_fullsized":"2024-03-21T13:04:06.244595+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:28.061868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:28.061868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":92,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.254283+0000","last_change":"2024-03-21T13:02:21.275708+0000","last_active":"2024-03-21T13:04:06.254283+0000","last_peered":"2024-03-21T13:04:06.254283+0000","last_clean":"2024-03-21T13:04:06.254283+0000","last_became_active":"2024-03-21T13:02:14.328020+0000","last_became_peered":"2024-03-21T13:02:14.328020+0000","last_unstale":"2024-03-21T13:04:06.254283+0000","last_undegraded":"2024-03-21T13:04:06.254283+0000","last_fullsized":"2024-03-21T13:04:06.254283+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:02:21.275321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:02:21.275321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100627+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":218,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.250718+0000","last_change":"2024-03-21T13:01:16.871013+0000","last_active":"2024-03-21T13:04:06.250718+0000","last_peered":"2024-03-21T13:04:06.250718+0000","last_clean":"2024-03-21T13:04:06.250718+0000","last_became_active":"2024-03-21T12:58:36.724067+0000","last_became_peered":"2024-03-21T12:58:36.724067+0000","last_unstale":"2024-03-21T13:04:06.250718+0000","last_undegraded":"2024-03-21T13:04:06.250718+0000","last_fullsized":"2024-03-21T13:04:06.250718+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:16.870806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:01:16.870806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100211+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0.000743678,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'13","reported_seq":132,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244447+0000","last_change":"2024-03-21T13:02:26.260934+0000","last_active":"2024-03-21T13:04:06.244447+0000","last_peered":"2024-03-21T13:04:06.244447+0000","last_clean":"2024-03-21T13:04:06.244447+0000","last_became_active":"2024-03-21T13:02:26.234230+0000","last_became_peered":"2024-03-21T13:02:26.234230+0000","last_unstale":"2024-03-21T13:04:06.244447+0000","last_undegraded":"2024-03-21T13:04:06.244447+0000","last_fullsized":"2024-03-21T13:04:06.244447+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'13","last_scrub_stamp":"2024-03-21T13:01:31.190053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.190053+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":207,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.251018+0000","last_change":"2024-03-21T13:02:26.228478+0000","last_active":"2024-03-21T13:04:06.251018+0000","last_peered":"2024-03-21T13:04:06.251018+0000","last_clean":"2024-03-21T13:04:06.251018+0000","last_became_active":"2024-03-21T13:02:26.228128+0000","last_became_peered":"2024-03-21T13:02:26.228128+0000","last_unstale":"2024-03-21T13:04:06.251018+0000","last_undegraded":"2024-03-21T13:04:06.251018+0000","last_fullsized":"2024-03-21T13:04:06.251018+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:25.668960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:25.668960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100421+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"154'14072","reported_seq":14212,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:09.700068+0000","last_change":"2024-03-21T13:01:48.219270+0000","last_active":"2024-03-21T13:04:09.700068+0000","last_peered":"2024-03-21T13:04:09.700068+0000","last_clean":"2024-03-21T13:04:09.700068+0000","last_became_active":"2024-03-21T13:01:48.213755+0000","last_became_peered":"2024-03-21T13:01:48.213755+0000","last_unstale":"2024-03-21T13:04:09.700068+0000","last_undegraded":"2024-03-21T13:04:09.700068+0000","last_fullsized":"2024-03-21T13:04:09.700068+0000","mapping_epoch":110,"log_start":"132'10062","ondisk_log_start":"132'10062","created":92,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4010,"log_dups_size":0,"ondisk_log_size":4010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.747409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36044800,"num_objects":550,"num_object_clones":0,"num_object_copies":1650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14072,"num_write_kb":467904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":72,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.250903+0000","last_change":"2024-03-21T13:02:26.236072+0000","last_active":"2024-03-21T13:04:06.250903+0000","last_peered":"2024-03-21T13:04:06.250903+0000","last_clean":"2024-03-21T13:04:06.250903+0000","last_became_active":"2024-03-21T13:02:26.235640+0000","last_became_peered":"2024-03-21T13:02:26.235640+0000","last_unstale":"2024-03-21T13:04:06.250903+0000","last_undegraded":"2024-03-21T13:04:06.250903+0000","last_fullsized":"2024-03-21T13:04:06.250903+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:52.658377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T12:59:52.658377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":455,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243637+0000","last_change":"2024-03-21T13:00:16.180260+0000","last_active":"2024-03-21T13:04:06.243637+0000","last_peered":"2024-03-21T13:04:06.243637+0000","last_clean":"2024-03-21T13:04:06.243637+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:04:06.243637+0000","last_undegraded":"2024-03-21T13:04:06.243637+0000","last_fullsized":"2024-03-21T13:04:06.243637+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:00:16.180126+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:00:16.180126+0000","last_clean_scrub_stamp":"2024-03-21T13:00:16.180126+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":206,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245564+0000","last_change":"2024-03-21T13:02:26.242969+0000","last_active":"2024-03-21T13:04:06.245564+0000","last_peered":"2024-03-21T13:04:06.245564+0000","last_clean":"2024-03-21T13:04:06.245564+0000","last_became_active":"2024-03-21T13:02:26.242456+0000","last_became_peered":"2024-03-21T13:02:26.242456+0000","last_unstale":"2024-03-21T13:04:06.245564+0000","last_undegraded":"2024-03-21T13:04:06.245564+0000","last_fullsized":"2024-03-21T13:04:06.245564+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:38.007486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.007486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.391741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":293,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.246156+0000","last_change":"2024-03-21T13:01:21.574857+0000","last_active":"2024-03-21T13:04:06.246156+0000","last_peered":"2024-03-21T13:04:06.246156+0000","last_clean":"2024-03-21T13:04:06.246156+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:04:06.246156+0000","last_undegraded":"2024-03-21T13:04:06.246156+0000","last_fullsized":"2024-03-21T13:04:06.246156+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-21T13:01:21.574712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:21.574712+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686713+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":204,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244234+0000","last_change":"2024-03-21T13:02:26.235494+0000","last_active":"2024-03-21T13:04:06.244234+0000","last_peered":"2024-03-21T13:04:06.244234+0000","last_clean":"2024-03-21T13:04:06.244234+0000","last_became_active":"2024-03-21T13:02:26.235140+0000","last_became_peered":"2024-03-21T13:02:26.235140+0000","last_unstale":"2024-03-21T13:04:06.244234+0000","last_undegraded":"2024-03-21T13:04:06.244234+0000","last_fullsized":"2024-03-21T13:04:06.244234+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.968175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"154'14205","reported_seq":14341,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:09.695740+0000","last_change":"2024-03-21T13:00:42.710083+0000","last_active":"2024-03-21T13:04:09.695740+0000","last_peered":"2024-03-21T13:04:09.695740+0000","last_clean":"2024-03-21T13:04:09.695740+0000","last_became_active":"2024-03-21T13:00:42.709753+0000","last_became_peered":"2024-03-21T13:00:42.709753+0000","last_unstale":"2024-03-21T13:04:09.695740+0000","last_undegraded":"2024-03-21T13:04:09.695740+0000","last_fullsized":"2024-03-21T13:04:09.695740+0000","mapping_epoch":92,"log_start":"132'10120","ondisk_log_start":"132'10120","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.747390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36765696,"num_objects":561,"num_object_clones":0,"num_object_copies":1683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14205,"num_write_kb":472512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":74,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.246749+0000","last_change":"2024-03-21T13:02:26.243121+0000","last_active":"2024-03-21T13:04:06.246749+0000","last_peered":"2024-03-21T13:04:06.246749+0000","last_clean":"2024-03-21T13:04:06.246749+0000","last_became_active":"2024-03-21T13:02:26.242708+0000","last_became_peered":"2024-03-21T13:02:26.242708+0000","last_unstale":"2024-03-21T13:04:06.246749+0000","last_undegraded":"2024-03-21T13:04:06.246749+0000","last_fullsized":"2024-03-21T13:04:06.246749+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:20.854917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.854917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.368184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":99,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.237811+0000","last_change":"2024-03-21T13:01:48.212127+0000","last_active":"2024-03-21T13:04:06.237811+0000","last_peered":"2024-03-21T13:04:06.237811+0000","last_clean":"2024-03-21T13:04:06.237811+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:04:06.237811+0000","last_undegraded":"2024-03-21T13:04:06.237811+0000","last_fullsized":"2024-03-21T13:04:06.237811+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:01:34.066532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.085152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":551,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:08.506243+0000","last_change":"2024-03-21T13:03:45.889240+0000","last_active":"2024-03-21T13:04:08.506243+0000","last_peered":"2024-03-21T13:04:08.506243+0000","last_clean":"2024-03-21T13:04:08.506243+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:04:08.506243+0000","last_undegraded":"2024-03-21T13:04:08.506243+0000","last_fullsized":"2024-03-21T13:04:08.506243+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:03:45.889083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:03:45.889083+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:05:13.432245+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":323,"num_read_kb":318,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":249,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243429+0000","last_change":"2024-03-21T13:03:31.347954+0000","last_active":"2024-03-21T13:04:06.243429+0000","last_peered":"2024-03-21T13:04:06.243429+0000","last_clean":"2024-03-21T13:04:06.243429+0000","last_became_active":"2024-03-21T13:02:14.333917+0000","last_became_peered":"2024-03-21T13:02:14.333917+0000","last_unstale":"2024-03-21T13:04:06.243429+0000","last_undegraded":"2024-03-21T13:04:06.243429+0000","last_fullsized":"2024-03-21T13:04:06.243429+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:03:31.347818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:03:31.347818+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:43.775209+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":71,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244793+0000","last_change":"2024-03-21T13:02:26.232971+0000","last_active":"2024-03-21T13:04:06.244793+0000","last_peered":"2024-03-21T13:04:06.244793+0000","last_clean":"2024-03-21T13:04:06.244793+0000","last_became_active":"2024-03-21T13:02:26.232625+0000","last_became_peered":"2024-03-21T13:02:26.232625+0000","last_unstale":"2024-03-21T13:04:06.244793+0000","last_undegraded":"2024-03-21T13:04:06.244793+0000","last_fullsized":"2024-03-21T13:04:06.244793+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:34.982531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:34.982531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"154'14292","reported_seq":22483,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:10.057692+0000","last_change":"2024-03-21T13:03:58.841061+0000","last_active":"2024-03-21T13:04:10.057692+0000","last_peered":"2024-03-21T13:04:10.057692+0000","last_clean":"2024-03-21T13:04:10.057692+0000","last_became_active":"2024-03-21T13:03:58.822016+0000","last_became_peered":"2024-03-21T13:03:58.822016+0000","last_unstale":"2024-03-21T13:04:10.057692+0000","last_undegraded":"2024-03-21T13:04:10.057692+0000","last_fullsized":"2024-03-21T13:04:10.057692+0000","mapping_epoch":150,"log_start":"136'10758","ondisk_log_start":"136'10758","created":92,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3534,"log_dups_size":0,"ondisk_log_size":3534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:13.404307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":512,"num_object_clones":0,"num_object_copies":1536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14292,"num_write_kb":473728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2676,"num_bytes_recovered":175374336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":84,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245379+0000","last_change":"2024-03-21T13:02:14.336240+0000","last_active":"2024-03-21T13:04:06.245379+0000","last_peered":"2024-03-21T13:04:06.245379+0000","last_clean":"2024-03-21T13:04:06.245379+0000","last_became_active":"2024-03-21T13:02:14.335897+0000","last_became_peered":"2024-03-21T13:02:14.335897+0000","last_unstale":"2024-03-21T13:04:06.245379+0000","last_undegraded":"2024-03-21T13:04:06.245379+0000","last_fullsized":"2024-03-21T13:04:06.245379+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:21.040252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:21.040252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":90,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.251234+0000","last_change":"2024-03-21T13:02:26.238041+0000","last_active":"2024-03-21T13:04:06.251234+0000","last_peered":"2024-03-21T13:04:06.251234+0000","last_clean":"2024-03-21T13:04:06.251234+0000","last_became_active":"2024-03-21T13:02:26.237683+0000","last_became_peered":"2024-03-21T13:02:26.237683+0000","last_unstale":"2024-03-21T13:04:06.251234+0000","last_undegraded":"2024-03-21T13:04:06.251234+0000","last_fullsized":"2024-03-21T13:04:06.251234+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:24.072225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:24.072225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"71'19","reported_seq":145,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244090+0000","last_change":"2024-03-21T13:02:26.274805+0000","last_active":"2024-03-21T13:04:06.244090+0000","last_peered":"2024-03-21T13:04:06.244090+0000","last_clean":"2024-03-21T13:04:06.244090+0000","last_became_active":"2024-03-21T13:02:26.239140+0000","last_became_peered":"2024-03-21T13:02:26.239140+0000","last_unstale":"2024-03-21T13:04:06.244090+0000","last_undegraded":"2024-03-21T13:04:06.244090+0000","last_fullsized":"2024-03-21T13:04:06.244090+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:01:29.106715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:29.106715+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":206,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245598+0000","last_change":"2024-03-21T13:02:26.252937+0000","last_active":"2024-03-21T13:04:06.245598+0000","last_peered":"2024-03-21T13:04:06.245598+0000","last_clean":"2024-03-21T13:04:06.245598+0000","last_became_active":"2024-03-21T13:02:26.252372+0000","last_became_peered":"2024-03-21T13:02:26.252372+0000","last_unstale":"2024-03-21T13:04:06.245598+0000","last_undegraded":"2024-03-21T13:04:06.245598+0000","last_fullsized":"2024-03-21T13:04:06.245598+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:22.601641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:22.601641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686498+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"154'14047","reported_seq":14242,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:09.673239+0000","last_change":"2024-03-21T13:01:47.452788+0000","last_active":"2024-03-21T13:04:09.673239+0000","last_peered":"2024-03-21T13:04:09.673239+0000","last_clean":"2024-03-21T13:04:09.673239+0000","last_became_active":"2024-03-21T13:00:42.707521+0000","last_became_peered":"2024-03-21T13:00:42.707521+0000","last_unstale":"2024-03-21T13:04:09.673239+0000","last_undegraded":"2024-03-21T13:04:09.673239+0000","last_fullsized":"2024-03-21T13:04:09.673239+0000","mapping_epoch":92,"log_start":"132'10058","ondisk_log_start":"132'10058","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'5147","last_scrub_stamp":"2024-03-21T13:01:47.452664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:01:47.452664+0000","objects_scrubbed":5137,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.085605+0000","scrub_duration":887,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35979264,"num_objects":549,"num_object_clones":0,"num_object_copies":1647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14047,"num_write_kb":467072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":99,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.248597+0000","last_change":"2024-03-21T13:01:48.209228+0000","last_active":"2024-03-21T13:04:06.248597+0000","last_peered":"2024-03-21T13:04:06.248597+0000","last_clean":"2024-03-21T13:04:06.248597+0000","last_became_active":"2024-03-21T13:01:48.208899+0000","last_became_peered":"2024-03-21T13:01:48.208899+0000","last_unstale":"2024-03-21T13:04:06.248597+0000","last_undegraded":"2024-03-21T13:04:06.248597+0000","last_fullsized":"2024-03-21T13:04:06.248597+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:31.121913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.121913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.391903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":72,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245962+0000","last_change":"2024-03-21T13:02:26.244688+0000","last_active":"2024-03-21T13:04:06.245962+0000","last_peered":"2024-03-21T13:04:06.245962+0000","last_clean":"2024-03-21T13:04:06.245962+0000","last_became_active":"2024-03-21T13:02:26.243417+0000","last_became_peered":"2024-03-21T13:02:26.243417+0000","last_unstale":"2024-03-21T13:04:06.245962+0000","last_undegraded":"2024-03-21T13:04:06.245962+0000","last_fullsized":"2024-03-21T13:04:06.245962+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:20.666893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:20.666893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"86'15","reported_seq":130,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238464+0000","last_change":"2024-03-21T13:02:26.263374+0000","last_active":"2024-03-21T13:04:06.238464+0000","last_peered":"2024-03-21T13:04:06.238464+0000","last_clean":"2024-03-21T13:04:06.238464+0000","last_became_active":"2024-03-21T13:02:26.232580+0000","last_became_peered":"2024-03-21T13:02:26.232580+0000","last_unstale":"2024-03-21T13:04:06.238464+0000","last_undegraded":"2024-03-21T13:04:06.238464+0000","last_fullsized":"2024-03-21T13:04:06.238464+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-03-21T13:01:27.655920+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:01:27.655920+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.084838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"125'12","reported_seq":165,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245547+0000","last_change":"2024-03-21T13:02:26.269099+0000","last_active":"2024-03-21T13:04:06.245547+0000","last_peered":"2024-03-21T13:04:06.245547+0000","last_clean":"2024-03-21T13:04:06.245547+0000","last_became_active":"2024-03-21T13:02:26.253720+0000","last_became_peered":"2024-03-21T13:02:26.253720+0000","last_unstale":"2024-03-21T13:04:06.245547+0000","last_undegraded":"2024-03-21T13:04:06.245547+0000","last_fullsized":"2024-03-21T13:04:06.245547+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:25.175225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:25.175225+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"154'14326","reported_seq":21472,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:10.101532+0000","last_change":"2024-03-21T13:03:42.534826+0000","last_active":"2024-03-21T13:04:10.101532+0000","last_peered":"2024-03-21T13:04:10.101532+0000","last_clean":"2024-03-21T13:04:10.101532+0000","last_became_active":"2024-03-21T13:03:42.465921+0000","last_became_peered":"2024-03-21T13:03:42.465921+0000","last_unstale":"2024-03-21T13:04:10.101532+0000","last_undegraded":"2024-03-21T13:04:10.101532+0000","last_fullsized":"2024-03-21T13:04:10.101532+0000","mapping_epoch":146,"log_start":"136'10783","ondisk_log_start":"136'10783","created":92,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3543,"log_dups_size":0,"ondisk_log_size":3543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:13.404335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32636928,"num_objects":498,"num_object_clones":0,"num_object_copies":1494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14326,"num_write_kb":474368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3491,"num_bytes_recovered":228720640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":194,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.244089+0000","last_change":"2024-03-21T13:01:50.086495+0000","last_active":"2024-03-21T13:04:06.244089+0000","last_peered":"2024-03-21T13:04:06.244089+0000","last_clean":"2024-03-21T13:04:06.244089+0000","last_became_active":"2024-03-21T12:58:43.353049+0000","last_became_peered":"2024-03-21T12:58:43.353049+0000","last_unstale":"2024-03-21T13:04:06.244089+0000","last_undegraded":"2024-03-21T13:04:06.244089+0000","last_fullsized":"2024-03-21T13:04:06.244089+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:50.086373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:50.086373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":226,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243537+0000","last_change":"2024-03-21T13:03:36.317825+0000","last_active":"2024-03-21T13:04:06.243537+0000","last_peered":"2024-03-21T13:04:06.243537+0000","last_clean":"2024-03-21T13:04:06.243537+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:04:06.243537+0000","last_undegraded":"2024-03-21T13:04:06.243537+0000","last_fullsized":"2024-03-21T13:04:06.243537+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:03:36.317684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:03:36.317684+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:05:02.927983+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"87'21","reported_seq":164,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245147+0000","last_change":"2024-03-21T13:02:26.263470+0000","last_active":"2024-03-21T13:04:06.245147+0000","last_peered":"2024-03-21T13:04:06.245147+0000","last_clean":"2024-03-21T13:04:06.245147+0000","last_became_active":"2024-03-21T13:02:26.227165+0000","last_became_peered":"2024-03-21T13:02:26.227165+0000","last_unstale":"2024-03-21T13:04:06.245147+0000","last_undegraded":"2024-03-21T13:04:06.245147+0000","last_fullsized":"2024-03-21T13:04:06.245147+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'21","last_scrub_stamp":"2024-03-21T13:01:29.911046+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:01:29.911046+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"119'9","reported_seq":278,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.249809+0000","last_change":"2024-03-21T13:02:14.375956+0000","last_active":"2024-03-21T13:04:06.249809+0000","last_peered":"2024-03-21T13:04:06.249809+0000","last_clean":"2024-03-21T13:04:06.249809+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:04:06.249809+0000","last_undegraded":"2024-03-21T13:04:06.249809+0000","last_fullsized":"2024-03-21T13:04:06.249809+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:25.700549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:25.700549+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"154'14101","reported_seq":14237,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:10.068519+0000","last_change":"2024-03-21T13:00:42.711111+0000","last_active":"2024-03-21T13:04:10.068519+0000","last_peered":"2024-03-21T13:04:10.068519+0000","last_clean":"2024-03-21T13:04:10.068519+0000","last_became_active":"2024-03-21T13:00:42.710829+0000","last_became_peered":"2024-03-21T13:00:42.710829+0000","last_unstale":"2024-03-21T13:04:10.068519+0000","last_undegraded":"2024-03-21T13:04:10.068519+0000","last_fullsized":"2024-03-21T13:04:10.068519+0000","mapping_epoch":92,"log_start":"136'10474","ondisk_log_start":"136'10474","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:13.404629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34406400,"num_objects":525,"num_object_clones":0,"num_object_copies":1575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14101,"num_write_kb":468032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":113,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245998+0000","last_change":"2024-03-21T13:01:27.382938+0000","last_active":"2024-03-21T13:04:06.245998+0000","last_peered":"2024-03-21T13:04:06.245998+0000","last_clean":"2024-03-21T13:04:06.245998+0000","last_became_active":"2024-03-21T13:01:27.382663+0000","last_became_peered":"2024-03-21T13:01:27.382663+0000","last_unstale":"2024-03-21T13:04:06.245998+0000","last_undegraded":"2024-03-21T13:04:06.245998+0000","last_fullsized":"2024-03-21T13:04:06.245998+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:00:08.823841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":200,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.250621+0000","last_change":"2024-03-21T12:59:43.240826+0000","last_active":"2024-03-21T13:04:06.250621+0000","last_peered":"2024-03-21T13:04:06.250621+0000","last_clean":"2024-03-21T13:04:06.250621+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:04:06.250621+0000","last_undegraded":"2024-03-21T13:04:06.250621+0000","last_fullsized":"2024-03-21T13:04:06.250621+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T12:59:43.240674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"71'18","reported_seq":228,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.249061+0000","last_change":"2024-03-21T12:59:48.179286+0000","last_active":"2024-03-21T13:04:06.249061+0000","last_peered":"2024-03-21T13:04:06.249061+0000","last_clean":"2024-03-21T13:04:06.249061+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:04:06.249061+0000","last_undegraded":"2024-03-21T13:04:06.249061+0000","last_fullsized":"2024-03-21T13:04:06.249061+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T12:59:48.179188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T12:59:48.179188+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367799+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":192,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.237486+0000","last_change":"2024-03-21T13:01:17.590965+0000","last_active":"2024-03-21T13:04:06.237486+0000","last_peered":"2024-03-21T13:04:06.237486+0000","last_clean":"2024-03-21T13:04:06.237486+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:04:06.237486+0000","last_undegraded":"2024-03-21T13:04:06.237486+0000","last_fullsized":"2024-03-21T13:04:06.237486+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:17.590831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:17.590831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.085095+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"154'14155","reported_seq":14291,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:09.684115+0000","last_change":"2024-03-21T13:00:42.705026+0000","last_active":"2024-03-21T13:04:09.684115+0000","last_peered":"2024-03-21T13:04:09.684115+0000","last_clean":"2024-03-21T13:04:09.684115+0000","last_became_active":"2024-03-21T13:00:42.704712+0000","last_became_peered":"2024-03-21T13:00:42.704712+0000","last_unstale":"2024-03-21T13:04:09.684115+0000","last_undegraded":"2024-03-21T13:04:09.684115+0000","last_fullsized":"2024-03-21T13:04:09.684115+0000","mapping_epoch":92,"log_start":"131'9968","ondisk_log_start":"131'9968","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33619968,"num_objects":513,"num_object_clones":0,"num_object_copies":1539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14155,"num_write_kb":469376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":192,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.251597+0000","last_change":"2024-03-21T13:02:26.236964+0000","last_active":"2024-03-21T13:04:06.251597+0000","last_peered":"2024-03-21T13:04:06.251597+0000","last_clean":"2024-03-21T13:04:06.251597+0000","last_became_active":"2024-03-21T13:02:26.236501+0000","last_became_peered":"2024-03-21T13:02:26.236501+0000","last_unstale":"2024-03-21T13:04:06.251597+0000","last_undegraded":"2024-03-21T13:04:06.251597+0000","last_fullsized":"2024-03-21T13:04:06.251597+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:24.875883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:24.875883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100291+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"125'10","reported_seq":130,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.243925+0000","last_change":"2024-03-21T13:03:29.367498+0000","last_active":"2024-03-21T13:04:06.243925+0000","last_peered":"2024-03-21T13:04:06.243925+0000","last_clean":"2024-03-21T13:04:06.243925+0000","last_became_active":"2024-03-21T13:02:26.231372+0000","last_became_peered":"2024-03-21T13:02:26.231372+0000","last_unstale":"2024-03-21T13:04:06.243925+0000","last_undegraded":"2024-03-21T13:04:06.243925+0000","last_fullsized":"2024-03-21T13:04:06.243925+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:03:29.367369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:03:29.367369+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:57.845682+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":206,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.248940+0000","last_change":"2024-03-21T13:02:14.352037+0000","last_active":"2024-03-21T13:04:06.248940+0000","last_peered":"2024-03-21T13:04:06.248940+0000","last_clean":"2024-03-21T13:04:06.248940+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:04:06.248940+0000","last_undegraded":"2024-03-21T13:04:06.248940+0000","last_fullsized":"2024-03-21T13:04:06.248940+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:59:41.231340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T12:59:41.231340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367549+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":144,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.250710+0000","last_change":"2024-03-21T13:02:14.377274+0000","last_active":"2024-03-21T13:04:06.250710+0000","last_peered":"2024-03-21T13:04:06.250710+0000","last_clean":"2024-03-21T13:04:06.250710+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:04:06.250710+0000","last_undegraded":"2024-03-21T13:04:06.250710+0000","last_fullsized":"2024-03-21T13:04:06.250710+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:01:46.167736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.167736+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.392143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"154'13903","reported_seq":24011,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:09.687122+0000","last_change":"2024-03-21T13:03:26.668086+0000","last_active":"2024-03-21T13:04:09.687122+0000","last_peered":"2024-03-21T13:04:09.687122+0000","last_clean":"2024-03-21T13:04:09.687122+0000","last_became_active":"2024-03-21T13:03:25.281931+0000","last_became_peered":"2024-03-21T13:03:25.281931+0000","last_unstale":"2024-03-21T13:04:09.687122+0000","last_undegraded":"2024-03-21T13:04:09.687122+0000","last_fullsized":"2024-03-21T13:04:09.687122+0000","mapping_epoch":137,"log_start":"131'9644","ondisk_log_start":"131'9644","created":92,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10932","last_scrub_stamp":"2024-03-21T13:03:26.667789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:03:26.667789+0000","objects_scrubbed":3563,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:33.333827+0000","scrub_duration":1030,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35913752,"num_objects":549,"num_object_clones":0,"num_object_copies":1647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13903,"num_write_kb":462401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4954,"num_bytes_recovered":324599832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":199,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.246759+0000","last_change":"2024-03-21T13:02:26.249048+0000","last_active":"2024-03-21T13:04:06.246759+0000","last_peered":"2024-03-21T13:04:06.246759+0000","last_clean":"2024-03-21T13:04:06.246759+0000","last_became_active":"2024-03-21T13:02:26.243985+0000","last_became_peered":"2024-03-21T13:02:26.243985+0000","last_unstale":"2024-03-21T13:04:06.246759+0000","last_undegraded":"2024-03-21T13:04:06.246759+0000","last_fullsized":"2024-03-21T13:04:06.246759+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:00:27.958396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:27.958396+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.381990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":72,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238813+0000","last_change":"2024-03-21T13:02:26.229605+0000","last_active":"2024-03-21T13:04:06.238813+0000","last_peered":"2024-03-21T13:04:06.238813+0000","last_clean":"2024-03-21T13:04:06.238813+0000","last_became_active":"2024-03-21T13:02:26.229296+0000","last_became_peered":"2024-03-21T13:02:26.229296+0000","last_unstale":"2024-03-21T13:04:06.238813+0000","last_undegraded":"2024-03-21T13:04:06.238813+0000","last_fullsized":"2024-03-21T13:04:06.238813+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:36.008632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:36.008632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.084988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.237592+0000","last_change":"2024-03-21T13:01:39.629358+0000","last_active":"2024-03-21T13:04:06.237592+0000","last_peered":"2024-03-21T13:04:06.237592+0000","last_clean":"2024-03-21T13:04:06.237592+0000","last_became_active":"2024-03-21T12:58:37.730858+0000","last_became_peered":"2024-03-21T12:58:37.730858+0000","last_unstale":"2024-03-21T13:04:06.237592+0000","last_undegraded":"2024-03-21T13:04:06.237592+0000","last_fullsized":"2024-03-21T13:04:06.237592+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:39.629248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.629248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:11.085181+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'11","reported_seq":241,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238135+0000","last_change":"2024-03-21T13:03:33.908220+0000","last_active":"2024-03-21T13:04:06.238135+0000","last_peered":"2024-03-21T13:04:06.238135+0000","last_clean":"2024-03-21T13:04:06.238135+0000","last_became_active":"2024-03-21T13:02:14.348499+0000","last_became_peered":"2024-03-21T13:02:14.348499+0000","last_unstale":"2024-03-21T13:04:06.238135+0000","last_undegraded":"2024-03-21T13:04:06.238135+0000","last_fullsized":"2024-03-21T13:04:06.238135+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:03:33.908122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:03:33.908122+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:34.304868+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"154'14214","reported_seq":24808,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:09.755885+0000","last_change":"2024-03-21T13:03:25.303683+0000","last_active":"2024-03-21T13:04:09.755885+0000","last_peered":"2024-03-21T13:04:09.755885+0000","last_clean":"2024-03-21T13:04:09.755885+0000","last_became_active":"2024-03-21T13:03:25.282195+0000","last_became_peered":"2024-03-21T13:03:25.282195+0000","last_unstale":"2024-03-21T13:04:09.755885+0000","last_undegraded":"2024-03-21T13:04:09.755885+0000","last_fullsized":"2024-03-21T13:04:09.755885+0000","mapping_epoch":137,"log_start":"128'9384","ondisk_log_start":"128'9384","created":92,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'7457","last_scrub_stamp":"2024-03-21T13:02:25.543659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:02:25.543659+0000","objects_scrubbed":7338,"log_size":4830,"log_dups_size":0,"ondisk_log_size":4830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34996224,"num_objects":534,"num_object_clones":0,"num_object_copies":1602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14214,"num_write_kb":471936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5179,"num_bytes_recovered":339410944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.252111+0000","last_change":"2024-03-21T13:01:47.562081+0000","last_active":"2024-03-21T13:04:06.252111+0000","last_peered":"2024-03-21T13:04:06.252111+0000","last_clean":"2024-03-21T13:04:06.252111+0000","last_became_active":"2024-03-21T12:58:43.183380+0000","last_became_peered":"2024-03-21T12:58:43.183380+0000","last_unstale":"2024-03-21T13:04:06.252111+0000","last_undegraded":"2024-03-21T13:04:06.252111+0000","last_fullsized":"2024-03-21T13:04:06.252111+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:47.561991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:01:47.561991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":92,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.238332+0000","last_change":"2024-03-21T13:03:43.965242+0000","last_active":"2024-03-21T13:04:06.238332+0000","last_peered":"2024-03-21T13:04:06.238332+0000","last_clean":"2024-03-21T13:04:06.238332+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:04:06.238332+0000","last_undegraded":"2024-03-21T13:04:06.238332+0000","last_fullsized":"2024-03-21T13:04:06.238332+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:43.965129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:03:43.965129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:58.668582+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":214,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.245662+0000","last_change":"2024-03-21T13:02:14.343797+0000","last_active":"2024-03-21T13:04:06.245662+0000","last_peered":"2024-03-21T13:04:06.245662+0000","last_clean":"2024-03-21T13:04:06.245662+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:04:06.245662+0000","last_undegraded":"2024-03-21T13:04:06.245662+0000","last_fullsized":"2024-03-21T13:04:06.245662+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:16.595403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:16.595403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:09.686368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":192,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.251688+0000","last_change":"2024-03-21T13:02:27.259863+0000","last_active":"2024-03-21T13:04:06.251688+0000","last_peered":"2024-03-21T13:04:06.251688+0000","last_clean":"2024-03-21T13:04:06.251688+0000","last_became_active":"2024-03-21T13:02:14.334882+0000","last_became_peered":"2024-03-21T13:02:14.334882+0000","last_unstale":"2024-03-21T13:04:06.251688+0000","last_undegraded":"2024-03-21T13:04:06.251688+0000","last_fullsized":"2024-03-21T13:04:06.251688+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-21T13:02:27.259731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:02:27.259731+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100602+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"154'14282","reported_seq":14428,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:10.092525+0000","last_change":"2024-03-21T13:01:48.274014+0000","last_active":"2024-03-21T13:04:10.092525+0000","last_peered":"2024-03-21T13:04:10.092525+0000","last_clean":"2024-03-21T13:04:10.092525+0000","last_became_active":"2024-03-21T13:01:48.220955+0000","last_became_peered":"2024-03-21T13:01:48.220955+0000","last_unstale":"2024-03-21T13:04:10.092525+0000","last_undegraded":"2024-03-21T13:04:10.092525+0000","last_fullsized":"2024-03-21T13:04:10.092525+0000","mapping_epoch":110,"log_start":"136'10739","ondisk_log_start":"136'10739","created":92,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:41.686636+0000","last_clean_scrub_stamp":"2024-03-21T13:00:41.686636+0000","objects_scrubbed":0,"log_size":3543,"log_dups_size":0,"ondisk_log_size":3543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:13.404600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29753344,"num_objects":454,"num_object_clones":0,"num_object_copies":1362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14282,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":92,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.250924+0000","last_change":"2024-03-21T13:02:26.228129+0000","last_active":"2024-03-21T13:04:06.250924+0000","last_peered":"2024-03-21T13:04:06.250924+0000","last_clean":"2024-03-21T13:04:06.250924+0000","last_became_active":"2024-03-21T13:02:26.227779+0000","last_became_peered":"2024-03-21T13:02:26.227779+0000","last_unstale":"2024-03-21T13:04:06.250924+0000","last_undegraded":"2024-03-21T13:04:06.250924+0000","last_fullsized":"2024-03-21T13:04:06.250924+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:32.225843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:32.225843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"125'10","reported_seq":287,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.251825+0000","last_change":"2024-03-21T13:02:26.263146+0000","last_active":"2024-03-21T13:04:06.251825+0000","last_peered":"2024-03-21T13:04:06.251825+0000","last_clean":"2024-03-21T13:04:06.251825+0000","last_became_active":"2024-03-21T13:02:26.235954+0000","last_became_peered":"2024-03-21T13:02:26.235954+0000","last_unstale":"2024-03-21T13:04:06.251825+0000","last_undegraded":"2024-03-21T13:04:06.251825+0000","last_fullsized":"2024-03-21T13:04:06.251825+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:41.660019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:41.660019+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:08.100473+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":200,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.248777+0000","last_change":"2024-03-21T13:00:25.960463+0000","last_active":"2024-03-21T13:04:06.248777+0000","last_peered":"2024-03-21T13:04:06.248777+0000","last_clean":"2024-03-21T13:04:06.248777+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:04:06.248777+0000","last_undegraded":"2024-03-21T13:04:06.248777+0000","last_fullsized":"2024-03-21T13:04:06.248777+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:25.960383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:25.960383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.381959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":147,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-21T13:04:06.246873+0000","last_change":"2024-03-21T13:01:27.363819+0000","last_active":"2024-03-21T13:04:06.246873+0000","last_peered":"2024-03-21T13:04:06.246873+0000","last_clean":"2024-03-21T13:04:06.246873+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:04:06.246873+0000","last_undegraded":"2024-03-21T13:04:06.246873+0000","last_fullsized":"2024-03-21T13:04:06.246873+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:00:10.785752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:10.785752+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:04:07.367994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":557252632,"num_objects":8504,"num_object_clones":0,"num_object_copies":25512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":264,"num_objects_unfound":0,"num_objects_dirty":8504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226374,"num_write_kb":7516033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35897,"num_bytes_recovered":2352414744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802317824,"data_stored":1802305608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64371,"ondisk_log_size":64371,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":323,"num_read_kb":318,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":684,"num_read_kb":441,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":3548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":424,"ondisk_log_size":424,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168647,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":436328,"kb_used_data":274656,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93308824,"statfs":{"total":95995035648,"available":95548235776,"internally_reserved":0,"allocated":281247744,"data_stored":280781750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":165542346},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462989,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":406836,"kb_used_data":259052,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93338316,"statfs":{"total":95995035648,"available":95578435584,"internally_reserved":0,"allocated":265269248,"data_stored":264790952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574949,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738475,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":435048,"kb_used_data":295456,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93310104,"statfs":{"total":95995035648,"available":95549546496,"internally_reserved":0,"allocated":302546944,"data_stored":302079765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":464712,"kb_used_data":311296,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93280440,"statfs":{"total":95995035648,"available":95519170560,"internally_reserved":0,"allocated":318767104,"data_stored":318331973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182492,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":651532,"kb_used_data":395396,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93093620,"statfs":{"total":95995035648,"available":95327866880,"internally_reserved":0,"allocated":404885504,"data_stored":404382552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":262207943},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018966,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":447128,"kb_used_data":271568,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93298024,"statfs":{"total":95995035648,"available":95537176576,"internally_reserved":0,"allocated":278085632,"data_stored":277634642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724625,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":275517440,"data_stored":275513368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34668544,"data_stored":34668544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":281018368,"data_stored":281018368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":363794432,"data_stored":363790360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":253296640,"data_stored":253296640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":116523008,"data_stored":116523008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":232722432,"data_stored":232718360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":244776960,"data_stored":244776960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:04:12.724 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-21T13:04:12.764 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:12.796 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:12.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:12.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:12.931 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:12.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:13.032 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:13.069 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:13.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:13.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:13.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:13.206 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:13.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:13.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:13.333 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:13.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:13.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:13.434 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:13.472 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:13.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:13.573 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:13.612 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:13.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:13.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:13.714 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:13.751 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:13.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:13.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:13.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:13 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:13.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:13 smithi204 ceph-mon[33153]: 4.2 scrub starts 2024-03-21T13:04:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:13 smithi204 ceph-mon[33153]: 4.2 scrub ok 2024-03-21T13:04:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:13 smithi204 ceph-mon[33153]: 3.12 scrub starts 2024-03-21T13:04:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:13 smithi204 ceph-mon[33153]: 3.12 scrub ok 2024-03-21T13:04:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:13 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:13 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:13 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[29621]: 4.2 scrub starts 2024-03-21T13:04:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[29621]: 4.2 scrub ok 2024-03-21T13:04:13.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[29621]: 3.12 scrub starts 2024-03-21T13:04:13.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[29621]: 3.12 scrub ok 2024-03-21T13:04:13.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:13.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:13.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[40675]: 4.2 scrub starts 2024-03-21T13:04:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[40675]: 4.2 scrub ok 2024-03-21T13:04:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[40675]: 3.12 scrub starts 2024-03-21T13:04:13.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[40675]: 3.12 scrub ok 2024-03-21T13:04:13.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:13.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:13.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:13 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:13.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:13.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:13.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:13.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:13.979 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:14.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:14.017 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:14.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:14.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:14.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:14.163 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:14.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:14.263 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:14.303 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:14.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:14.404 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:14.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:14.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:14.537 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:14.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:14.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:14.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:14.671 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:14.707 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:14.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:14 smithi204 ceph-mon[33153]: pgmap v226: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 532 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 688 B/s rd, 0 B/s wr, 858 op/s; 264/26106 objects misplaced (1.011%) 2024-03-21T13:04:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:14 smithi204 ceph-mon[33153]: 3.3 scrub starts 2024-03-21T13:04:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:14 smithi204 ceph-mon[33153]: 3.3 scrub ok 2024-03-21T13:04:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:14 smithi204 ceph-mon[33153]: 4.11 scrub starts 2024-03-21T13:04:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:14 smithi204 ceph-mon[33153]: 4.11 scrub ok 2024-03-21T13:04:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:14 smithi204 ceph-mon[33153]: 4.15 scrub starts 2024-03-21T13:04:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:14 smithi204 ceph-mon[33153]: 4.15 scrub ok 2024-03-21T13:04:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[29621]: pgmap v226: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 532 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 688 B/s rd, 0 B/s wr, 858 op/s; 264/26106 objects misplaced (1.011%) 2024-03-21T13:04:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[29621]: 3.3 scrub starts 2024-03-21T13:04:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[29621]: 3.3 scrub ok 2024-03-21T13:04:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[29621]: 4.11 scrub starts 2024-03-21T13:04:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[29621]: 4.11 scrub ok 2024-03-21T13:04:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[29621]: 4.15 scrub starts 2024-03-21T13:04:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[29621]: 4.15 scrub ok 2024-03-21T13:04:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[40675]: pgmap v226: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 532 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 688 B/s rd, 0 B/s wr, 858 op/s; 264/26106 objects misplaced (1.011%) 2024-03-21T13:04:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[40675]: 3.3 scrub starts 2024-03-21T13:04:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[40675]: 3.3 scrub ok 2024-03-21T13:04:14.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[40675]: 4.11 scrub starts 2024-03-21T13:04:14.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[40675]: 4.11 scrub ok 2024-03-21T13:04:14.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[40675]: 4.15 scrub starts 2024-03-21T13:04:14.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:14 smithi163 ceph-mon[40675]: 4.15 scrub ok 2024-03-21T13:04:14.809 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:14.847 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:14.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:14.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:14.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:14.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:14.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:15.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:15.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:15.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:15.216 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:15.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:15.259 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:15.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:15.360 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:15.397 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:15.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:15.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:15.499 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:15.535 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:15.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:15.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:15.637 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:15.675 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:15.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:15.776 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:15.815 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:15.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:15.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:15.948 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:15.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:16.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:04:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:04:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:04:16.049 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:16.074 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:16.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:16.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:16.176 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:16.194 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Removed 117438 objects 2024-03-21T13:04:16.194 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Clean up completed and total clean up time :112.099 2024-03-21T13:04:16.202 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:16.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:16.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:16.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:16.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:16.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:16.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:16.430 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:16.453 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:16.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:16.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:16.555 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:16.579 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:16.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:16.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:16.680 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:16.704 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:16.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:16 smithi204 ceph-mon[33153]: pgmap v227: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 297 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 72/14817 objects misplaced (0.486%) 2024-03-21T13:04:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:16 smithi204 ceph-mon[33153]: 3.1a scrub starts 2024-03-21T13:04:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:16 smithi204 ceph-mon[33153]: 3.1a scrub ok 2024-03-21T13:04:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:16 smithi204 ceph-mon[33153]: osdmap e156: 8 total, 8 up, 6 in 2024-03-21T13:04:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:16 smithi163 ceph-mon[29621]: pgmap v227: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 297 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 72/14817 objects misplaced (0.486%) 2024-03-21T13:04:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:16 smithi163 ceph-mon[29621]: 3.1a scrub starts 2024-03-21T13:04:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:16 smithi163 ceph-mon[29621]: 3.1a scrub ok 2024-03-21T13:04:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:16 smithi163 ceph-mon[29621]: osdmap e156: 8 total, 8 up, 6 in 2024-03-21T13:04:16.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:16 smithi163 ceph-mon[40675]: pgmap v227: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 297 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 72/14817 objects misplaced (0.486%) 2024-03-21T13:04:16.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:16 smithi163 ceph-mon[40675]: 3.1a scrub starts 2024-03-21T13:04:16.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:16 smithi163 ceph-mon[40675]: 3.1a scrub ok 2024-03-21T13:04:16.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:16 smithi163 ceph-mon[40675]: osdmap e156: 8 total, 8 up, 6 in 2024-03-21T13:04:16.806 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:16.830 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:16.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:16.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:16.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:16.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:17.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:17.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:17.083 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:17.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:17.184 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:17.208 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:17.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:17.310 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:17.334 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:17.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:17.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:17.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:17.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:17.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:17 smithi163 ceph-mon[29621]: 5.1d scrub starts 2024-03-21T13:04:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:17 smithi163 ceph-mon[29621]: 5.1d scrub ok 2024-03-21T13:04:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:17 smithi163 ceph-mon[29621]: 5.9 scrub starts 2024-03-21T13:04:17.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:17 smithi163 ceph-mon[29621]: 5.9 scrub ok 2024-03-21T13:04:17.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:17 smithi163 ceph-mon[40675]: 5.1d scrub starts 2024-03-21T13:04:17.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:17 smithi163 ceph-mon[40675]: 5.1d scrub ok 2024-03-21T13:04:17.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:17 smithi163 ceph-mon[40675]: 5.9 scrub starts 2024-03-21T13:04:17.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:17 smithi163 ceph-mon[40675]: 5.9 scrub ok 2024-03-21T13:04:17.563 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:17.587 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:17.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:17.689 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:17.715 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:17.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:17.725 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:04:17.725 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:04:17.725 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-21T13:04:17.725 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 1.000000 2024-03-21T13:04:17.725 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-21T13:04:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:17 smithi204 ceph-mon[33153]: 5.1d scrub starts 2024-03-21T13:04:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:17 smithi204 ceph-mon[33153]: 5.1d scrub ok 2024-03-21T13:04:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:17 smithi204 ceph-mon[33153]: 5.9 scrub starts 2024-03-21T13:04:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:17 smithi204 ceph-mon[33153]: 5.9 scrub ok 2024-03-21T13:04:17.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:17.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:17.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:17.946 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:17.971 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:17.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:18.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:18.097 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:18.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:18.198 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:18.221 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:18.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:18.323 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:18.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:18.348 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:18.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:18.449 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:18.475 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:18.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:18.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[29621]: pgmap v229: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 117 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s; 72/6183 objects misplaced (1.164%) 2024-03-21T13:04:18.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[29621]: 3.1c scrub starts 2024-03-21T13:04:18.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[29621]: 3.1c scrub ok 2024-03-21T13:04:18.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[29621]: 2.1 scrub starts 2024-03-21T13:04:18.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[29621]: 2.1 scrub ok 2024-03-21T13:04:18.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[29621]: 7.5 scrub starts 2024-03-21T13:04:18.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[29621]: 7.5 scrub ok 2024-03-21T13:04:18.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[40675]: pgmap v229: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 117 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s; 72/6183 objects misplaced (1.164%) 2024-03-21T13:04:18.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[40675]: 3.1c scrub starts 2024-03-21T13:04:18.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[40675]: 3.1c scrub ok 2024-03-21T13:04:18.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[40675]: 2.1 scrub starts 2024-03-21T13:04:18.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[40675]: 2.1 scrub ok 2024-03-21T13:04:18.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[40675]: 7.5 scrub starts 2024-03-21T13:04:18.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:18 smithi163 ceph-mon[40675]: 7.5 scrub ok 2024-03-21T13:04:18.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:18.604 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:18.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:18.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:18.706 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:18.732 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:18.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:18 smithi204 ceph-mon[33153]: pgmap v229: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 117 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s; 72/6183 objects misplaced (1.164%) 2024-03-21T13:04:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:18 smithi204 ceph-mon[33153]: 3.1c scrub starts 2024-03-21T13:04:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:18 smithi204 ceph-mon[33153]: 3.1c scrub ok 2024-03-21T13:04:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:18 smithi204 ceph-mon[33153]: 2.1 scrub starts 2024-03-21T13:04:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:18 smithi204 ceph-mon[33153]: 2.1 scrub ok 2024-03-21T13:04:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:18 smithi204 ceph-mon[33153]: 7.5 scrub starts 2024-03-21T13:04:18.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:18 smithi204 ceph-mon[33153]: 7.5 scrub ok 2024-03-21T13:04:18.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:18.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:18.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:18.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:18.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:18.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:19.033 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:04:19.033 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:04:19.093 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:19.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:04:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:19.119 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:19.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:19.221 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:19.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:19.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:19.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:19.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:19.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:19.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:19.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:19.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:19.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:19.508 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:19.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:19.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[29621]: 3.1f scrub starts 2024-03-21T13:04:19.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[29621]: 3.1f scrub ok 2024-03-21T13:04:19.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-03-21T13:04:19.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]: dispatch 2024-03-21T13:04:19.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]: dispatch 2024-03-21T13:04:19.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [7, 0]}]: dispatch 2024-03-21T13:04:19.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:19.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[40675]: 3.1f scrub starts 2024-03-21T13:04:19.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[40675]: 3.1f scrub ok 2024-03-21T13:04:19.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-03-21T13:04:19.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]: dispatch 2024-03-21T13:04:19.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]: dispatch 2024-03-21T13:04:19.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [7, 0]}]: dispatch 2024-03-21T13:04:19.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:19.610 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:19.633 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:19.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:19.734 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:19.741 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:04:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:19 smithi204 ceph-mon[33153]: 3.1f scrub starts 2024-03-21T13:04:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:19 smithi204 ceph-mon[33153]: 3.1f scrub ok 2024-03-21T13:04:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-03-21T13:04:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]: dispatch 2024-03-21T13:04:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]: dispatch 2024-03-21T13:04:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [7, 0]}]: dispatch 2024-03-21T13:04:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:19.760 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:19.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:19.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:19.762 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:04:19.781 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:04:19.801 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:04:19.822 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:04:19.842 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:04:19.861 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:19.862 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:04:19.885 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:04:19.885 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:19.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:19.886 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:04:19.887 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:04:19.887 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:04:19.887 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:04:19.887 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:04:19.887 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:04:19.887 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:04:19.887 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:04:19.987 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:20.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:20.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:20.144 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:20.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:20.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:20.273 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:20.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:20.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:20.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-21T13:04:20.375 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:20.401 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:20.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:20.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:20.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:20.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:20.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:20.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:20.653 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[29621]: pgmap v230: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 117 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 830 op/s; 72/6183 objects misplaced (1.164%) 2024-03-21T13:04:20.653 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[29621]: 5.f scrub starts 2024-03-21T13:04:20.653 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[29621]: 5.f scrub ok 2024-03-21T13:04:20.653 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-03-21T13:04:20.653 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]': finished 2024-03-21T13:04:20.653 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]': finished 2024-03-21T13:04:20.653 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [7, 0]}]': finished 2024-03-21T13:04:20.653 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[29621]: osdmap e157: 8 total, 8 up, 6 in 2024-03-21T13:04:20.653 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[29621]: 3.8 scrub starts 2024-03-21T13:04:20.653 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[29621]: 3.8 scrub ok 2024-03-21T13:04:20.653 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/943658236' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:04:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[40675]: pgmap v230: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 117 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 830 op/s; 72/6183 objects misplaced (1.164%) 2024-03-21T13:04:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[40675]: 5.f scrub starts 2024-03-21T13:04:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[40675]: 5.f scrub ok 2024-03-21T13:04:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-03-21T13:04:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]': finished 2024-03-21T13:04:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]': finished 2024-03-21T13:04:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [7, 0]}]': finished 2024-03-21T13:04:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[40675]: osdmap e157: 8 total, 8 up, 6 in 2024-03-21T13:04:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[40675]: 3.8 scrub starts 2024-03-21T13:04:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[40675]: 3.8 scrub ok 2024-03-21T13:04:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:20 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/943658236' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:04:20.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:20.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:20.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:20 smithi204 ceph-mon[33153]: pgmap v230: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 117 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 830 op/s; 72/6183 objects misplaced (1.164%) 2024-03-21T13:04:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:20 smithi204 ceph-mon[33153]: 5.f scrub starts 2024-03-21T13:04:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:20 smithi204 ceph-mon[33153]: 5.f scrub ok 2024-03-21T13:04:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-03-21T13:04:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]': finished 2024-03-21T13:04:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]': finished 2024-03-21T13:04:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [7, 0]}]': finished 2024-03-21T13:04:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:20 smithi204 ceph-mon[33153]: osdmap e157: 8 total, 8 up, 6 in 2024-03-21T13:04:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:20 smithi204 ceph-mon[33153]: 3.8 scrub starts 2024-03-21T13:04:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:20 smithi204 ceph-mon[33153]: 3.8 scrub ok 2024-03-21T13:04:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:20 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/943658236' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:04:20.761 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:20.785 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:20.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:20.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:20.914 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:20.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:20.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:21.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:21.043 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:21.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:21.145 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:21.168 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:21.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:21.195 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:21.213 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:21.242 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:21.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:21.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:21.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:21.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:21.418 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-21T13:04:21.418 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-21T13:04:21.428 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:21.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:21.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:21 smithi163 ceph-mon[29621]: osdmap e158: 8 total, 8 up, 6 in 2024-03-21T13:04:21.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:21 smithi163 ceph-mon[29621]: 3.7 scrub starts 2024-03-21T13:04:21.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:21 smithi163 ceph-mon[29621]: 3.7 scrub ok 2024-03-21T13:04:21.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:21 smithi163 ceph-mon[29621]: osdmap e159: 8 total, 8 up, 6 in 2024-03-21T13:04:21.443 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:21 smithi163 ceph-mon[40675]: osdmap e158: 8 total, 8 up, 6 in 2024-03-21T13:04:21.443 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:21 smithi163 ceph-mon[40675]: 3.7 scrub starts 2024-03-21T13:04:21.443 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:21 smithi163 ceph-mon[40675]: 3.7 scrub ok 2024-03-21T13:04:21.443 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:21 smithi163 ceph-mon[40675]: osdmap e159: 8 total, 8 up, 6 in 2024-03-21T13:04:21.530 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:21.554 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:21.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:21.656 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:21.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:21.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:21.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:21 smithi204 ceph-mon[33153]: osdmap e158: 8 total, 8 up, 6 in 2024-03-21T13:04:21.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:21 smithi204 ceph-mon[33153]: 3.7 scrub starts 2024-03-21T13:04:21.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:21 smithi204 ceph-mon[33153]: 3.7 scrub ok 2024-03-21T13:04:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:21 smithi204 ceph-mon[33153]: osdmap e159: 8 total, 8 up, 6 in 2024-03-21T13:04:21.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:04:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:04:21.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:21.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:21.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:21.916 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:21.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:21.942 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:21.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:22.043 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:22.072 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:22.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:22.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:22.174 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:22.198 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:22.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:22.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:22.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:22.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:22.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:22.430 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:22.454 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:22.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:22.555 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:22.579 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:22.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:22.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:22.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:22 smithi163 ceph-mon[29621]: pgmap v233: 148 pgs: 1 active+remapped+backfill_toofull, 4 peering, 143 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 790 op/s 2024-03-21T13:04:22.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:22 smithi163 ceph-mon[29621]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T13:04:22.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:22 smithi163 ceph-mon[29621]: 7.6 scrub starts 2024-03-21T13:04:22.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:22 smithi163 ceph-mon[29621]: 7.6 scrub ok 2024-03-21T13:04:22.615 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:22 smithi163 ceph-mon[40675]: pgmap v233: 148 pgs: 1 active+remapped+backfill_toofull, 4 peering, 143 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 790 op/s 2024-03-21T13:04:22.615 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:22 smithi163 ceph-mon[40675]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T13:04:22.615 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:22 smithi163 ceph-mon[40675]: 7.6 scrub starts 2024-03-21T13:04:22.615 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:22 smithi163 ceph-mon[40675]: 7.6 scrub ok 2024-03-21T13:04:22.680 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:22.704 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:22.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:22.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:22.725 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:22 smithi204 ceph-mon[33153]: pgmap v233: 148 pgs: 1 active+remapped+backfill_toofull, 4 peering, 143 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 790 op/s 2024-03-21T13:04:22.725 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:22 smithi204 ceph-mon[33153]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T13:04:22.725 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:22 smithi204 ceph-mon[33153]: 7.6 scrub starts 2024-03-21T13:04:22.726 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:22 smithi204 ceph-mon[33153]: 7.6 scrub ok 2024-03-21T13:04:22.806 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:22.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:22.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:22.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:22.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:22.962 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:22.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:23.063 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:23.088 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:23.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:23.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:23.097 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:23.139 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:04:23.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:23.218 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:23.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:23.320 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:23.345 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:23.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:23.446 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:23.470 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:23.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:23.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:23 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:23.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:23 smithi163 ceph-mon[29621]: 4.17 scrub starts 2024-03-21T13:04:23.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:23 smithi163 ceph-mon[29621]: 4.17 scrub ok 2024-03-21T13:04:23.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:23 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:23.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:23 smithi163 ceph-mon[40675]: 4.17 scrub starts 2024-03-21T13:04:23.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:23 smithi163 ceph-mon[40675]: 4.17 scrub ok 2024-03-21T13:04:23.573 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:23.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:23.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:23.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:23.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:23.724 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:23.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:23.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:23 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:23.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:23 smithi204 ceph-mon[33153]: 4.17 scrub starts 2024-03-21T13:04:23.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:23 smithi204 ceph-mon[33153]: 4.17 scrub ok 2024-03-21T13:04:23.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:23.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:23.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:23.957 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:23.982 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:23.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:24.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:24.107 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:24.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:24.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:24.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:24.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:24.239 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:24.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:24.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:24.367 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:24.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:24.510 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:24.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:24.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:24.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:24.641 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:24 smithi163 ceph-mon[29621]: pgmap v235: 148 pgs: 1 active+remapped+backfill_toofull, 4 peering, 143 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 311 op/s 2024-03-21T13:04:24.641 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:24 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:04:24.641 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:24 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:04:24.641 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:24 smithi163 ceph-mon[29621]: osdmap e160: 8 total, 8 up, 6 in 2024-03-21T13:04:24.641 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:24 smithi163 ceph-mon[40675]: pgmap v235: 148 pgs: 1 active+remapped+backfill_toofull, 4 peering, 143 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 311 op/s 2024-03-21T13:04:24.641 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:24 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:04:24.641 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:24 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:04:24.641 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:24 smithi163 ceph-mon[40675]: osdmap e160: 8 total, 8 up, 6 in 2024-03-21T13:04:24.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:24.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:24 smithi204 ceph-mon[33153]: pgmap v235: 148 pgs: 1 active+remapped+backfill_toofull, 4 peering, 143 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 311 op/s 2024-03-21T13:04:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:24 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:04:24.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:24 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:04:24.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:24 smithi204 ceph-mon[33153]: osdmap e160: 8 total, 8 up, 6 in 2024-03-21T13:04:24.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:24.796 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:24.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:24.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:24.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:24.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:25.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:25.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:25.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:25.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:25.157 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:25.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:25.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:25.311 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:25.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:25.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:25.440 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:25.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:25.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:25.542 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:25.566 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:25.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:25.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:25 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/386745310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:04:25.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:25 smithi163 ceph-mon[29621]: pgmap v237: 148 pgs: 1 active+remapped+backfill_toofull, 3 peering, 144 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 348 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T13:04:25.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:25 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/386745310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:04:25.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:25 smithi163 ceph-mon[40675]: pgmap v237: 148 pgs: 1 active+remapped+backfill_toofull, 3 peering, 144 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 348 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T13:04:25.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:25.696 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:25.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:25.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:25.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:25 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/386745310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:04:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:25 smithi204 ceph-mon[33153]: pgmap v237: 148 pgs: 1 active+remapped+backfill_toofull, 3 peering, 144 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 348 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T13:04:25.798 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:25.825 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:25.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:25.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:25.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:25.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:25.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:26.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:04:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:04:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:04:26.057 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:26.082 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:26.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:26.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:26.184 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:26.208 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:26.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:26.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:26.288 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:04:26.310 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:26.333 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:26.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:26.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:26.379 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-21T13:04:26.392 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:26 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:04:26.392 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:26 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/386745310' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:04:26.393 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:26 smithi163 ceph-mon[29621]: osdmap e161: 8 total, 8 up, 6 in 2024-03-21T13:04:26.393 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:26 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/366580399' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:04:26.393 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:26 smithi163 ceph-mon[29621]: 4.8 scrub starts 2024-03-21T13:04:26.393 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:26 smithi163 ceph-mon[29621]: 4.8 scrub ok 2024-03-21T13:04:26.393 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:26 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:04:26.393 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:26 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/386745310' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:04:26.393 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:26 smithi163 ceph-mon[40675]: osdmap e161: 8 total, 8 up, 6 in 2024-03-21T13:04:26.393 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:26 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/366580399' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:04:26.393 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:26 smithi163 ceph-mon[40675]: 4.8 scrub starts 2024-03-21T13:04:26.393 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:26 smithi163 ceph-mon[40675]: 4.8 scrub ok 2024-03-21T13:04:26.436 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:26.459 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:26.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:26.561 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:26.584 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:26.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:26.686 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:26.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:26.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:26 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:04:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:26 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/386745310' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:04:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:26 smithi204 ceph-mon[33153]: osdmap e161: 8 total, 8 up, 6 in 2024-03-21T13:04:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:26 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/366580399' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:04:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:26 smithi204 ceph-mon[33153]: 4.8 scrub starts 2024-03-21T13:04:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:26 smithi204 ceph-mon[33153]: 4.8 scrub ok 2024-03-21T13:04:26.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:26.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:26.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:26.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:26.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:26.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:26.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:27.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:27.099 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:27.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:27.201 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:27.227 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:27.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:27.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:27.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:27.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:27.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:27.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:27.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:27.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:27.482 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:27.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:27.583 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:27.608 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:27.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:27.709 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:27.734 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:27.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:27.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:27 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/366580399' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:04:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:27 smithi204 ceph-mon[33153]: osdmap e162: 8 total, 8 up, 6 in 2024-03-21T13:04:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:27 smithi204 ceph-mon[33153]: pgmap v240: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 933 B/s rd, 0 op/s; 112 B/s, 1 objects/s recovering 2024-03-21T13:04:27.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:27 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/366580399' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:04:27.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:27 smithi163 ceph-mon[29621]: osdmap e162: 8 total, 8 up, 6 in 2024-03-21T13:04:27.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:27 smithi163 ceph-mon[29621]: pgmap v240: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 933 B/s rd, 0 op/s; 112 B/s, 1 objects/s recovering 2024-03-21T13:04:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:27 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/366580399' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:04:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:27 smithi163 ceph-mon[40675]: osdmap e162: 8 total, 8 up, 6 in 2024-03-21T13:04:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:27 smithi163 ceph-mon[40675]: pgmap v240: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 933 B/s rd, 0 op/s; 112 B/s, 1 objects/s recovering 2024-03-21T13:04:27.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:27.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:27.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:27.962 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:27.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:27.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:27.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:28.089 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:28.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:28.115 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:28.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:28.216 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:28.240 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:28.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:28.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:28.342 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:28.367 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:28.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:28.469 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:28.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:28.594 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:28.617 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:28.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:28.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:28.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:28.745 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:28.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:28 smithi204 ceph-mon[33153]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-21T13:04:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:28 smithi204 ceph-mon[33153]: osdmap e163: 8 total, 8 up, 6 in 2024-03-21T13:04:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:28 smithi204 ceph-mon[33153]: 6.7 scrub starts 2024-03-21T13:04:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:28 smithi204 ceph-mon[33153]: 6.7 scrub ok 2024-03-21T13:04:28.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:28 smithi163 ceph-mon[29621]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-21T13:04:28.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:28 smithi163 ceph-mon[29621]: osdmap e163: 8 total, 8 up, 6 in 2024-03-21T13:04:28.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:28 smithi163 ceph-mon[29621]: 6.7 scrub starts 2024-03-21T13:04:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:28 smithi163 ceph-mon[29621]: 6.7 scrub ok 2024-03-21T13:04:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:28 smithi163 ceph-mon[40675]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-21T13:04:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:28 smithi163 ceph-mon[40675]: osdmap e163: 8 total, 8 up, 6 in 2024-03-21T13:04:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:28 smithi163 ceph-mon[40675]: 6.7 scrub starts 2024-03-21T13:04:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:28 smithi163 ceph-mon[40675]: 6.7 scrub ok 2024-03-21T13:04:28.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:28.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:28.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:28.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:28.995 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:04:29.000 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:29.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:29.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:29.102 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:29.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:29.127 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:29.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:29.229 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:29.253 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:29.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:29.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:29.354 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:29.379 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:29.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:29.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:29.398 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:04:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:04:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-03-21T13:04:29.398 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:04:29.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:29.507 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:29.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:29.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:29.609 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:29.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:29.635 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:29.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:29.736 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:29 smithi204 ceph-mon[33153]: 5.18 scrub starts 2024-03-21T13:04:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:29 smithi204 ceph-mon[33153]: 5.18 scrub ok 2024-03-21T13:04:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:29 smithi204 ceph-mon[33153]: 7.4 scrub starts 2024-03-21T13:04:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:29 smithi204 ceph-mon[33153]: 7.4 scrub ok 2024-03-21T13:04:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:29 smithi204 ceph-mon[33153]: pgmap v242: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 955 B/s rd, 0 op/s; 115 B/s, 1 objects/s recovering 2024-03-21T13:04:29.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:29 smithi204 ceph-mon[33153]: 7.9 scrub starts 2024-03-21T13:04:29.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:29 smithi204 ceph-mon[33153]: 7.9 scrub ok 2024-03-21T13:04:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[29621]: 5.18 scrub starts 2024-03-21T13:04:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[29621]: 5.18 scrub ok 2024-03-21T13:04:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[29621]: 7.4 scrub starts 2024-03-21T13:04:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[29621]: 7.4 scrub ok 2024-03-21T13:04:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[29621]: pgmap v242: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 955 B/s rd, 0 op/s; 115 B/s, 1 objects/s recovering 2024-03-21T13:04:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[29621]: 7.9 scrub starts 2024-03-21T13:04:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[29621]: 7.9 scrub ok 2024-03-21T13:04:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[40675]: 5.18 scrub starts 2024-03-21T13:04:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[40675]: 5.18 scrub ok 2024-03-21T13:04:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[40675]: 7.4 scrub starts 2024-03-21T13:04:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[40675]: 7.4 scrub ok 2024-03-21T13:04:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[40675]: pgmap v242: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 955 B/s rd, 0 op/s; 115 B/s, 1 objects/s recovering 2024-03-21T13:04:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[40675]: 7.9 scrub starts 2024-03-21T13:04:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:29 smithi163 ceph-mon[40675]: 7.9 scrub ok 2024-03-21T13:04:29.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:29.763 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:29.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:29.864 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:29.890 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:29.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:29.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:30.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:30.019 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:30.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:30.120 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:30.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:30.148 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:30.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:30.208 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-21T13:04:30.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:30.274 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:30.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:30.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:30.403 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:30.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:30.505 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:30.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:30 smithi163 ceph-mon[29621]: 6.14 scrub starts 2024-03-21T13:04:30.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:30 smithi163 ceph-mon[29621]: 6.14 scrub ok 2024-03-21T13:04:30.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:30 smithi163 ceph-mon[29621]: 6.1d scrub starts 2024-03-21T13:04:30.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:30 smithi163 ceph-mon[29621]: 6.1d scrub ok 2024-03-21T13:04:30.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:30 smithi163 ceph-mon[29621]: 5.1c scrub starts 2024-03-21T13:04:30.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:30 smithi163 ceph-mon[29621]: 5.1c scrub ok 2024-03-21T13:04:30.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:30 smithi163 ceph-mon[40675]: 6.14 scrub starts 2024-03-21T13:04:30.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:30 smithi163 ceph-mon[40675]: 6.14 scrub ok 2024-03-21T13:04:30.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:30 smithi163 ceph-mon[40675]: 6.1d scrub starts 2024-03-21T13:04:30.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:30 smithi163 ceph-mon[40675]: 6.1d scrub ok 2024-03-21T13:04:30.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:30 smithi163 ceph-mon[40675]: 5.1c scrub starts 2024-03-21T13:04:30.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:30 smithi163 ceph-mon[40675]: 5.1c scrub ok 2024-03-21T13:04:30.529 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:30.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:30.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:30.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:30.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:30.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:30 smithi204 ceph-mon[33153]: 6.14 scrub starts 2024-03-21T13:04:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:30 smithi204 ceph-mon[33153]: 6.14 scrub ok 2024-03-21T13:04:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:30 smithi204 ceph-mon[33153]: 6.1d scrub starts 2024-03-21T13:04:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:30 smithi204 ceph-mon[33153]: 6.1d scrub ok 2024-03-21T13:04:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:30 smithi204 ceph-mon[33153]: 5.1c scrub starts 2024-03-21T13:04:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:30 smithi204 ceph-mon[33153]: 5.1c scrub ok 2024-03-21T13:04:30.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:30.787 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:30.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:30.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:30.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:30.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:31.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:31.043 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:31.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:31.145 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:31.169 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:31.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:31.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:31.186 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:31.187 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:31.271 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:31.295 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:31.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:31.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:31.423 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'unique_pool_0' removed 2024-03-21T13:04:31.441 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:31.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:31.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:31.542 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:31.573 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:31.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:31.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:31.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:31.701 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:31.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:31.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[29621]: 7.7 scrub starts 2024-03-21T13:04:31.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[29621]: 7.7 scrub ok 2024-03-21T13:04:31.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[29621]: 6.1e scrub starts 2024-03-21T13:04:31.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[29621]: 6.1e scrub ok 2024-03-21T13:04:31.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2531934104' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:04:31.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:04:31.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[29621]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 87 B/s, 1 objects/s recovering 2024-03-21T13:04:31.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[29621]: 5.1a scrub starts 2024-03-21T13:04:31.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[29621]: 5.1a scrub ok 2024-03-21T13:04:31.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[40675]: 7.7 scrub starts 2024-03-21T13:04:31.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[40675]: 7.7 scrub ok 2024-03-21T13:04:31.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[40675]: 6.1e scrub starts 2024-03-21T13:04:31.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[40675]: 6.1e scrub ok 2024-03-21T13:04:31.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2531934104' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:04:31.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:04:31.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[40675]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 87 B/s, 1 objects/s recovering 2024-03-21T13:04:31.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[40675]: 5.1a scrub starts 2024-03-21T13:04:31.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:31 smithi163 ceph-mon[40675]: 5.1a scrub ok 2024-03-21T13:04:31.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:04:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:04:31.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:31 smithi204 ceph-mon[33153]: 7.7 scrub starts 2024-03-21T13:04:31.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:31 smithi204 ceph-mon[33153]: 7.7 scrub ok 2024-03-21T13:04:31.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:31 smithi204 ceph-mon[33153]: 6.1e scrub starts 2024-03-21T13:04:31.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:31 smithi204 ceph-mon[33153]: 6.1e scrub ok 2024-03-21T13:04:31.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:31 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2531934104' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:04:31.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:31 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:04:31.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:31 smithi204 ceph-mon[33153]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 87 B/s, 1 objects/s recovering 2024-03-21T13:04:31.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:31 smithi204 ceph-mon[33153]: 5.1a scrub starts 2024-03-21T13:04:31.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:31 smithi204 ceph-mon[33153]: 5.1a scrub ok 2024-03-21T13:04:31.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:31.825 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:31.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:31.926 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:31.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:31.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:32.052 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:32.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:32.078 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:32.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:32.179 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:32.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:32.203 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:32.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:32.304 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:32.327 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:32.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:32.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:32.429 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:32.453 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:32.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:32.555 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:32.579 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:32.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:32.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:32.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:32.682 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[40675]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:04:32.682 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:04:32.682 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:04:32.682 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[40675]: osdmap e164: 8 total, 8 up, 6 in 2024-03-21T13:04:32.682 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:32.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[40675]: 6.16 scrub starts 2024-03-21T13:04:32.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[40675]: 6.16 scrub ok 2024-03-21T13:04:32.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[29621]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:04:32.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:04:32.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:04:32.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[29621]: osdmap e164: 8 total, 8 up, 6 in 2024-03-21T13:04:32.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:32.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[29621]: 6.16 scrub starts 2024-03-21T13:04:32.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:32 smithi163 ceph-mon[29621]: 6.16 scrub ok 2024-03-21T13:04:32.706 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:32.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:32 smithi204 ceph-mon[33153]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:04:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:32 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:04:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:32 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:04:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:32 smithi204 ceph-mon[33153]: osdmap e164: 8 total, 8 up, 6 in 2024-03-21T13:04:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:32 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:32 smithi204 ceph-mon[33153]: 6.16 scrub starts 2024-03-21T13:04:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:32 smithi204 ceph-mon[33153]: 6.16 scrub ok 2024-03-21T13:04:32.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:32.832 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:32.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:32.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:32.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:32.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:33.061 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:33.085 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:33.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:33.187 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:33.211 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:33.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:33.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:33.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:33.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:33.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:33.438 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:33.462 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:33.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:33.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:33.564 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:33.588 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:33.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:33.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:33.613 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:33 smithi163 ceph-mon[29621]: 3.14 scrub starts 2024-03-21T13:04:33.614 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:33 smithi163 ceph-mon[29621]: 3.14 scrub ok 2024-03-21T13:04:33.614 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:33 smithi163 ceph-mon[29621]: pgmap v245: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 645 B/s rd, 0 op/s 2024-03-21T13:04:33.614 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:33 smithi163 ceph-mon[29621]: 6.18 scrub starts 2024-03-21T13:04:33.614 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:33 smithi163 ceph-mon[29621]: 6.18 scrub ok 2024-03-21T13:04:33.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:33 smithi163 ceph-mon[40675]: 3.14 scrub starts 2024-03-21T13:04:33.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:33 smithi163 ceph-mon[40675]: 3.14 scrub ok 2024-03-21T13:04:33.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:33 smithi163 ceph-mon[40675]: pgmap v245: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 645 B/s rd, 0 op/s 2024-03-21T13:04:33.615 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:33 smithi163 ceph-mon[40675]: 6.18 scrub starts 2024-03-21T13:04:33.615 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:33 smithi163 ceph-mon[40675]: 6.18 scrub ok 2024-03-21T13:04:33.690 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:33.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:33.715 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:33.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:33.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:33 smithi204 ceph-mon[33153]: 3.14 scrub starts 2024-03-21T13:04:33.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:33 smithi204 ceph-mon[33153]: 3.14 scrub ok 2024-03-21T13:04:33.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:33 smithi204 ceph-mon[33153]: pgmap v245: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 645 B/s rd, 0 op/s 2024-03-21T13:04:33.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:33 smithi204 ceph-mon[33153]: 6.18 scrub starts 2024-03-21T13:04:33.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:33 smithi204 ceph-mon[33153]: 6.18 scrub ok 2024-03-21T13:04:33.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:33.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:33.842 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:33.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:33.943 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:33.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:33.969 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:33.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:33.975 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:04:33.994 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:04:34.014 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:04:34.034 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:04:34.054 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:04:34.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:34.073 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:04:34.094 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:04:34.095 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:34.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:34.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:34.114 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:04:34.115 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:04:34.115 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:04:34.115 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:04:34.115 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:04:34.116 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:04:34.116 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:04:34.116 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:04:34.116 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:04:34.180 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:04:34.180 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-21T13:04:34.197 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:34.221 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:34.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:34.323 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:34.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:34.348 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:34.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:34.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:34.479 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:34.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:34.580 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:34.604 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:34.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:34.632 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-21T13:04:34.633 INFO:tasks.radosbench:Beginning radosbench... 2024-03-21T13:04:34.633 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-21T13:04:34.633 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-21T13:04:34.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[29621]: 3.a scrub starts 2024-03-21T13:04:34.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[29621]: 3.a scrub ok 2024-03-21T13:04:34.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[29621]: 6.1c scrub starts 2024-03-21T13:04:34.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[29621]: 6.1c scrub ok 2024-03-21T13:04:34.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1450252558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:04:34.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:34.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:34.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[40675]: 3.a scrub starts 2024-03-21T13:04:34.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[40675]: 3.a scrub ok 2024-03-21T13:04:34.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[40675]: 6.1c scrub starts 2024-03-21T13:04:34.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[40675]: 6.1c scrub ok 2024-03-21T13:04:34.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1450252558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:04:34.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:34.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:34 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:34.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:34.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:34.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:34.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:34 smithi204 ceph-mon[33153]: 3.a scrub starts 2024-03-21T13:04:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:34 smithi204 ceph-mon[33153]: 3.a scrub ok 2024-03-21T13:04:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:34 smithi204 ceph-mon[33153]: 6.1c scrub starts 2024-03-21T13:04:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:34 smithi204 ceph-mon[33153]: 6.1c scrub ok 2024-03-21T13:04:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:34 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1450252558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:04:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:34 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:34 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:34.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:34.857 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:34.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:34.959 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:34.984 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:34.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:35.085 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:35.109 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:35.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:35.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:35.211 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:35.235 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:35.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:35.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:35.337 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:35.364 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:35.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:35.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:35.412 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:04:35.466 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:35.490 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:35.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:35.592 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:35.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:35.616 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:35.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:35.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:35 smithi163 ceph-mon[29621]: 6.d scrub starts 2024-03-21T13:04:35.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:35 smithi163 ceph-mon[29621]: 6.d scrub ok 2024-03-21T13:04:35.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:35 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3770702901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:04:35.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:35 smithi163 ceph-mon[29621]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:35.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:35 smithi163 ceph-mon[29621]: 6.8 deep-scrub starts 2024-03-21T13:04:35.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:35 smithi163 ceph-mon[29621]: 6.8 deep-scrub ok 2024-03-21T13:04:35.691 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:04:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:04:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:04:35.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:35 smithi163 ceph-mon[40675]: 6.d scrub starts 2024-03-21T13:04:35.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:35 smithi163 ceph-mon[40675]: 6.d scrub ok 2024-03-21T13:04:35.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:35 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3770702901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:04:35.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:35 smithi163 ceph-mon[40675]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:35.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:35 smithi163 ceph-mon[40675]: 6.8 deep-scrub starts 2024-03-21T13:04:35.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:35 smithi163 ceph-mon[40675]: 6.8 deep-scrub ok 2024-03-21T13:04:35.717 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:35.740 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:35.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:35.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:35.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:35 smithi204 ceph-mon[33153]: 6.d scrub starts 2024-03-21T13:04:35.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:35 smithi204 ceph-mon[33153]: 6.d scrub ok 2024-03-21T13:04:35.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:35 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3770702901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:04:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:35 smithi204 ceph-mon[33153]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:35 smithi204 ceph-mon[33153]: 6.8 deep-scrub starts 2024-03-21T13:04:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:35 smithi204 ceph-mon[33153]: 6.8 deep-scrub ok 2024-03-21T13:04:35.842 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:35.866 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:35.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:35.967 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:35.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:36.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:36.121 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:36.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:36.223 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:36.250 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:36.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:36.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:36.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:36.378 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:36.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:36.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:36.504 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:36.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:36.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:36.606 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:36.629 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:36.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:36.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:36 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:04:36.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:36 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3770702901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:04:36.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:36 smithi163 ceph-mon[29621]: osdmap e165: 8 total, 8 up, 6 in 2024-03-21T13:04:36.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:36 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:04:36.661 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:36 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3770702901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:04:36.661 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:36 smithi163 ceph-mon[40675]: osdmap e165: 8 total, 8 up, 6 in 2024-03-21T13:04:36.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:36 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:04:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:36 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3770702901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:04:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:36 smithi204 ceph-mon[33153]: osdmap e165: 8 total, 8 up, 6 in 2024-03-21T13:04:36.756 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:36.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:36.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:36.858 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:36.883 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:36.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:36.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:37.012 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:37.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:37.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:37.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:37.141 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:37.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:37.242 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:37.269 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:37.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:37.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:37.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:37.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:37.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:37.500 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:37.524 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:37.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:37.626 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:37.650 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:37.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:37 smithi163 ceph-mon[40675]: pgmap v248: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:37.650 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:37 smithi163 ceph-mon[29621]: pgmap v248: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:37.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:37.752 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:37.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:37 smithi204 ceph-mon[33153]: pgmap v248: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:37.780 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:37.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:37.881 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:37.909 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:37.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:37.929 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:04:37.929 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:04:37.931 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:38.010 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:38.034 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:38.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:38.121 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:04:38.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:38.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:38.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:38.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:38.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:38.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:38.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:38.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:38.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:38.419 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:38.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:38.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:38.521 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:38.545 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:38.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:38.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:38.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:38.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:38.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:38.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:38.777 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:38.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:38.802 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:38.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:38.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:38.926 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:38.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:38.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:38.966 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-21T13:04:39.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:39.057 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:39.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:39.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:39.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:39.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:39.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:39.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:39.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:39.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:39.417 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:39.442 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:39.443 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:39.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:39.545 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:39.573 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:39.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:39.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:39.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:04:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:39.700 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:39.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:39.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:39.832 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:39.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:39.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:39.963 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:39.964 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:39 smithi163 ceph-mon[29621]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:04:39.964 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:39 smithi163 ceph-mon[40675]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:04:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:39.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:40.067 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:40.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:40.092 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:40.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:40.193 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:40.219 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:40.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:40.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:39 smithi204 ceph-mon[33153]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:04:40.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:40.348 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:40.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:40.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:40.450 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:40.476 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:40.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:40.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:40.606 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:40.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:40.708 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:40.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:40.733 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:40.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:40.835 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:40.863 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:40.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:40.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:40.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:40.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:41.027 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:41.031 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:41.031 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:41.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:41.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:41.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:41.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:41.225 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:41.250 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:41.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:41.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:41.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:41.380 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:41.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:41.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:41.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:41.506 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:41.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:41.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:41.609 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:41.637 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:41.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:41.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:41.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:04:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:04:41.767 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:41.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:41.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:41.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:41.892 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:41.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:41.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:41.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:41 smithi163 ceph-mon[29621]: pgmap v250: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:41.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:41 smithi163 ceph-mon[40675]: pgmap v250: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:41.994 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:42.018 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:42.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:42.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:42.027 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:41 smithi204 ceph-mon[33153]: pgmap v250: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:42.120 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:42.144 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:42.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:42.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:42.246 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:42.270 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:42.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:42.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:42.372 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:42.396 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:42.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:42.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:42.522 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:42.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:42.623 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:42.647 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:42.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:42.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:42.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:42.773 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:42.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:42.801 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:42 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:42.801 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:42 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3613778317' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:04:42.801 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:42 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:04:42.801 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:42 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:42.801 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:42 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3613778317' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:04:42.801 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:42 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:04:42.875 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:42.898 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:42.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:43.000 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:43.024 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:43.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:43.125 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:43.149 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:43.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:43.251 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:43.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:42 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:43.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:42 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3613778317' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:04:43.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:42 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:04:43.274 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:43.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:43.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:43.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:43.403 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:43.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:43.505 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:43.531 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:43.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:43.632 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:43.660 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:43.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:43.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:43.762 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:43.785 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:43.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:43.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:43.791 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:04:43.791 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:04:43.792 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[29621]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:04:43.792 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:04:43.792 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:04:43.792 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:04:43.792 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[29621]: osdmap e166: 8 total, 8 up, 6 in 2024-03-21T13:04:43.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:43.911 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:43.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:43.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:44.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:44.038 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:44.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:44.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:44.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:44.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:44.165 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:44.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:44.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:04:44.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:04:44.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[40675]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:04:44.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:04:44.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:04:44.174 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:04:44.175 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:43 smithi163 ceph-mon[40675]: osdmap e166: 8 total, 8 up, 6 in 2024-03-21T13:04:44.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:43 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:04:44.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:43 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:04:44.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:43 smithi204 ceph-mon[33153]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:04:44.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:43 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:04:44.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:43 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:04:44.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:43 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:04:44.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:43 smithi204 ceph-mon[33153]: osdmap e166: 8 total, 8 up, 6 in 2024-03-21T13:04:44.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:44.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:44.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:44.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:44.392 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:44.417 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:44.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:44.518 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:44.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:44.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:44.552 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.0 to 1 (10000) 2024-03-21T13:04:44.644 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:44.669 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:44.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:44.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:44.771 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:44.794 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:44.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:44.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:44.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:44.924 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:44.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:45.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:45.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:45.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:45.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:45.155 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:45.182 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:45.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:45.202 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-21T13:04:45.285 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:45.310 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:45.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:45.411 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:45.434 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:45.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:45.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:45.564 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:45.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:45.665 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:45.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:45.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:45.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:45.795 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:45.819 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:45.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:45.921 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:45.927 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:04:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:04:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:04:45.944 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:45.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:46.035 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'unique_pool_1' created 2024-03-21T13:04:46.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:46.074 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:46.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:46.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:46.175 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:46.202 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:46.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:46.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:45 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:04:46.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:45 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:04:46.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:45 smithi204 ceph-mon[33153]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:46.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:45 smithi204 ceph-mon[33153]: osdmap e167: 8 total, 8 up, 6 in 2024-03-21T13:04:46.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:45 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:04:46.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:45 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:04:46.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:45 smithi163 ceph-mon[40675]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:46.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:45 smithi163 ceph-mon[40675]: osdmap e167: 8 total, 8 up, 6 in 2024-03-21T13:04:46.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:45 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:04:46.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:45 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:04:46.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:45 smithi163 ceph-mon[29621]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:46.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:45 smithi163 ceph-mon[29621]: osdmap e167: 8 total, 8 up, 6 in 2024-03-21T13:04:46.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:46.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:46.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:46.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:46.431 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:46.456 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:46.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:46.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:46.509 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:04:46.529 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:04:46.550 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:04:46.557 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:46.567 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:04:46.583 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:46.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:46.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:46.586 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:04:46.608 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:04:46.627 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:04:46.647 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:04:46.648 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:04:46.648 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:04:46.648 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:04:46.648 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:04:46.648 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:04:46.649 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:04:46.649 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:04:46.649 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:04:46.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:46.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:46.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:46.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:46.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:46.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:46.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:46.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:46.940 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:46.966 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:46.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:46.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:46.976 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-21T13:04:47.067 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:47.090 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:47.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:47.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:47.192 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:47.217 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:47.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:47.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:47.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:46 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4230819985' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-21T13:04:47.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:46 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-21T13:04:47.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:46 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-21T13:04:47.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:46 smithi204 ceph-mon[33153]: osdmap e168: 8 total, 8 up, 6 in 2024-03-21T13:04:47.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:46 smithi204 ceph-mon[33153]: osdmap e169: 8 total, 8 up, 6 in 2024-03-21T13:04:47.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:46 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2106369154' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:04:47.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:46 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4230819985' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-21T13:04:47.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:46 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-21T13:04:47.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:46 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-21T13:04:47.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:46 smithi163 ceph-mon[29621]: osdmap e168: 8 total, 8 up, 6 in 2024-03-21T13:04:47.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:46 smithi163 ceph-mon[29621]: osdmap e169: 8 total, 8 up, 6 in 2024-03-21T13:04:47.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:46 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2106369154' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:04:47.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:46 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4230819985' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-21T13:04:47.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:46 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-21T13:04:47.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:46 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-21T13:04:47.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:46 smithi163 ceph-mon[40675]: osdmap e168: 8 total, 8 up, 6 in 2024-03-21T13:04:47.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:46 smithi163 ceph-mon[40675]: osdmap e169: 8 total, 8 up, 6 in 2024-03-21T13:04:47.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:46 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2106369154' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:04:47.319 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:47.347 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:47.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:47.448 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:47.471 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:47.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:47.573 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:47.596 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:47.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:47.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:47.722 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:47.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:47.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:47.823 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:47.847 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:47.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:47.948 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:47.972 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:47.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:47.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:48.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:47 smithi163 ceph-mon[29621]: pgmap v257: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:48.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:47 smithi163 ceph-mon[29621]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:04:48.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:47 smithi163 ceph-mon[29621]: osdmap e170: 8 total, 8 up, 6 in 2024-03-21T13:04:48.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:47 smithi163 ceph-mon[40675]: pgmap v257: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:48.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:47 smithi163 ceph-mon[40675]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:04:48.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:47 smithi163 ceph-mon[40675]: osdmap e170: 8 total, 8 up, 6 in 2024-03-21T13:04:48.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:48.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:48.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:48.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:48.204 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:48.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:48.229 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:48.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:48.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:47 smithi204 ceph-mon[33153]: pgmap v257: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:48.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:47 smithi204 ceph-mon[33153]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:04:48.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:47 smithi204 ceph-mon[33153]: osdmap e170: 8 total, 8 up, 6 in 2024-03-21T13:04:48.330 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:48.355 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:48.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:48.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:48.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:48.480 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:48.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:48.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:48.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:48.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:48.611 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:48.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:48.712 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:48.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:48.736 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:48.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:48.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:48.865 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:48.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:48.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:48.967 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:48.995 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:48.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:49.097 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:49.123 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:49.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:49.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:49.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:49.253 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:49.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:49.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:49.355 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:49.383 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:49.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:49.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:49.485 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:49.512 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:49.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:49.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:49.596 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:49.614 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:49.639 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:49.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:49.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:49.740 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:49.767 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:04:49.769 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:49.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:49.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:49.831 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:49.871 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:49.872 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5929 7.b 7:d2f6a783:::benchmark_data_smithi163_80312_object5928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.234297+0000", 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 233.63205108700001, 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.080282460999999999, 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5929 2024-03-21T13:04:49.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234297+0000", 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234297+0000", 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234297+0000", 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.61e-07 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234319+0000", 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1543999999999999e-05 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234326+0000", 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2200000000000003e-06 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234332+0000", 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9399999999999999e-06 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235470+0000", 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00113791 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235585+0000", 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115172 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235718+0000", 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013329999999999999 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237771+0000", 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002052267 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237826+0000", 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4988000000000003e-05 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238146+0000", 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032077100000000002 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238181+0000", 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4523000000000001e-05 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314579+0000", 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076398165000000004 2024-03-21T13:04:49.878 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.879 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.879 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.879 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5930 7.b 7:dfc262d2:::benchmark_data_smithi163_80312_object5929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:04:49.879 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.234341+0000", 2024-03-21T13:04:49.879 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 233.63200686100001, 2024-03-21T13:04:49.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.080402360000000006, 2024-03-21T13:04:49.879 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.879 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.879 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:49.879 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:04:49.879 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5930 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234341+0000", 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234341+0000", 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234341+0000", 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4400000000000001e-07 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234363+0000", 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1693999999999999e-05 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234369+0000", 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1109999999999996e-06 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.234376+0000", 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5989999999999998e-06 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236182+0000", 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018059250000000001 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236319+0000", 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013760500000000001 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236468+0000", 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014911899999999999 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237778+0000", 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013097219999999999 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238067+0000", 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00028868900000000001 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314635+0000", 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076568339999999999 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314702+0000", 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6563000000000006e-05 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314743+0000", 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1548999999999998e-05 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5935 7.b 7:d67b008b:::benchmark_data_smithi163_80312_object5934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.235598+0000", 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 233.63074995900001, 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.079317974999999999, 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5935 2024-03-21T13:04:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235598+0000", 2024-03-21T13:04:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235598+0000", 2024-03-21T13:04:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235599+0000", 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3200000000000004e-07 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235625+0000", 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6478999999999998e-05 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235634+0000", 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7979999999999995e-06 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.235646+0000", 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2685e-05 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236993+0000", 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001346185 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237079+0000", 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6819999999999999e-05 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237248+0000", 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016860900000000001 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238104+0000", 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00085547599999999998 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314784+0000", 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076680188999999996 2024-03-21T13:04:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314827+0000", 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3439999999999997e-05 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314880+0000", 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2515000000000001e-05 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314916+0000", 2024-03-21T13:04:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6047000000000002e-05 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5938 7.b 7:d7840c27:::benchmark_data_smithi163_80312_object5937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.236773+0000", 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 233.629574534, 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.080271274000000004, 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:04:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5938 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236773+0000", 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236773+0000", 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236774+0000", 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.879999999999999e-07 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236796+0000", 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2181e-05 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236816+0000", 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.967e-05 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236825+0000", 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9029999999999993e-06 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314950+0000", 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078125147000000006 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.315105+0000", 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000154537 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.315253+0000", 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000148267 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.316839+0000", 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015863030000000001 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.316880+0000", 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0299000000000002e-05 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.316946+0000", 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6766000000000004e-05 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.316997+0000", 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0674000000000003e-05 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.317044+0000", 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7539e-05 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:5939 7.b 7:d562aafc:::benchmark_data_smithi163_80312_object5938:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.236919+0000", 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 233.62942909700001, 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.080813425999999994, 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5939 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236919+0000", 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236919+0000", 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236919+0000", 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9999999999999996e-07 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236944+0000", 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4477e-05 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236953+0000", 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8219999999999996e-06 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.236975+0000", 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2062999999999999e-05 2024-03-21T13:04:49.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.315716+0000", 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078741252999999997 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.315827+0000", 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000111068 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.316426+0000", 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059925199999999999 2024-03-21T13:04:49.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.317054+0000", 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00062737999999999997 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.317165+0000", 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011152799999999999 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.317638+0000", 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047229900000000003 2024-03-21T13:04:49.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.317696+0000", 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7960000000000001e-05 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.317732+0000", 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6523999999999998e-05 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.14820.0:5929 7.b e97/92 ondisk, result = 0)", 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.237471+0000", 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 233.62887633, 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077120584000000006, 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237471+0000", 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237471+0000", 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237472+0000", 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9299999999999998e-07 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237476+0000", 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3209999999999998e-06 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237496+0000", 2024-03-21T13:04:49.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9340000000000001e-05 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237503+0000", 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1359999999999999e-06 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238138+0000", 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00063532499999999995 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238144+0000", 2024-03-21T13:04:49.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.502e-06 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314592+0000", 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076448367000000003 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.14820.0:5930 7.b e97/92 ondisk, result = 0)", 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.237520+0000", 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 233.62882761, 2024-03-21T13:04:49.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077228064999999999, 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237520+0000", 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237520+0000", 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237520+0000", 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2300000000000002e-07 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237523+0000", 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.695e-06 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237529+0000", 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6690000000000003e-06 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.237534+0000", 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9749999999999998e-06 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314622+0000", 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077087951000000002 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314629+0000", 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7200000000000006e-06 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314748+0000", 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118732 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.14820.0:5935 7.b e97/92 ondisk, result = 0)", 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.238090+0000", 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 233.62825783, 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076700827999999999, 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:04:49.910 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238090+0000", 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238090+0000", 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238090+0000", 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4899999999999995e-07 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238094+0000", 2024-03-21T13:04:49.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9770000000000002e-06 2024-03-21T13:04:49.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238102+0000", 2024-03-21T13:04:49.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.52e-06 2024-03-21T13:04:49.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238108+0000", 2024-03-21T13:04:49.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1229999999999997e-06 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314772+0000", 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076663696000000003 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314779+0000", 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7750000000000004e-06 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314791+0000", 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2187999999999999e-05 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.14820.0:5935 7.b e97/92 ondisk, result = 0)", 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:56.238227+0000", 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 233.628120362, 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076694090000000006, 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238227+0000", 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238227+0000", 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238228+0000", 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2399999999999998e-07 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.238231+0000", 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4259999999999999e-06 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314601+0000", 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076369211000000006 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314611+0000", 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0232e-05 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314817+0000", 2024-03-21T13:04:49.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020572400000000001 2024-03-21T13:04:49.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314822+0000", 2024-03-21T13:04:49.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9340000000000003e-06 2024-03-21T13:04:49.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:56.314921+0000", 2024-03-21T13:04:49.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9038999999999998e-05 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:9967 7.d e97/92 7:b814f2b4:::benchmark_data_smithi163_80312_object9966:head v 97'622, mlcod=97'620)", 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:57.825906+0000", 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 232.040441689, 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073277348000000006, 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.825906+0000", 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.825906+0000", 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.825907+0000", 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5000000000000002e-07 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.826014+0000", 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107049 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.826055+0000", 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1072999999999998e-05 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.826066+0000", 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0767e-05 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.826131+0000", 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5259000000000007e-05 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.826152+0000", 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0826999999999999e-05 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.899144+0000", 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.072991923 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.899183+0000", 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9799999999999998e-05 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:9975 7.0 e97/92 7:05211790:::benchmark_data_smithi163_80312_object9974:head v 97'608, mlcod=97'606)", 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:57.827271+0000", 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 232.03907678499999, 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071878264999999997, 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.827271+0000", 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.827271+0000", 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.827272+0000", 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.827367+0000", 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5440999999999996e-05 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.897956+0000", 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.070589003999999997 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.897965+0000", 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3659999999999992e-06 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.898011+0000", 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6689e-05 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.898038+0000", 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.641e-05 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.899108+0000", 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010697289999999999 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.899149+0000", 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1705000000000001e-05 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:9972 7.2 e97/92 7:47544506:::benchmark_data_smithi163_80312_object9971:head v 97'653, mlcod=97'651)", 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:00:57.827471+0000", 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 232.03887668900001, 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071810505999999996, 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.927 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.827471+0000", 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.827471+0000", 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.827472+0000", 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8700000000000005e-07 2024-03-21T13:04:49.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.827525+0000", 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2926000000000001e-05 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.897916+0000", 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.070391130999999996 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.897932+0000", 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.647e-05 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.897995+0000", 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2581000000000003e-05 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.898023+0000", 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7929999999999999e-05 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.899249+0000", 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012256770000000001 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:00:57.899282+0000", 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3003999999999998e-05 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:89863 7.9 7:901f84a5:::benchmark_data_smithi163_80312_object89862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:54.714348+0000", 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 175.15199987700001, 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067425181000000001, 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.931 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 89863 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714348+0000", 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714348+0000", 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714349+0000", 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4219999999999999e-06 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714368+0000", 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8301e-05 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714378+0000", 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0594000000000001e-05 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714386+0000", 2024-03-21T13:04:49.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6639999999999998e-06 2024-03-21T13:04:49.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714461+0000", 2024-03-21T13:04:49.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4826999999999994e-05 2024-03-21T13:04:49.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714602+0000", 2024-03-21T13:04:49.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141686 2024-03-21T13:04:49.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714759+0000", 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015627600000000001 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.721998+0000", 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0072397199999999998 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723198+0000", 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001200095 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.781671+0000", 2024-03-21T13:04:49.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.058472203 2024-03-21T13:04:49.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.781725+0000", 2024-03-21T13:04:49.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4168e-05 2024-03-21T13:04:49.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.781773+0000", 2024-03-21T13:04:49.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8225e-05 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:89877 7.9 7:9aa81cc8:::benchmark_data_smithi163_80312_object89876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:54.723093+0000", 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 175.143254641, 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.070265444999999996, 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:04:49.937 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 89877 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723093+0000", 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723093+0000", 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723095+0000", 2024-03-21T13:04:49.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.564e-06 2024-03-21T13:04:49.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723117+0000", 2024-03-21T13:04:49.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:49.940 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:49.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:49.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2660999999999999e-05 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723132+0000", 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5092999999999999e-05 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723140+0000", 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8669999999999994e-06 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723227+0000", 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6644999999999995e-05 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723339+0000", 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011218900000000001 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723472+0000", 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013277499999999999 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726766+0000", 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032936649999999999 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.730039+0000", 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032733950000000001 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.793215+0000", 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063175855000000003 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.793312+0000", 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7078999999999997e-05 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.793359+0000", 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6656999999999997e-05 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:89880 7.9 7:9d968a4b:::benchmark_data_smithi163_80312_object89879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:54.726832+0000", 2024-03-21T13:04:49.944 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 175.139515983, 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.066681890999999993, 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 89880 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726832+0000", 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726832+0000", 2024-03-21T13:04:49.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726833+0000", 2024-03-21T13:04:49.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5300000000000002e-07 2024-03-21T13:04:49.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726851+0000", 2024-03-21T13:04:49.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8651000000000002e-05 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726863+0000", 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1615e-05 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726871+0000", 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3629999999999994e-06 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726926+0000", 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4891e-05 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.727174+0000", 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024773599999999997 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.727352+0000", 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017773600000000001 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:49.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.730133+0000", 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027817240000000002 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.730525+0000", 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039137999999999998 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.793397+0000", 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062871705999999999 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.793476+0000", 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9856000000000003e-05 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.793514+0000", 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7280000000000002e-05 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:91006 7.b 7:d6eb55a7:::benchmark_data_smithi163_80312_object91005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:04:49.950 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:55.792977+0000", 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 174.07337078, 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062755527000000005, 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91006 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.792977+0000", 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.792977+0000", 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.792978+0000", 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7700000000000004e-07 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.792997+0000", 2024-03-21T13:04:49.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9125000000000001e-05 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.793007+0000", 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6590000000000002e-06 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.793013+0000", 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9489999999999999e-06 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.798124+0000", 2024-03-21T13:04:49.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0051106149999999998 2024-03-21T13:04:49.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.798240+0000", 2024-03-21T13:04:49.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115652 2024-03-21T13:04:49.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-03-21T13:04:49.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.798382+0000", 2024-03-21T13:04:49.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141748 2024-03-21T13:04:49.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.801043+0000", 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026610359999999999 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.809641+0000", 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085982579999999992 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.855609+0000", 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045968588999999997 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.855678+0000", 2024-03-21T13:04:49.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8659999999999997e-05 2024-03-21T13:04:49.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.855733+0000", 2024-03-21T13:04:49.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4459000000000002e-05 2024-03-21T13:04:49.956 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.956 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.956 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:91014 7.b 7:da4ce39b:::benchmark_data_smithi163_80312_object91013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:55.799579+0000", 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 174.066769206, 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063691170000000005, 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91014 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.799579+0000", 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.799579+0000", 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.799580+0000", 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.319e-06 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.799602+0000", 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2589e-05 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.799612+0000", 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4919999999999999e-06 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.799619+0000", 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6420000000000001e-06 2024-03-21T13:04:49.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.801356+0000", 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001737904 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.801469+0000", 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011238099999999999 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.801609+0000", 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014058300000000001 2024-03-21T13:04:49.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.809681+0000", 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0080712089999999993 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.812820+0000", 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031392690000000001 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.863124+0000", 2024-03-21T13:04:49.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.050303632000000001 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.863219+0000", 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5036000000000003e-05 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.863270+0000", 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1113999999999999e-05 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:91025 7.b 7:d35260cc:::benchmark_data_smithi163_80312_object91024:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:55.808912+0000", 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 174.057435785, 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055675924000000002, 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91025 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.808912+0000", 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.808912+0000", 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.808914+0000", 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.606e-06 2024-03-21T13:04:49.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.808933+0000", 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9772999999999999e-05 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.808945+0000", 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2098e-05 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.808954+0000", 2024-03-21T13:04:49.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1079999999999993e-06 2024-03-21T13:04:49.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.810074+0000", 2024-03-21T13:04:49.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001120832 2024-03-21T13:04:49.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.810235+0000", 2024-03-21T13:04:49.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000160338 2024-03-21T13:04:49.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-03-21T13:04:49.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.810340+0000", 2024-03-21T13:04:49.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010535300000000001 2024-03-21T13:04:49.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.812963+0000", 2024-03-21T13:04:49.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026229090000000001 2024-03-21T13:04:49.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:49.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.812976+0000", 2024-03-21T13:04:49.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3481e-05 2024-03-21T13:04:49.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.864500+0000", 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051523878000000002 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.864551+0000", 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0519000000000002e-05 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.864588+0000", 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7029000000000001e-05 2024-03-21T13:04:49.968 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.969 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.969 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.969 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:91194 7.9 7:940b1401:::benchmark_data_smithi163_80312_object91193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:04:49.969 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:55.990360+0000", 2024-03-21T13:04:49.969 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 173.87598788099999, 2024-03-21T13:04:49.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054572715000000001, 2024-03-21T13:04:49.969 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.969 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.969 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:49.969 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:04:49.969 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91194 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990360+0000", 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990360+0000", 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990362+0000", 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8479999999999999e-06 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990386+0000", 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4181000000000001e-05 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990402+0000", 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5821e-05 2024-03-21T13:04:49.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990410+0000", 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4970000000000001e-06 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990486+0000", 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5362000000000001e-05 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990616+0000", 2024-03-21T13:04:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013085700000000001 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990912+0000", 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029580900000000001 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.994547+0000", 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0036350950000000001 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.999053+0000", 2024-03-21T13:04:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045052060000000003 2024-03-21T13:04:49.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.044850+0000", 2024-03-21T13:04:49.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045797787 2024-03-21T13:04:49.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.044889+0000", 2024-03-21T13:04:49.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8739999999999998e-05 2024-03-21T13:04:49.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.044933+0000", 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3511999999999999e-05 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:92024 7.b 7:def0ff7d:::benchmark_data_smithi163_80312_object92023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:56.751202+0000", 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 173.11514550699999, 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.058877783000000003, 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:04:49.975 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92024 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.751202+0000", 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.751202+0000", 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:49.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:49.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.751204+0000", 2024-03-21T13:04:49.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5340000000000001e-06 2024-03-21T13:04:49.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:49.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.751226+0000", 2024-03-21T13:04:49.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2104e-05 2024-03-21T13:04:49.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:49.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.751238+0000", 2024-03-21T13:04:49.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2072999999999999e-05 2024-03-21T13:04:49.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:49.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.751246+0000", 2024-03-21T13:04:49.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3100000000000001e-06 2024-03-21T13:04:49.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:49.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.752009+0000", 2024-03-21T13:04:49.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00076259600000000004 2024-03-21T13:04:49.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.752143+0000", 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013382499999999999 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.752284+0000", 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141025 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.755341+0000", 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0030574339999999999 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.755507+0000", 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000165556 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.809940+0000", 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054433606000000002 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.810017+0000", 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6401000000000003e-05 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:49.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:49.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:04:49.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.810080+0000", 2024-03-21T13:04:49.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3318999999999995e-05 2024-03-21T13:04:49.981 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.981 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.981 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.981 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:49.981 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:49.981 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:04:50.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:50.065 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:50.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:50.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:50.116 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:04:50.166 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:50.190 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:50.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:50.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:50.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:50 smithi163 ceph-mon[40675]: pgmap v259: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-21T13:04:50.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:50.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:50.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:50 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2643162970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:04:50.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:50 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:04:50.212 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:50 smithi163 ceph-mon[29621]: pgmap v259: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-21T13:04:50.212 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:50.212 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:50.212 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:50 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2643162970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:04:50.212 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:50 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:04:50.291 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:50.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:50.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:50.417 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:50.445 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:50.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:50.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:50 smithi204 ceph-mon[33153]: pgmap v259: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-21T13:04:50.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:04:50.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:50.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:50 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2643162970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:04:50.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:50 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:04:50.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:50.570 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:50.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:50.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:50.699 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:50.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:50.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:50.801 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:50.826 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:50.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:50.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:50.927 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:50.950 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:50.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:50.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:51.052 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:51.075 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:51.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:51.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:51.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:51.204 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:51.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:51.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:51.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:51 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:04:51.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:51 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:04:51.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:51 smithi163 ceph-mon[29621]: osdmap e171: 8 total, 8 up, 6 in 2024-03-21T13:04:51.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:51 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:04:51.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:51 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:04:51.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:51 smithi163 ceph-mon[40675]: osdmap e171: 8 total, 8 up, 6 in 2024-03-21T13:04:51.305 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:51.330 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:51.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:51.431 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:51.455 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:51.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:51.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:51 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:04:51.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:51 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:04:51.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:51 smithi204 ceph-mon[33153]: osdmap e171: 8 total, 8 up, 6 in 2024-03-21T13:04:51.556 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:51.579 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:51.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:51.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:51.681 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:51.705 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:51.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:51.807 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:51.830 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:51.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:51.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:51.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:51.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:51.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:52.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:04:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:04:52.026 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-21T13:04:52.062 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:52.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:52.089 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:52.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:52.190 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:52.214 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:52.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:52.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:52 smithi163 ceph-mon[29621]: pgmap v261: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:52.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:52 smithi163 ceph-mon[40675]: pgmap v261: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:52.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:52.339 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:52.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:52.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:52.441 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:52.465 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:52.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:52.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:52 smithi204 ceph-mon[33153]: pgmap v261: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:52.565 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:52.589 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:52.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:52.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:52.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:52.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:52.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:52.821 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:52.845 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:52.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:52.946 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:52.970 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:52.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:52.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:53.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:53.097 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:53.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:53.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:53.223 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:53.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:53.325 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:53.349 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:53.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:53.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:53.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:53.350 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:53.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:53.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:53.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:53.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:53.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:53.578 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:53.602 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:53.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:53.703 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:53.727 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:53.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:53.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:53.829 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:53.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:53.853 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:53.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:53.954 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:53.978 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:53.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:53.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:54.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:54.108 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:54.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:54.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:54.210 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:54.235 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:54.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:54.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:54 smithi163 ceph-mon[29621]: pgmap v262: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 789 B/s rd, 0 op/s 2024-03-21T13:04:54.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:54 smithi163 ceph-mon[40675]: pgmap v262: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 789 B/s rd, 0 op/s 2024-03-21T13:04:54.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:54.360 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:54.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:54.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:54.462 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:54.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:54.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:54.593 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:54.617 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:54.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:54.632 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:54.677 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:54.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:54.746 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:54.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:54.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:54 smithi204 ceph-mon[33153]: pgmap v262: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 789 B/s rd, 0 op/s 2024-03-21T13:04:54.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:54.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:54.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:54.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:54.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:55.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:55.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:55.107 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:55.108 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:04:55.134 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:55.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:55.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:55.236 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:55.265 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:55.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:55.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:55.273 INFO:teuthology.orchestra.run.smithi163.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-21T13:04:55.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:55.401 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:55.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:55.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:55.503 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:55.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:55 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:04:55.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:55 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:04:55.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:55 smithi163 ceph-mon[29621]: osdmap e172: 8 total, 8 up, 6 in 2024-03-21T13:04:55.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:55 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1463770911' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:04:55.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:55 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:04:55.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:55 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:04:55.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:55 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:04:55.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:55 smithi163 ceph-mon[40675]: osdmap e172: 8 total, 8 up, 6 in 2024-03-21T13:04:55.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:55 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1463770911' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:04:55.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:55 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:04:55.526 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:55.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:55.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:55.628 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:55.652 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:55.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:55.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:55 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:04:55.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:55 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:04:55.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:55 smithi204 ceph-mon[33153]: osdmap e172: 8 total, 8 up, 6 in 2024-03-21T13:04:55.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:55 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1463770911' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:04:55.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:55 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:04:55.754 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:55.782 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:55.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:55.884 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:55.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:55.909 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:55.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:55.994 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:04:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:04:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:04:56.010 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:56.033 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:56.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:56.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:56.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:56.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:56.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:56.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:56.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:56.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:56.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:56.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:56.423 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:56.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:56.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:56.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:56.553 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:56.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:56.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:56.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:56 smithi163 ceph-mon[29621]: pgmap v264: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:04:56.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:56 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:04:56.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:56 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:04:56.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:56 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:04:56.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:56 smithi163 ceph-mon[29621]: osdmap e173: 8 total, 8 up, 6 in 2024-03-21T13:04:56.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:56 smithi163 ceph-mon[40675]: pgmap v264: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:04:56.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:56 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:04:56.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:56 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:04:56.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:56 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:04:56.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:56 smithi163 ceph-mon[40675]: osdmap e173: 8 total, 8 up, 6 in 2024-03-21T13:04:56.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:04:56.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:56.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:04:56.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:56 smithi204 ceph-mon[33153]: pgmap v264: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:04:56.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:56 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:04:56.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:56 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:04:56.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:56 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:04:56.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:56 smithi204 ceph-mon[33153]: osdmap e173: 8 total, 8 up, 6 in 2024-03-21T13:04:56.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:56.813 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:56.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:56.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:56.915 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:56.940 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:56.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:56.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:57.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:57.071 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:57.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:57.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:57.172 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:57.198 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:57.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:57.299 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:57.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:57.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:57.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:57.430 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:57.458 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:57.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:57.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:57.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:57 smithi163 ceph-mon[29621]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:04:57.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:57 smithi163 ceph-mon[40675]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:04:57.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:57.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:57.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:57.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:57.713 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:57.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:57.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:57 smithi204 ceph-mon[33153]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:04:57.815 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:57.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:57.839 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:57.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:57.940 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:57.964 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:57.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:58.020 DEBUG:teuthology.orchestra.run.smithi163:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-21T13:04:58.022 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-21T13:04:58.023 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-21T13:04:58.065 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:58.070 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:hints = 1 2024-03-21T13:04:58.070 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-21T13:04:58.070 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Object prefix: benchmark_data_smithi163_107079 2024-03-21T13:04:58.071 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:04:58.071 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 0 0 0 0 0 0 - 0 2024-03-21T13:04:58.107 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:58.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:58.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:58.210 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:04:58.252 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:58.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:04:58.329 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.2 to 1 (10000) 2024-03-21T13:04:58.355 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:58.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:58.402 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:58.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:58.503 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:58.546 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:58.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:58.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:58.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:58.685 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:58.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:58.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:58 smithi163 ceph-mon[29621]: pgmap v266: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 772 B/s rd, 0 op/s 2024-03-21T13:04:58.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:58 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3304115333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:04:58.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:58 smithi163 ceph-mon[40675]: pgmap v266: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 772 B/s rd, 0 op/s 2024-03-21T13:04:58.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:58 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3304115333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:04:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:58 smithi204 ceph-mon[33153]: pgmap v266: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 772 B/s rd, 0 op/s 2024-03-21T13:04:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:58 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3304115333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:04:58.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:58.820 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:58.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:58.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:04:58.860 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:04:58.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:58.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:58.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:59.058 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:04:59.070 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 1 15 3434 3419 213.675 213.688 0.00348403 0.00466093 2024-03-21T13:04:59.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:59.101 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:59.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:04:59.202 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:04:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:59.237 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:59.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:04:59.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:59.370 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:59.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:59.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:59.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:04:59.503 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:59.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:04:59.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:59 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3304115333' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:04:59.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:04:59 smithi163 ceph-mon[40675]: osdmap e174: 8 total, 8 up, 6 in 2024-03-21T13:04:59.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:59 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3304115333' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:04:59.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:04:59 smithi163 ceph-mon[29621]: osdmap e174: 8 total, 8 up, 6 in 2024-03-21T13:04:59.511 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:04:59.511 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:04:59.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.511 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:4791 8.1 8:8d3e3c97:::benchmark_data_smithi163_107079_object4790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-03-21T13:04:59.511 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:04:59.507466+0000", 2024-03-21T13:04:59.511 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.0026519, 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002688934, 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 4791 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.507466+0000", 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.507466+0000", 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.507468+0000", 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6270000000000001e-06 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.507504+0000", 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5794999999999999e-05 2024-03-21T13:04:59.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:59.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.507517+0000", 2024-03-21T13:04:59.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3047999999999999e-05 2024-03-21T13:04:59.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:59.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.507525+0000", 2024-03-21T13:04:59.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3899999999999993e-06 2024-03-21T13:04:59.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:59.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.507579+0000", 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3607999999999999e-05 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.507839+0000", 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025969200000000001 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.508079+0000", 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024008699999999999 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:04:59.516 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.509007+0000", 2024-03-21T13:04:59.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092851200000000004 2024-03-21T13:04:59.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:59.516 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:59.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:59.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.516 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15078.0:4796 8.7 e174/168 8:e1e51904:::benchmark_data_smithi163_107079_object4795:head v 174'287, mlcod=174'285)", 2024-03-21T13:04:59.516 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:04:59.509288+0000", 2024-03-21T13:04:59.516 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.00082988199999999999, 2024-03-21T13:04:59.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00096337300000000001, 2024-03-21T13:04:59.516 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.509288+0000", 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.509288+0000", 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:59.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.509289+0000", 2024-03-21T13:04:59.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0900000000000001e-07 2024-03-21T13:04:59.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:59.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.509308+0000", 2024-03-21T13:04:59.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9017000000000001e-05 2024-03-21T13:04:59.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:59.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.509320+0000", 2024-03-21T13:04:59.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1963e-05 2024-03-21T13:04:59.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:59.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.509328+0000", 2024-03-21T13:04:59.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5240000000000003e-06 2024-03-21T13:04:59.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:59.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.509382+0000", 2024-03-21T13:04:59.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4255000000000001e-05 2024-03-21T13:04:59.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:59.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.509401+0000", 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8833000000000001e-05 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15078.0:4797 8.7 e174/168 8:eb45c414:::benchmark_data_smithi163_107079_object4796:head v 174'288, mlcod=174'285)", 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:04:59.510078+0000", 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.9844e-05, 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024773500000000001, 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:04:59.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.510078+0000", 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.510078+0000", 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.510079+0000", 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6400000000000002e-07 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:04:59.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.510101+0000", 2024-03-21T13:04:59.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1895999999999998e-05 2024-03-21T13:04:59.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:04:59.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.510113+0000", 2024-03-21T13:04:59.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1892999999999999e-05 2024-03-21T13:04:59.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:04:59.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.510120+0000", 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0110000000000003e-06 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.510226+0000", 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000106038 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:04:59.510245+0000", 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9270999999999999e-05 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:04:59.523 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:59.524 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:04:59.524 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:04:59.524 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 3 2024-03-21T13:04:59.524 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:04:59.605 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:04:59.638 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:59.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:04:59.740 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:04:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:59 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3304115333' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:04:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:04:59 smithi204 ceph-mon[33153]: osdmap e174: 8 total, 8 up, 6 in 2024-03-21T13:04:59.781 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:04:59.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:59.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:04:59.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:04:59.914 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:04:59.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:04:59.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:00.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:00.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:00.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:00.071 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 2 16 6501 6485 202.642 191.625 0.00251931 0.00492206 2024-03-21T13:05:00.155 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:00.196 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:00.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:00.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:00.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:00.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:00.400 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:05:00.400 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:05:00.400 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:05:00.400 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:05:00.400 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:05:00.400 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:05:00.400 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:05:00.400 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:05:00.401 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:05:00.401 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:05:00.401 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:05:00.401 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:05:00.401 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:05:00.401 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:05:00.401 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:05:00.401 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:05:00.431 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:00.471 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:00.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:00.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:00.573 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:00.607 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:00.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:00.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:00.709 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:00.749 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:00.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:00.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:00 smithi204 ceph-mon[33153]: pgmap v268: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:05:00.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:00 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1314050005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:05:00.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:00 smithi163 ceph-mon[29621]: pgmap v268: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:05:00.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:00 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1314050005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:05:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:00 smithi163 ceph-mon[40675]: pgmap v268: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:05:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:00 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1314050005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:05:00.850 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:00.892 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:00.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:00.993 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:01.035 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:01.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:01.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:01.071 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 3 16 9631 9615 200.297 195.625 0.00539403 0.00498451 2024-03-21T13:05:01.137 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:01.186 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:01.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:01.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:01.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:01.323 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:01.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:01.424 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:01.462 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:01.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:01.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:01.600 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:01.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:01.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:01.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:01.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:01.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:01.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:01.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:01.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:01.870 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:01.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:01.928 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-21T13:05:01.970 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:02.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:05:01 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:05:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:02.016 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:02.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:02.071 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 4 15 12569 12554 196.14 183.688 0.0131798 0.00508767 2024-03-21T13:05:02.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-21T13:05:02.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:02.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:02.148 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:02.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:02.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:02.301 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:02.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:02.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:02.395 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:02 smithi163 ceph-mon[29621]: pgmap v269: 148 pgs: 148 active+clean; 369 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 57 MiB/s wr, 912 op/s 2024-03-21T13:05:02.395 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:02 smithi163 ceph-mon[29621]: osdmap e175: 8 total, 8 up, 6 in 2024-03-21T13:05:02.396 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:02 smithi163 ceph-mon[40675]: pgmap v269: 148 pgs: 148 active+clean; 369 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 57 MiB/s wr, 912 op/s 2024-03-21T13:05:02.396 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:02 smithi163 ceph-mon[40675]: osdmap e175: 8 total, 8 up, 6 in 2024-03-21T13:05:02.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:02.452 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:02.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:02.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:02.587 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:02.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:02.688 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:02.728 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:02.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:02 smithi204 ceph-mon[33153]: pgmap v269: 148 pgs: 148 active+clean; 369 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 57 MiB/s wr, 912 op/s 2024-03-21T13:05:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:02 smithi204 ceph-mon[33153]: osdmap e175: 8 total, 8 up, 6 in 2024-03-21T13:05:02.830 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:02.872 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:02.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:02.974 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:03.025 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:03.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:03.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:03.071 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 5 16 15211 15195 189.922 165.062 0.00397576 0.00525911 2024-03-21T13:05:03.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:03.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:03.167 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:03.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:03.182 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:05:03.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:03.302 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:03.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:03.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:03.404 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:03.450 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:03.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:03.505 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:03.505 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:03.551 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:03.575 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:03.598 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:03.612 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:03.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:03.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:03.745 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:03.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:03.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:03.773 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:03.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:03.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:03.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:03.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:04.017 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:04.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:04.071 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 6 16 17724 17708 184.443 157.062 0.00339918 0.00541508 2024-03-21T13:05:04.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:04.159 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:04.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:04.261 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:04.302 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:04.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:04.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:04 smithi163 ceph-mon[29621]: pgmap v271: 148 pgs: 148 active+clean; 369 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 685 B/s rd, 49 MiB/s wr, 790 op/s 2024-03-21T13:05:04.351 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:04 smithi163 ceph-mon[40675]: pgmap v271: 148 pgs: 148 active+clean; 369 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 685 B/s rd, 49 MiB/s wr, 790 op/s 2024-03-21T13:05:04.404 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:04.438 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:04.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:04.540 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:04.577 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:04.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:04.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:04.679 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:04.722 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:04.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:04.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:04.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:04 smithi204 ceph-mon[33153]: pgmap v271: 148 pgs: 148 active+clean; 369 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 685 B/s rd, 49 MiB/s wr, 790 op/s 2024-03-21T13:05:04.824 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:04.862 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:04.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:04.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:04.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:04.994 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:04.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:04.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:05.071 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 7 16 20187 20171 180.083 153.938 0.00972438 0.00554692 2024-03-21T13:05:05.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:05.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:05.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:05.229 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:05.271 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:05.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:05.372 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:05.413 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:05.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:05.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:05.515 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:05.544 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:05.545 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:05.545 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:05 smithi163 ceph-mon[29621]: pgmap v272: 148 pgs: 148 active+clean; 750 MiB data, 2.9 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-21T13:05:05.545 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:05.545 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:05.545 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:05 smithi163 ceph-mon[40675]: pgmap v272: 148 pgs: 148 active+clean; 750 MiB data, 2.9 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-21T13:05:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:05.554 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:05.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:05.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:05.688 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:05.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:05 smithi204 ceph-mon[33153]: pgmap v272: 148 pgs: 148 active+clean; 750 MiB data, 2.9 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-21T13:05:05.790 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:05.830 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:05.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:05.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:05.900 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:05:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:05:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:05:05.932 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:05.970 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:05.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:06.071 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 8 15 22574 22559 176.227 149.25 0.00497729 0.00566956 2024-03-21T13:05:06.073 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:06.110 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:06.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:06.212 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:06.247 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:06.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:06.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:06.349 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:06.394 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:06.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:06.421 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:05:06.495 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:06.531 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:06.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:06.633 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:06.683 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:06.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:06.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:06 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3722573163' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:05:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:06 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3722573163' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:05:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:06 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3722573163' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:05:06.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:06.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:06.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:06.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:06.911 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:06.949 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:06.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:06.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:07.051 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:07.071 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 9 16 24791 24775 172.031 138.5 0.0102152 0.00580641 2024-03-21T13:05:07.089 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:07.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:07.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:07.222 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:07.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:07.324 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:07.362 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:07.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:07.464 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:07.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:07.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:07.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:07.599 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:07.642 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:07.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:07.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:07.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:07 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:05:07.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:07 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3722573163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:05:07.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:07 smithi204 ceph-mon[33153]: osdmap e176: 8 total, 8 up, 6 in 2024-03-21T13:05:07.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:07 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:05:07.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:07 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:05:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:07 smithi204 ceph-mon[33153]: pgmap v274: 148 pgs: 148 active+clean; 1.2 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 154 MiB/s wr, 2.47k op/s 2024-03-21T13:05:07.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:07 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:05:07.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:07 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3722573163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:05:07.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:07 smithi163 ceph-mon[29621]: osdmap e176: 8 total, 8 up, 6 in 2024-03-21T13:05:07.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:07 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:05:07.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:07 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:05:07.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:07 smithi163 ceph-mon[29621]: pgmap v274: 148 pgs: 148 active+clean; 1.2 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 154 MiB/s wr, 2.47k op/s 2024-03-21T13:05:07.755 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:07 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:05:07.755 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:07 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3722573163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:05:07.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:07 smithi163 ceph-mon[40675]: osdmap e176: 8 total, 8 up, 6 in 2024-03-21T13:05:07.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:07 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:05:07.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:07 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:05:07.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:07 smithi163 ceph-mon[40675]: pgmap v274: 148 pgs: 148 active+clean; 1.2 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 154 MiB/s wr, 2.47k op/s 2024-03-21T13:05:07.773 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:07.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:07.875 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:07.907 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:07.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:07.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:08.009 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:08.032 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:08.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:08.071 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 10 16 26517 26501 165.615 107.875 0.00830029 0.0060316 2024-03-21T13:05:08.133 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:08.173 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:08.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:08.275 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:08.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:08.315 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:08.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:08.416 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:08.439 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:08.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:08.541 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:08.580 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:08.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:08.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:08.682 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:08.723 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:08.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:08 smithi204 ceph-mon[33153]: 5.1 scrub starts 2024-03-21T13:05:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:08 smithi204 ceph-mon[33153]: 5.1 scrub ok 2024-03-21T13:05:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:08 smithi204 ceph-mon[33153]: 6.1b scrub starts 2024-03-21T13:05:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:08 smithi204 ceph-mon[33153]: 3.4 scrub starts 2024-03-21T13:05:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:08 smithi204 ceph-mon[33153]: 3.4 scrub ok 2024-03-21T13:05:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:08 smithi204 ceph-mon[33153]: 6.1b scrub ok 2024-03-21T13:05:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:08 smithi204 ceph-mon[33153]: osdmap e177: 8 total, 8 up, 6 in 2024-03-21T13:05:08.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[29621]: 5.1 scrub starts 2024-03-21T13:05:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[29621]: 5.1 scrub ok 2024-03-21T13:05:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[29621]: 6.1b scrub starts 2024-03-21T13:05:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[29621]: 3.4 scrub starts 2024-03-21T13:05:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[29621]: 3.4 scrub ok 2024-03-21T13:05:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[29621]: 6.1b scrub ok 2024-03-21T13:05:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[29621]: osdmap e177: 8 total, 8 up, 6 in 2024-03-21T13:05:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[40675]: 5.1 scrub starts 2024-03-21T13:05:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[40675]: 5.1 scrub ok 2024-03-21T13:05:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[40675]: 6.1b scrub starts 2024-03-21T13:05:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[40675]: 3.4 scrub starts 2024-03-21T13:05:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[40675]: 3.4 scrub ok 2024-03-21T13:05:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[40675]: 6.1b scrub ok 2024-03-21T13:05:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:08 smithi163 ceph-mon[40675]: osdmap e177: 8 total, 8 up, 6 in 2024-03-21T13:05:08.824 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:08.869 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:08.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:08.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:09.006 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:09.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:09.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:09.072 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 11 16 28287 28271 160.615 110.625 0.0103311 0.0062192 2024-03-21T13:05:09.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:09.144 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:09.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:09.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:09.269 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:09.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:09.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:09.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:09.415 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:09.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:09.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:09.517 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:09.550 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:09.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:09.652 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:09.692 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:09.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:09.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:09 smithi204 ceph-mon[33153]: 4.b scrub starts 2024-03-21T13:05:09.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:09 smithi204 ceph-mon[33153]: 4.b scrub ok 2024-03-21T13:05:09.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:09 smithi204 ceph-mon[33153]: 6.5 scrub starts 2024-03-21T13:05:09.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:09 smithi204 ceph-mon[33153]: 6.5 scrub ok 2024-03-21T13:05:09.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:09 smithi204 ceph-mon[33153]: pgmap v276: 148 pgs: 148 active+clean; 1.2 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 684 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-21T13:05:09.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:09 smithi163 ceph-mon[29621]: 4.b scrub starts 2024-03-21T13:05:09.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:09 smithi163 ceph-mon[29621]: 4.b scrub ok 2024-03-21T13:05:09.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:09 smithi163 ceph-mon[29621]: 6.5 scrub starts 2024-03-21T13:05:09.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:09 smithi163 ceph-mon[29621]: 6.5 scrub ok 2024-03-21T13:05:09.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:09 smithi163 ceph-mon[29621]: pgmap v276: 148 pgs: 148 active+clean; 1.2 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 684 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-21T13:05:09.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:09 smithi163 ceph-mon[40675]: 4.b scrub starts 2024-03-21T13:05:09.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:09 smithi163 ceph-mon[40675]: 4.b scrub ok 2024-03-21T13:05:09.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:09 smithi163 ceph-mon[40675]: 6.5 scrub starts 2024-03-21T13:05:09.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:09 smithi163 ceph-mon[40675]: 6.5 scrub ok 2024-03-21T13:05:09.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:09 smithi163 ceph-mon[40675]: pgmap v276: 148 pgs: 148 active+clean; 1.2 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 684 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-21T13:05:09.793 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:09.829 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:09.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:09.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:09.966 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:09.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:10.069 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:10.072 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 12 16 30201 30185 157.198 119.625 0.00340699 0.00635576 2024-03-21T13:05:10.098 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:10.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:10.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:10.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:10.236 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:10.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:10.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:10.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:10.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:10.464 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:10.499 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:10.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:10.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:10.601 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:10.632 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:10.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:10.734 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:10.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:10 smithi204 ceph-mon[33153]: 3.e scrub starts 2024-03-21T13:05:10.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:10 smithi204 ceph-mon[33153]: 3.e scrub ok 2024-03-21T13:05:10.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:10 smithi204 ceph-mon[33153]: 3.18 scrub starts 2024-03-21T13:05:10.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:10 smithi204 ceph-mon[33153]: 3.18 scrub ok 2024-03-21T13:05:10.758 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:10.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:10.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:10.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:10 smithi163 ceph-mon[29621]: 3.e scrub starts 2024-03-21T13:05:10.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:10 smithi163 ceph-mon[29621]: 3.e scrub ok 2024-03-21T13:05:10.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:10 smithi163 ceph-mon[29621]: 3.18 scrub starts 2024-03-21T13:05:10.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:10 smithi163 ceph-mon[29621]: 3.18 scrub ok 2024-03-21T13:05:10.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:10 smithi163 ceph-mon[40675]: 3.e scrub starts 2024-03-21T13:05:10.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:10 smithi163 ceph-mon[40675]: 3.e scrub ok 2024-03-21T13:05:10.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:10 smithi163 ceph-mon[40675]: 3.18 scrub starts 2024-03-21T13:05:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:10 smithi163 ceph-mon[40675]: 3.18 scrub ok 2024-03-21T13:05:10.860 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:10.900 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:10.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:10.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:11.003 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:11.026 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:11.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:11.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:11.072 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 13 16 31724 31708 152.427 95.1875 0.00221782 0.00653987 2024-03-21T13:05:11.128 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:11.166 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:11.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:11.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:11.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:11.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:11.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:11.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:11.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:11.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:11.537 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:11.572 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:11.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:11.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:11.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:11.710 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:11.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:11.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:11.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:11 smithi204 ceph-mon[33153]: pgmap v277: 148 pgs: 148 active+clean; 1.8 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 186 MiB/s wr, 2.98k op/s 2024-03-21T13:05:11.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:05:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:05:11.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:11 smithi163 ceph-mon[29621]: pgmap v277: 148 pgs: 148 active+clean; 1.8 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 186 MiB/s wr, 2.98k op/s 2024-03-21T13:05:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:11 smithi163 ceph-mon[40675]: pgmap v277: 148 pgs: 148 active+clean; 1.8 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 186 MiB/s wr, 2.98k op/s 2024-03-21T13:05:11.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:11.848 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:11.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:11.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:11.950 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:11.974 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:11.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:12.072 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 14 16 33595 33579 149.891 116.938 0.0032883 0.00666533 2024-03-21T13:05:12.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:12.114 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:12.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:12.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:12.216 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:12.259 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:12.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:12.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:12.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:12.403 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:12.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:12.504 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:12.543 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:12.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:12.644 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:12.681 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:12.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:12.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:12 smithi204 ceph-mon[33153]: 3.1 scrub starts 2024-03-21T13:05:12.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:12 smithi204 ceph-mon[33153]: 3.1 scrub ok 2024-03-21T13:05:12.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:12 smithi204 ceph-mon[33153]: 4.6 scrub starts 2024-03-21T13:05:12.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:12 smithi204 ceph-mon[33153]: 4.6 scrub ok 2024-03-21T13:05:12.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:12 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:12.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:12 smithi163 ceph-mon[29621]: 3.1 scrub starts 2024-03-21T13:05:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:12 smithi163 ceph-mon[29621]: 3.1 scrub ok 2024-03-21T13:05:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:12 smithi163 ceph-mon[29621]: 4.6 scrub starts 2024-03-21T13:05:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:12 smithi163 ceph-mon[29621]: 4.6 scrub ok 2024-03-21T13:05:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:12 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:12 smithi163 ceph-mon[40675]: 3.1 scrub starts 2024-03-21T13:05:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:12 smithi163 ceph-mon[40675]: 3.1 scrub ok 2024-03-21T13:05:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:12 smithi163 ceph-mon[40675]: 4.6 scrub starts 2024-03-21T13:05:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:12 smithi163 ceph-mon[40675]: 4.6 scrub ok 2024-03-21T13:05:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:12 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:12.784 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:12.825 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:12.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:12.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:12.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:12.954 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:12.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:13.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:13.072 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 15 16 35451 35435 147.631 116 0.00665424 0.00676796 2024-03-21T13:05:13.092 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:13.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:13.193 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:13.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:13.237 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:13.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:13.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:13.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:13.376 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:13.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:13.477 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:13.512 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:13.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:13.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:13.614 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:13.648 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:13.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:13.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:13.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:13 smithi204 ceph-mon[33153]: 4.0 scrub starts 2024-03-21T13:05:13.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:13 smithi204 ceph-mon[33153]: 4.0 scrub ok 2024-03-21T13:05:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:13 smithi204 ceph-mon[33153]: 5.1b scrub starts 2024-03-21T13:05:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:13 smithi204 ceph-mon[33153]: 5.1b scrub ok 2024-03-21T13:05:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:13 smithi204 ceph-mon[33153]: pgmap v278: 148 pgs: 148 active+clean; 1.8 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 139 MiB/s wr, 2.22k op/s 2024-03-21T13:05:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:13 smithi204 ceph-mon[33153]: 5.7 scrub starts 2024-03-21T13:05:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:13 smithi204 ceph-mon[33153]: 5.7 scrub ok 2024-03-21T13:05:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:13 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[29621]: 4.0 scrub starts 2024-03-21T13:05:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[29621]: 4.0 scrub ok 2024-03-21T13:05:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[29621]: 5.1b scrub starts 2024-03-21T13:05:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[29621]: 5.1b scrub ok 2024-03-21T13:05:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[29621]: pgmap v278: 148 pgs: 148 active+clean; 1.8 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 139 MiB/s wr, 2.22k op/s 2024-03-21T13:05:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[29621]: 5.7 scrub starts 2024-03-21T13:05:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[29621]: 5.7 scrub ok 2024-03-21T13:05:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[40675]: 4.0 scrub starts 2024-03-21T13:05:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[40675]: 4.0 scrub ok 2024-03-21T13:05:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[40675]: 5.1b scrub starts 2024-03-21T13:05:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[40675]: 5.1b scrub ok 2024-03-21T13:05:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[40675]: pgmap v278: 148 pgs: 148 active+clean; 1.8 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 139 MiB/s wr, 2.22k op/s 2024-03-21T13:05:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[40675]: 5.7 scrub starts 2024-03-21T13:05:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[40675]: 5.7 scrub ok 2024-03-21T13:05:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:13 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:13.787 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:13.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:13.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:13.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:13.929 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:13.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:14.030 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:14.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:14.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:14.072 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 16 16 36984 36968 144.392 95.8125 0.00590003 0.00691972 2024-03-21T13:05:14.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:14.201 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:14.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:14.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:14.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:14.339 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:14.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:14.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:14.480 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:14.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:14.581 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:14.628 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:14.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:14.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:14.782 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:14.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:14.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[29621]: 3.5 scrub starts 2024-03-21T13:05:14.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[29621]: 3.5 scrub ok 2024-03-21T13:05:14.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:14.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:14.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[29621]: 5.3 scrub starts 2024-03-21T13:05:14.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[29621]: 5.3 scrub ok 2024-03-21T13:05:14.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:14.827 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:14.827 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:14.884 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:14.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:14.931 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:14.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:14.957 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:05:14.957 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:05:15.032 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:15.064 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:15.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:15.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:15.072 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 17 16 38766 38750 142.45 111.375 0.00791867 0.0070146 2024-03-21T13:05:15.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[40675]: 3.5 scrub starts 2024-03-21T13:05:15.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[40675]: 3.5 scrub ok 2024-03-21T13:05:15.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:15.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:15.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[40675]: 5.3 scrub starts 2024-03-21T13:05:15.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[40675]: 5.3 scrub ok 2024-03-21T13:05:15.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:15.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:15.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:14 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:15.166 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:15.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:15.208 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:15.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:15.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:14 smithi204 ceph-mon[33153]: 3.5 scrub starts 2024-03-21T13:05:15.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:14 smithi204 ceph-mon[33153]: 3.5 scrub ok 2024-03-21T13:05:15.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:14 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:15.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:14 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:15.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:14 smithi204 ceph-mon[33153]: 5.3 scrub starts 2024-03-21T13:05:15.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:14 smithi204 ceph-mon[33153]: 5.3 scrub ok 2024-03-21T13:05:15.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:14 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:15.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:14 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:15.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:14 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:15.310 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:15.346 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:15.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:15.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:15.448 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:15.493 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:15.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:15.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:15.596 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:15.634 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:15.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:15.737 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:15.776 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:15.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:15.867 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:05:15.867 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:15 smithi163 ceph-mon[29621]: 3.9 scrub starts 2024-03-21T13:05:15.867 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:15 smithi163 ceph-mon[29621]: 3.9 scrub ok 2024-03-21T13:05:15.868 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:15 smithi163 ceph-mon[29621]: pgmap v279: 148 pgs: 148 active+clean; 2.1 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-21T13:05:15.868 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:15 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1345636810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:05:15.868 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:05:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:05:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:05:15.868 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:15 smithi163 ceph-mon[40675]: 3.9 scrub starts 2024-03-21T13:05:15.868 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:15 smithi163 ceph-mon[40675]: 3.9 scrub ok 2024-03-21T13:05:15.868 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:15 smithi163 ceph-mon[40675]: pgmap v279: 148 pgs: 148 active+clean; 2.1 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-21T13:05:15.868 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:15 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1345636810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:05:15.878 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:15.919 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:05:15.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:15.923 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:15.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:15.967 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:05:16.021 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:05:16.026 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:16.065 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:16.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:16.068 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:05:16.072 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 18 16 40523 40507 140.636 109.812 0.0147663 0.00710544 2024-03-21T13:05:16.110 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:05:16.155 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:05:16.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:16.197 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:05:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:16.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:16.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:05:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:05:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:05:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:05:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:05:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:05:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:05:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:05:16.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:15 smithi204 ceph-mon[33153]: 3.9 scrub starts 2024-03-21T13:05:16.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:15 smithi204 ceph-mon[33153]: 3.9 scrub ok 2024-03-21T13:05:16.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:15 smithi204 ceph-mon[33153]: pgmap v279: 148 pgs: 148 active+clean; 2.1 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-21T13:05:16.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:15 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1345636810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:05:16.299 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:16.330 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:16.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:16.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:16.432 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:16.478 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:16.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:16.580 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:16.617 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-21T13:05:16.623 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:16.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:16.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:16.726 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:16.773 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:16.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:16.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:16.906 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:16.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:16.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:17.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:17.038 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:17.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:17.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:17.072 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 19 15 42149 42134 138.586 101.688 0.00551015 0.00721034 2024-03-21T13:05:17.140 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:17.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:17.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:17.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:16 smithi204 ceph-mon[33153]: 4.7 scrub starts 2024-03-21T13:05:17.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:16 smithi204 ceph-mon[33153]: 4.7 scrub ok 2024-03-21T13:05:17.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:16 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1233547375' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:05:17.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:16 smithi163 ceph-mon[29621]: 4.7 scrub starts 2024-03-21T13:05:17.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:16 smithi163 ceph-mon[29621]: 4.7 scrub ok 2024-03-21T13:05:17.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:16 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1233547375' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:05:17.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:16 smithi163 ceph-mon[40675]: 4.7 scrub starts 2024-03-21T13:05:17.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:16 smithi163 ceph-mon[40675]: 4.7 scrub ok 2024-03-21T13:05:17.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:16 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1233547375' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:05:17.271 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.3 to 1 (10000) 2024-03-21T13:05:17.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:17.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:17.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:17.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:17.408 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:17.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:17.461 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:17.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:17.562 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:17.603 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:17.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:17.705 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:17.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:17.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:17.852 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:17.891 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:17.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:17.994 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:18.034 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:18.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:18.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:18.072 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:05:18.073296+0000 min lat: 0.00120788 max lat: 0.149754 avg lat: 0.00728376 2024-03-21T13:05:18.072 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:05:18.072 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 20 16 43921 43905 137.19 110.688 0.0138629 0.00728376 2024-03-21T13:05:18.104 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[29621]: 4.1b scrub starts 2024-03-21T13:05:18.104 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[29621]: 4.1b scrub ok 2024-03-21T13:05:18.104 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[29621]: pgmap v280: 148 pgs: 148 active+clean; 2.3 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-21T13:05:18.104 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[29621]: 6.10 scrub starts 2024-03-21T13:05:18.104 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[29621]: 6.10 scrub ok 2024-03-21T13:05:18.104 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:05:18.105 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:05:18.105 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1233547375' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:05:18.105 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[29621]: osdmap e178: 8 total, 8 up, 6 in 2024-03-21T13:05:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[40675]: 4.1b scrub starts 2024-03-21T13:05:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[40675]: 4.1b scrub ok 2024-03-21T13:05:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[40675]: pgmap v280: 148 pgs: 148 active+clean; 2.3 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-21T13:05:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[40675]: 6.10 scrub starts 2024-03-21T13:05:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[40675]: 6.10 scrub ok 2024-03-21T13:05:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:05:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:05:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1233547375' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:05:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:17 smithi163 ceph-mon[40675]: osdmap e178: 8 total, 8 up, 6 in 2024-03-21T13:05:18.136 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:18.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:18.178 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:18.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:18.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:17 smithi204 ceph-mon[33153]: 4.1b scrub starts 2024-03-21T13:05:18.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:17 smithi204 ceph-mon[33153]: 4.1b scrub ok 2024-03-21T13:05:18.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:17 smithi204 ceph-mon[33153]: pgmap v280: 148 pgs: 148 active+clean; 2.3 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-21T13:05:18.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:17 smithi204 ceph-mon[33153]: 6.10 scrub starts 2024-03-21T13:05:18.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:17 smithi204 ceph-mon[33153]: 6.10 scrub ok 2024-03-21T13:05:18.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:17 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:05:18.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:17 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:05:18.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:17 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1233547375' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:05:18.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:17 smithi204 ceph-mon[33153]: osdmap e178: 8 total, 8 up, 6 in 2024-03-21T13:05:18.280 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:18.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:18.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:18.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:18.415 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:18.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:18.447 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:18.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:18.548 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:18.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:18.587 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:18.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:18.688 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:18.702 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-21T13:05:18.724 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:18.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:18.733 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:18.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:18.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:18.866 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:18.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:18.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:18 smithi163 ceph-mon[29621]: 4.18 scrub starts 2024-03-21T13:05:18.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:18 smithi163 ceph-mon[29621]: 4.18 scrub ok 2024-03-21T13:05:18.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:18 smithi163 ceph-mon[29621]: 5.17 scrub starts 2024-03-21T13:05:18.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:18 smithi163 ceph-mon[29621]: 5.17 scrub ok 2024-03-21T13:05:18.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:18 smithi163 ceph-mon[29621]: 5.1e scrub starts 2024-03-21T13:05:18.903 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:18 smithi163 ceph-mon[40675]: 4.18 scrub starts 2024-03-21T13:05:18.903 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:18 smithi163 ceph-mon[40675]: 4.18 scrub ok 2024-03-21T13:05:18.903 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:18 smithi163 ceph-mon[40675]: 5.17 scrub starts 2024-03-21T13:05:18.903 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:18 smithi163 ceph-mon[40675]: 5.17 scrub ok 2024-03-21T13:05:18.903 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:18 smithi163 ceph-mon[40675]: 5.1e scrub starts 2024-03-21T13:05:18.968 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:19.013 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:19.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:19.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 21 16 45543 45527 135.47 101.375 0.0059533 0.00737625 2024-03-21T13:05:19.115 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:19.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:19.161 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:19.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:19.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:18 smithi204 ceph-mon[33153]: 4.18 scrub starts 2024-03-21T13:05:19.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:18 smithi204 ceph-mon[33153]: 4.18 scrub ok 2024-03-21T13:05:19.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:18 smithi204 ceph-mon[33153]: 5.17 scrub starts 2024-03-21T13:05:19.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:18 smithi204 ceph-mon[33153]: 5.17 scrub ok 2024-03-21T13:05:19.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:18 smithi204 ceph-mon[33153]: 5.1e scrub starts 2024-03-21T13:05:19.262 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:19.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:19.305 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:19.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:19.406 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:19.446 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:19.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:19.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:19.466 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:05:19.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:19.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:19.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:19.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:19.680 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:19.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:19.732 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:19.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:19.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:19.869 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:19.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:19.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:20.007 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:20.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:20.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:20.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 22 16 47187 47171 133.983 102.75 0.00746235 0.00745915 2024-03-21T13:05:20.109 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:20.116 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:19 smithi163 ceph-mon[29621]: 5.1e scrub ok 2024-03-21T13:05:20.116 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:19 smithi163 ceph-mon[29621]: 6.2 scrub starts 2024-03-21T13:05:20.116 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:19 smithi163 ceph-mon[29621]: 6.2 scrub ok 2024-03-21T13:05:20.117 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:19 smithi163 ceph-mon[29621]: pgmap v282: 148 pgs: 148 active+clean; 2.3 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-21T13:05:20.117 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:20.117 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:19 smithi163 ceph-mon[40675]: 5.1e scrub ok 2024-03-21T13:05:20.117 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:19 smithi163 ceph-mon[40675]: 6.2 scrub starts 2024-03-21T13:05:20.117 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:19 smithi163 ceph-mon[40675]: 6.2 scrub ok 2024-03-21T13:05:20.117 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:19 smithi163 ceph-mon[40675]: pgmap v282: 148 pgs: 148 active+clean; 2.3 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-21T13:05:20.117 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:20.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:20.162 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:20.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:20.162 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:05:20.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:19 smithi204 ceph-mon[33153]: 5.1e scrub ok 2024-03-21T13:05:20.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:19 smithi204 ceph-mon[33153]: 6.2 scrub starts 2024-03-21T13:05:20.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:19 smithi204 ceph-mon[33153]: 6.2 scrub ok 2024-03-21T13:05:20.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:19 smithi204 ceph-mon[33153]: pgmap v282: 148 pgs: 148 active+clean; 2.3 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-21T13:05:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:20.263 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:20.273 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:20.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:20.346 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:20.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:20.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:20.478 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:20.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:20.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:20.580 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:20.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:20.653 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:20.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:20.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:20.797 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:20.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:20.897 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:20.937 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:20.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:20.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:20 smithi163 ceph-mon[29621]: 6.13 scrub starts 2024-03-21T13:05:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:20 smithi163 ceph-mon[29621]: 6.13 scrub ok 2024-03-21T13:05:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:20 smithi163 ceph-mon[29621]: 5.4 scrub starts 2024-03-21T13:05:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:20 smithi163 ceph-mon[29621]: 5.4 scrub ok 2024-03-21T13:05:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:20 smithi163 ceph-mon[29621]: 5.e scrub starts 2024-03-21T13:05:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:20 smithi163 ceph-mon[29621]: 5.e scrub ok 2024-03-21T13:05:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:20 smithi163 ceph-mon[40675]: 6.13 scrub starts 2024-03-21T13:05:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:20 smithi163 ceph-mon[40675]: 6.13 scrub ok 2024-03-21T13:05:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:20 smithi163 ceph-mon[40675]: 5.4 scrub starts 2024-03-21T13:05:21.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:20 smithi163 ceph-mon[40675]: 5.4 scrub ok 2024-03-21T13:05:21.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:20 smithi163 ceph-mon[40675]: 5.e scrub starts 2024-03-21T13:05:21.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:20 smithi163 ceph-mon[40675]: 5.e scrub ok 2024-03-21T13:05:21.039 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:21.072 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:21.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:21.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 23 16 48700 48684 132.269 94.5625 0.00705071 0.00755536 2024-03-21T13:05:21.174 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:21.206 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:21.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:21.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:20 smithi204 ceph-mon[33153]: 6.13 scrub starts 2024-03-21T13:05:21.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:20 smithi204 ceph-mon[33153]: 6.13 scrub ok 2024-03-21T13:05:21.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:20 smithi204 ceph-mon[33153]: 5.4 scrub starts 2024-03-21T13:05:21.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:20 smithi204 ceph-mon[33153]: 5.4 scrub ok 2024-03-21T13:05:21.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:20 smithi204 ceph-mon[33153]: 5.e scrub starts 2024-03-21T13:05:21.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:20 smithi204 ceph-mon[33153]: 5.e scrub ok 2024-03-21T13:05:21.308 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:21.342 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:21.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:21.438 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:21.443 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:21.501 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:21.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:21.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:21.635 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:21.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:21.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:21.736 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:21.779 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:21.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:21.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:21.881 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:21.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:05:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:05:21.912 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:21.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:21.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:22.014 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:22.068 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:22.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:22.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 24 16 50372 50356 131.111 104.5 0.00593436 0.00762181 2024-03-21T13:05:22.170 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:22.171 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:21 smithi163 ceph-mon[29621]: 6.1a scrub starts 2024-03-21T13:05:22.171 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:21 smithi163 ceph-mon[29621]: 6.1a scrub ok 2024-03-21T13:05:22.171 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:21 smithi163 ceph-mon[29621]: 4.1c scrub starts 2024-03-21T13:05:22.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:21 smithi163 ceph-mon[29621]: 4.1c scrub ok 2024-03-21T13:05:22.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:21 smithi163 ceph-mon[29621]: pgmap v283: 148 pgs: 148 active+clean; 2.9 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-21T13:05:22.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:21 smithi163 ceph-mon[29621]: osdmap e179: 8 total, 8 up, 6 in 2024-03-21T13:05:22.172 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:21 smithi163 ceph-mon[40675]: 6.1a scrub starts 2024-03-21T13:05:22.172 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:21 smithi163 ceph-mon[40675]: 6.1a scrub ok 2024-03-21T13:05:22.172 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:21 smithi163 ceph-mon[40675]: 4.1c scrub starts 2024-03-21T13:05:22.172 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:21 smithi163 ceph-mon[40675]: 4.1c scrub ok 2024-03-21T13:05:22.172 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:21 smithi163 ceph-mon[40675]: pgmap v283: 148 pgs: 148 active+clean; 2.9 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-21T13:05:22.172 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:21 smithi163 ceph-mon[40675]: osdmap e179: 8 total, 8 up, 6 in 2024-03-21T13:05:22.205 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:22.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:22.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:22.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:21 smithi204 ceph-mon[33153]: 6.1a scrub starts 2024-03-21T13:05:22.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:21 smithi204 ceph-mon[33153]: 6.1a scrub ok 2024-03-21T13:05:22.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:21 smithi204 ceph-mon[33153]: 4.1c scrub starts 2024-03-21T13:05:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:21 smithi204 ceph-mon[33153]: 4.1c scrub ok 2024-03-21T13:05:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:21 smithi204 ceph-mon[33153]: pgmap v283: 148 pgs: 148 active+clean; 2.9 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-21T13:05:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:21 smithi204 ceph-mon[33153]: osdmap e179: 8 total, 8 up, 6 in 2024-03-21T13:05:22.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:22.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:22.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:22.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:22.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:22.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:22.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:22.574 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:22.612 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:22.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:22.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:22.714 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:22.754 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:22.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:22.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:22.889 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:22.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:22.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:22.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:23.028 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:23.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:23.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 25 16 52026 52010 130.001 103.375 0.0106295 0.00768685 2024-03-21T13:05:23.129 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:23.166 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:23.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:23.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:23.176 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[29621]: 6.17 scrub starts 2024-03-21T13:05:23.176 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[29621]: 4.1e scrub starts 2024-03-21T13:05:23.176 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[29621]: 4.1e scrub ok 2024-03-21T13:05:23.176 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[29621]: 6.17 scrub ok 2024-03-21T13:05:23.176 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[29621]: 5.8 scrub starts 2024-03-21T13:05:23.177 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[29621]: 5.8 scrub ok 2024-03-21T13:05:23.177 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:23.177 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[29621]: 3.d scrub starts 2024-03-21T13:05:23.177 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[29621]: 3.d scrub ok 2024-03-21T13:05:23.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[40675]: 6.17 scrub starts 2024-03-21T13:05:23.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[40675]: 4.1e scrub starts 2024-03-21T13:05:23.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[40675]: 4.1e scrub ok 2024-03-21T13:05:23.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[40675]: 6.17 scrub ok 2024-03-21T13:05:23.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[40675]: 5.8 scrub starts 2024-03-21T13:05:23.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[40675]: 5.8 scrub ok 2024-03-21T13:05:23.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:23.178 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[40675]: 3.d scrub starts 2024-03-21T13:05:23.178 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:22 smithi163 ceph-mon[40675]: 3.d scrub ok 2024-03-21T13:05:23.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:22 smithi204 ceph-mon[33153]: 6.17 scrub starts 2024-03-21T13:05:23.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:22 smithi204 ceph-mon[33153]: 4.1e scrub starts 2024-03-21T13:05:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:22 smithi204 ceph-mon[33153]: 4.1e scrub ok 2024-03-21T13:05:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:22 smithi204 ceph-mon[33153]: 6.17 scrub ok 2024-03-21T13:05:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:22 smithi204 ceph-mon[33153]: 5.8 scrub starts 2024-03-21T13:05:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:22 smithi204 ceph-mon[33153]: 5.8 scrub ok 2024-03-21T13:05:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:22 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:22 smithi204 ceph-mon[33153]: 3.d scrub starts 2024-03-21T13:05:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:22 smithi204 ceph-mon[33153]: 3.d scrub ok 2024-03-21T13:05:23.268 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:23.307 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:23.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:23.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:23.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:23.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:23.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:23.542 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:23.573 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:23.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:23.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:23.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:23.722 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:23.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:23.823 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:23.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:23.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:23.957 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:23.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:23.992 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:23.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:24.040 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[29621]: 4.4 scrub starts 2024-03-21T13:05:24.040 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[29621]: 4.4 scrub ok 2024-03-21T13:05:24.040 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[29621]: 4.c scrub starts 2024-03-21T13:05:24.040 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[29621]: 4.c scrub ok 2024-03-21T13:05:24.040 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[29621]: pgmap v285: 148 pgs: 148 active+clean; 2.9 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-21T13:05:24.040 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[29621]: 3.11 scrub starts 2024-03-21T13:05:24.041 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[29621]: 3.11 scrub ok 2024-03-21T13:05:24.041 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[29621]: 4.14 scrub starts 2024-03-21T13:05:24.041 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[29621]: 4.14 scrub ok 2024-03-21T13:05:24.041 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[40675]: 4.4 scrub starts 2024-03-21T13:05:24.041 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[40675]: 4.4 scrub ok 2024-03-21T13:05:24.041 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[40675]: 4.c scrub starts 2024-03-21T13:05:24.041 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[40675]: 4.c scrub ok 2024-03-21T13:05:24.041 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[40675]: pgmap v285: 148 pgs: 148 active+clean; 2.9 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-21T13:05:24.041 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[40675]: 3.11 scrub starts 2024-03-21T13:05:24.041 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[40675]: 3.11 scrub ok 2024-03-21T13:05:24.041 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[40675]: 4.14 scrub starts 2024-03-21T13:05:24.042 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:23 smithi163 ceph-mon[40675]: 4.14 scrub ok 2024-03-21T13:05:24.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 26 16 53569 53553 128.71 96.4375 0.0035532 0.00776387 2024-03-21T13:05:24.093 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:24.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:24.137 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:24.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:24.238 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:24.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:23 smithi204 ceph-mon[33153]: 4.4 scrub starts 2024-03-21T13:05:24.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:23 smithi204 ceph-mon[33153]: 4.4 scrub ok 2024-03-21T13:05:24.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:23 smithi204 ceph-mon[33153]: 4.c scrub starts 2024-03-21T13:05:24.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:23 smithi204 ceph-mon[33153]: 4.c scrub ok 2024-03-21T13:05:24.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:23 smithi204 ceph-mon[33153]: pgmap v285: 148 pgs: 148 active+clean; 2.9 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-21T13:05:24.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:23 smithi204 ceph-mon[33153]: 3.11 scrub starts 2024-03-21T13:05:24.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:23 smithi204 ceph-mon[33153]: 3.11 scrub ok 2024-03-21T13:05:24.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:23 smithi204 ceph-mon[33153]: 4.14 scrub starts 2024-03-21T13:05:24.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:23 smithi204 ceph-mon[33153]: 4.14 scrub ok 2024-03-21T13:05:24.276 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:24.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:24.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:24.378 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:24.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:24.419 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:24.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:24.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:24.549 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:24.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:24.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:24.650 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:24.691 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:24.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:24.792 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:24.799 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:05:24.799 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-03-21T13:05:24.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-03-21T13:05:24.799 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:05:24.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.799 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:116224 7.e 7:79293ee5:::benchmark_data_smithi163_80312_object116223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:05:24.799 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:22.703641+0000", 2024-03-21T13:05:24.800 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 182.090168162, 2024-03-21T13:05:24.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.14740398099999999, 2024-03-21T13:05:24.800 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.800 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.800 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.800 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.800 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.800 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 116224 2024-03-21T13:05:24.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.800 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.703641+0000", 2024-03-21T13:05:24.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.703641+0000", 2024-03-21T13:05:24.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.703643+0000", 2024-03-21T13:05:24.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.364e-06 2024-03-21T13:05:24.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.703668+0000", 2024-03-21T13:05:24.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5629000000000001e-05 2024-03-21T13:05:24.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.703703+0000", 2024-03-21T13:05:24.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4869000000000001e-05 2024-03-21T13:05:24.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.703713+0000", 2024-03-21T13:05:24.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.499e-06 2024-03-21T13:05:24.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.714459+0000", 2024-03-21T13:05:24.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010746695000000001 2024-03-21T13:05:24.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.714585+0000", 2024-03-21T13:05:24.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000125406 2024-03-21T13:05:24.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-03-21T13:05:24.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.714732+0000", 2024-03-21T13:05:24.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014734399999999999 2024-03-21T13:05:24.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.718208+0000", 2024-03-21T13:05:24.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034759769999999999 2024-03-21T13:05:24.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.721508+0000", 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003299457 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.850871+0000", 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.12936341400000001 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.850977+0000", 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105609 2024-03-21T13:05:24.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.851045+0000", 2024-03-21T13:05:24.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8718000000000002e-05 2024-03-21T13:05:24.806 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.806 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.806 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.806 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:116230 7.e 7:72c04ea8:::benchmark_data_smithi163_80312_object116229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:05:24.806 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:22.711246+0000", 2024-03-21T13:05:24.806 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 182.08256345500001, 2024-03-21T13:05:24.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.145862349, 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 116230 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.711246+0000", 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.711246+0000", 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.711248+0000", 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5680000000000001e-06 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.711272+0000", 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3988000000000001e-05 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.711285+0000", 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2897e-05 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.711294+0000", 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5310000000000007e-06 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.718238+0000", 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0069434279999999998 2024-03-21T13:05:24.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.718466+0000", 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00022805899999999999 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.718613+0000", 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014732600000000001 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.850819+0000", 2024-03-21T13:05:24.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.132205776 2024-03-21T13:05:24.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.851092+0000", 2024-03-21T13:05:24.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00027295500000000002 2024-03-21T13:05:24.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.856942+0000", 2024-03-21T13:05:24.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00585046 2024-03-21T13:05:24.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.857047+0000", 2024-03-21T13:05:24.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104434 2024-03-21T13:05:24.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.857108+0000", 2024-03-21T13:05:24.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1927000000000005e-05 2024-03-21T13:05:24.812 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.812 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.812 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.812 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:116232 7.e 7:79c2cda6:::benchmark_data_smithi163_80312_object116231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:22.711892+0000", 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 182.081917293, 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.15047192700000001, 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 116232 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.711892+0000", 2024-03-21T13:05:24.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.711892+0000", 2024-03-21T13:05:24.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.711893+0000", 2024-03-21T13:05:24.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8299999999999995e-07 2024-03-21T13:05:24.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.711916+0000", 2024-03-21T13:05:24.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2951000000000001e-05 2024-03-21T13:05:24.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.711928+0000", 2024-03-21T13:05:24.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1603999999999999e-05 2024-03-21T13:05:24.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.711937+0000", 2024-03-21T13:05:24.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7329999999999992e-06 2024-03-21T13:05:24.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.721539+0000", 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0096023370000000007 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.721673+0000", 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013433799999999999 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.721821+0000", 2024-03-21T13:05:24.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014795499999999999 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.850824+0000", 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.12900304600000001 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.851131+0000", 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030704199999999999 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.862041+0000", 2024-03-21T13:05:24.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010910173 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.862298+0000", 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025688499999999999 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.862364+0000", 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5879999999999999e-05 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.819 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:116237 7.e 7:7b83d4d6:::benchmark_data_smithi163_80312_object116236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:05:24.819 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:22.718292+0000", 2024-03-21T13:05:24.819 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 182.07551785199999, 2024-03-21T13:05:24.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.14458668499999999, 2024-03-21T13:05:24.819 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.819 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.819 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.819 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.819 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.819 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 116237 2024-03-21T13:05:24.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.819 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.718292+0000", 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.718292+0000", 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.718293+0000", 2024-03-21T13:05:24.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5090000000000001e-06 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.718317+0000", 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3343e-05 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.718326+0000", 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1349999999999998e-06 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.718337+0000", 2024-03-21T13:05:24.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0821e-05 2024-03-21T13:05:24.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.847552+0000", 2024-03-21T13:05:24.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.129215883 2024-03-21T13:05:24.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.847709+0000", 2024-03-21T13:05:24.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015632600000000001 2024-03-21T13:05:24.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-03-21T13:05:24.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.847885+0000", 2024-03-21T13:05:24.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000176365 2024-03-21T13:05:24.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.856843+0000", 2024-03-21T13:05:24.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089581929999999997 2024-03-21T13:05:24.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.856891+0000", 2024-03-21T13:05:24.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7549999999999997e-05 2024-03-21T13:05:24.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.862411+0000", 2024-03-21T13:05:24.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055206500000000002 2024-03-21T13:05:24.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.862725+0000", 2024-03-21T13:05:24.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00031384799999999998 2024-03-21T13:05:24.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.862878+0000", 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015306200000000001 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.14820.0:116243 7.2 e122/118 7:4a4b4daf:::benchmark_data_smithi163_80312_object116242:head v 122'7340, mlcod=122'7338)", 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:22.724830+0000", 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 182.06897958299999, 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.13572020900000001, 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.827 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.724830+0000", 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.724830+0000", 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.724831+0000", 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6000000000000003e-07 2024-03-21T13:05:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.724857+0000", 2024-03-21T13:05:24.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5846999999999998e-05 2024-03-21T13:05:24.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:24.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.847468+0000", 2024-03-21T13:05:24.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.122611688 2024-03-21T13:05:24.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.847476+0000", 2024-03-21T13:05:24.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.345e-06 2024-03-21T13:05:24.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.848450+0000", 2024-03-21T13:05:24.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00097467199999999999 2024-03-21T13:05:24.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.848473+0000", 2024-03-21T13:05:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3034999999999999e-05 2024-03-21T13:05:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.860350+0000", 2024-03-21T13:05:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011876513 2024-03-21T13:05:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.860550+0000", 2024-03-21T13:05:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000200349 2024-03-21T13:05:24.834 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.834 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.834 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.834 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:116246 7.8 7:162aacad:::benchmark_data_smithi163_80312_object116245:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:05:24.834 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:22.725707+0000", 2024-03-21T13:05:24.834 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 182.06810276499999, 2024-03-21T13:05:24.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.143343359, 2024-03-21T13:05:24.834 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.834 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.834 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.834 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 116246 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.725707+0000", 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.725707+0000", 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.725708+0000", 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1599999999999999e-06 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.725727+0000", 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9111e-05 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.847010+0000", 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.121282635 2024-03-21T13:05:24.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.847046+0000", 2024-03-21T13:05:24.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6086e-05 2024-03-21T13:05:24.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.852788+0000", 2024-03-21T13:05:24.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057421360000000001 2024-03-21T13:05:24.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.852927+0000", 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000139195 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.853089+0000", 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000161791 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.857045+0000", 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039556499999999998 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.858012+0000", 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00096714900000000005 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.866313+0000", 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0083014349999999994 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.868948+0000", 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026347039999999999 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:22.869050+0000", 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010230699999999999 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118934 7.e 7:73cbb615:::benchmark_data_smithi163_80312_object1493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.733675+0000", 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 179.06013429500001, 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5687331979999999, 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:05:24.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118934 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.733675+0000", 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.733675+0000", 2024-03-21T13:05:24.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.733677+0000", 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.934e-06 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.733683+0000", 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1360000000000002e-06 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.733695+0000", 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1691e-05 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.733704+0000", 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.733911+0000", 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020701500000000001 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.734050+0000", 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013966 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.734178+0000", 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012779299999999999 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.765948+0000", 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.031769955000000002 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.765962+0000", 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3587999999999999e-05 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.220385+0000", 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.45442343099999999 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.220399+0000", 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3818000000000001e-05 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.302325+0000", 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.081925647 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.302408+0000", 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3879000000000001e-05 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118987 7.e 7:73908788:::benchmark_data_smithi163_80312_object1546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:05:24.846 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.836218+0000", 2024-03-21T13:05:24.847 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 178.957591885, 2024-03-21T13:05:24.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4809060139999999, 2024-03-21T13:05:24.847 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.847 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.847 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.847 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.847 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.847 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118987 2024-03-21T13:05:24.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.847 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.836218+0000", 2024-03-21T13:05:24.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.836218+0000", 2024-03-21T13:05:24.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.836219+0000", 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.206e-06 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.836225+0000", 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7440000000000004e-06 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.836235+0000", 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0353999999999999e-05 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.836242+0000", 2024-03-21T13:05:24.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1330000000000001e-06 2024-03-21T13:05:24.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.836318+0000", 2024-03-21T13:05:24.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5741000000000005e-05 2024-03-21T13:05:24.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.836330+0000", 2024-03-21T13:05:24.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2337999999999999e-05 2024-03-21T13:05:24.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.220428+0000", 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.38409759599999999 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.220436+0000", 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8499999999999994e-06 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.302738+0000", 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0823026950000001 2024-03-21T13:05:24.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.302866+0000", 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012801300000000001 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,5", 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.303296+0000", 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00042974899999999999 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.315636+0000", 2024-03-21T13:05:24.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012340352000000001 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.315681+0000", 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4163999999999999e-05 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.316528+0000", 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000847276 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.316906+0000", 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00037839699999999999 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.316969+0000", 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2793999999999995e-05 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317124+0000", 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015461199999999999 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.854 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118990 7.e 7:7fa4d517:::benchmark_data_smithi163_80312_object1549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.844344+0000", 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 178.949465514, 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4730408850000001, 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118990 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.855 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.844344+0000", 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.844344+0000", 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.844345+0000", 2024-03-21T13:05:24.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.217e-06 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.844350+0000", 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8940000000000001e-06 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.844361+0000", 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0851e-05 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.844369+0000", 2024-03-21T13:05:24.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4570000000000005e-06 2024-03-21T13:05:24.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.844441+0000", 2024-03-21T13:05:24.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1390999999999995e-05 2024-03-21T13:05:24.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.844450+0000", 2024-03-21T13:05:24.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9959999999999999e-06 2024-03-21T13:05:24.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.220461+0000", 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.37601073499999998 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.220469+0000", 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0669999999999992e-06 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.313161+0000", 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0926922269999999 2024-03-21T13:05:24.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.313281+0000", 2024-03-21T13:05:24.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119821 2024-03-21T13:05:24.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,5", 2024-03-21T13:05:24.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.313503+0000", 2024-03-21T13:05:24.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00022230899999999999 2024-03-21T13:05:24.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.316534+0000", 2024-03-21T13:05:24.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0030308319999999998 2024-03-21T13:05:24.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.316861+0000", 2024-03-21T13:05:24.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032696400000000001 2024-03-21T13:05:24.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317169+0000", 2024-03-21T13:05:24.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000308609 2024-03-21T13:05:24.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317213+0000", 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4032000000000001e-05 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317273+0000", 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9376999999999997e-05 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317385+0000", 2024-03-21T13:05:24.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112106 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:127780 7.e 7:7b1a4715:::benchmark_data_smithi163_80312_object10339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:35.555174+0000", 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 169.23863514600001, 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.14042528100000001, 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.863 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 127780 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.555174+0000", 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.555174+0000", 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.555176+0000", 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1480000000000001e-06 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.555181+0000", 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6019999999999999e-06 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.555192+0000", 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.084e-05 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.555200+0000", 2024-03-21T13:05:24.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8059999999999995e-06 2024-03-21T13:05:24.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.673883+0000", 2024-03-21T13:05:24.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.11868335100000001 2024-03-21T13:05:24.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.674083+0000", 2024-03-21T13:05:24.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019984500000000001 2024-03-21T13:05:24.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,5", 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.674232+0000", 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000148601 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.678313+0000", 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0040815299999999999 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.682990+0000", 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0046768749999999996 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.690255+0000", 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0072654629999999998 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.695364+0000", 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0051085549999999999 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.695465+0000", 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000100933 2024-03-21T13:05:24.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.695600+0000", 2024-03-21T13:05:24.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013473200000000001 2024-03-21T13:05:24.869 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.869 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.869 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.869 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:127786 7.e 7:76c90635:::benchmark_data_smithi163_80312_object10345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-21T13:05:24.869 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:35.558234+0000", 2024-03-21T13:05:24.869 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 169.23557606899999, 2024-03-21T13:05:24.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.137627733, 2024-03-21T13:05:24.870 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.870 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.870 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.870 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.870 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.870 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 127786 2024-03-21T13:05:24.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.870 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.558234+0000", 2024-03-21T13:05:24.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.558234+0000", 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.558235+0000", 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.266e-06 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.558241+0000", 2024-03-21T13:05:24.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3249999999999996e-06 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.558252+0000", 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0845e-05 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.558260+0000", 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9179999999999994e-06 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.678350+0000", 2024-03-21T13:05:24.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.120090086 2024-03-21T13:05:24.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:24.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:24.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.678500+0000", 2024-03-21T13:05:24.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014989100000000001 2024-03-21T13:05:24.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,5", 2024-03-21T13:05:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.678651+0000", 2024-03-21T13:05:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015153200000000001 2024-03-21T13:05:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.682997+0000", 2024-03-21T13:05:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0043452500000000002 2024-03-21T13:05:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.683748+0000", 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00075093299999999996 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.691307+0000", 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0075599500000000002 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.695651+0000", 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004343965 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.695744+0000", 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2893999999999995e-05 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:35.695861+0000", 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000116878 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175770 7.e 7:78fafcef:::benchmark_data_smithi163_80312_object58329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.228931+0000", 2024-03-21T13:05:24.877 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 120.56487828, 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0743425600000001, 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175770 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.228931+0000", 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.228931+0000", 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.228933+0000", 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.643e-06 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.228949+0000", 2024-03-21T13:05:24.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5934e-05 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.228970+0000", 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0761e-05 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.228979+0000", 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0000000000000002e-06 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.229178+0000", 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019924100000000001 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.229299+0000", 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000121453 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,5", 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.229415+0000", 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011579 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.232533+0000", 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031179269999999999 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.232583+0000", 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9795e-05 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.240425+0000", 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078425779999999994 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.284990+0000", 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044564430000000002 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285004+0000", 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4148000000000001e-05 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.303201+0000", 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0181971030000001 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.303274+0000", 2024-03-21T13:05:24.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2756999999999995e-05 2024-03-21T13:05:24.884 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.884 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.884 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.884 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175774 7.e 7:7df3ee0e:::benchmark_data_smithi163_80312_object58333:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T13:05:24.884 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.242901+0000", 2024-03-21T13:05:24.884 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 120.550908931, 2024-03-21T13:05:24.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.061355786, 2024-03-21T13:05:24.884 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.884 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175774 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.242901+0000", 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.242901+0000", 2024-03-21T13:05:24.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.242902+0000", 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5069999999999999e-06 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.242909+0000", 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.99e-06 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.242922+0000", 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2479e-05 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.242931+0000", 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9199999999999993e-06 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.244240+0000", 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00130948 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.244703+0000", 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00046305100000000002 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,5", 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.245050+0000", 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00034666999999999999 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.249144+0000", 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004094019 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265019+0000", 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015874935999999999 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265037+0000", 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8360999999999999e-05 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285302+0000", 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020265004999999999 2024-03-21T13:05:24.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285312+0000", 2024-03-21T13:05:24.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7899999999999994e-06 2024-03-21T13:05:24.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304189+0000", 2024-03-21T13:05:24.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0188767400000001 2024-03-21T13:05:24.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304256+0000", 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7837999999999997e-05 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175778 7.e 7:7258fe02:::benchmark_data_smithi163_80312_object58337:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.252579+0000", 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 120.541230728, 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.051759933, 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.891 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.892 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175778 2024-03-21T13:05:24.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.892 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252579+0000", 2024-03-21T13:05:24.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252579+0000", 2024-03-21T13:05:24.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252580+0000", 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.573e-06 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252588+0000", 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9950000000000005e-06 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252605+0000", 2024-03-21T13:05:24.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6659000000000001e-05 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252615+0000", 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0421000000000001e-05 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252739+0000", 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012360100000000001 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252887+0000", 2024-03-21T13:05:24.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000148133 2024-03-21T13:05:24.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,5", 2024-03-21T13:05:24.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.253037+0000", 2024-03-21T13:05:24.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000149357 2024-03-21T13:05:24.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.256167+0000", 2024-03-21T13:05:24.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031299349999999999 2024-03-21T13:05:24.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265042+0000", 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0088755320000000002 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265074+0000", 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2274999999999998e-05 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285342+0000", 2024-03-21T13:05:24.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020267252999999999 2024-03-21T13:05:24.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285351+0000", 2024-03-21T13:05:24.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9260000000000006e-06 2024-03-21T13:05:24.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304289+0000", 2024-03-21T13:05:24.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0189389820000001 2024-03-21T13:05:24.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304339+0000", 2024-03-21T13:05:24.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9291e-05 2024-03-21T13:05:24.898 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.898 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.898 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.898 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175781 7.e 7:70cd84ea:::benchmark_data_smithi163_80312_object58340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T13:05:24.898 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.262490+0000", 2024-03-21T13:05:24.898 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 120.531319504, 2024-03-21T13:05:24.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.042246081, 2024-03-21T13:05:24.898 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.899 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:05:24.899 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.899 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.899 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.899 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175781 2024-03-21T13:05:24.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.899 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.262490+0000", 2024-03-21T13:05:24.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.262490+0000", 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.262492+0000", 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0430000000000002e-06 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.262498+0000", 2024-03-21T13:05:24.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3239999999999999e-06 2024-03-21T13:05:24.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.262511+0000", 2024-03-21T13:05:24.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2934e-05 2024-03-21T13:05:24.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.262520+0000", 2024-03-21T13:05:24.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4519999999999997e-06 2024-03-21T13:05:24.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265104+0000", 2024-03-21T13:05:24.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002584392 2024-03-21T13:05:24.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265233+0000", 2024-03-21T13:05:24.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000128823 2024-03-21T13:05:24.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,5", 2024-03-21T13:05:24.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265363+0000", 2024-03-21T13:05:24.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013014999999999999 2024-03-21T13:05:24.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285381+0000", 2024-03-21T13:05:24.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020017521999999999 2024-03-21T13:05:24.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285389+0000", 2024-03-21T13:05:24.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5459999999999992e-06 2024-03-21T13:05:24.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304470+0000", 2024-03-21T13:05:24.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.019080553 2024-03-21T13:05:24.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304736+0000", 2024-03-21T13:05:24.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00026634199999999998 2024-03-21T13:05:24.904 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.904 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.904 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.904 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175796 7.e 7:7e6aa0fe:::benchmark_data_smithi163_80312_object58355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.292110+0000", 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 120.501699496, 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.031751093, 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175796 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.292110+0000", 2024-03-21T13:05:24.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.292110+0000", 2024-03-21T13:05:24.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.292112+0000", 2024-03-21T13:05:24.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.826e-06 2024-03-21T13:05:24.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.292120+0000", 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8499999999999994e-06 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295305+0000", 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031855289999999999 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295317+0000", 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1387e-05 2024-03-21T13:05:24.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295911+0000", 2024-03-21T13:05:24.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059404100000000001 2024-03-21T13:05:24.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295919+0000", 2024-03-21T13:05:24.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3659999999999992e-06 2024-03-21T13:05:24.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304767+0000", 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0088483269999999 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304886+0000", 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119027 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304966+0000", 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9462999999999995e-05 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.316206+0000", 2024-03-21T13:05:24.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011240529000000001 2024-03-21T13:05:24.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.319434+0000", 2024-03-21T13:05:24.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003227415 2024-03-21T13:05:24.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.323761+0000", 2024-03-21T13:05:24.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0043271450000000001 2024-03-21T13:05:24.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.323816+0000", 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5157000000000001e-05 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.323861+0000", 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5031000000000003e-05 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.911 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175802 7.e 7:76aff798:::benchmark_data_smithi163_80312_object58361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.295348+0000", 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 120.498461187, 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0286435650000001, 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175802 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295348+0000", 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295348+0000", 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295350+0000", 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2839999999999999e-06 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295356+0000", 2024-03-21T13:05:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9959999999999999e-06 2024-03-21T13:05:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295363+0000", 2024-03-21T13:05:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3549999999999999e-06 2024-03-21T13:05:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295369+0000", 2024-03-21T13:05:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4979999999999997e-06 2024-03-21T13:05:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295939+0000", 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057073699999999998 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295945+0000", 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3079999999999998e-06 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.307185+0000", 2024-03-21T13:05:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0112405719999999 2024-03-21T13:05:24.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.307303+0000", 2024-03-21T13:05:24.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118151 2024-03-21T13:05:24.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:05:24.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.307426+0000", 2024-03-21T13:05:24.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000122302 2024-03-21T13:05:24.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.319353+0000", 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011927713 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.319478+0000", 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012420500000000001 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.323906+0000", 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044284820000000001 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.323955+0000", 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9020999999999997e-05 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.323992+0000", 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6940999999999997e-05 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.918 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175826 7.e 7:707dd3cd:::benchmark_data_smithi163_80312_object58385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T13:05:24.919 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.326877+0000", 2024-03-21T13:05:24.919 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 120.466932374, 2024-03-21T13:05:24.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.011345951, 2024-03-21T13:05:24.919 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.919 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.919 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.919 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.919 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.919 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175826 2024-03-21T13:05:24.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.919 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.326877+0000", 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.326877+0000", 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.326879+0000", 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1969999999999999e-06 2024-03-21T13:05:24.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.326887+0000", 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6639999999999998e-06 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.326907+0000", 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0026000000000001e-05 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.326918+0000", 2024-03-21T13:05:24.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1051e-05 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.327127+0000", 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020895399999999999 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.327139+0000", 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1640999999999999e-05 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.313333+0000", 2024-03-21T13:05:24.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98619378800000002 2024-03-21T13:05:24.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.313440+0000", 2024-03-21T13:05:24.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107896 2024-03-21T13:05:24.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:05:24.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.313556+0000", 2024-03-21T13:05:24.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115229 2024-03-21T13:05:24.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.319392+0000", 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0058364580000000001 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.324041+0000", 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0046487569999999999 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338129+0000", 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01408769 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338181+0000", 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1984000000000002e-05 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338223+0000", 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2616e-05 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.925 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175840 7.e 7:7d14a16e:::benchmark_data_smithi163_80312_object58399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T13:05:24.926 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.351435+0000", 2024-03-21T13:05:24.926 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 120.44237433000001, 2024-03-21T13:05:24.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98690400499999997, 2024-03-21T13:05:24.926 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.926 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.926 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.926 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.926 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.926 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175840 2024-03-21T13:05:24.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.926 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351435+0000", 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351435+0000", 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351437+0000", 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4699999999999999e-06 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351444+0000", 2024-03-21T13:05:24.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9489999999999999e-06 2024-03-21T13:05:24.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351459+0000", 2024-03-21T13:05:24.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5057e-05 2024-03-21T13:05:24.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351468+0000", 2024-03-21T13:05:24.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2699999999999993e-06 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351526+0000", 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7726999999999998e-05 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351532+0000", 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9699999999999996e-06 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.316235+0000", 2024-03-21T13:05:24.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.96470319000000004 2024-03-21T13:05:24.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.316345+0000", 2024-03-21T13:05:24.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011040900000000001 2024-03-21T13:05:24.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:05:24.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.316473+0000", 2024-03-21T13:05:24.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012779800000000001 2024-03-21T13:05:24.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.319490+0000", 2024-03-21T13:05:24.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0030170319999999998 2024-03-21T13:05:24.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.324867+0000", 2024-03-21T13:05:24.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0053769639999999997 2024-03-21T13:05:24.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338262+0000", 2024-03-21T13:05:24.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0133945 2024-03-21T13:05:24.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338306+0000", 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4622000000000001e-05 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338339+0000", 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3046999999999997e-05 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175843 7.e 7:7008cba1:::benchmark_data_smithi163_80312_object58402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.360203+0000", 2024-03-21T13:05:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 120.433606807, 2024-03-21T13:05:24.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.97825061400000002, 2024-03-21T13:05:24.934 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:24.934 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:24.934 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:24.934 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:24.934 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:24.934 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175843 2024-03-21T13:05:24.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.934 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:24.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:24.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360203+0000", 2024-03-21T13:05:24.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360203+0000", 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360204+0000", 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5030000000000001e-06 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360210+0000", 2024-03-21T13:05:24.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0669999999999997e-06 2024-03-21T13:05:24.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:24.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360225+0000", 2024-03-21T13:05:24.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4739000000000001e-05 2024-03-21T13:05:24.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:24.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360234+0000", 2024-03-21T13:05:24.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2119999999999992e-06 2024-03-21T13:05:24.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360303+0000", 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8526000000000001e-05 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360312+0000", 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4539999999999996e-06 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.320443+0000", 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.96013029599999999 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:24.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.320547+0000", 2024-03-21T13:05:24.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010440700000000001 2024-03-21T13:05:24.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:05:24.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.320898+0000", 2024-03-21T13:05:24.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035098699999999998 2024-03-21T13:05:24.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:24.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.324884+0000", 2024-03-21T13:05:24.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039858139999999999 2024-03-21T13:05:24.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.325424+0000", 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053980900000000004 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338376+0000", 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012952823 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338421+0000", 2024-03-21T13:05:24.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4348999999999998e-05 2024-03-21T13:05:24.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:24.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:24.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:24.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338453+0000", 2024-03-21T13:05:24.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2628e-05 2024-03-21T13:05:24.940 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.940 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.940 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.940 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:24.940 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:24.940 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:05:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.4 to 1 (10000) 2024-03-21T13:05:24.951 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[29621]: 5.6 scrub starts 2024-03-21T13:05:24.951 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[29621]: 5.6 scrub ok 2024-03-21T13:05:24.951 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:05:24.951 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:05:24.951 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[29621]: 4.1 scrub starts 2024-03-21T13:05:24.952 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[29621]: 4.1 scrub ok 2024-03-21T13:05:24.952 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:05:24.952 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:05:24.952 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1390932836' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:05:24.952 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[40675]: 5.6 scrub starts 2024-03-21T13:05:24.952 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[40675]: 5.6 scrub ok 2024-03-21T13:05:24.952 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:05:24.952 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:05:24.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[40675]: 4.1 scrub starts 2024-03-21T13:05:24.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[40675]: 4.1 scrub ok 2024-03-21T13:05:24.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:05:24.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:05:24.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:24 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1390932836' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:05:24.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:25.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:25.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:25.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:25.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 27 16 54849 54833 126.906 80 0.00844042 0.00787408 2024-03-21T13:05:25.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:25.138 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:25.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:25.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:25.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:25.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:24 smithi204 ceph-mon[33153]: 5.6 scrub starts 2024-03-21T13:05:25.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:24 smithi204 ceph-mon[33153]: 5.6 scrub ok 2024-03-21T13:05:25.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:24 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:05:25.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:24 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:05:25.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:24 smithi204 ceph-mon[33153]: 4.1 scrub starts 2024-03-21T13:05:25.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:24 smithi204 ceph-mon[33153]: 4.1 scrub ok 2024-03-21T13:05:25.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:24 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:05:25.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:24 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:05:25.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:24 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1390932836' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:05:25.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:25.277 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:25.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:25.378 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:25.423 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:25.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:25.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:25.525 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:25.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:25.569 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:25.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:25.670 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:25.708 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:25.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:25.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:25.810 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:25.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:25.852 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:25.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:25.918 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:05:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:05:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:05:25.919 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:05:25.919 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:05:25.919 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:05:25.919 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:05:25.919 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:05:25.919 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:05:25.920 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:05:25.920 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:05:25.920 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:05:25.920 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:05:25.920 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:05:25.920 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:05:25.920 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:05:25.920 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:05:25.920 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:05:25.920 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:05:25.953 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:25.991 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:25.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:25.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:26.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 28 15 56369 56354 125.768 95.0625 0.00782134 0.00794572 2024-03-21T13:05:26.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:26.216 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:26.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:26.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:25 smithi204 ceph-mon[33153]: pgmap v286: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T13:05:26.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:25 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1390932836' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:05:26.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:25 smithi204 ceph-mon[33153]: osdmap e180: 8 total, 8 up, 6 in 2024-03-21T13:05:26.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:25 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1185378369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:05:26.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:25 smithi204 ceph-mon[33153]: 5.16 scrub starts 2024-03-21T13:05:26.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:25 smithi204 ceph-mon[33153]: 5.16 scrub ok 2024-03-21T13:05:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:25 smithi163 ceph-mon[29621]: pgmap v286: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T13:05:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:25 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1390932836' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:05:26.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:25 smithi163 ceph-mon[29621]: osdmap e180: 8 total, 8 up, 6 in 2024-03-21T13:05:26.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:25 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1185378369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:05:26.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:25 smithi163 ceph-mon[29621]: 5.16 scrub starts 2024-03-21T13:05:26.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:25 smithi163 ceph-mon[29621]: 5.16 scrub ok 2024-03-21T13:05:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:25 smithi163 ceph-mon[40675]: pgmap v286: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T13:05:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:25 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1390932836' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:05:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:25 smithi163 ceph-mon[40675]: osdmap e180: 8 total, 8 up, 6 in 2024-03-21T13:05:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:25 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1185378369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:05:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:25 smithi163 ceph-mon[40675]: 5.16 scrub starts 2024-03-21T13:05:26.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:25 smithi163 ceph-mon[40675]: 5.16 scrub ok 2024-03-21T13:05:26.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:26.351 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:26.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:26.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:26.454 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:26.497 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:26.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:26.599 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:26.638 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:26.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:26.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:26.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:26.772 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:26.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:26.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:26.873 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:26.914 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:26.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:27.015 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:27.053 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:27.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:27.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:27.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 29 16 57911 57895 124.753 96.3125 0.0120128 0.00801082 2024-03-21T13:05:27.155 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:27.189 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:27.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:27.290 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:27.310 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:27.331 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:27.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:27.364 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-21T13:05:27.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:27.466 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:27.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:27.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:27.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:27.599 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:27.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:27.701 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:27.752 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:27 smithi204 ceph-mon[33153]: 4.e scrub starts 2024-03-21T13:05:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:27 smithi204 ceph-mon[33153]: 4.e scrub ok 2024-03-21T13:05:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:27 smithi204 ceph-mon[33153]: osdmap e181: 8 total, 8 up, 6 in 2024-03-21T13:05:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:27 smithi204 ceph-mon[33153]: 4.19 scrub starts 2024-03-21T13:05:27.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:27 smithi204 ceph-mon[33153]: 4.19 scrub ok 2024-03-21T13:05:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:27.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:27 smithi163 ceph-mon[29621]: 4.e scrub starts 2024-03-21T13:05:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:27 smithi163 ceph-mon[29621]: 4.e scrub ok 2024-03-21T13:05:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:27 smithi163 ceph-mon[29621]: osdmap e181: 8 total, 8 up, 6 in 2024-03-21T13:05:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:27 smithi163 ceph-mon[29621]: 4.19 scrub starts 2024-03-21T13:05:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:27 smithi163 ceph-mon[29621]: 4.19 scrub ok 2024-03-21T13:05:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:27 smithi163 ceph-mon[40675]: 4.e scrub starts 2024-03-21T13:05:27.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:27 smithi163 ceph-mon[40675]: 4.e scrub ok 2024-03-21T13:05:27.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:27 smithi163 ceph-mon[40675]: osdmap e181: 8 total, 8 up, 6 in 2024-03-21T13:05:27.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:27 smithi163 ceph-mon[40675]: 4.19 scrub starts 2024-03-21T13:05:27.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:27 smithi163 ceph-mon[40675]: 4.19 scrub ok 2024-03-21T13:05:27.855 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:27.901 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:27.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:27.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:28.002 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:28.043 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:28.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:28.075 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 30 16 59327 59311 123.544 88.5 0.014344 0.00808926 2024-03-21T13:05:28.145 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:28.147 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-21T13:05:28.188 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:28.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:28.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:28.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:28.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:28.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:28.422 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:28.461 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:28 smithi163 ceph-mon[29621]: pgmap v289: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T13:05:28.461 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:28 smithi163 ceph-mon[29621]: 5.2 scrub starts 2024-03-21T13:05:28.461 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:28 smithi163 ceph-mon[29621]: 5.2 scrub ok 2024-03-21T13:05:28.461 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:28 smithi163 ceph-mon[29621]: 4.3 scrub starts 2024-03-21T13:05:28.461 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:28 smithi163 ceph-mon[29621]: 4.3 scrub ok 2024-03-21T13:05:28.461 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:28 smithi163 ceph-mon[40675]: pgmap v289: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T13:05:28.461 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:28 smithi163 ceph-mon[40675]: 5.2 scrub starts 2024-03-21T13:05:28.461 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:28 smithi163 ceph-mon[40675]: 5.2 scrub ok 2024-03-21T13:05:28.461 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:28 smithi163 ceph-mon[40675]: 4.3 scrub starts 2024-03-21T13:05:28.462 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:28 smithi163 ceph-mon[40675]: 4.3 scrub ok 2024-03-21T13:05:28.474 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:28.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:28.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:28.606 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:28.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:28.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:28.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:28.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:28 smithi204 ceph-mon[33153]: pgmap v289: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T13:05:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:28 smithi204 ceph-mon[33153]: 5.2 scrub starts 2024-03-21T13:05:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:28 smithi204 ceph-mon[33153]: 5.2 scrub ok 2024-03-21T13:05:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:28 smithi204 ceph-mon[33153]: 4.3 scrub starts 2024-03-21T13:05:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:28 smithi204 ceph-mon[33153]: 4.3 scrub ok 2024-03-21T13:05:28.840 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:28.871 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:28.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:28.885 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:05:28.973 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:29.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:29.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:29.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:29.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 31 16 60716 60700 122.359 86.8125 0.0147384 0.00816605 2024-03-21T13:05:29.079 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:29.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:29.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:29.132 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:29.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:29.233 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:29.263 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:29.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:29.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:29.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:29.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:29.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:29.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:29.533 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:29.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:29.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:29.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:29.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:29 smithi204 ceph-mon[33153]: 5.b scrub starts 2024-03-21T13:05:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:29 smithi204 ceph-mon[33153]: 5.b scrub ok 2024-03-21T13:05:29.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:29 smithi204 ceph-mon[33153]: 4.13 scrub starts 2024-03-21T13:05:29.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:29 smithi204 ceph-mon[33153]: 4.13 scrub ok 2024-03-21T13:05:29.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:29 smithi204 ceph-mon[33153]: 3.17 scrub starts 2024-03-21T13:05:29.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:29 smithi204 ceph-mon[33153]: 3.17 scrub ok 2024-03-21T13:05:29.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:29 smithi204 ceph-mon[33153]: 6.11 scrub starts 2024-03-21T13:05:29.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:29 smithi204 ceph-mon[33153]: 6.11 scrub ok 2024-03-21T13:05:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[29621]: 5.b scrub starts 2024-03-21T13:05:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[29621]: 5.b scrub ok 2024-03-21T13:05:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[29621]: 4.13 scrub starts 2024-03-21T13:05:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[29621]: 4.13 scrub ok 2024-03-21T13:05:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[29621]: 3.17 scrub starts 2024-03-21T13:05:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[29621]: 3.17 scrub ok 2024-03-21T13:05:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[29621]: 6.11 scrub starts 2024-03-21T13:05:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[29621]: 6.11 scrub ok 2024-03-21T13:05:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[40675]: 5.b scrub starts 2024-03-21T13:05:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[40675]: 5.b scrub ok 2024-03-21T13:05:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[40675]: 4.13 scrub starts 2024-03-21T13:05:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[40675]: 4.13 scrub ok 2024-03-21T13:05:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[40675]: 3.17 scrub starts 2024-03-21T13:05:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[40675]: 3.17 scrub ok 2024-03-21T13:05:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[40675]: 6.11 scrub starts 2024-03-21T13:05:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:29 smithi163 ceph-mon[40675]: 6.11 scrub ok 2024-03-21T13:05:29.775 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:29.813 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:29.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:29.914 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:29.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:29.953 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:29.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:30.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:30.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 32 16 62206 62190 121.445 93.125 0.00884289 0.00822876 2024-03-21T13:05:30.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:30.086 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:30.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:30.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:30.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:30.228 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:30.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:30.329 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:30.365 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:30.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:30.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:30.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:30.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:30.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:30.597 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:30.631 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:30 smithi163 ceph-mon[40675]: pgmap v290: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 683 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T13:05:30.632 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:30 smithi163 ceph-mon[40675]: 6.4 scrub starts 2024-03-21T13:05:30.632 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:30 smithi163 ceph-mon[40675]: 6.4 scrub ok 2024-03-21T13:05:30.632 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:30 smithi163 ceph-mon[29621]: pgmap v290: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 683 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T13:05:30.632 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:30 smithi163 ceph-mon[29621]: 6.4 scrub starts 2024-03-21T13:05:30.632 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:30 smithi163 ceph-mon[29621]: 6.4 scrub ok 2024-03-21T13:05:30.639 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:30.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:30.741 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:30 smithi204 ceph-mon[33153]: pgmap v290: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 683 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T13:05:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:30 smithi204 ceph-mon[33153]: 6.4 scrub starts 2024-03-21T13:05:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:30 smithi204 ceph-mon[33153]: 6.4 scrub ok 2024-03-21T13:05:30.780 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:30.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:30.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:30.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:31.017 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:31.058 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:31.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:31.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:31.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 33 15 63562 63547 120.335 84.8125 0.00783072 0.00830566 2024-03-21T13:05:31.160 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:31.202 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:31.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:31.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:31.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:31.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:31.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:31.385 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:31.439 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:31.481 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:31.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:31.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:31.552 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:31 smithi204 ceph-mon[33153]: 4.a scrub starts 2024-03-21T13:05:31.552 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:31 smithi204 ceph-mon[33153]: 4.a scrub ok 2024-03-21T13:05:31.552 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:31 smithi204 ceph-mon[33153]: 5.d scrub starts 2024-03-21T13:05:31.552 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:31 smithi204 ceph-mon[33153]: 5.d scrub ok 2024-03-21T13:05:31.552 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:31 smithi204 ceph-mon[33153]: 3.13 scrub starts 2024-03-21T13:05:31.552 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:31 smithi204 ceph-mon[33153]: 3.13 scrub ok 2024-03-21T13:05:31.583 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:31.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:31 smithi163 ceph-mon[29621]: 4.a scrub starts 2024-03-21T13:05:31.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:31 smithi163 ceph-mon[29621]: 4.a scrub ok 2024-03-21T13:05:31.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:31 smithi163 ceph-mon[29621]: 5.d scrub starts 2024-03-21T13:05:31.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:31 smithi163 ceph-mon[29621]: 5.d scrub ok 2024-03-21T13:05:31.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:31 smithi163 ceph-mon[29621]: 3.13 scrub starts 2024-03-21T13:05:31.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:31 smithi163 ceph-mon[29621]: 3.13 scrub ok 2024-03-21T13:05:31.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:31 smithi163 ceph-mon[40675]: 4.a scrub starts 2024-03-21T13:05:31.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:31 smithi163 ceph-mon[40675]: 4.a scrub ok 2024-03-21T13:05:31.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:31 smithi163 ceph-mon[40675]: 5.d scrub starts 2024-03-21T13:05:31.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:31 smithi163 ceph-mon[40675]: 5.d scrub ok 2024-03-21T13:05:31.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:31 smithi163 ceph-mon[40675]: 3.13 scrub starts 2024-03-21T13:05:31.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:31 smithi163 ceph-mon[40675]: 3.13 scrub ok 2024-03-21T13:05:31.631 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:31.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:31.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:31.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:31.766 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:31.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:31.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:31.905 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:31.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:32.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:05:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:05:32.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:32.041 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:32.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:32.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 34 16 64989 64973 119.417 89.125 0.0116476 0.00836892 2024-03-21T13:05:32.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:32.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:32.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:32.278 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:32.308 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:32.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:32.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:32.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:32.440 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:32.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:32.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:32.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:32.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:32.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:32.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:32 smithi163 ceph-mon[29621]: pgmap v291: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-21T13:05:32.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:32 smithi163 ceph-mon[29621]: 5.13 scrub starts 2024-03-21T13:05:32.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:32 smithi163 ceph-mon[29621]: 5.13 scrub ok 2024-03-21T13:05:32.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:32 smithi163 ceph-mon[40675]: pgmap v291: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-21T13:05:32.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:32 smithi163 ceph-mon[40675]: 5.13 scrub starts 2024-03-21T13:05:32.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:32 smithi163 ceph-mon[40675]: 5.13 scrub ok 2024-03-21T13:05:32.676 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:32.719 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:32.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:32 smithi204 ceph-mon[33153]: pgmap v291: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-21T13:05:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:32 smithi204 ceph-mon[33153]: 5.13 scrub starts 2024-03-21T13:05:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:32 smithi204 ceph-mon[33153]: 5.13 scrub ok 2024-03-21T13:05:32.821 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:32.860 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:32.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:32.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:32.962 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:32.994 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:32.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:32.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:33.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 35 16 66354 66338 118.442 85.3125 0.0090229 0.00843786 2024-03-21T13:05:33.095 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:33.137 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:33.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:33.238 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:33.266 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:33.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:33.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:33.398 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:33.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:33.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:33.499 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:33.536 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:33.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:33.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:33.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:33.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:33.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:33.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:33 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:33.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:33 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:33.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:33 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:33.768 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:33.797 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:33.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:33.804 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:33.900 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:33.940 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:33.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:34.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:34.071 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:34.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:34.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 36 16 67736 67720 117.55 86.375 0.00742846 0.00850219 2024-03-21T13:05:34.174 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:34.199 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:34.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:34.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:34.331 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:34.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:34.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:34.333 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:05:34.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:34.460 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:34.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:34.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:34.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:34.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:34.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:34.727 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:34.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:34.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:34 smithi204 ceph-mon[33153]: pgmap v292: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-21T13:05:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:34 smithi204 ceph-mon[33153]: 5.c scrub starts 2024-03-21T13:05:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:34 smithi204 ceph-mon[33153]: 5.c scrub ok 2024-03-21T13:05:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:34 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/643493216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:05:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:34 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:05:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:34 smithi163 ceph-mon[29621]: pgmap v292: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-21T13:05:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:34 smithi163 ceph-mon[29621]: 5.c scrub starts 2024-03-21T13:05:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:34 smithi163 ceph-mon[29621]: 5.c scrub ok 2024-03-21T13:05:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:34 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/643493216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:05:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:34 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:05:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:34 smithi163 ceph-mon[40675]: pgmap v292: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-21T13:05:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:34 smithi163 ceph-mon[40675]: 5.c scrub starts 2024-03-21T13:05:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:34 smithi163 ceph-mon[40675]: 5.c scrub ok 2024-03-21T13:05:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:34 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/643493216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:05:34.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:34 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:05:34.829 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:34.870 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:34.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:34.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:35.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:35.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:35.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:35.076 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 37 16 69054 69038 116.6 82.375 0.00676658 0.00857191 2024-03-21T13:05:35.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:35.145 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:35.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:35.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:35.247 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:35.292 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:35.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:35.393 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:35.417 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:35.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:35.519 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:35.555 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:35.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:35.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:35.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[29621]: 4.12 scrub starts 2024-03-21T13:05:35.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[29621]: 4.12 scrub ok 2024-03-21T13:05:35.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[29621]: 6.e scrub starts 2024-03-21T13:05:35.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[29621]: 6.e scrub ok 2024-03-21T13:05:35.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[29621]: 5.5 scrub starts 2024-03-21T13:05:35.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[29621]: 5.5 scrub ok 2024-03-21T13:05:35.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:05:35.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:05:35.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:05:35.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[29621]: osdmap e182: 8 total, 8 up, 6 in 2024-03-21T13:05:35.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:35.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[40675]: 4.12 scrub starts 2024-03-21T13:05:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[40675]: 4.12 scrub ok 2024-03-21T13:05:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[40675]: 6.e scrub starts 2024-03-21T13:05:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[40675]: 6.e scrub ok 2024-03-21T13:05:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[40675]: 5.5 scrub starts 2024-03-21T13:05:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[40675]: 5.5 scrub ok 2024-03-21T13:05:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:05:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:05:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:05:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[40675]: osdmap e182: 8 total, 8 up, 6 in 2024-03-21T13:05:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:35.657 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:35.695 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:35.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:35.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:35.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:35 smithi204 ceph-mon[33153]: 4.12 scrub starts 2024-03-21T13:05:35.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:35 smithi204 ceph-mon[33153]: 4.12 scrub ok 2024-03-21T13:05:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:35 smithi204 ceph-mon[33153]: 6.e scrub starts 2024-03-21T13:05:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:35 smithi204 ceph-mon[33153]: 6.e scrub ok 2024-03-21T13:05:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:35 smithi204 ceph-mon[33153]: 5.5 scrub starts 2024-03-21T13:05:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:35 smithi204 ceph-mon[33153]: 5.5 scrub ok 2024-03-21T13:05:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:35 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:05:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:35 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:05:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:35 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:05:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:35 smithi204 ceph-mon[33153]: osdmap e182: 8 total, 8 up, 6 in 2024-03-21T13:05:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:05:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:35.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:35.831 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:35.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:35.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:35.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:35.966 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:35.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:36.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:05:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:05:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:05:36.067 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:36.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 38 16 70240 70224 115.48 74.125 0.00705138 0.0086546 2024-03-21T13:05:36.104 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:36.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:36.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:36.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:36.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:36.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:36.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:36.343 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:36.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:36.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:36.477 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:36.501 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:36.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:36.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:36.633 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:36.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:36.734 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:36 smithi204 ceph-mon[33153]: pgmap v294: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-21T13:05:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:36 smithi204 ceph-mon[33153]: 6.1 scrub starts 2024-03-21T13:05:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:36 smithi204 ceph-mon[33153]: 6.1 scrub ok 2024-03-21T13:05:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:36 smithi204 ceph-mon[33153]: osdmap e183: 8 total, 8 up, 6 in 2024-03-21T13:05:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:36 smithi163 ceph-mon[29621]: pgmap v294: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-21T13:05:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:36 smithi163 ceph-mon[29621]: 6.1 scrub starts 2024-03-21T13:05:36.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:36 smithi163 ceph-mon[29621]: 6.1 scrub ok 2024-03-21T13:05:36.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:36 smithi163 ceph-mon[29621]: osdmap e183: 8 total, 8 up, 6 in 2024-03-21T13:05:36.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:36 smithi163 ceph-mon[40675]: pgmap v294: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-21T13:05:36.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:36 smithi163 ceph-mon[40675]: 6.1 scrub starts 2024-03-21T13:05:36.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:36 smithi163 ceph-mon[40675]: 6.1 scrub ok 2024-03-21T13:05:36.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:36 smithi163 ceph-mon[40675]: osdmap e183: 8 total, 8 up, 6 in 2024-03-21T13:05:36.773 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:36.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:36.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:36.875 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:36.916 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:36.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:36.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:37.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:37.043 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:37.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:37.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 39 16 71347 71331 114.293 69.1875 0.00847004 0.00874486 2024-03-21T13:05:37.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:37.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:37.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:37.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:37.282 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:37.308 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:37.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:37.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:37.439 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:37.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:37.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:37.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:37.577 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:37.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:37.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:37.679 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:37.712 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:37.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:37.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:37.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:37 smithi204 ceph-mon[33153]: 6.f scrub starts 2024-03-21T13:05:37.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:37 smithi204 ceph-mon[33153]: 6.f scrub ok 2024-03-21T13:05:37.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:37 smithi163 ceph-mon[29621]: 6.f scrub starts 2024-03-21T13:05:37.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:37 smithi163 ceph-mon[29621]: 6.f scrub ok 2024-03-21T13:05:37.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:37 smithi163 ceph-mon[40675]: 6.f scrub starts 2024-03-21T13:05:37.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:37 smithi163 ceph-mon[40675]: 6.f scrub ok 2024-03-21T13:05:37.814 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:37.838 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:37.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:37.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:37.940 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:37.976 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:37.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:37.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:38.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:05:38.078247+0000 min lat: 0.00120788 max lat: 0.163103 avg lat: 0.0088151 2024-03-21T13:05:38.078 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:05:38.078 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 40 16 72571 72555 113.348 76.5 0.00529726 0.0088151 2024-03-21T13:05:38.078 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:38.101 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:38.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:38.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:38.203 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:38.243 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:38.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:38.345 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:38.368 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:38.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:38.470 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:38.506 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:38.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:38.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:38.608 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:38.647 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:38.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:38.749 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:38.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:38 smithi204 ceph-mon[33153]: pgmap v296: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-21T13:05:38.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:38 smithi204 ceph-mon[33153]: 5.a scrub starts 2024-03-21T13:05:38.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:38 smithi204 ceph-mon[33153]: 5.a scrub ok 2024-03-21T13:05:38.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:38 smithi204 ceph-mon[33153]: 5.1d scrub starts 2024-03-21T13:05:38.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:38 smithi204 ceph-mon[33153]: 5.1d scrub ok 2024-03-21T13:05:38.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:38 smithi163 ceph-mon[29621]: pgmap v296: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-21T13:05:38.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:38 smithi163 ceph-mon[29621]: 5.a scrub starts 2024-03-21T13:05:38.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:38 smithi163 ceph-mon[29621]: 5.a scrub ok 2024-03-21T13:05:38.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:38 smithi163 ceph-mon[29621]: 5.1d scrub starts 2024-03-21T13:05:38.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:38 smithi163 ceph-mon[29621]: 5.1d scrub ok 2024-03-21T13:05:38.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:38 smithi163 ceph-mon[40675]: pgmap v296: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-21T13:05:38.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:38 smithi163 ceph-mon[40675]: 5.a scrub starts 2024-03-21T13:05:38.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:38 smithi163 ceph-mon[40675]: 5.a scrub ok 2024-03-21T13:05:38.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:38 smithi163 ceph-mon[40675]: 5.1d scrub starts 2024-03-21T13:05:38.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:38 smithi163 ceph-mon[40675]: 5.1d scrub ok 2024-03-21T13:05:38.783 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:38.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:38.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:38.885 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:38.916 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:38.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:38.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:39.018 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:39.064 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:39.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:39.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:39.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 41 16 73595 73579 112.144 64 0.0152216 0.0089116 2024-03-21T13:05:39.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:39.200 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:39.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:39.301 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:39.338 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:39.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:39.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:39.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:39.481 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:39.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:39.582 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:39.619 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:39.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:39.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:39.641 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[29621]: 5.1f scrub starts 2024-03-21T13:05:39.641 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[29621]: 5.1f scrub ok 2024-03-21T13:05:39.642 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[29621]: 4.d scrub starts 2024-03-21T13:05:39.642 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[29621]: 4.d scrub ok 2024-03-21T13:05:39.642 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[29621]: 5.f scrub starts 2024-03-21T13:05:39.642 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[29621]: 5.f scrub ok 2024-03-21T13:05:39.642 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[29621]: 3.b scrub starts 2024-03-21T13:05:39.642 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[29621]: 3.b scrub ok 2024-03-21T13:05:39.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[40675]: 5.1f scrub starts 2024-03-21T13:05:39.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[40675]: 5.1f scrub ok 2024-03-21T13:05:39.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[40675]: 4.d scrub starts 2024-03-21T13:05:39.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[40675]: 4.d scrub ok 2024-03-21T13:05:39.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[40675]: 5.f scrub starts 2024-03-21T13:05:39.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[40675]: 5.f scrub ok 2024-03-21T13:05:39.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[40675]: 3.b scrub starts 2024-03-21T13:05:39.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:39 smithi163 ceph-mon[40675]: 3.b scrub ok 2024-03-21T13:05:39.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:39.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:39 smithi204 ceph-mon[33153]: 5.1f scrub starts 2024-03-21T13:05:39.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:39 smithi204 ceph-mon[33153]: 5.1f scrub ok 2024-03-21T13:05:39.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:39 smithi204 ceph-mon[33153]: 4.d scrub starts 2024-03-21T13:05:39.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:39 smithi204 ceph-mon[33153]: 4.d scrub ok 2024-03-21T13:05:39.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:39 smithi204 ceph-mon[33153]: 5.f scrub starts 2024-03-21T13:05:39.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:39 smithi204 ceph-mon[33153]: 5.f scrub ok 2024-03-21T13:05:39.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:39 smithi204 ceph-mon[33153]: 3.b scrub starts 2024-03-21T13:05:39.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:39 smithi204 ceph-mon[33153]: 3.b scrub ok 2024-03-21T13:05:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:39.758 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:39.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:39.859 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:39.900 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:39.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:40.002 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:40.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:40.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:40.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 42 16 75015 74999 111.587 88.75 0.0245663 0.00895661 2024-03-21T13:05:40.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:40.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:40.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:40.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:40.266 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:40.304 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:40.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:40.406 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:40.439 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:40.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:40.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[29621]: pgmap v297: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 54 MiB/s wr, 859 op/s 2024-03-21T13:05:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[29621]: 3.0 scrub starts 2024-03-21T13:05:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[29621]: 3.0 scrub ok 2024-03-21T13:05:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[29621]: 3.1a scrub starts 2024-03-21T13:05:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[29621]: 3.1a scrub ok 2024-03-21T13:05:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[29621]: 6.c scrub starts 2024-03-21T13:05:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[29621]: 6.c scrub ok 2024-03-21T13:05:40.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[29621]: 6.9 scrub starts 2024-03-21T13:05:40.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[29621]: 6.9 scrub ok 2024-03-21T13:05:40.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[40675]: pgmap v297: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 54 MiB/s wr, 859 op/s 2024-03-21T13:05:40.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[40675]: 3.0 scrub starts 2024-03-21T13:05:40.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[40675]: 3.0 scrub ok 2024-03-21T13:05:40.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[40675]: 3.1a scrub starts 2024-03-21T13:05:40.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[40675]: 3.1a scrub ok 2024-03-21T13:05:40.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[40675]: 6.c scrub starts 2024-03-21T13:05:40.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[40675]: 6.c scrub ok 2024-03-21T13:05:40.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[40675]: 6.9 scrub starts 2024-03-21T13:05:40.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:40 smithi163 ceph-mon[40675]: 6.9 scrub ok 2024-03-21T13:05:40.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:40.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:40.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:40.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:40.729 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:40.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:40.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:40 smithi204 ceph-mon[33153]: pgmap v297: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 54 MiB/s wr, 859 op/s 2024-03-21T13:05:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:40 smithi204 ceph-mon[33153]: 3.0 scrub starts 2024-03-21T13:05:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:40 smithi204 ceph-mon[33153]: 3.0 scrub ok 2024-03-21T13:05:40.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:40 smithi204 ceph-mon[33153]: 3.1a scrub starts 2024-03-21T13:05:40.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:40 smithi204 ceph-mon[33153]: 3.1a scrub ok 2024-03-21T13:05:40.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:40 smithi204 ceph-mon[33153]: 6.c scrub starts 2024-03-21T13:05:40.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:40 smithi204 ceph-mon[33153]: 6.c scrub ok 2024-03-21T13:05:40.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:40 smithi204 ceph-mon[33153]: 6.9 scrub starts 2024-03-21T13:05:40.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:40 smithi204 ceph-mon[33153]: 6.9 scrub ok 2024-03-21T13:05:40.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:40.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:40.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:40.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:40.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:40.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:41.078 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 43 16 76428 76412 111.046 88.3125 0.00650377 0.00899957 2024-03-21T13:05:41.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:41.130 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:41.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:41.232 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:41.280 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:41.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:41.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:41.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:41.419 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:41.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:41.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:41.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:41.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:41.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:41.653 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:41 smithi204 ceph-mon[33153]: 4.11 scrub starts 2024-03-21T13:05:41.653 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:41 smithi204 ceph-mon[33153]: 4.11 scrub ok 2024-03-21T13:05:41.653 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:05:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:05:41.656 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:41.694 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:41.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:41.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:41 smithi163 ceph-mon[29621]: 4.11 scrub starts 2024-03-21T13:05:41.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:41 smithi163 ceph-mon[29621]: 4.11 scrub ok 2024-03-21T13:05:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:41 smithi163 ceph-mon[40675]: 4.11 scrub starts 2024-03-21T13:05:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:41 smithi163 ceph-mon[40675]: 4.11 scrub ok 2024-03-21T13:05:41.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:41.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:41.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:41.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:41.957 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:41.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:42.059 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:42.077 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 44 16 77783 77767 110.446 84.6875 0.0159868 0.00904918 2024-03-21T13:05:42.089 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:42.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:42.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:42.223 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:42.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:42.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:42.303 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:05:42.303 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:05:42.325 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:42.364 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:42.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:42.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:42.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:42.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:42.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:42.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:42.639 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:42.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:42.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[29621]: pgmap v298: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-21T13:05:42.691 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[29621]: 3.10 scrub starts 2024-03-21T13:05:42.691 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[29621]: 3.10 scrub ok 2024-03-21T13:05:42.691 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[29621]: 3.1c scrub starts 2024-03-21T13:05:42.691 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[29621]: 3.1c scrub ok 2024-03-21T13:05:42.691 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[29621]: 6.3 scrub starts 2024-03-21T13:05:42.691 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[29621]: 6.3 scrub ok 2024-03-21T13:05:42.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[40675]: pgmap v298: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-21T13:05:42.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[40675]: 3.10 scrub starts 2024-03-21T13:05:42.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[40675]: 3.10 scrub ok 2024-03-21T13:05:42.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[40675]: 3.1c scrub starts 2024-03-21T13:05:42.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[40675]: 3.1c scrub ok 2024-03-21T13:05:42.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[40675]: 6.3 scrub starts 2024-03-21T13:05:42.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:42 smithi163 ceph-mon[40675]: 6.3 scrub ok 2024-03-21T13:05:42.741 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:42 smithi204 ceph-mon[33153]: pgmap v298: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-21T13:05:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:42 smithi204 ceph-mon[33153]: 3.10 scrub starts 2024-03-21T13:05:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:42 smithi204 ceph-mon[33153]: 3.10 scrub ok 2024-03-21T13:05:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:42 smithi204 ceph-mon[33153]: 3.1c scrub starts 2024-03-21T13:05:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:42 smithi204 ceph-mon[33153]: 3.1c scrub ok 2024-03-21T13:05:42.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:42 smithi204 ceph-mon[33153]: 6.3 scrub starts 2024-03-21T13:05:42.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:42 smithi204 ceph-mon[33153]: 6.3 scrub ok 2024-03-21T13:05:42.781 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:42.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:42.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:42.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:42.914 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:42.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:42.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:43.016 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:43.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:43.058 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:43.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:43.078 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 45 16 79129 79113 109.861 84.125 0.0102566 0.0090977 2024-03-21T13:05:43.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:43.191 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:43.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:43.293 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:43.367 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:43.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:43.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:43.500 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:43.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:43.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:43.561 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:43.561 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[29621]: 4.5 deep-scrub starts 2024-03-21T13:05:43.561 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[29621]: 4.5 deep-scrub ok 2024-03-21T13:05:43.561 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[29621]: 6.14 scrub starts 2024-03-21T13:05:43.561 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[29621]: 6.14 scrub ok 2024-03-21T13:05:43.561 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[29621]: 4.2 scrub starts 2024-03-21T13:05:43.561 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[29621]: 4.2 scrub ok 2024-03-21T13:05:43.562 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:43.562 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[40675]: 4.5 deep-scrub starts 2024-03-21T13:05:43.562 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[40675]: 4.5 deep-scrub ok 2024-03-21T13:05:43.562 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[40675]: 6.14 scrub starts 2024-03-21T13:05:43.562 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[40675]: 6.14 scrub ok 2024-03-21T13:05:43.562 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[40675]: 4.2 scrub starts 2024-03-21T13:05:43.562 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:43 smithi163 ceph-mon[40675]: 4.2 scrub ok 2024-03-21T13:05:43.602 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:43.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:43.643 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:43.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:43.744 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:43 smithi204 ceph-mon[33153]: 4.5 deep-scrub starts 2024-03-21T13:05:43.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:43 smithi204 ceph-mon[33153]: 4.5 deep-scrub ok 2024-03-21T13:05:43.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:43 smithi204 ceph-mon[33153]: 6.14 scrub starts 2024-03-21T13:05:43.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:43 smithi204 ceph-mon[33153]: 6.14 scrub ok 2024-03-21T13:05:43.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:43 smithi204 ceph-mon[33153]: 4.2 scrub starts 2024-03-21T13:05:43.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:43 smithi204 ceph-mon[33153]: 4.2 scrub ok 2024-03-21T13:05:43.773 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-21T13:05:43.775 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:43.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:43.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:43.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:43.906 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:43.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:43.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:44.008 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:44.053 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:44.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:44.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 46 16 80436 80420 109.244 81.6875 0.0239402 0.00914651 2024-03-21T13:05:44.154 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:44.195 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:44.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:44.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:44.297 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:44.347 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:44.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:44.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:44.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:44.482 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:44.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:44.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:44 smithi163 ceph-mon[29621]: pgmap v299: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-21T13:05:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:44 smithi163 ceph-mon[29621]: 5.11 scrub starts 2024-03-21T13:05:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:44 smithi163 ceph-mon[29621]: 5.11 scrub ok 2024-03-21T13:05:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:44 smithi163 ceph-mon[29621]: 6.b deep-scrub starts 2024-03-21T13:05:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:44 smithi163 ceph-mon[29621]: 6.b deep-scrub ok 2024-03-21T13:05:44.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:44 smithi163 ceph-mon[40675]: pgmap v299: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-21T13:05:44.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:44 smithi163 ceph-mon[40675]: 5.11 scrub starts 2024-03-21T13:05:44.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:44 smithi163 ceph-mon[40675]: 5.11 scrub ok 2024-03-21T13:05:44.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:44 smithi163 ceph-mon[40675]: 6.b deep-scrub starts 2024-03-21T13:05:44.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:44 smithi163 ceph-mon[40675]: 6.b deep-scrub ok 2024-03-21T13:05:44.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:44.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:44.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:44.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:44.735 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:44.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:44.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:44.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:44.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:44 smithi204 ceph-mon[33153]: pgmap v299: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-21T13:05:44.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:44 smithi204 ceph-mon[33153]: 5.11 scrub starts 2024-03-21T13:05:44.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:44 smithi204 ceph-mon[33153]: 5.11 scrub ok 2024-03-21T13:05:44.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:44 smithi204 ceph-mon[33153]: 6.b deep-scrub starts 2024-03-21T13:05:44.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:44 smithi204 ceph-mon[33153]: 6.b deep-scrub ok 2024-03-21T13:05:44.773 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:05:44.818 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:05:44.850 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:44.869 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:05:44.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:44.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:44.907 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:05:44.952 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:05:44.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:44.993 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:05:45.013 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:45.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:45.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:45.037 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:05:45.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 47 16 81462 81446 108.284 64.125 0.0159213 0.0092297 2024-03-21T13:05:45.082 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:05:45.090 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:05:45.090 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:05:45.090 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:05:45.090 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:05:45.090 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:05:45.090 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:05:45.090 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:05:45.090 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:05:45.115 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:45.160 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:45.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:45.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:45.291 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:05:45.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:45.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:45.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:45.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:45.435 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:45.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:45.462 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.6 to 1 (10000) 2024-03-21T13:05:45.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:45.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:45.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:45.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:45.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[29621]: 2.2 deep-scrub starts 2024-03-21T13:05:45.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[29621]: 2.2 deep-scrub ok 2024-03-21T13:05:45.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:05:45.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:05:45.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[29621]: osdmap e184: 8 total, 8 up, 6 in 2024-03-21T13:05:45.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3358069650' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:05:45.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:05:45.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2487169432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:05:45.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[29621]: pgmap v301: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-21T13:05:45.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[29621]: 3.f scrub starts 2024-03-21T13:05:45.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[29621]: 3.f scrub ok 2024-03-21T13:05:45.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[40675]: 2.2 deep-scrub starts 2024-03-21T13:05:45.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[40675]: 2.2 deep-scrub ok 2024-03-21T13:05:45.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:05:45.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:05:45.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[40675]: osdmap e184: 8 total, 8 up, 6 in 2024-03-21T13:05:45.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3358069650' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:05:45.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:05:45.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2487169432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:05:45.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[40675]: pgmap v301: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-21T13:05:45.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[40675]: 3.f scrub starts 2024-03-21T13:05:45.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-mon[40675]: 3.f scrub ok 2024-03-21T13:05:45.668 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:45.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:45.715 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:45.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:45.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:45 smithi204 ceph-mon[33153]: 2.2 deep-scrub starts 2024-03-21T13:05:45.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:45 smithi204 ceph-mon[33153]: 2.2 deep-scrub ok 2024-03-21T13:05:45.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:45 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:05:45.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:45 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:05:45.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:45 smithi204 ceph-mon[33153]: osdmap e184: 8 total, 8 up, 6 in 2024-03-21T13:05:45.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:45 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3358069650' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:05:45.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:45 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:05:45.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:45 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2487169432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:05:45.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:45 smithi204 ceph-mon[33153]: pgmap v301: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-21T13:05:45.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:45 smithi204 ceph-mon[33153]: 3.f scrub starts 2024-03-21T13:05:45.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:45 smithi204 ceph-mon[33153]: 3.f scrub ok 2024-03-21T13:05:45.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:45.865 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:45.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:45.919 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:05:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:05:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:05:45.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:45.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:45.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:45.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:46.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 48 16 82599 82583 107.509 71.0625 0.00440892 0.00929648 2024-03-21T13:05:46.099 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:46.143 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:46.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:46.245 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:46.278 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:46.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:46.379 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-21T13:05:46.380 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:46.420 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:46.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:46.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:46 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:05:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:46 smithi163 ceph-mon[29621]: osdmap e185: 8 total, 8 up, 6 in 2024-03-21T13:05:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:46 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:05:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:46 smithi163 ceph-mon[40675]: osdmap e185: 8 total, 8 up, 6 in 2024-03-21T13:05:46.523 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:46.556 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:46.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:46.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:46.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:46.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:46.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:46.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:46 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:05:46.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:46 smithi204 ceph-mon[33153]: osdmap e185: 8 total, 8 up, 6 in 2024-03-21T13:05:46.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:46.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:46.842 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:46.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:46.943 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:46.980 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:46.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:46.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:47.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 49 16 83940 83924 107.025 83.8125 0.00753601 0.00933869 2024-03-21T13:05:47.083 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:47.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:47.127 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:47.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:47.228 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:47.267 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:47.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:47.369 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:47.411 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:47.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:47.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:47.513 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:47.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:47.545 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:47.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:47.646 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:47.673 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:47.710 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:47.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:47.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:47 smithi204 ceph-mon[33153]: 3.1d scrub starts 2024-03-21T13:05:47.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:47 smithi204 ceph-mon[33153]: 3.1d scrub ok 2024-03-21T13:05:47.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:47 smithi204 ceph-mon[33153]: 2.1 scrub starts 2024-03-21T13:05:47.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:47 smithi204 ceph-mon[33153]: 2.1 scrub ok 2024-03-21T13:05:47.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:47 smithi204 ceph-mon[33153]: pgmap v303: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s 2024-03-21T13:05:47.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:47 smithi163 ceph-mon[29621]: 3.1d scrub starts 2024-03-21T13:05:47.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:47 smithi163 ceph-mon[29621]: 3.1d scrub ok 2024-03-21T13:05:47.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:47 smithi163 ceph-mon[29621]: 2.1 scrub starts 2024-03-21T13:05:47.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:47 smithi163 ceph-mon[29621]: 2.1 scrub ok 2024-03-21T13:05:47.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:47 smithi163 ceph-mon[29621]: pgmap v303: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s 2024-03-21T13:05:47.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:47 smithi163 ceph-mon[40675]: 3.1d scrub starts 2024-03-21T13:05:47.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:47 smithi163 ceph-mon[40675]: 3.1d scrub ok 2024-03-21T13:05:47.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:47 smithi163 ceph-mon[40675]: 2.1 scrub starts 2024-03-21T13:05:47.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:47 smithi163 ceph-mon[40675]: 2.1 scrub ok 2024-03-21T13:05:47.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:47 smithi163 ceph-mon[40675]: pgmap v303: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s 2024-03-21T13:05:47.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:47.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:47.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:47.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:47.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:47.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:48.058 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:48.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:48.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 50 16 85272 85256 106.55 83.25 0.00879212 0.00937886 2024-03-21T13:05:48.109 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:48.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:48.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:48.211 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:48.243 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:48.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:48.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:48.346 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:48.390 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:48.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:48.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:48.406 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:05:48.406 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:05:48.406 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-03-21T13:05:48.407 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-03-21T13:05:48.407 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:05:48.491 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:48.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:48.534 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:48.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:48.636 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:48.672 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:48.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:48.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:48.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:48 smithi204 ceph-mon[33153]: 6.d scrub starts 2024-03-21T13:05:48.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:48 smithi204 ceph-mon[33153]: 6.d scrub ok 2024-03-21T13:05:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:48 smithi163 ceph-mon[29621]: 6.d scrub starts 2024-03-21T13:05:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:48 smithi163 ceph-mon[29621]: 6.d scrub ok 2024-03-21T13:05:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:48 smithi163 ceph-mon[40675]: 6.d scrub starts 2024-03-21T13:05:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:48 smithi163 ceph-mon[40675]: 6.d scrub ok 2024-03-21T13:05:48.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:48.805 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:48.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:48.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:48.907 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:48.944 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:48.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:48.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:49.046 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:49.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 51 16 86534 86518 106.007 78.875 0.0104962 0.00942831 2024-03-21T13:05:49.087 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:49.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:49.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:49.189 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:49.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:49.223 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:49.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:49.323 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:49.371 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:49.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:49.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:49.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:49.501 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:49.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:49.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:49.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:49.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:49.659 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:05:49.738 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:49.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:49 smithi204 ceph-mon[33153]: 6.a scrub starts 2024-03-21T13:05:49.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:49 smithi204 ceph-mon[33153]: 6.a scrub ok 2024-03-21T13:05:49.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:49 smithi204 ceph-mon[33153]: 3.3 scrub starts 2024-03-21T13:05:49.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:49 smithi204 ceph-mon[33153]: 3.3 scrub ok 2024-03-21T13:05:49.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:49 smithi204 ceph-mon[33153]: pgmap v304: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 50 MiB/s wr, 807 op/s 2024-03-21T13:05:49.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:49 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:49.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:49 smithi163 ceph-mon[29621]: 6.a scrub starts 2024-03-21T13:05:49.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:49 smithi163 ceph-mon[29621]: 6.a scrub ok 2024-03-21T13:05:49.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:49 smithi163 ceph-mon[29621]: 3.3 scrub starts 2024-03-21T13:05:49.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:49 smithi163 ceph-mon[29621]: 3.3 scrub ok 2024-03-21T13:05:49.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:49 smithi163 ceph-mon[29621]: pgmap v304: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 50 MiB/s wr, 807 op/s 2024-03-21T13:05:49.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:49 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:49.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:49 smithi163 ceph-mon[40675]: 6.a scrub starts 2024-03-21T13:05:49.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:49 smithi163 ceph-mon[40675]: 6.a scrub ok 2024-03-21T13:05:49.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:49 smithi163 ceph-mon[40675]: 3.3 scrub starts 2024-03-21T13:05:49.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:49 smithi163 ceph-mon[40675]: 3.3 scrub ok 2024-03-21T13:05:49.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:49 smithi163 ceph-mon[40675]: pgmap v304: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 50 MiB/s wr, 807 op/s 2024-03-21T13:05:49.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:49 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:49.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:49.793 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:49.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:49.894 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:49.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:49.943 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:49.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:50.044 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:50.076 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:50.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:50.080 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 52 16 87825 87809 105.52 80.6875 0.0107616 0.00947162 2024-03-21T13:05:50.120 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-21T13:05:50.178 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:50.207 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:50.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:50.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:50.309 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:50.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:50.362 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:50.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:50.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:50.505 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:50.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:50.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:50.609 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:50.651 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:50.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:50.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:50.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:50 smithi204 ceph-mon[33153]: 3.1f scrub starts 2024-03-21T13:05:50.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:50 smithi204 ceph-mon[33153]: 3.1f scrub ok 2024-03-21T13:05:50.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:50 smithi163 ceph-mon[29621]: 3.1f scrub starts 2024-03-21T13:05:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:50 smithi163 ceph-mon[29621]: 3.1f scrub ok 2024-03-21T13:05:50.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:50 smithi163 ceph-mon[40675]: 3.1f scrub starts 2024-03-21T13:05:50.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:50 smithi163 ceph-mon[40675]: 3.1f scrub ok 2024-03-21T13:05:50.790 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:50.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:50.892 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:50.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:50.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:51.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:51.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:51.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:51.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:51.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 53 16 89166 89150 105.109 83.8125 0.0223688 0.00950877 2024-03-21T13:05:51.159 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:51.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:51.201 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:51.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:51.302 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:51.342 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:51.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:51.443 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:51.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:51.497 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:51.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:51.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:51.628 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:51.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:51 smithi163 ceph-mon[29621]: 4.1f scrub starts 2024-03-21T13:05:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:51 smithi163 ceph-mon[29621]: 4.1f scrub ok 2024-03-21T13:05:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:51 smithi163 ceph-mon[29621]: 6.1c scrub starts 2024-03-21T13:05:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:51 smithi163 ceph-mon[29621]: 6.1c scrub ok 2024-03-21T13:05:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:51 smithi163 ceph-mon[29621]: pgmap v305: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-21T13:05:51.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:51 smithi163 ceph-mon[40675]: 4.1f scrub starts 2024-03-21T13:05:51.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:51 smithi163 ceph-mon[40675]: 4.1f scrub ok 2024-03-21T13:05:51.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:51 smithi163 ceph-mon[40675]: 6.1c scrub starts 2024-03-21T13:05:51.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:51 smithi163 ceph-mon[40675]: 6.1c scrub ok 2024-03-21T13:05:51.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:51 smithi163 ceph-mon[40675]: pgmap v305: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-21T13:05:51.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:51.774 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:51.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:51.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:51.909 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:51.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:52.003 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:05:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:05:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:51 smithi204 ceph-mon[33153]: 4.1f scrub starts 2024-03-21T13:05:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:51 smithi204 ceph-mon[33153]: 4.1f scrub ok 2024-03-21T13:05:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:51 smithi204 ceph-mon[33153]: 6.1c scrub starts 2024-03-21T13:05:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:51 smithi204 ceph-mon[33153]: 6.1c scrub ok 2024-03-21T13:05:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:51 smithi204 ceph-mon[33153]: pgmap v305: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-21T13:05:52.012 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:52.050 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:52.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:52.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 54 15 90474 90459 104.677 81.8125 0.0115012 0.00954843 2024-03-21T13:05:52.151 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:52.191 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:52.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:52.292 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:52.342 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:52.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:52.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:52.443 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:52.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:52.489 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:52.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:52.546 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:05:52.590 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:52.630 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:52.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:52.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[29621]: 5.18 scrub starts 2024-03-21T13:05:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[29621]: 5.18 scrub ok 2024-03-21T13:05:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[29621]: 4.1a scrub starts 2024-03-21T13:05:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[29621]: 4.1a scrub ok 2024-03-21T13:05:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/540085550' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:05:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:05:52.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[40675]: 5.18 scrub starts 2024-03-21T13:05:52.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[40675]: 5.18 scrub ok 2024-03-21T13:05:52.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:52.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[40675]: 4.1a scrub starts 2024-03-21T13:05:52.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[40675]: 4.1a scrub ok 2024-03-21T13:05:52.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/540085550' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:05:52.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:52 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:05:52.770 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:52.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:52.872 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:52.921 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:52.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:52.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:53.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:52 smithi204 ceph-mon[33153]: 5.18 scrub starts 2024-03-21T13:05:53.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:52 smithi204 ceph-mon[33153]: 5.18 scrub ok 2024-03-21T13:05:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:52 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:52 smithi204 ceph-mon[33153]: 4.1a scrub starts 2024-03-21T13:05:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:52 smithi204 ceph-mon[33153]: 4.1a scrub ok 2024-03-21T13:05:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:52 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/540085550' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:05:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:52 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:05:53.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:53.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:53.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:53.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 55 16 91817 91801 104.299 83.875 0.0070622 0.00958316 2024-03-21T13:05:53.127 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:53.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:53.191 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:53.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:53.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:53.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:53.326 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:53.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:53.428 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:53.472 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:53.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:53.573 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:53.603 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:53.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:53.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:53.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:53.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:53.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:53.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:53 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:05:53.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:53 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:05:53.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:53 smithi163 ceph-mon[29621]: osdmap e186: 8 total, 8 up, 6 in 2024-03-21T13:05:53.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:53 smithi163 ceph-mon[29621]: pgmap v307: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-21T13:05:53.811 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:53 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:05:53.811 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:53 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:05:53.811 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:53 smithi163 ceph-mon[40675]: osdmap e186: 8 total, 8 up, 6 in 2024-03-21T13:05:53.811 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:53 smithi163 ceph-mon[40675]: pgmap v307: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-21T13:05:53.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:53.870 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:53.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:53.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:54.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:54.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:53 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:05:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:53 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:05:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:53 smithi204 ceph-mon[33153]: osdmap e186: 8 total, 8 up, 6 in 2024-03-21T13:05:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:53 smithi204 ceph-mon[33153]: pgmap v307: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-21T13:05:54.082 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 56 16 93242 93226 104.027 89.0625 0.0156034 0.00960867 2024-03-21T13:05:54.103 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:54.140 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:54.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:54.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:54.242 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:54.278 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:54.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:54.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:54.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:54.412 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:54.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:54.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:54.514 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:54.553 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:54.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:54.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:54.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:54.685 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:54.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:54.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:54.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:54.820 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:54.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:54.831 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:05:54.922 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:54.968 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:54.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:55.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:55.081 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 57 16 94556 94540 103.642 82.125 0.00879951 0.00964403 2024-03-21T13:05:55.104 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:55.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:55.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:55.255 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:55.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:55.356 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:55.397 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:55.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:55.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:55.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:55.531 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:55.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:55.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:55.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:55.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:55.758 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:05:55.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:55.800 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:55.801 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:05:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:05:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:05:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:55.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:55.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:55.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:55.947 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:55.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:56.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:56.061 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:55 smithi163 ceph-mon[29621]: pgmap v308: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-21T13:05:56.062 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:55 smithi163 ceph-mon[40675]: pgmap v308: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-21T13:05:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:56.080 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:56.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:56.082 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 58 16 95885 95869 103.285 83.0625 0.00880435 0.0096768 2024-03-21T13:05:56.181 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:56.222 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:56.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:56.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:56.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:55 smithi204 ceph-mon[33153]: pgmap v308: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-21T13:05:56.324 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:56.354 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:56.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:56.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:56.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:56.487 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:56.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:56.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:56.589 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:56.640 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:56.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:56.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:56.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:56.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:56.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:56.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:56.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:56.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:57.006 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:57.053 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:57.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:57.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:57.083 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 59 16 97181 97165 102.908 81 0.0129129 0.00971146 2024-03-21T13:05:57.155 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:05:57.205 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:57.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:57.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:05:57.284 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.7 to 1 (10000) 2024-03-21T13:05:57.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:57.357 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:57.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:57.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:57.403 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:05:57.403 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-03-21T13:05:57.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-03-21T13:05:57.403 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:05:57.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.403 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118954 7.a 7:5b69434d:::benchmark_data_smithi163_80312_object1513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:05:57.404 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.773489+0000", 2024-03-21T13:05:57.404 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 211.624828944, 2024-03-21T13:05:57.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5438778449999999, 2024-03-21T13:05:57.404 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.404 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.404 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.404 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.404 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.404 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118954 2024-03-21T13:05:57.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.404 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.773489+0000", 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.773489+0000", 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.773489+0000", 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9900000000000001e-07 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.773492+0000", 2024-03-21T13:05:57.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2500000000000001e-06 2024-03-21T13:05:57.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.773496+0000", 2024-03-21T13:05:57.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7230000000000004e-06 2024-03-21T13:05:57.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.773501+0000", 2024-03-21T13:05:57.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5979999999999999e-06 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786364+0000", 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012863400000000001 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786376+0000", 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2014000000000001e-05 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.215878+0000", 2024-03-21T13:05:57.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.42950202599999998 2024-03-21T13:05:57.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.215891+0000", 2024-03-21T13:05:57.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2928999999999999e-05 2024-03-21T13:05:57.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.298444+0000", 2024-03-21T13:05:57.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0825528470000001 2024-03-21T13:05:57.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.298577+0000", 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013321000000000001 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4,5", 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.298934+0000", 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035652299999999997 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.308602+0000", 2024-03-21T13:05:57.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0096679880000000006 2024-03-21T13:05:57.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.308663+0000", 2024-03-21T13:05:57.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1314000000000003e-05 2024-03-21T13:05:57.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.310765+0000", 2024-03-21T13:05:57.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021017190000000002 2024-03-21T13:05:57.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317198+0000", 2024-03-21T13:05:57.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.006432615 2024-03-21T13:05:57.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317257+0000", 2024-03-21T13:05:57.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9392999999999999e-05 2024-03-21T13:05:57.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317367+0000", 2024-03-21T13:05:57.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010989699999999999 2024-03-21T13:05:57.411 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.412 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.412 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.412 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118961 7.a 7:560b0fdf:::benchmark_data_smithi163_80312_object1520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:05:57.412 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.786480+0000", 2024-03-21T13:05:57.412 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 211.61183786800001, 2024-03-21T13:05:57.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5310859999999999, 2024-03-21T13:05:57.412 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.412 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.412 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.412 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.412 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.413 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118961 2024-03-21T13:05:57.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.413 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786480+0000", 2024-03-21T13:05:57.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786480+0000", 2024-03-21T13:05:57.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786482+0000", 2024-03-21T13:05:57.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.871e-06 2024-03-21T13:05:57.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786486+0000", 2024-03-21T13:05:57.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2980000000000002e-06 2024-03-21T13:05:57.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786495+0000", 2024-03-21T13:05:57.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9630000000000004e-06 2024-03-21T13:05:57.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786502+0000", 2024-03-21T13:05:57.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1559999999999998e-06 2024-03-21T13:05:57.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.790972+0000", 2024-03-21T13:05:57.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044696629999999996 2024-03-21T13:05:57.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.790982+0000", 2024-03-21T13:05:57.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.044e-05 2024-03-21T13:05:57.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.215919+0000", 2024-03-21T13:05:57.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.42493610500000001 2024-03-21T13:05:57.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.215929+0000", 2024-03-21T13:05:57.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9559999999999992e-06 2024-03-21T13:05:57.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.304457+0000", 2024-03-21T13:05:57.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.088528666 2024-03-21T13:05:57.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.304563+0000", 2024-03-21T13:05:57.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105543 2024-03-21T13:05:57.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4,5", 2024-03-21T13:05:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.304702+0000", 2024-03-21T13:05:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013904999999999999 2024-03-21T13:05:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.308719+0000", 2024-03-21T13:05:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0040172130000000004 2024-03-21T13:05:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.310771+0000", 2024-03-21T13:05:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020521760000000002 2024-03-21T13:05:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.312810+0000", 2024-03-21T13:05:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020392409999999998 2024-03-21T13:05:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317409+0000", 2024-03-21T13:05:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00459897 2024-03-21T13:05:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317460+0000", 2024-03-21T13:05:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1007000000000001e-05 2024-03-21T13:05:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317566+0000", 2024-03-21T13:05:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105682 2024-03-21T13:05:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118962 7.a 7:5f152353:::benchmark_data_smithi163_80312_object1521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:05:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.786514+0000", 2024-03-21T13:05:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 211.611803721, 2024-03-21T13:05:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5321943259999999, 2024-03-21T13:05:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118962 2024-03-21T13:05:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786514+0000", 2024-03-21T13:05:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786514+0000", 2024-03-21T13:05:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786515+0000", 2024-03-21T13:05:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.82e-07 2024-03-21T13:05:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786518+0000", 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1480000000000002e-06 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786524+0000", 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9989999999999997e-06 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.786529+0000", 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9989999999999999e-06 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.791005+0000", 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044763759999999998 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.791011+0000", 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2160000000000003e-06 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.215953+0000", 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.42494156799999999 2024-03-21T13:05:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.215961+0000", 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3499999999999997e-06 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.306672+0000", 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.090710691 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.306785+0000", 2024-03-21T13:05:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011281 2024-03-21T13:05:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4,5", 2024-03-21T13:05:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.306886+0000", 2024-03-21T13:05:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010084799999999999 2024-03-21T13:05:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.310777+0000", 2024-03-21T13:05:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0038913509999999999 2024-03-21T13:05:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.312867+0000", 2024-03-21T13:05:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020901579999999999 2024-03-21T13:05:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317858+0000", 2024-03-21T13:05:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049912070000000001 2024-03-21T13:05:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.318036+0000", 2024-03-21T13:05:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000177749 2024-03-21T13:05:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.318096+0000", 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0127999999999998e-05 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.318709+0000", 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061224599999999999 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118963 7.a 7:533d2a7e:::benchmark_data_smithi163_80312_object1522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.789277+0000", 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 211.609041018, 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5338491250000001, 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118963 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.789277+0000", 2024-03-21T13:05:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.789277+0000", 2024-03-21T13:05:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.789278+0000", 2024-03-21T13:05:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1349999999999999e-06 2024-03-21T13:05:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.789282+0000", 2024-03-21T13:05:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6550000000000002e-06 2024-03-21T13:05:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.789289+0000", 2024-03-21T13:05:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7139999999999993e-06 2024-03-21T13:05:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.789295+0000", 2024-03-21T13:05:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0229999999999998e-06 2024-03-21T13:05:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.791043+0000", 2024-03-21T13:05:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0017477529999999999 2024-03-21T13:05:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.791050+0000", 2024-03-21T13:05:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0419999999999997e-06 2024-03-21T13:05:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.215987+0000", 2024-03-21T13:05:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.42493687800000002 2024-03-21T13:05:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.215995+0000", 2024-03-21T13:05:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1850000000000004e-06 2024-03-21T13:05:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.308766+0000", 2024-03-21T13:05:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.092770402 2024-03-21T13:05:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.308884+0000", 2024-03-21T13:05:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118374 2024-03-21T13:05:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4,5", 2024-03-21T13:05:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.309016+0000", 2024-03-21T13:05:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000131674 2024-03-21T13:05:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.312884+0000", 2024-03-21T13:05:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0038680939999999999 2024-03-21T13:05:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.312908+0000", 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3971000000000001e-05 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317921+0000", 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050134910000000001 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.318807+0000", 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00088602799999999997 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.318986+0000", 2024-03-21T13:05:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000178596 2024-03-21T13:05:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.323126+0000", 2024-03-21T13:05:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041401099999999998 2024-03-21T13:05:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118968 7.a 7:599f3b62:::benchmark_data_smithi163_80312_object1527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.794791+0000", 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 211.603526472, 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.528656061, 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118968 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.794791+0000", 2024-03-21T13:05:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.794791+0000", 2024-03-21T13:05:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.794793+0000", 2024-03-21T13:05:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.415e-06 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.794800+0000", 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0700000000000001e-06 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.794819+0000", 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9315000000000001e-05 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.794828+0000", 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9609999999999994e-06 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.794969+0000", 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014057699999999999 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.794979+0000", 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0560999999999999e-05 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.216022+0000", 2024-03-21T13:05:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.421042317 2024-03-21T13:05:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.216036+0000", 2024-03-21T13:05:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4371e-05 2024-03-21T13:05:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.310807+0000", 2024-03-21T13:05:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0947708700000001 2024-03-21T13:05:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.310921+0000", 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000113813 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4,5", 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.311051+0000", 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000130297 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.314897+0000", 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0038457000000000001 2024-03-21T13:05:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.317980+0000", 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003082759 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.323186+0000", 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052067420000000003 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.323274+0000", 2024-03-21T13:05:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7799000000000004e-05 2024-03-21T13:05:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.323335+0000", 2024-03-21T13:05:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1130000000000006e-05 2024-03-21T13:05:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.323448+0000", 2024-03-21T13:05:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112364 2024-03-21T13:05:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118969 7.a 7:53f4fcaf:::benchmark_data_smithi163_80312_object1528:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.796195+0000", 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 211.602123003, 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5298895969999999, 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118969 2024-03-21T13:05:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.796195+0000", 2024-03-21T13:05:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.796195+0000", 2024-03-21T13:05:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.796196+0000", 2024-03-21T13:05:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.052e-06 2024-03-21T13:05:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.796202+0000", 2024-03-21T13:05:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6520000000000003e-06 2024-03-21T13:05:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.796217+0000", 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4918e-05 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.796225+0000", 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0350000000000001e-06 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.796299+0000", 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4684000000000005e-05 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.796311+0000", 2024-03-21T13:05:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2038000000000001e-05 2024-03-21T13:05:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.216061+0000", 2024-03-21T13:05:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.41974935000000002 2024-03-21T13:05:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.216069+0000", 2024-03-21T13:05:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.354e-06 2024-03-21T13:05:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.312946+0000", 2024-03-21T13:05:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0968767690000001 2024-03-21T13:05:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.313068+0000", 2024-03-21T13:05:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012260299999999999 2024-03-21T13:05:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4,5", 2024-03-21T13:05:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.313207+0000", 2024-03-21T13:05:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013810100000000001 2024-03-21T13:05:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.318763+0000", 2024-03-21T13:05:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.005556198 2024-03-21T13:05:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.323490+0000", 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047271700000000002 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.323576+0000", 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5896000000000006e-05 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.325905+0000", 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002329107 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.325953+0000", 2024-03-21T13:05:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8210000000000001e-05 2024-03-21T13:05:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.326085+0000", 2024-03-21T13:05:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013145999999999999 2024-03-21T13:05:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118972 7.a 7:5a45e77c:::benchmark_data_smithi163_80312_object1531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:05:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.797076+0000", 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 211.60124217800001, 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5291578019999998, 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118972 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.797076+0000", 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.797076+0000", 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.797076+0000", 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9699999999999995e-07 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.797082+0000", 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9610000000000002e-06 2024-03-21T13:05:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.797094+0000", 2024-03-21T13:05:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1591e-05 2024-03-21T13:05:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.797102+0000", 2024-03-21T13:05:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2109999999999998e-06 2024-03-21T13:05:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.797163+0000", 2024-03-21T13:05:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1047e-05 2024-03-21T13:05:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.797177+0000", 2024-03-21T13:05:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3244999999999999e-05 2024-03-21T13:05:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.216096+0000", 2024-03-21T13:05:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.41891990099999998 2024-03-21T13:05:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.216105+0000", 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0700000000000007e-06 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.315003+0000", 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.098898938 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.315124+0000", 2024-03-21T13:05:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012078000000000001 2024-03-21T13:05:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4,5", 2024-03-21T13:05:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.315254+0000", 2024-03-21T13:05:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012990500000000001 2024-03-21T13:05:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.323231+0000", 2024-03-21T13:05:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079773710000000005 2024-03-21T13:05:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.323533+0000", 2024-03-21T13:05:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030132299999999999 2024-03-21T13:05:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.461 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:57 smithi163 ceph-mon[29621]: osdmap e187: 8 total, 8 up, 6 in 2024-03-21T13:05:57.461 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:57 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1434583784' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:05:57.461 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:57 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:05:57.462 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:57 smithi163 ceph-mon[40675]: osdmap e187: 8 total, 8 up, 6 in 2024-03-21T13:05:57.462 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:57 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1434583784' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:05:57.462 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:57 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:05:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.323619+0000", 2024-03-21T13:05:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6625999999999994e-05 2024-03-21T13:05:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.326092+0000", 2024-03-21T13:05:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024726129999999998 2024-03-21T13:05:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.326135+0000", 2024-03-21T13:05:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3135e-05 2024-03-21T13:05:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.326234+0000", 2024-03-21T13:05:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8387999999999994e-05 2024-03-21T13:05:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118975 7.a 7:5441d719:::benchmark_data_smithi163_80312_object1534:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:05:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:25.804609+0000", 2024-03-21T13:05:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 211.593708822, 2024-03-21T13:05:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.529318277, 2024-03-21T13:05:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118975 2024-03-21T13:05:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.804609+0000", 2024-03-21T13:05:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.804609+0000", 2024-03-21T13:05:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.804611+0000", 2024-03-21T13:05:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.606e-06 2024-03-21T13:05:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.804617+0000", 2024-03-21T13:05:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.913e-06 2024-03-21T13:05:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.804629+0000", 2024-03-21T13:05:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2808e-05 2024-03-21T13:05:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.804638+0000", 2024-03-21T13:05:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0900000000000005e-06 2024-03-21T13:05:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.804709+0000", 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1934000000000006e-05 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:25.804721+0000", 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1528999999999999e-05 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.216130+0000", 2024-03-21T13:05:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.41140882699999998 2024-03-21T13:05:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.216138+0000", 2024-03-21T13:05:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7270000000000007e-06 2024-03-21T13:05:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.323655+0000", 2024-03-21T13:05:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.107517233 2024-03-21T13:05:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.323785+0000", 2024-03-21T13:05:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000129861 2024-03-21T13:05:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4,5", 2024-03-21T13:05:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.323932+0000", 2024-03-21T13:05:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147362 2024-03-21T13:05:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.328726+0000", 2024-03-21T13:05:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047943669999999999 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.330817+0000", 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020909520000000001 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.330859+0000", 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1464999999999997e-05 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.333711+0000", 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028519219999999998 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.333815+0000", 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104002 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.333927+0000", 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112679 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118902 7.0 7:007d3f07:::benchmark_data_smithi163_80312_object1461:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-03-21T13:05:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:26.221197+0000", 2024-03-21T13:05:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 211.177121422, 2024-03-21T13:05:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1029988239999999, 2024-03-21T13:05:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:05:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118902 2024-03-21T13:05:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221197+0000", 2024-03-21T13:05:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221197+0000", 2024-03-21T13:05:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221198+0000", 2024-03-21T13:05:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4330000000000001e-06 2024-03-21T13:05:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221203+0000", 2024-03-21T13:05:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6110000000000004e-06 2024-03-21T13:05:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221213+0000", 2024-03-21T13:05:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9899999999999992e-06 2024-03-21T13:05:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221223+0000", 2024-03-21T13:05:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0676000000000001e-05 2024-03-21T13:05:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221601+0000", 2024-03-21T13:05:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00037807599999999998 2024-03-21T13:05:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221612+0000", 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0625000000000001e-05 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.322320+0000", 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1007079850000001 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.324195+0000", 2024-03-21T13:05:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018754279999999999 2024-03-21T13:05:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118942 7.0 7:002109f8:::benchmark_data_smithi163_80312_object1501:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-03-21T13:05:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:26.221244+0000", 2024-03-21T13:05:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 211.177074347, 2024-03-21T13:05:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1604169, 2024-03-21T13:05:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118942 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221244+0000", 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221244+0000", 2024-03-21T13:05:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221244+0000", 2024-03-21T13:05:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8800000000000003e-07 2024-03-21T13:05:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221247+0000", 2024-03-21T13:05:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0900000000000001e-06 2024-03-21T13:05:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221254+0000", 2024-03-21T13:05:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3400000000000003e-06 2024-03-21T13:05:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221260+0000", 2024-03-21T13:05:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3319999999999997e-06 2024-03-21T13:05:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221640+0000", 2024-03-21T13:05:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00037987900000000001 2024-03-21T13:05:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221647+0000", 2024-03-21T13:05:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4089999999999997e-06 2024-03-21T13:05:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.324508+0000", 2024-03-21T13:05:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.10286069 2024-03-21T13:05:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.325067+0000", 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00055885400000000001 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,2,5,6", 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.325338+0000", 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00027088200000000001 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.372034+0000", 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046696143000000002 2024-03-21T13:05:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.372076+0000", 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2014000000000001e-05 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.372115+0000", 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9544e-05 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.372159+0000", 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3624000000000001e-05 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.381440+0000", 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092809480000000007 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.381537+0000", 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6788999999999998e-05 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.381661+0000", 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000123874 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118952 7.0 7:08d5a09c:::benchmark_data_smithi163_80312_object1511:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:26.221268+0000", 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 211.17704991400001, 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.160781673, 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118952 2024-03-21T13:05:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221268+0000", 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221268+0000", 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221269+0000", 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5400000000000002e-07 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221271+0000", 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2809999999999998e-06 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221276+0000", 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3909999999999997e-06 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221282+0000", 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7069999999999997e-06 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221671+0000", 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000389491 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221679+0000", 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2479999999999999e-06 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.372190+0000", 2024-03-21T13:05:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.150510989 2024-03-21T13:05:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.372304+0000", 2024-03-21T13:05:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114355 2024-03-21T13:05:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,2,5,6", 2024-03-21T13:05:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.372434+0000", 2024-03-21T13:05:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000130474 2024-03-21T13:05:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.381669+0000", 2024-03-21T13:05:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.009234473 2024-03-21T13:05:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.381713+0000", 2024-03-21T13:05:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3967000000000001e-05 2024-03-21T13:05:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.381770+0000", 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6969000000000003e-05 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.381815+0000", 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4666999999999998e-05 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.381858+0000", 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3609999999999998e-05 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.381916+0000", 2024-03-21T13:05:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7695000000000002e-05 2024-03-21T13:05:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.382050+0000", 2024-03-21T13:05:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013390200000000001 2024-03-21T13:05:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:118953 7.0 7:07ae2bc1:::benchmark_data_smithi163_80312_object1512:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-03-21T13:05:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:02:26.221289+0000", 2024-03-21T13:05:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 211.17702911500001, 2024-03-21T13:05:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.169101768, 2024-03-21T13:05:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 118953 2024-03-21T13:05:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221289+0000", 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221289+0000", 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221289+0000", 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3200000000000001e-07 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221291+0000", 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2910000000000002e-06 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221297+0000", 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4410000000000001e-06 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221303+0000", 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9070000000000004e-06 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221703+0000", 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040040099999999999 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:26.221710+0000", 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2049999999999996e-06 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.376979+0000", 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.155268269 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.377130+0000", 2024-03-21T13:05:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015090500000000001 2024-03-21T13:05:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,2,5,6", 2024-03-21T13:05:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.377449+0000", 2024-03-21T13:05:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00031931199999999998 2024-03-21T13:05:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.381727+0000", 2024-03-21T13:05:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0042776929999999999 2024-03-21T13:05:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.382100+0000", 2024-03-21T13:05:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00037295900000000002 2024-03-21T13:05:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.382144+0000", 2024-03-21T13:05:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4094000000000002e-05 2024-03-21T13:05:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.382186+0000", 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2539e-05 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.390196+0000", 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0080095040000000006 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.390266+0000", 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9973000000000002e-05 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:02:27.390391+0000", 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124943 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:211482 7.0 7:01296f40:::benchmark_data_smithi163_80312_object94041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:57.810917+0000", 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.587400937, 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.032653117, 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 211482 2024-03-21T13:05:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.810917+0000", 2024-03-21T13:05:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.810917+0000", 2024-03-21T13:05:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.810918+0000", 2024-03-21T13:05:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3399999999999999e-07 2024-03-21T13:05:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.810921+0000", 2024-03-21T13:05:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.371e-06 2024-03-21T13:05:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.810928+0000", 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2089999999999999e-06 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.810935+0000", 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5660000000000003e-06 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.818259+0000", 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0073246609999999997 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.818268+0000", 2024-03-21T13:05:57.505 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8689999999999993e-06 2024-03-21T13:05:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.838711+0000", 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0204426710000001 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.838836+0000", 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012539499999999999 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,6", 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.838921+0000", 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4293000000000001e-05 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.843431+0000", 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045108290000000001 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.843480+0000", 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9018999999999999e-05 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.843492+0000", 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1946e-05 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.843525+0000", 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2431000000000002e-05 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.843570+0000", 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5323e-05 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:211487 7.0 7:00881968:::benchmark_data_smithi163_80312_object94046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:57.821557+0000", 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.57676093800001, 2024-03-21T13:05:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0247014860000001, 2024-03-21T13:05:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 211487 2024-03-21T13:05:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.821557+0000", 2024-03-21T13:05:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.821557+0000", 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.821558+0000", 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.389e-06 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.821563+0000", 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8799999999999999e-06 2024-03-21T13:05:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.821573+0000", 2024-03-21T13:05:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9029999999999999e-06 2024-03-21T13:05:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.821581+0000", 2024-03-21T13:05:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3259999999999998e-06 2024-03-21T13:05:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.821833+0000", 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025230800000000002 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.821841+0000", 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.055e-06 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.841241+0000", 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0194001029999999 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.841361+0000", 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119586 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,6", 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.841447+0000", 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6891000000000005e-05 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.843695+0000", 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022476689999999999 2024-03-21T13:05:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.843706+0000", 2024-03-21T13:05:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1338000000000001e-05 2024-03-21T13:05:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.846139+0000", 2024-03-21T13:05:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024320829999999998 2024-03-21T13:05:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.846207+0000", 2024-03-21T13:05:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8936000000000005e-05 2024-03-21T13:05:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.846259+0000", 2024-03-21T13:05:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1019000000000001e-05 2024-03-21T13:05:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:211498 7.0 7:0de644bf:::benchmark_data_smithi163_80312_object94057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:57.829342+0000", 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.568975792, 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.020731869, 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 211498 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.829342+0000", 2024-03-21T13:05:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.829342+0000", 2024-03-21T13:05:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.829344+0000", 2024-03-21T13:05:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.375e-06 2024-03-21T13:05:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.829350+0000", 2024-03-21T13:05:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1959999999999996e-06 2024-03-21T13:05:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.829363+0000", 2024-03-21T13:05:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2842e-05 2024-03-21T13:05:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.829371+0000", 2024-03-21T13:05:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9899999999999997e-06 2024-03-21T13:05:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.829458+0000", 2024-03-21T13:05:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7817000000000003e-05 2024-03-21T13:05:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.829467+0000", 2024-03-21T13:05:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8210000000000008e-06 2024-03-21T13:05:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.843732+0000", 2024-03-21T13:05:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0142650719999999 2024-03-21T13:05:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.843863+0000", 2024-03-21T13:05:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013041800000000001 2024-03-21T13:05:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,6", 2024-03-21T13:05:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.843982+0000", 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011947000000000001 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.849873+0000", 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0058909729999999999 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.849887+0000", 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3684e-05 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.849926+0000", 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9172999999999999e-05 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.849982+0000", 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6155e-05 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.850074+0000", 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1883000000000001e-05 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:211503 7.0 7:0678b138:::benchmark_data_smithi163_80312_object94062:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:57.830885+0000", 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.56743322200001, 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.027959906, 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 211503 2024-03-21T13:05:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.830885+0000", 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.830885+0000", 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.830886+0000", 2024-03-21T13:05:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9299999999999997e-07 2024-03-21T13:05:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.830890+0000", 2024-03-21T13:05:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6829999999999999e-06 2024-03-21T13:05:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.830897+0000", 2024-03-21T13:05:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9439999999999999e-06 2024-03-21T13:05:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.830902+0000", 2024-03-21T13:05:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1560000000000003e-06 2024-03-21T13:05:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.830949+0000", 2024-03-21T13:05:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6384999999999997e-05 2024-03-21T13:05:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.830953+0000", 2024-03-21T13:05:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7829999999999998e-06 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.846292+0000", 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0153383520000001 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.846418+0000", 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012658100000000001 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,6", 2024-03-21T13:05:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.846542+0000", 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012339600000000001 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.854610+0000", 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0080686419999999991 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.854642+0000", 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1878999999999998e-05 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.858713+0000", 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0040708109999999997 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.858800+0000", 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6782999999999999e-05 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.858845+0000", 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4718000000000001e-05 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.530 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:211506 7.0 7:00e2d9a3:::benchmark_data_smithi163_80312_object94065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:57.832305+0000", 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.566013357, 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.026729572, 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 211506 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.531 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.832305+0000", 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.832305+0000", 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.832305+0000", 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7599999999999996e-07 2024-03-21T13:05:57.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.832310+0000", 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.437e-06 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.832319+0000", 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4970000000000007e-06 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.832327+0000", 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4989999999999997e-06 2024-03-21T13:05:57.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.832376+0000", 2024-03-21T13:05:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8989000000000001e-05 2024-03-21T13:05:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.832383+0000", 2024-03-21T13:05:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.25e-06 2024-03-21T13:05:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.850124+0000", 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.017741029 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.850249+0000", 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124957 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,6", 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.850405+0000", 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015594099999999999 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.854671+0000", 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0042661890000000001 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.854690+0000", 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9279e-05 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.858896+0000", 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0042052319999999997 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.536 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.858972+0000", 2024-03-21T13:05:57.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6631999999999994e-05 2024-03-21T13:05:57.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.859034+0000", 2024-03-21T13:05:57.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1865000000000005e-05 2024-03-21T13:05:57.537 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.537 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.537 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.537 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:211508 7.0 7:0b03df30:::benchmark_data_smithi163_80312_object94067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:57.834995+0000", 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.563323329, 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0241970629999999, 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 211508 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.834995+0000", 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.834995+0000", 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.834996+0000", 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1829999999999999e-06 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.835001+0000", 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0350000000000001e-06 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.835015+0000", 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.464e-05 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.835023+0000", 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9310000000000008e-06 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.835259+0000", 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023535799999999999 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.835267+0000", 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7649999999999993e-06 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.852563+0000", 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0172960339999999 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.541 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.852690+0000", 2024-03-21T13:05:57.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012704200000000001 2024-03-21T13:05:57.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,6", 2024-03-21T13:05:57.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.852817+0000", 2024-03-21T13:05:57.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012712199999999999 2024-03-21T13:05:57.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.854716+0000", 2024-03-21T13:05:57.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001898945 2024-03-21T13:05:57.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.859041+0000", 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0043257119999999998 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.859080+0000", 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9100000000000002e-05 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.859156+0000", 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5568999999999994e-05 2024-03-21T13:05:57.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.859192+0000", 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5627000000000003e-05 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:211517 7.0 7:0e54554c:::benchmark_data_smithi163_80312_object94076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:57.838000+0000", 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.560317509, 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0290471000000001, 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.544 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 211517 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.838000+0000", 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.838000+0000", 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.546 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.838001+0000", 2024-03-21T13:05:57.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9299999999999997e-07 2024-03-21T13:05:57.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.546 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.838006+0000", 2024-03-21T13:05:57.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2089999999999996e-06 2024-03-21T13:05:57.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.546 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.838017+0000", 2024-03-21T13:05:57.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0216e-05 2024-03-21T13:05:57.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.547 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.838029+0000", 2024-03-21T13:05:57.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2532e-05 2024-03-21T13:05:57.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.547 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.838149+0000", 2024-03-21T13:05:57.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012000100000000001 2024-03-21T13:05:57.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.548 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.548 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.838158+0000", 2024-03-21T13:05:57.548 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9269999999999994e-06 2024-03-21T13:05:57.548 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.548 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.548 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.548 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.854741+0000", 2024-03-21T13:05:57.548 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0165827709999999 2024-03-21T13:05:57.548 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.548 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.548 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.548 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.854806+0000", 2024-03-21T13:05:57.548 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5240000000000006e-05 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,6", 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.854870+0000", 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3602999999999994e-05 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.858884+0000", 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0040145220000000004 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.859046+0000", 2024-03-21T13:05:57.550 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016198300000000001 2024-03-21T13:05:57.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.550 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.550 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.866949+0000", 2024-03-21T13:05:57.550 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079032129999999992 2024-03-21T13:05:57.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.550 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.550 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.866998+0000", 2024-03-21T13:05:57.550 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8223999999999998e-05 2024-03-21T13:05:57.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.551 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.551 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.867048+0000", 2024-03-21T13:05:57.551 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9866e-05 2024-03-21T13:05:57.551 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.551 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.551 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.551 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:211522 7.0 7:0d16f964:::benchmark_data_smithi163_80312_object94081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-21T13:05:57.551 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:57.841739+0000", 2024-03-21T13:05:57.551 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.556579074, 2024-03-21T13:05:57.551 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0254621820000001, 2024-03-21T13:05:57.551 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 211522 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.841739+0000", 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.552 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.841739+0000", 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.841740+0000", 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7300000000000005e-07 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.841744+0000", 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5800000000000002e-06 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:05:57.553 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.841754+0000", 2024-03-21T13:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3780000000000007e-06 2024-03-21T13:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.841761+0000", 2024-03-21T13:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3390000000000004e-06 2024-03-21T13:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.841817+0000", 2024-03-21T13:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6023999999999998e-05 2024-03-21T13:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:57.841826+0000", 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7490000000000004e-06 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.856700+0000", 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0148738100000001 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.856814+0000", 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114264 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,6", 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.856929+0000", 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011491 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.859200+0000", 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022711820000000001 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.859235+0000", 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5247000000000003e-05 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.867126+0000", 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078904400000000003 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.867169+0000", 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3229999999999998e-05 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:58.867201+0000", 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2255999999999997e-05 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.557 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.558 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:05:57.558 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:05:57.558 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:05:57.595 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:57.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:57.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:57.696 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:57.733 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:57.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:57 smithi204 ceph-mon[33153]: osdmap e187: 8 total, 8 up, 6 in 2024-03-21T13:05:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:57 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1434583784' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:05:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:57 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:05:57.835 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:05:57.865 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:57.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:57.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:05:57.966 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:58.011 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:58.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:58.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:58.083 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:05:58.083807+0000 min lat: 0.00120788 max lat: 0.163103 avg lat: 0.00975437 2024-03-21T13:05:58.083 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:05:58.083 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 60 16 98408 98392 102.471 76.6875 0.00548855 0.00975437 2024-03-21T13:05:58.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:58.154 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:58.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:58.256 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:58.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:58.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:58.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:58.430 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:58.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:58.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:58.562 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:58.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:58.664 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:05:58.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:58 smithi163 ceph-mon[29621]: pgmap v310: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-21T13:05:58.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:58 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:05:58.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:05:58 smithi163 ceph-mon[29621]: osdmap e188: 8 total, 8 up, 6 in 2024-03-21T13:05:58.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:58 smithi163 ceph-mon[40675]: pgmap v310: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-21T13:05:58.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:58 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:05:58.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:05:58 smithi163 ceph-mon[40675]: osdmap e188: 8 total, 8 up, 6 in 2024-03-21T13:05:58.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:58.706 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:58.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:05:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:58 smithi204 ceph-mon[33153]: pgmap v310: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-21T13:05:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:58 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:05:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:05:58 smithi204 ceph-mon[33153]: osdmap e188: 8 total, 8 up, 6 in 2024-03-21T13:05:58.807 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:58.847 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:58.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:58.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:58.949 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:58.981 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:58.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:05:59.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:05:59.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 61 16 99850 99834 102.268 90.125 0.00866142 0.00977345 2024-03-21T13:05:59.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:59.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:05:59.217 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:59.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-21T13:05:59.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:59.247 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:59.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:59.349 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:05:59.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:59.391 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:59.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:05:59.492 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:59.539 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:59.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:59.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:59.626 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:05:59.640 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:05:59.678 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:59.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:59.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:05:59.779 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:05:59.825 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:05:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:59.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:05:59.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:05:59.964 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:05:59.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:05:59.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:00.066 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:00.083 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 62 16 101203 101187 101.982 84.5625 0.00900874 0.00980017 2024-03-21T13:06:00.105 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:00.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:00.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:00.207 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:00.251 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:00.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:00.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:00.353 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:00.404 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:00.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:00.505 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:00.533 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:00.533 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:00 smithi163 ceph-mon[29621]: pgmap v312: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 822 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-21T13:06:00.534 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:00 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1646706495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:00.534 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:00 smithi163 ceph-mon[40675]: pgmap v312: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 822 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-21T13:06:00.534 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:00 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1646706495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:00.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:00.635 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:00.677 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:00.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:00.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:00.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:00 smithi204 ceph-mon[33153]: pgmap v312: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 822 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-21T13:06:00.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:00 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1646706495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:00.779 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:00.816 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:00.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:00.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:00.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:00.959 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:00.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:00.979 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:06:00.979 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:06:00.979 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:06:00.979 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:06:00.979 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:06:00.979 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:06:00.980 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:06:00.980 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:06:00.980 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:06:00.980 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:06:00.980 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:06:00.980 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:06:00.980 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:06:00.980 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:06:00.980 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:06:00.980 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:06:01.061 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:01.083 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 63 16 102315 102299 101.467 69.5 0.00998817 0.00985121 2024-03-21T13:06:01.104 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:01.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:01.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:01.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:01.236 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:01.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:01.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:01.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:01.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:01.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:01.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:01.407 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:01 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:06:01.408 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:01 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:06:01.408 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:01 smithi204 ceph-mon[33153]: osdmap e189: 8 total, 8 up, 6 in 2024-03-21T13:06:01.464 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:01.502 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:01.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:01.586 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:06:01.586 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:06:01.604 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:01.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:01.633 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:01.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:01.734 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:01.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:06:01 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:06:01.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:01 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:06:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:01 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:06:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:01 smithi163 ceph-mon[29621]: osdmap e189: 8 total, 8 up, 6 in 2024-03-21T13:06:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:01 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:06:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:01 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:06:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:01 smithi163 ceph-mon[40675]: osdmap e189: 8 total, 8 up, 6 in 2024-03-21T13:06:01.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:01.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:01.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:01.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:01.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:01.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:01.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:02.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:02.044 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:02.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:02.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:02.083 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 64 16 103497 103481 101.036 73.875 0.00738988 0.00989299 2024-03-21T13:06:02.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:02.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:02.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:02.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:02.271 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:02.310 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:02.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:02.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:02.412 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:02.440 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:02.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:02.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-21T13:06:02.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:02.554 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:02 smithi204 ceph-mon[33153]: pgmap v313: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-21T13:06:02.595 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:02.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:02.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:02.697 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:02.714 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:02.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:02.742 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:02.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:02 smithi163 ceph-mon[29621]: pgmap v313: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-21T13:06:02.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:02 smithi163 ceph-mon[40675]: pgmap v313: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-21T13:06:02.826 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:06:02.843 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:02.882 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:02.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:02.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:03.021 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:03.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:03.083 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 65 16 104576 104560 100.519 67.4375 0.009239 0.00994389 2024-03-21T13:06:03.123 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:03.162 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:03.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:03.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:03.264 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:03.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:03.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:03.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:03.403 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-21T13:06:03.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:06:03.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:03.450 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:03.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:03.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:03.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:03.575 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:03.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:03.677 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:03.713 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:03.714 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:03.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:03.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:03.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:03.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:03.866 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:03.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:03.967 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:03.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:03.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:03.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:04.083 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 66 15 105694 105679 100.055 69.9375 0.0116524 0.00999023 2024-03-21T13:06:04.094 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:04.131 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:04.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:04.231 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:04.269 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:04.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:04.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:04.409 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:04.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:04.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:04.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:04 smithi204 ceph-mon[33153]: pgmap v315: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 789 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-21T13:06:04.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:04.548 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:04.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:04.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:04.694 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:04.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:04 smithi163 ceph-mon[29621]: pgmap v315: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 789 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-21T13:06:04.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:04 smithi163 ceph-mon[40675]: pgmap v315: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 789 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-21T13:06:04.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:04.824 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:04.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:04.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:04.926 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:04.967 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:04.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:04.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:04.984 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:06:04.984 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:06:05.069 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:05.083 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 67 16 106869 106853 99.657 73.375 0.00698017 0.0100299 2024-03-21T13:06:05.107 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:05.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:05.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:05.209 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:05.252 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:05.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:05.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:05.354 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:05.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:05.395 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:05.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:05.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:05.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:05.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:05.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:05.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:06:05.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:05.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:05 smithi163 ceph-mon[29621]: pgmap v316: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-21T13:06:05.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:06:05.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:05.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:05 smithi163 ceph-mon[40675]: pgmap v316: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-21T13:06:05.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:05.661 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:05.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:05.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:06:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:05.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:05 smithi204 ceph-mon[33153]: pgmap v316: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-21T13:06:05.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:05.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:05.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:05.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:05.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-21T13:06:05.895 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:05.937 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:05.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:05.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:06.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:06:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:06:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:06:06.039 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:06.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 68 16 108292 108276 99.4992 88.9375 0.0106058 0.010046 2024-03-21T13:06:06.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:06.085 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:06.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:06.186 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:06.221 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:06.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:06.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:06.226 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:06:06.323 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:06.361 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:06.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:06.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:06.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:06.500 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:06.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:06.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:06.602 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:06.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:06.641 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:06.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:06.742 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:06.774 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:06.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:06.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:06.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:06.905 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:06.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:06.952 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:07.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:07.038 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:07.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:07.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 69 16 109654 109638 99.2908 85.125 0.0117523 0.010067 2024-03-21T13:06:07.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:07.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:07.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:07.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:07.276 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:07.323 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:07.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:07.425 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:07.456 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:07.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:07.558 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:07.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:07.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:07.700 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:07.737 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:07.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:07.839 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:07.876 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:07.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:07.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:07.978 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:08.021 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:08.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:08.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:08.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 70 16 111004 110988 99.0776 84.375 0.00330974 0.0100862 2024-03-21T13:06:08.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:08.165 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:08.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:08.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:08.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:08.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:08.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:08.384 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:06:08.384 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:06:08.401 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:08.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:08.442 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:08.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:08.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:08 smithi204 ceph-mon[33153]: pgmap v317: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-21T13:06:08.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:08 smithi163 ceph-mon[29621]: pgmap v317: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-21T13:06:08.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:08 smithi163 ceph-mon[40675]: pgmap v317: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-21T13:06:08.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:08.578 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:08.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:08.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:08.680 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:08.724 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:08.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:08.729 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:06:08.825 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:08.892 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:08.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:08.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:08.994 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:09.023 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:09.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:09.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 71 16 112302 112286 98.8247 81.125 0.00737192 0.0101143 2024-03-21T13:06:09.125 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:09.166 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:09.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:09.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:09.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:09.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:09.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:09.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:09.407 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-21T13:06:09.433 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:09.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:09.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:09.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:09.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:09.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:09.686 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:09.707 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:09.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:09.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:09.782 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:06:09.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:09.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:09.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:09.957 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:10.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:10.003 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:10.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:10.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 72 16 113529 113513 98.5172 76.6875 0.0145014 0.0101461 2024-03-21T13:06:10.097 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:06:10.104 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:10.110 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:06:10.136 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:10.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:10.238 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:10.275 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:10.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:10.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:10.302 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:06:10.377 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:10.417 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:10.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:10.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:10 smithi163 ceph-mon[29621]: pgmap v318: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-21T13:06:10.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:10 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3182395817' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:06:10.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:10 smithi163 ceph-mon[40675]: pgmap v318: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-21T13:06:10.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:10 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3182395817' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:06:10.519 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:10.550 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:10.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:10.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:10.651 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:10.689 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:10.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:10.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:10 smithi204 ceph-mon[33153]: pgmap v318: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-21T13:06:10.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:10 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3182395817' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:06:10.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:10.830 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:10.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:10.932 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:10.971 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:10.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:11.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:11.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 73 16 114864 114848 98.3105 83.4375 0.0108634 0.0101673 2024-03-21T13:06:11.111 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:11.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:11.213 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:11.256 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:11.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:11.358 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:11.396 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:11.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:11.498 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:11.533 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:11.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:11.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:11 smithi204 ceph-mon[33153]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:06:11.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:11 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:06:11.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:11 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3182395817' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:06:11.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:11 smithi204 ceph-mon[33153]: osdmap e190: 8 total, 8 up, 6 in 2024-03-21T13:06:11.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:11 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:06:11.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:11 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:06:11.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:11 smithi204 ceph-mon[33153]: osdmap e191: 8 total, 8 up, 6 in 2024-03-21T13:06:11.635 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:11.670 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:11.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:11.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[29621]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:06:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:06:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3182395817' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:06:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[29621]: osdmap e190: 8 total, 8 up, 6 in 2024-03-21T13:06:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:06:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:06:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[29621]: osdmap e191: 8 total, 8 up, 6 in 2024-03-21T13:06:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[40675]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:06:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:06:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3182395817' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:06:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[40675]: osdmap e190: 8 total, 8 up, 6 in 2024-03-21T13:06:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:06:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:06:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:11 smithi163 ceph-mon[40675]: osdmap e191: 8 total, 8 up, 6 in 2024-03-21T13:06:11.772 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:11.817 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:11.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:11.914 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:06:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:06:11.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:11.954 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:11.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:11.958 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:06:11.958 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:06:12.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:12.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 74 16 116105 116089 98.0301 77.5625 0.0120057 0.0101967 2024-03-21T13:06:12.095 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:12.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:12.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:12.197 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:12.237 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:12.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:12.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:12.339 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:12.378 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:12.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:12.480 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:12.515 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:12.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:12.617 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:12.628 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:12 smithi204 ceph-mon[33153]: pgmap v320: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T13:06:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:12.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:12.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:12.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-21T13:06:12.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:12 smithi163 ceph-mon[29621]: pgmap v320: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T13:06:12.685 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:12 smithi163 ceph-mon[40675]: pgmap v320: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T13:06:12.753 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:12.784 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:12.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:12.886 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:12.910 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:12.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:12.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:12.975 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:06:13.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:13.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:13.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:13.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:13.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 75 16 117351 117335 97.7613 77.875 0.0260353 0.0102239 2024-03-21T13:06:13.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:13.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:13.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:13.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:13.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:13.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:13.324 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:13.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:13.424 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:13 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:13.425 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:13.448 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:13.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:13.549 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:13.588 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:13.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:13.689 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:13.735 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:13.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:13.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:13 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:13.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:13 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:13.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:13.869 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:13.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:13.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:13.971 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:14.010 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:14.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:14.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:14.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 76 16 118464 118448 97.3902 69.5625 0.0110576 0.0102633 2024-03-21T13:06:14.112 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:14.153 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:14.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:14.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:14.255 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:14.298 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:14.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:14.399 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:14.441 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:14.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:14.542 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:14.601 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:14.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:14.601 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:14 smithi163 ceph-mon[29621]: pgmap v322: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-21T13:06:14.601 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:14 smithi163 ceph-mon[40675]: pgmap v322: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-21T13:06:14.602 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-03-21T13:06:08.757584+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6995581352,"num_objects":106933,"num_object_clones":0,"num_object_copies":320799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106933,"num_whiteouts":0,"num_read":1358,"num_read_kb":1204,"num_write":107350,"num_write_kb":6833501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1775783,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65190,"ondisk_log_size":65190,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":22056276,"kb_used_data":20351844,"kb_used_omap":9,"kb_used_meta":1704374,"kb_avail":540414636,"statfs":{"total":575970213888,"available":553384587264,"internally_reserved":0,"allocated":20840288256,"data_stored":20836351956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":1745279654},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":812449792,"num_objects":12397,"num_object_clones":0,"num_object_copies":37191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12397,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12397,"num_write_kb":793408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002846"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":286,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.270531+0000","last_change":"2024-03-21T13:05:20.083213+0000","last_active":"2024-03-21T13:06:01.270531+0000","last_peered":"2024-03-21T13:06:01.270531+0000","last_clean":"2024-03-21T13:06:01.270531+0000","last_became_active":"2024-03-21T13:02:26.223292+0000","last_became_peered":"2024-03-21T13:02:26.223292+0000","last_unstale":"2024-03-21T13:06:01.270531+0000","last_undegraded":"2024-03-21T13:06:01.270531+0000","last_fullsized":"2024-03-21T13:06:01.270531+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:20.083111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.083111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:37.095715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":153,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275334+0000","last_change":"2024-03-21T13:03:40.962728+0000","last_active":"2024-03-21T13:06:01.275334+0000","last_peered":"2024-03-21T13:06:01.275334+0000","last_clean":"2024-03-21T13:06:01.275334+0000","last_became_active":"2024-03-21T13:02:26.233709+0000","last_became_peered":"2024-03-21T13:02:26.233709+0000","last_unstale":"2024-03-21T13:06:01.275334+0000","last_undegraded":"2024-03-21T13:06:01.275334+0000","last_fullsized":"2024-03-21T13:06:01.275334+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:40.962617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:03:40.962617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:03.988588+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":161,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274398+0000","last_change":"2024-03-21T13:05:49.260935+0000","last_active":"2024-03-21T13:06:01.274398+0000","last_peered":"2024-03-21T13:06:01.274398+0000","last_clean":"2024-03-21T13:06:01.274398+0000","last_became_active":"2024-03-21T13:02:26.237501+0000","last_became_peered":"2024-03-21T13:02:26.237501+0000","last_unstale":"2024-03-21T13:06:01.274398+0000","last_undegraded":"2024-03-21T13:06:01.274398+0000","last_fullsized":"2024-03-21T13:06:01.274398+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:49.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.260693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:01.973518+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"89'13","reported_seq":335,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.270937+0000","last_change":"2024-03-21T13:05:17.080137+0000","last_active":"2024-03-21T13:06:01.270937+0000","last_peered":"2024-03-21T13:06:01.270937+0000","last_clean":"2024-03-21T13:06:01.270937+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:06:01.270937+0000","last_undegraded":"2024-03-21T13:06:01.270937+0000","last_fullsized":"2024-03-21T13:06:01.270937+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-03-21T13:05:17.079993+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:05:17.079993+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:25.868022+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":266,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272174+0000","last_change":"2024-03-21T13:05:07.334438+0000","last_active":"2024-03-21T13:06:01.272174+0000","last_peered":"2024-03-21T13:06:01.272174+0000","last_clean":"2024-03-21T13:06:01.272174+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:06:01.272174+0000","last_undegraded":"2024-03-21T13:06:01.272174+0000","last_fullsized":"2024-03-21T13:06:01.272174+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.330296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.330296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:22.315386+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":205,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.271961+0000","last_change":"2024-03-21T13:05:50.991814+0000","last_active":"2024-03-21T13:06:01.271961+0000","last_peered":"2024-03-21T13:06:01.271961+0000","last_clean":"2024-03-21T13:06:01.271961+0000","last_became_active":"2024-03-21T13:02:14.330467+0000","last_became_peered":"2024-03-21T13:02:14.330467+0000","last_unstale":"2024-03-21T13:06:01.271961+0000","last_undegraded":"2024-03-21T13:06:01.271961+0000","last_fullsized":"2024-03-21T13:06:01.271961+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:50.991735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.991735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:06.675583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":278,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272499+0000","last_change":"2024-03-21T13:01:38.766164+0000","last_active":"2024-03-21T13:06:01.272499+0000","last_peered":"2024-03-21T13:06:01.272499+0000","last_clean":"2024-03-21T13:06:01.272499+0000","last_became_active":"2024-03-21T12:58:37.728383+0000","last_became_peered":"2024-03-21T12:58:37.728383+0000","last_unstale":"2024-03-21T13:06:01.272499+0000","last_undegraded":"2024-03-21T13:06:01.272499+0000","last_fullsized":"2024-03-21T13:06:01.272499+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_clean_scrub_stamp":"2024-03-21T13:01:38.765896+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"71'15","reported_seq":300,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275953+0000","last_change":"2024-03-21T13:05:26.476181+0000","last_active":"2024-03-21T13:06:01.275953+0000","last_peered":"2024-03-21T13:06:01.275953+0000","last_clean":"2024-03-21T13:06:01.275953+0000","last_became_active":"2024-03-21T12:58:39.750396+0000","last_became_peered":"2024-03-21T12:58:39.750396+0000","last_unstale":"2024-03-21T13:06:01.275953+0000","last_undegraded":"2024-03-21T13:06:01.275953+0000","last_fullsized":"2024-03-21T13:06:01.275953+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:26.476043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.476043+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:46.996849+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":165,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274937+0000","last_change":"2024-03-21T13:04:32.834762+0000","last_active":"2024-03-21T13:06:01.274937+0000","last_peered":"2024-03-21T13:06:01.274937+0000","last_clean":"2024-03-21T13:06:01.274937+0000","last_became_active":"2024-03-21T13:02:14.334221+0000","last_became_peered":"2024-03-21T13:02:14.334221+0000","last_unstale":"2024-03-21T13:06:01.274937+0000","last_undegraded":"2024-03-21T13:06:01.274937+0000","last_fullsized":"2024-03-21T13:06:01.274937+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:32.834646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:32.834646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:02.710078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":159,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.279397+0000","last_change":"2024-03-21T13:05:12.353129+0000","last_active":"2024-03-21T13:06:01.279397+0000","last_peered":"2024-03-21T13:06:01.279397+0000","last_clean":"2024-03-21T13:06:01.279397+0000","last_became_active":"2024-03-21T13:02:26.236707+0000","last_became_peered":"2024-03-21T13:02:26.236707+0000","last_unstale":"2024-03-21T13:06:01.279397+0000","last_undegraded":"2024-03-21T13:06:01.279397+0000","last_fullsized":"2024-03-21T13:06:01.279397+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:12.353011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:05:12.353011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:41.590667+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":151,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274098+0000","last_change":"2024-03-21T13:05:45.989199+0000","last_active":"2024-03-21T13:06:01.274098+0000","last_peered":"2024-03-21T13:06:01.274098+0000","last_clean":"2024-03-21T13:06:01.274098+0000","last_became_active":"2024-03-21T13:02:26.245402+0000","last_became_peered":"2024-03-21T13:02:26.245402+0000","last_unstale":"2024-03-21T13:06:01.274098+0000","last_undegraded":"2024-03-21T13:06:01.274098+0000","last_fullsized":"2024-03-21T13:06:01.274098+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:45.989114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:45.989114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:01.207660+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":268,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272848+0000","last_change":"2024-03-21T13:05:51.580203+0000","last_active":"2024-03-21T13:06:01.272848+0000","last_peered":"2024-03-21T13:06:01.272848+0000","last_clean":"2024-03-21T13:06:01.272848+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:06:01.272848+0000","last_undegraded":"2024-03-21T13:06:01.272848+0000","last_fullsized":"2024-03-21T13:06:01.272848+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-21T13:05:51.580074+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:05:51.580074+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:09.467583+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":263,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274718+0000","last_change":"2024-03-21T13:02:26.229621+0000","last_active":"2024-03-21T13:06:01.274718+0000","last_peered":"2024-03-21T13:06:01.274718+0000","last_clean":"2024-03-21T13:06:01.274718+0000","last_became_active":"2024-03-21T13:02:26.229397+0000","last_became_peered":"2024-03-21T13:02:26.229397+0000","last_unstale":"2024-03-21T13:06:01.274718+0000","last_undegraded":"2024-03-21T13:06:01.274718+0000","last_fullsized":"2024-03-21T13:06:01.274718+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:32.971665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:32.971665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275272+0000","last_change":"2024-03-21T13:04:30.831274+0000","last_active":"2024-03-21T13:06:01.275272+0000","last_peered":"2024-03-21T13:06:01.275272+0000","last_clean":"2024-03-21T13:06:01.275272+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:06:01.275272+0000","last_undegraded":"2024-03-21T13:06:01.275272+0000","last_fullsized":"2024-03-21T13:06:01.275272+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:30.831143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:04:30.831143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:03.988727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":291,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.271311+0000","last_change":"2024-03-21T13:05:41.337232+0000","last_active":"2024-03-21T13:06:01.271311+0000","last_peered":"2024-03-21T13:06:01.271311+0000","last_clean":"2024-03-21T13:06:01.271311+0000","last_became_active":"2024-03-21T13:02:26.230057+0000","last_became_peered":"2024-03-21T13:02:26.230057+0000","last_unstale":"2024-03-21T13:06:01.271311+0000","last_undegraded":"2024-03-21T13:06:01.271311+0000","last_fullsized":"2024-03-21T13:06:01.271311+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.337144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.337144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:52.877759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":270,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.271916+0000","last_change":"2024-03-21T13:05:16.384862+0000","last_active":"2024-03-21T13:06:01.271916+0000","last_peered":"2024-03-21T13:06:01.271916+0000","last_clean":"2024-03-21T13:06:01.271916+0000","last_became_active":"2024-03-21T13:02:14.349072+0000","last_became_peered":"2024-03-21T13:02:14.349072+0000","last_unstale":"2024-03-21T13:06:01.271916+0000","last_undegraded":"2024-03-21T13:06:01.271916+0000","last_fullsized":"2024-03-21T13:06:01.271916+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T13:05:16.384676+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:05:16.384676+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:29.918580+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":281,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.269465+0000","last_change":"2024-03-21T13:04:30.145245+0000","last_active":"2024-03-21T13:06:01.269465+0000","last_peered":"2024-03-21T13:06:01.269465+0000","last_clean":"2024-03-21T13:06:01.269465+0000","last_became_active":"2024-03-21T13:02:26.226675+0000","last_became_peered":"2024-03-21T13:02:26.226675+0000","last_unstale":"2024-03-21T13:06:01.269465+0000","last_undegraded":"2024-03-21T13:06:01.269465+0000","last_fullsized":"2024-03-21T13:06:01.269465+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:30.145090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:30.145090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":177,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.271171+0000","last_change":"2024-03-21T13:05:37.359133+0000","last_active":"2024-03-21T13:06:01.271171+0000","last_peered":"2024-03-21T13:06:01.271171+0000","last_clean":"2024-03-21T13:06:01.271171+0000","last_became_active":"2024-03-21T13:02:26.231879+0000","last_became_peered":"2024-03-21T13:02:26.231879+0000","last_unstale":"2024-03-21T13:06:01.271171+0000","last_undegraded":"2024-03-21T13:06:01.271171+0000","last_fullsized":"2024-03-21T13:06:01.271171+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:37.358974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:05:37.358974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:56.298751+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":169,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275515+0000","last_change":"2024-03-21T13:01:48.210250+0000","last_active":"2024-03-21T13:06:01.275515+0000","last_peered":"2024-03-21T13:06:01.275515+0000","last_clean":"2024-03-21T13:06:01.275515+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:06:01.275515+0000","last_undegraded":"2024-03-21T13:06:01.275515+0000","last_fullsized":"2024-03-21T13:06:01.275515+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:44.005379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:44.005379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:03.988487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":330,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273527+0000","last_change":"2024-03-21T13:05:20.158435+0000","last_active":"2024-03-21T13:06:01.273527+0000","last_peered":"2024-03-21T13:06:01.273527+0000","last_clean":"2024-03-21T13:06:01.273527+0000","last_became_active":"2024-03-21T13:02:14.350563+0000","last_became_peered":"2024-03-21T13:02:14.350563+0000","last_unstale":"2024-03-21T13:06:01.273527+0000","last_undegraded":"2024-03-21T13:06:01.273527+0000","last_fullsized":"2024-03-21T13:06:01.273527+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:20.158184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.158184+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:44.893552+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":265,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273411+0000","last_change":"2024-03-21T13:02:26.243438+0000","last_active":"2024-03-21T13:06:01.273411+0000","last_peered":"2024-03-21T13:06:01.273411+0000","last_clean":"2024-03-21T13:06:01.273411+0000","last_became_active":"2024-03-21T13:02:26.242482+0000","last_became_peered":"2024-03-21T13:02:26.242482+0000","last_unstale":"2024-03-21T13:06:01.273411+0000","last_undegraded":"2024-03-21T13:06:01.273411+0000","last_fullsized":"2024-03-21T13:06:01.273411+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:35.036335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.036335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":191,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272784+0000","last_change":"2024-03-21T13:04:29.464933+0000","last_active":"2024-03-21T13:06:01.272784+0000","last_peered":"2024-03-21T13:06:01.272784+0000","last_clean":"2024-03-21T13:06:01.272784+0000","last_became_active":"2024-03-21T13:01:27.366846+0000","last_became_peered":"2024-03-21T13:01:27.366846+0000","last_unstale":"2024-03-21T13:06:01.272784+0000","last_undegraded":"2024-03-21T13:06:01.272784+0000","last_fullsized":"2024-03-21T13:06:01.272784+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:29.464856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:04:29.464856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":301,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.270294+0000","last_change":"2024-03-21T13:05:39.354730+0000","last_active":"2024-03-21T13:06:01.270294+0000","last_peered":"2024-03-21T13:06:01.270294+0000","last_clean":"2024-03-21T13:06:01.270294+0000","last_became_active":"2024-03-21T13:02:14.329156+0000","last_became_peered":"2024-03-21T13:02:14.329156+0000","last_unstale":"2024-03-21T13:06:01.270294+0000","last_undegraded":"2024-03-21T13:06:01.270294+0000","last_fullsized":"2024-03-21T13:06:01.270294+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.354575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.354575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:04.801630+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'12","reported_seq":325,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272782+0000","last_change":"2024-03-21T13:02:14.362313+0000","last_active":"2024-03-21T13:06:01.272782+0000","last_peered":"2024-03-21T13:06:01.272782+0000","last_clean":"2024-03-21T13:06:01.272782+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:06:01.272782+0000","last_undegraded":"2024-03-21T13:06:01.272782+0000","last_fullsized":"2024-03-21T13:06:01.272782+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:01:39.755345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.755345+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":106,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273468+0000","last_change":"2024-03-21T13:05:50.295455+0000","last_active":"2024-03-21T13:06:01.273468+0000","last_peered":"2024-03-21T13:06:01.273468+0000","last_clean":"2024-03-21T13:06:01.273468+0000","last_became_active":"2024-03-21T13:04:20.359494+0000","last_became_peered":"2024-03-21T13:04:20.359494+0000","last_unstale":"2024-03-21T13:06:01.273468+0000","last_undegraded":"2024-03-21T13:06:01.273468+0000","last_fullsized":"2024-03-21T13:06:01.273468+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:50.295057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.295057+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:18.189590+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":380,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273042+0000","last_change":"2024-03-21T13:05:38.022572+0000","last_active":"2024-03-21T13:06:01.273042+0000","last_peered":"2024-03-21T13:06:01.273042+0000","last_clean":"2024-03-21T13:06:01.273042+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:06:01.273042+0000","last_undegraded":"2024-03-21T13:06:01.273042+0000","last_fullsized":"2024-03-21T13:06:01.273042+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:05:38.022439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.022439+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:06.821939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":177,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.276077+0000","last_change":"2024-03-21T13:04:20.361810+0000","last_active":"2024-03-21T13:06:01.276077+0000","last_peered":"2024-03-21T13:06:01.276077+0000","last_clean":"2024-03-21T13:06:01.276077+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:06:01.276077+0000","last_undegraded":"2024-03-21T13:06:01.276077+0000","last_fullsized":"2024-03-21T13:06:01.276077+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:01:37.244557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:37.244557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"71'10","reported_seq":212,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.276555+0000","last_change":"2024-03-21T13:05:21.117502+0000","last_active":"2024-03-21T13:06:01.276555+0000","last_peered":"2024-03-21T13:06:01.276555+0000","last_clean":"2024-03-21T13:06:01.276555+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:06:01.276555+0000","last_undegraded":"2024-03-21T13:06:01.276555+0000","last_fullsized":"2024-03-21T13:06:01.276555+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:21.117370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.117370+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:47.662070+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":153,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.270495+0000","last_change":"2024-03-21T13:04:29.180370+0000","last_active":"2024-03-21T13:06:01.270495+0000","last_peered":"2024-03-21T13:06:01.270495+0000","last_clean":"2024-03-21T13:06:01.270495+0000","last_became_active":"2024-03-21T13:02:26.229188+0000","last_became_peered":"2024-03-21T13:02:26.229188+0000","last_unstale":"2024-03-21T13:06:01.270495+0000","last_undegraded":"2024-03-21T13:06:01.270495+0000","last_fullsized":"2024-03-21T13:06:01.270495+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:29.180240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:29.180240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":178,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.276402+0000","last_change":"2024-03-21T13:05:18.088915+0000","last_active":"2024-03-21T13:06:01.276402+0000","last_peered":"2024-03-21T13:06:01.276402+0000","last_clean":"2024-03-21T13:06:01.276402+0000","last_became_active":"2024-03-21T13:01:48.215160+0000","last_became_peered":"2024-03-21T13:01:48.215160+0000","last_unstale":"2024-03-21T13:06:01.276402+0000","last_undegraded":"2024-03-21T13:06:01.276402+0000","last_fullsized":"2024-03-21T13:06:01.276402+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.088812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.088812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:30.428137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":285,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275009+0000","last_change":"2024-03-21T13:05:09.349604+0000","last_active":"2024-03-21T13:06:01.275009+0000","last_peered":"2024-03-21T13:06:01.275009+0000","last_clean":"2024-03-21T13:06:01.275009+0000","last_became_active":"2024-03-21T13:02:14.341286+0000","last_became_peered":"2024-03-21T13:02:14.341286+0000","last_unstale":"2024-03-21T13:06:01.275009+0000","last_undegraded":"2024-03-21T13:06:01.275009+0000","last_fullsized":"2024-03-21T13:06:01.275009+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:09.348646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:09.348646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:20.112225+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'11","reported_seq":338,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273093+0000","last_change":"2024-03-21T13:05:49.982963+0000","last_active":"2024-03-21T13:06:01.273093+0000","last_peered":"2024-03-21T13:06:01.273093+0000","last_clean":"2024-03-21T13:06:01.273093+0000","last_became_active":"2024-03-21T13:02:26.254436+0000","last_became_peered":"2024-03-21T13:02:26.254436+0000","last_unstale":"2024-03-21T13:06:01.273093+0000","last_undegraded":"2024-03-21T13:06:01.273093+0000","last_fullsized":"2024-03-21T13:06:01.273093+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:49.982812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.982812+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:14.345457+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":265,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275336+0000","last_change":"2024-03-21T13:01:35.005181+0000","last_active":"2024-03-21T13:06:01.275336+0000","last_peered":"2024-03-21T13:06:01.275336+0000","last_clean":"2024-03-21T13:06:01.275336+0000","last_became_active":"2024-03-21T12:58:43.183475+0000","last_became_peered":"2024-03-21T12:58:43.183475+0000","last_unstale":"2024-03-21T13:06:01.275336+0000","last_undegraded":"2024-03-21T13:06:01.275336+0000","last_fullsized":"2024-03-21T13:06:01.275336+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:01:35.005026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:03.988458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273369+0000","last_change":"2024-03-21T13:05:42.990138+0000","last_active":"2024-03-21T13:06:01.273369+0000","last_peered":"2024-03-21T13:06:01.273369+0000","last_clean":"2024-03-21T13:06:01.273369+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:06:01.273369+0000","last_undegraded":"2024-03-21T13:06:01.273369+0000","last_fullsized":"2024-03-21T13:06:01.273369+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:42.990022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.990022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:49.038464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":153,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273500+0000","last_change":"2024-03-21T13:05:28.499322+0000","last_active":"2024-03-21T13:06:01.273500+0000","last_peered":"2024-03-21T13:06:01.273500+0000","last_clean":"2024-03-21T13:06:01.273500+0000","last_became_active":"2024-03-21T13:02:26.242973+0000","last_became_peered":"2024-03-21T13:02:26.242973+0000","last_unstale":"2024-03-21T13:06:01.273500+0000","last_undegraded":"2024-03-21T13:06:01.273500+0000","last_fullsized":"2024-03-21T13:06:01.273500+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.499185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.499185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:37.620969+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":285,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275109+0000","last_change":"2024-03-21T13:02:14.369668+0000","last_active":"2024-03-21T13:06:01.275109+0000","last_peered":"2024-03-21T13:06:01.275109+0000","last_clean":"2024-03-21T13:06:01.275109+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:06:01.275109+0000","last_undegraded":"2024-03-21T13:06:01.275109+0000","last_fullsized":"2024-03-21T13:06:01.275109+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-21T13:01:30.807979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:30.807979+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":267,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272345+0000","last_change":"2024-03-21T13:05:19.431337+0000","last_active":"2024-03-21T13:06:01.272345+0000","last_peered":"2024-03-21T13:06:01.272345+0000","last_clean":"2024-03-21T13:06:01.272345+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:06:01.272345+0000","last_undegraded":"2024-03-21T13:06:01.272345+0000","last_fullsized":"2024-03-21T13:06:01.272345+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:19.430947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:19.430947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:23.467977+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":269,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275441+0000","last_change":"2024-03-21T13:03:43.905627+0000","last_active":"2024-03-21T13:06:01.275441+0000","last_peered":"2024-03-21T13:06:01.275441+0000","last_clean":"2024-03-21T13:06:01.275441+0000","last_became_active":"2024-03-21T12:58:41.164784+0000","last_became_peered":"2024-03-21T12:58:41.164784+0000","last_unstale":"2024-03-21T13:06:01.275441+0000","last_undegraded":"2024-03-21T13:06:01.275441+0000","last_fullsized":"2024-03-21T13:06:01.275441+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:43.905509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:03:43.905509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:03.988614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":267,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.276218+0000","last_change":"2024-03-21T13:02:26.236549+0000","last_active":"2024-03-21T13:06:01.276218+0000","last_peered":"2024-03-21T13:06:01.276218+0000","last_clean":"2024-03-21T13:06:01.276218+0000","last_became_active":"2024-03-21T13:02:26.236235+0000","last_became_peered":"2024-03-21T13:02:26.236235+0000","last_unstale":"2024-03-21T13:06:01.276218+0000","last_undegraded":"2024-03-21T13:06:01.276218+0000","last_fullsized":"2024-03-21T13:06:01.276218+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:02:21.087308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:02:21.087308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:03.988537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"71'11","reported_seq":93,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274110+0000","last_change":"2024-03-21T13:05:40.262398+0000","last_active":"2024-03-21T13:06:01.274110+0000","last_peered":"2024-03-21T13:06:01.274110+0000","last_clean":"2024-03-21T13:06:01.274110+0000","last_became_active":"2024-03-21T13:04:20.357539+0000","last_became_peered":"2024-03-21T13:04:20.357539+0000","last_unstale":"2024-03-21T13:06:01.274110+0000","last_undegraded":"2024-03-21T13:06:01.274110+0000","last_fullsized":"2024-03-21T13:06:01.274110+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:40.262277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:40.262277+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:05.063873+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":174,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.271843+0000","last_change":"2024-03-21T13:05:16.850442+0000","last_active":"2024-03-21T13:06:01.271843+0000","last_peered":"2024-03-21T13:06:01.271843+0000","last_clean":"2024-03-21T13:06:01.271843+0000","last_became_active":"2024-03-21T13:02:26.252488+0000","last_became_peered":"2024-03-21T13:02:26.252488+0000","last_unstale":"2024-03-21T13:06:01.271843+0000","last_undegraded":"2024-03-21T13:06:01.271843+0000","last_fullsized":"2024-03-21T13:06:01.271843+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:16.850307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:16.850307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:22.767349+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":283,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272097+0000","last_change":"2024-03-21T13:05:31.496412+0000","last_active":"2024-03-21T13:06:01.272097+0000","last_peered":"2024-03-21T13:06:01.272097+0000","last_clean":"2024-03-21T13:06:01.272097+0000","last_became_active":"2024-03-21T13:02:26.232682+0000","last_became_peered":"2024-03-21T13:02:26.232682+0000","last_unstale":"2024-03-21T13:06:01.272097+0000","last_undegraded":"2024-03-21T13:06:01.272097+0000","last_fullsized":"2024-03-21T13:06:01.272097+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.495639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.495639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:54.630189+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":277,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275940+0000","last_change":"2024-03-21T13:01:43.982673+0000","last_active":"2024-03-21T13:06:01.275940+0000","last_peered":"2024-03-21T13:06:01.275940+0000","last_clean":"2024-03-21T13:06:01.275940+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:06:01.275940+0000","last_undegraded":"2024-03-21T13:06:01.275940+0000","last_fullsized":"2024-03-21T13:06:01.275940+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:43.982553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:43.982553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:03.988512+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"86'13","reported_seq":242,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272465+0000","last_change":"2024-03-21T13:05:34.036587+0000","last_active":"2024-03-21T13:06:01.272465+0000","last_peered":"2024-03-21T13:06:01.272465+0000","last_clean":"2024-03-21T13:06:01.272465+0000","last_became_active":"2024-03-21T13:02:14.330196+0000","last_became_peered":"2024-03-21T13:02:14.330196+0000","last_unstale":"2024-03-21T13:06:01.272465+0000","last_undegraded":"2024-03-21T13:06:01.272465+0000","last_fullsized":"2024-03-21T13:06:01.272465+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:05:34.036430+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.036430+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:42.427384+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":112,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275011+0000","last_change":"2024-03-21T13:05:29.031235+0000","last_active":"2024-03-21T13:06:01.275011+0000","last_peered":"2024-03-21T13:06:01.275011+0000","last_clean":"2024-03-21T13:06:01.275011+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:06:01.275011+0000","last_undegraded":"2024-03-21T13:06:01.275011+0000","last_fullsized":"2024-03-21T13:06:01.275011+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.031062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.031062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:51.141442+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":145,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272939+0000","last_change":"2024-03-21T13:02:26.242867+0000","last_active":"2024-03-21T13:06:01.272939+0000","last_peered":"2024-03-21T13:06:01.272939+0000","last_clean":"2024-03-21T13:06:01.272939+0000","last_became_active":"2024-03-21T13:02:26.242491+0000","last_became_peered":"2024-03-21T13:02:26.242491+0000","last_unstale":"2024-03-21T13:06:01.272939+0000","last_undegraded":"2024-03-21T13:06:01.272939+0000","last_fullsized":"2024-03-21T13:06:01.272939+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.029629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.029629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":287,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.271358+0000","last_change":"2024-03-21T13:04:32.151296+0000","last_active":"2024-03-21T13:06:01.271358+0000","last_peered":"2024-03-21T13:06:01.271358+0000","last_clean":"2024-03-21T13:06:01.271358+0000","last_became_active":"2024-03-21T12:58:37.731659+0000","last_became_peered":"2024-03-21T12:58:37.731659+0000","last_unstale":"2024-03-21T13:06:01.271358+0000","last_undegraded":"2024-03-21T13:06:01.271358+0000","last_fullsized":"2024-03-21T13:06:01.271358+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:32.151217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:04:32.151217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":338,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273163+0000","last_change":"2024-03-21T13:05:28.260436+0000","last_active":"2024-03-21T13:06:01.273163+0000","last_peered":"2024-03-21T13:06:01.273163+0000","last_clean":"2024-03-21T13:06:01.273163+0000","last_became_active":"2024-03-21T13:02:26.225627+0000","last_became_peered":"2024-03-21T13:02:26.225627+0000","last_unstale":"2024-03-21T13:06:01.273163+0000","last_undegraded":"2024-03-21T13:06:01.273163+0000","last_fullsized":"2024-03-21T13:06:01.273163+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:28.260333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.260333+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:28.630975+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":177,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.276104+0000","last_change":"2024-03-21T13:04:31.805808+0000","last_active":"2024-03-21T13:06:01.276104+0000","last_peered":"2024-03-21T13:06:01.276104+0000","last_clean":"2024-03-21T13:06:01.276104+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:06:01.276104+0000","last_undegraded":"2024-03-21T13:06:01.276104+0000","last_fullsized":"2024-03-21T13:06:01.276104+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:31.805690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:31.805690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"125'10","reported_seq":370,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272476+0000","last_change":"2024-03-21T13:02:26.247826+0000","last_active":"2024-03-21T13:06:01.272476+0000","last_peered":"2024-03-21T13:06:01.272476+0000","last_clean":"2024-03-21T13:06:01.272476+0000","last_became_active":"2024-03-21T13:02:26.234494+0000","last_became_peered":"2024-03-21T13:02:26.234494+0000","last_unstale":"2024-03-21T13:06:01.272476+0000","last_undegraded":"2024-03-21T13:06:01.272476+0000","last_fullsized":"2024-03-21T13:06:01.272476+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:23.867564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:23.867564+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:07.522751+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":285,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275392+0000","last_change":"2024-03-21T13:05:31.042470+0000","last_active":"2024-03-21T13:06:01.275392+0000","last_peered":"2024-03-21T13:06:01.275392+0000","last_clean":"2024-03-21T13:06:01.275392+0000","last_became_active":"2024-03-21T12:58:37.731988+0000","last_became_peered":"2024-03-21T12:58:37.731988+0000","last_unstale":"2024-03-21T13:06:01.275392+0000","last_undegraded":"2024-03-21T13:06:01.275392+0000","last_fullsized":"2024-03-21T13:06:01.275392+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.042372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.042372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:38.456469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"71'10","reported_seq":308,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273073+0000","last_change":"2024-03-21T13:05:23.379150+0000","last_active":"2024-03-21T13:06:01.273073+0000","last_peered":"2024-03-21T13:06:01.273073+0000","last_clean":"2024-03-21T13:06:01.273073+0000","last_became_active":"2024-03-21T13:02:14.353938+0000","last_became_peered":"2024-03-21T13:02:14.353938+0000","last_unstale":"2024-03-21T13:06:01.273073+0000","last_undegraded":"2024-03-21T13:06:01.273073+0000","last_fullsized":"2024-03-21T13:06:01.273073+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:23.379004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.379004+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:45.392478+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":281,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.270646+0000","last_change":"2024-03-21T13:05:21.157416+0000","last_active":"2024-03-21T13:06:01.270646+0000","last_peered":"2024-03-21T13:06:01.270646+0000","last_clean":"2024-03-21T13:06:01.270646+0000","last_became_active":"2024-03-21T13:02:26.233049+0000","last_became_peered":"2024-03-21T13:02:26.233049+0000","last_unstale":"2024-03-21T13:06:01.270646+0000","last_undegraded":"2024-03-21T13:06:01.270646+0000","last_fullsized":"2024-03-21T13:06:01.270646+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:21.157295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.157295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:33.608380+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":361,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273204+0000","last_change":"2024-03-21T13:01:18.526684+0000","last_active":"2024-03-21T13:06:01.273204+0000","last_peered":"2024-03-21T13:06:01.273204+0000","last_clean":"2024-03-21T13:06:01.273204+0000","last_became_active":"2024-03-21T12:58:41.167826+0000","last_became_peered":"2024-03-21T12:58:41.167826+0000","last_unstale":"2024-03-21T13:06:01.273204+0000","last_undegraded":"2024-03-21T13:06:01.273204+0000","last_fullsized":"2024-03-21T13:06:01.273204+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:18.526428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.526428+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:07.522362+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":177,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275160+0000","last_change":"2024-03-21T13:04:12.819697+0000","last_active":"2024-03-21T13:06:01.275160+0000","last_peered":"2024-03-21T13:06:01.275160+0000","last_clean":"2024-03-21T13:06:01.275160+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:06:01.275160+0000","last_undegraded":"2024-03-21T13:06:01.275160+0000","last_fullsized":"2024-03-21T13:06:01.275160+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:12.819556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:04:12.819556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:03.988782+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"71'9","reported_seq":196,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.276277+0000","last_change":"2024-03-21T13:04:13.827838+0000","last_active":"2024-03-21T13:06:01.276277+0000","last_peered":"2024-03-21T13:06:01.276277+0000","last_clean":"2024-03-21T13:06:01.276277+0000","last_became_active":"2024-03-21T13:02:26.239161+0000","last_became_peered":"2024-03-21T13:02:26.239161+0000","last_unstale":"2024-03-21T13:06:01.276277+0000","last_undegraded":"2024-03-21T13:06:01.276277+0000","last_fullsized":"2024-03-21T13:06:01.276277+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:04:13.827615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:13.827615+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:03.988702+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":284,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273953+0000","last_change":"2024-03-21T13:05:42.187570+0000","last_active":"2024-03-21T13:06:01.273953+0000","last_peered":"2024-03-21T13:06:01.273953+0000","last_clean":"2024-03-21T13:06:01.273953+0000","last_became_active":"2024-03-21T12:58:43.182126+0000","last_became_peered":"2024-03-21T12:58:43.182126+0000","last_unstale":"2024-03-21T13:06:01.273953+0000","last_undegraded":"2024-03-21T13:06:01.273953+0000","last_fullsized":"2024-03-21T13:06:01.273953+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:42.187461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.187461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:07.563556+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":277,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273718+0000","last_change":"2024-03-21T13:05:17.347764+0000","last_active":"2024-03-21T13:06:01.273718+0000","last_peered":"2024-03-21T13:06:01.273718+0000","last_clean":"2024-03-21T13:06:01.273718+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:06:01.273718+0000","last_undegraded":"2024-03-21T13:06:01.273718+0000","last_fullsized":"2024-03-21T13:06:01.273718+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:17.347604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:17.347604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:21.115130+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":291,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275207+0000","last_change":"2024-03-21T13:05:23.049951+0000","last_active":"2024-03-21T13:06:01.275207+0000","last_peered":"2024-03-21T13:06:01.275207+0000","last_clean":"2024-03-21T13:06:01.275207+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:06:01.275207+0000","last_undegraded":"2024-03-21T13:06:01.275207+0000","last_fullsized":"2024-03-21T13:06:01.275207+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:05:23.049811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.049811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:26.798813+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'9","reported_seq":204,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272074+0000","last_change":"2024-03-21T13:01:48.210983+0000","last_active":"2024-03-21T13:06:01.272074+0000","last_peered":"2024-03-21T13:06:01.272074+0000","last_clean":"2024-03-21T13:06:01.272074+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:06:01.272074+0000","last_undegraded":"2024-03-21T13:06:01.272074+0000","last_fullsized":"2024-03-21T13:06:01.272074+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:01:40.964615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.964615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":265,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275803+0000","last_change":"2024-03-21T13:03:41.952266+0000","last_active":"2024-03-21T13:06:01.275803+0000","last_peered":"2024-03-21T13:06:01.275803+0000","last_clean":"2024-03-21T13:06:01.275803+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:06:01.275803+0000","last_undegraded":"2024-03-21T13:06:01.275803+0000","last_fullsized":"2024-03-21T13:06:01.275803+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:41.952189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:03:41.952189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:03.988563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":267,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272293+0000","last_change":"2024-03-21T13:05:25.466812+0000","last_active":"2024-03-21T13:06:01.272293+0000","last_peered":"2024-03-21T13:06:01.272293+0000","last_clean":"2024-03-21T13:06:01.272293+0000","last_became_active":"2024-03-21T13:02:26.242502+0000","last_became_peered":"2024-03-21T13:02:26.242502+0000","last_unstale":"2024-03-21T13:06:01.272293+0000","last_undegraded":"2024-03-21T13:06:01.272293+0000","last_fullsized":"2024-03-21T13:06:01.272293+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:25.466695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:25.466695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:39.968442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":293,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272039+0000","last_change":"2024-03-21T13:05:41.004652+0000","last_active":"2024-03-21T13:06:01.272039+0000","last_peered":"2024-03-21T13:06:01.272039+0000","last_clean":"2024-03-21T13:06:01.272039+0000","last_became_active":"2024-03-21T13:02:26.252406+0000","last_became_peered":"2024-03-21T13:02:26.252406+0000","last_unstale":"2024-03-21T13:06:01.272039+0000","last_undegraded":"2024-03-21T13:06:01.272039+0000","last_fullsized":"2024-03-21T13:06:01.272039+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.004412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.004412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:02.118773+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'6","reported_seq":298,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273632+0000","last_change":"2024-03-21T13:04:22.563945+0000","last_active":"2024-03-21T13:06:01.273632+0000","last_peered":"2024-03-21T13:06:01.273632+0000","last_clean":"2024-03-21T13:06:01.273632+0000","last_became_active":"2024-03-21T13:02:14.352117+0000","last_became_peered":"2024-03-21T13:02:14.352117+0000","last_unstale":"2024-03-21T13:06:01.273632+0000","last_undegraded":"2024-03-21T13:06:01.273632+0000","last_fullsized":"2024-03-21T13:06:01.273632+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T13:04:22.563861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:22.563861+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"189'6695","reported_seq":6746,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:04.731301+0000","last_change":"2024-03-21T13:04:46.254330+0000","last_active":"2024-03-21T13:06:04.731301+0000","last_peered":"2024-03-21T13:06:04.731301+0000","last_clean":"2024-03-21T13:06:04.731301+0000","last_became_active":"2024-03-21T13:04:46.252954+0000","last_became_peered":"2024-03-21T13:04:46.252954+0000","last_unstale":"2024-03-21T13:06:04.731301+0000","last_undegraded":"2024-03-21T13:06:04.731301+0000","last_fullsized":"2024-03-21T13:06:04.731301+0000","mapping_epoch":168,"log_start":"178'2700","ondisk_log_start":"178'2700","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438763520,"num_objects":6695,"num_object_clones":0,"num_object_copies":20085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6695,"num_write_kb":428480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":151,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272156+0000","last_change":"2024-03-21T13:05:47.973945+0000","last_active":"2024-03-21T13:06:01.272156+0000","last_peered":"2024-03-21T13:06:01.272156+0000","last_clean":"2024-03-21T13:06:01.272156+0000","last_became_active":"2024-03-21T13:02:26.252904+0000","last_became_peered":"2024-03-21T13:02:26.252904+0000","last_unstale":"2024-03-21T13:06:01.272156+0000","last_undegraded":"2024-03-21T13:06:01.272156+0000","last_fullsized":"2024-03-21T13:06:01.272156+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.973796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.973796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:13.525530+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":309,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.276219+0000","last_change":"2024-03-21T13:05:44.816505+0000","last_active":"2024-03-21T13:06:01.276219+0000","last_peered":"2024-03-21T13:06:01.276219+0000","last_clean":"2024-03-21T13:06:01.276219+0000","last_became_active":"2024-03-21T13:04:20.359078+0000","last_became_peered":"2024-03-21T13:04:20.359078+0000","last_unstale":"2024-03-21T13:06:01.276219+0000","last_undegraded":"2024-03-21T13:06:01.276219+0000","last_fullsized":"2024-03-21T13:06:01.276219+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:05:44.816353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:44.816353+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:45.988130+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'15","reported_seq":211,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275336+0000","last_change":"2024-03-21T13:04:25.787896+0000","last_active":"2024-03-21T13:06:01.275336+0000","last_peered":"2024-03-21T13:06:01.275336+0000","last_clean":"2024-03-21T13:06:01.275336+0000","last_became_active":"2024-03-21T13:02:26.237800+0000","last_became_peered":"2024-03-21T13:02:26.237800+0000","last_unstale":"2024-03-21T13:06:01.275336+0000","last_undegraded":"2024-03-21T13:06:01.275336+0000","last_fullsized":"2024-03-21T13:06:01.275336+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:04:25.787703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:25.787703+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:03.988757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":368,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275449+0000","last_change":"2024-03-21T13:04:16.810359+0000","last_active":"2024-03-21T13:06:01.275449+0000","last_peered":"2024-03-21T13:06:01.275449+0000","last_clean":"2024-03-21T13:06:01.275449+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:06:01.275449+0000","last_undegraded":"2024-03-21T13:06:01.275449+0000","last_fullsized":"2024-03-21T13:06:01.275449+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:04:16.810269+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:04:16.810269+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:03.988811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"189'6602","reported_seq":6654,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:04.725836+0000","last_change":"2024-03-21T13:04:46.259252+0000","last_active":"2024-03-21T13:06:04.725836+0000","last_peered":"2024-03-21T13:06:04.725836+0000","last_clean":"2024-03-21T13:06:04.725836+0000","last_became_active":"2024-03-21T13:04:46.258930+0000","last_became_peered":"2024-03-21T13:04:46.258930+0000","last_unstale":"2024-03-21T13:06:04.725836+0000","last_undegraded":"2024-03-21T13:06:04.725836+0000","last_fullsized":"2024-03-21T13:06:04.725836+0000","mapping_epoch":168,"log_start":"177'2400","ondisk_log_start":"177'2400","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4202,"log_dups_size":0,"ondisk_log_size":4202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:06.060765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432668672,"num_objects":6602,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6602,"num_write_kb":422528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":273,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272259+0000","last_change":"2024-03-21T13:05:43.596914+0000","last_active":"2024-03-21T13:06:01.272259+0000","last_peered":"2024-03-21T13:06:01.272259+0000","last_clean":"2024-03-21T13:06:01.272259+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:06:01.272259+0000","last_undegraded":"2024-03-21T13:06:01.272259+0000","last_fullsized":"2024-03-21T13:06:01.272259+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.596691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:11.157166+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":291,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.276312+0000","last_change":"2024-03-21T13:05:09.138059+0000","last_active":"2024-03-21T13:06:01.276312+0000","last_peered":"2024-03-21T13:06:01.276312+0000","last_clean":"2024-03-21T13:06:01.276312+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:06:01.276312+0000","last_undegraded":"2024-03-21T13:06:01.276312+0000","last_fullsized":"2024-03-21T13:06:01.276312+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:09.137868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:09.137868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:30.933204+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"71'12","reported_seq":209,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275800+0000","last_change":"2024-03-21T13:02:14.383361+0000","last_active":"2024-03-21T13:06:01.275800+0000","last_peered":"2024-03-21T13:06:01.275800+0000","last_clean":"2024-03-21T13:06:01.275800+0000","last_became_active":"2024-03-21T13:02:14.347102+0000","last_became_peered":"2024-03-21T13:02:14.347102+0000","last_unstale":"2024-03-21T13:06:01.275800+0000","last_undegraded":"2024-03-21T13:06:01.275800+0000","last_fullsized":"2024-03-21T13:06:01.275800+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:00:32.864608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.864608+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":292,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273809+0000","last_change":"2024-03-21T13:05:21.171124+0000","last_active":"2024-03-21T13:06:01.273809+0000","last_peered":"2024-03-21T13:06:01.273809+0000","last_clean":"2024-03-21T13:06:01.273809+0000","last_became_active":"2024-03-21T13:02:26.235083+0000","last_became_peered":"2024-03-21T13:02:26.235083+0000","last_unstale":"2024-03-21T13:06:01.273809+0000","last_undegraded":"2024-03-21T13:06:01.273809+0000","last_fullsized":"2024-03-21T13:06:01.273809+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:21.170999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.170999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:38.625718+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"189'6707","reported_seq":6759,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:04.686965+0000","last_change":"2024-03-21T13:04:46.257959+0000","last_active":"2024-03-21T13:06:04.686965+0000","last_peered":"2024-03-21T13:06:04.686965+0000","last_clean":"2024-03-21T13:06:04.686965+0000","last_became_active":"2024-03-21T13:04:46.257488+0000","last_became_peered":"2024-03-21T13:04:46.257488+0000","last_unstale":"2024-03-21T13:06:04.686965+0000","last_undegraded":"2024-03-21T13:06:04.686965+0000","last_fullsized":"2024-03-21T13:06:04.686965+0000","mapping_epoch":168,"log_start":"177'2500","ondisk_log_start":"177'2500","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4207,"log_dups_size":0,"ondisk_log_size":4207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:06.060735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439549952,"num_objects":6707,"num_object_clones":0,"num_object_copies":20121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6707,"num_write_kb":429248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":273,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275000+0000","last_change":"2024-03-21T13:04:34.848696+0000","last_active":"2024-03-21T13:06:01.275000+0000","last_peered":"2024-03-21T13:06:01.275000+0000","last_clean":"2024-03-21T13:06:01.275000+0000","last_became_active":"2024-03-21T12:58:43.179007+0000","last_became_peered":"2024-03-21T12:58:43.179007+0000","last_unstale":"2024-03-21T13:06:01.275000+0000","last_undegraded":"2024-03-21T13:06:01.275000+0000","last_fullsized":"2024-03-21T13:06:01.275000+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:34.848616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:34.848616+0000","last_clean_scrub_stamp":"2024-03-21T13:04:34.848616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":286,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274423+0000","last_change":"2024-03-21T13:05:28.216804+0000","last_active":"2024-03-21T13:06:01.274423+0000","last_peered":"2024-03-21T13:06:01.274423+0000","last_clean":"2024-03-21T13:06:01.274423+0000","last_became_active":"2024-03-21T13:02:26.238116+0000","last_became_peered":"2024-03-21T13:02:26.238116+0000","last_unstale":"2024-03-21T13:06:01.274423+0000","last_undegraded":"2024-03-21T13:06:01.274423+0000","last_fullsized":"2024-03-21T13:06:01.274423+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.216547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.216547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:43.002064+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":291,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.276188+0000","last_change":"2024-03-21T13:05:22.078652+0000","last_active":"2024-03-21T13:06:01.276188+0000","last_peered":"2024-03-21T13:06:01.276188+0000","last_clean":"2024-03-21T13:06:01.276188+0000","last_became_active":"2024-03-21T13:02:26.236780+0000","last_became_peered":"2024-03-21T13:02:26.236780+0000","last_unstale":"2024-03-21T13:06:01.276188+0000","last_undegraded":"2024-03-21T13:06:01.276188+0000","last_fullsized":"2024-03-21T13:06:01.276188+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:22.078546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.078546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:48.094404+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"71'19","reported_seq":366,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272257+0000","last_change":"2024-03-21T13:05:29.992376+0000","last_active":"2024-03-21T13:06:01.272257+0000","last_peered":"2024-03-21T13:06:01.272257+0000","last_clean":"2024-03-21T13:06:01.272257+0000","last_became_active":"2024-03-21T13:02:26.240350+0000","last_became_peered":"2024-03-21T13:02:26.240350+0000","last_unstale":"2024-03-21T13:06:01.272257+0000","last_undegraded":"2024-03-21T13:06:01.272257+0000","last_fullsized":"2024-03-21T13:06:01.272257+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:05:29.992207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.992207+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:55.257135+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"189'6554","reported_seq":6605,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:05.279801+0000","last_change":"2024-03-21T13:04:46.263341+0000","last_active":"2024-03-21T13:06:05.279801+0000","last_peered":"2024-03-21T13:06:05.279801+0000","last_clean":"2024-03-21T13:06:05.279801+0000","last_became_active":"2024-03-21T13:04:46.259554+0000","last_became_peered":"2024-03-21T13:04:46.259554+0000","last_unstale":"2024-03-21T13:06:05.279801+0000","last_undegraded":"2024-03-21T13:06:05.279801+0000","last_fullsized":"2024-03-21T13:06:05.279801+0000","mapping_epoch":168,"log_start":"178'2900","ondisk_log_start":"178'2900","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3654,"log_dups_size":0,"ondisk_log_size":3654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:07.526359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429522944,"num_objects":6554,"num_object_clones":0,"num_object_copies":19662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6554,"num_write_kb":419456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":177,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275088+0000","last_change":"2024-03-21T13:05:39.926952+0000","last_active":"2024-03-21T13:06:01.275088+0000","last_peered":"2024-03-21T13:06:01.275088+0000","last_clean":"2024-03-21T13:06:01.275088+0000","last_became_active":"2024-03-21T13:01:48.213995+0000","last_became_peered":"2024-03-21T13:01:48.213995+0000","last_unstale":"2024-03-21T13:06:01.275088+0000","last_undegraded":"2024-03-21T13:06:01.275088+0000","last_fullsized":"2024-03-21T13:06:01.275088+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.926828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.926828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:57.400154+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273607+0000","last_change":"2024-03-21T13:05:37.225792+0000","last_active":"2024-03-21T13:06:01.273607+0000","last_peered":"2024-03-21T13:06:01.273607+0000","last_clean":"2024-03-21T13:06:01.273607+0000","last_became_active":"2024-03-21T12:58:41.163843+0000","last_became_peered":"2024-03-21T12:58:41.163843+0000","last_unstale":"2024-03-21T13:06:01.273607+0000","last_undegraded":"2024-03-21T13:06:01.273607+0000","last_fullsized":"2024-03-21T13:06:01.273607+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:37.225558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:37.225558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:44.100517+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":143,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273312+0000","last_change":"2024-03-21T13:02:26.244355+0000","last_active":"2024-03-21T13:06:01.273312+0000","last_peered":"2024-03-21T13:06:01.273312+0000","last_clean":"2024-03-21T13:06:01.273312+0000","last_became_active":"2024-03-21T13:02:26.243974+0000","last_became_peered":"2024-03-21T13:02:26.243974+0000","last_unstale":"2024-03-21T13:06:01.273312+0000","last_undegraded":"2024-03-21T13:06:01.273312+0000","last_fullsized":"2024-03-21T13:06:01.273312+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:33.998382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:33.998382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"86'13","reported_seq":241,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272820+0000","last_change":"2024-03-21T13:05:08.081782+0000","last_active":"2024-03-21T13:06:01.272820+0000","last_peered":"2024-03-21T13:06:01.272820+0000","last_clean":"2024-03-21T13:06:01.272820+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:06:01.272820+0000","last_undegraded":"2024-03-21T13:06:01.272820+0000","last_fullsized":"2024-03-21T13:06:01.272820+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:05:08.081600+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:05:08.081600+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:19.313431+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.8","version":"189'6723","reported_seq":6775,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:05.233661+0000","last_change":"2024-03-21T13:04:46.257489+0000","last_active":"2024-03-21T13:06:05.233661+0000","last_peered":"2024-03-21T13:06:05.233661+0000","last_clean":"2024-03-21T13:06:05.233661+0000","last_became_active":"2024-03-21T13:04:46.257160+0000","last_became_peered":"2024-03-21T13:04:46.257160+0000","last_unstale":"2024-03-21T13:06:05.233661+0000","last_undegraded":"2024-03-21T13:06:05.233661+0000","last_fullsized":"2024-03-21T13:06:05.233661+0000","mapping_epoch":168,"log_start":"179'3100","ondisk_log_start":"179'3100","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3623,"log_dups_size":0,"ondisk_log_size":3623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:14.786568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440598528,"num_objects":6723,"num_object_clones":0,"num_object_copies":20169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6723,"num_write_kb":430272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":259,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272374+0000","last_change":"2024-03-21T13:01:46.738822+0000","last_active":"2024-03-21T13:06:01.272374+0000","last_peered":"2024-03-21T13:06:01.272374+0000","last_clean":"2024-03-21T13:06:01.272374+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:06:01.272374+0000","last_undegraded":"2024-03-21T13:06:01.272374+0000","last_fullsized":"2024-03-21T13:06:01.272374+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.738659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":182,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272220+0000","last_change":"2024-03-21T13:05:43.961181+0000","last_active":"2024-03-21T13:06:01.272220+0000","last_peered":"2024-03-21T13:06:01.272220+0000","last_clean":"2024-03-21T13:06:01.272220+0000","last_became_active":"2024-03-21T13:02:26.243692+0000","last_became_peered":"2024-03-21T13:02:26.243692+0000","last_unstale":"2024-03-21T13:06:01.272220+0000","last_undegraded":"2024-03-21T13:06:01.272220+0000","last_fullsized":"2024-03-21T13:06:01.272220+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.961057+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:04.863341+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":295,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272948+0000","last_change":"2024-03-21T13:05:48.266178+0000","last_active":"2024-03-21T13:06:01.272948+0000","last_peered":"2024-03-21T13:06:01.272948+0000","last_clean":"2024-03-21T13:06:01.272948+0000","last_became_active":"2024-03-21T12:58:37.729454+0000","last_became_peered":"2024-03-21T12:58:37.729454+0000","last_unstale":"2024-03-21T13:06:01.272948+0000","last_undegraded":"2024-03-21T13:06:01.272948+0000","last_fullsized":"2024-03-21T13:06:01.272948+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:48.265982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:48.265982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:50.943082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"119'29","reported_seq":301,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273849+0000","last_change":"2024-03-21T13:05:22.202039+0000","last_active":"2024-03-21T13:06:01.273849+0000","last_peered":"2024-03-21T13:06:01.273849+0000","last_clean":"2024-03-21T13:06:01.273849+0000","last_became_active":"2024-03-21T13:02:14.351114+0000","last_became_peered":"2024-03-21T13:02:14.351114+0000","last_unstale":"2024-03-21T13:06:01.273849+0000","last_undegraded":"2024-03-21T13:06:01.273849+0000","last_fullsized":"2024-03-21T13:06:01.273849+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'29","last_scrub_stamp":"2024-03-21T13:05:22.201879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.201879+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:32.239279+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":173,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.270842+0000","last_change":"2024-03-21T13:05:34.312717+0000","last_active":"2024-03-21T13:06:01.270842+0000","last_peered":"2024-03-21T13:06:01.270842+0000","last_clean":"2024-03-21T13:06:01.270842+0000","last_became_active":"2024-03-21T13:02:14.336177+0000","last_became_peered":"2024-03-21T13:02:14.336177+0000","last_unstale":"2024-03-21T13:06:01.270842+0000","last_undegraded":"2024-03-21T13:06:01.270842+0000","last_fullsized":"2024-03-21T13:06:01.270842+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.312583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.312583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:40.632357+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.b","version":"189'6669","reported_seq":6721,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:05.284063+0000","last_change":"2024-03-21T13:04:46.259472+0000","last_active":"2024-03-21T13:06:05.284063+0000","last_peered":"2024-03-21T13:06:05.284063+0000","last_clean":"2024-03-21T13:06:05.284063+0000","last_became_active":"2024-03-21T13:04:46.259164+0000","last_became_peered":"2024-03-21T13:04:46.259164+0000","last_unstale":"2024-03-21T13:06:05.284063+0000","last_undegraded":"2024-03-21T13:06:05.284063+0000","last_fullsized":"2024-03-21T13:06:05.284063+0000","mapping_epoch":168,"log_start":"178'3000","ondisk_log_start":"178'3000","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:07.526245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437059584,"num_objects":6669,"num_object_clones":0,"num_object_copies":20007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6669,"num_write_kb":426816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":273,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.276373+0000","last_change":"2024-03-21T13:05:08.172825+0000","last_active":"2024-03-21T13:06:01.276373+0000","last_peered":"2024-03-21T13:06:01.276373+0000","last_clean":"2024-03-21T13:06:01.276373+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:06:01.276373+0000","last_undegraded":"2024-03-21T13:06:01.276373+0000","last_fullsized":"2024-03-21T13:06:01.276373+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:08.172705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:08.172705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:13.347754+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":171,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274023+0000","last_change":"2024-03-21T13:05:39.247223+0000","last_active":"2024-03-21T13:06:01.274023+0000","last_peered":"2024-03-21T13:06:01.274023+0000","last_clean":"2024-03-21T13:06:01.274023+0000","last_became_active":"2024-03-21T13:02:14.328020+0000","last_became_peered":"2024-03-21T13:02:14.328020+0000","last_unstale":"2024-03-21T13:06:01.274023+0000","last_undegraded":"2024-03-21T13:06:01.274023+0000","last_fullsized":"2024-03-21T13:06:01.274023+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.247107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.247107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:49.706890+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":305,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274278+0000","last_change":"2024-03-21T13:05:46.263820+0000","last_active":"2024-03-21T13:06:01.274278+0000","last_peered":"2024-03-21T13:06:01.274278+0000","last_clean":"2024-03-21T13:06:01.274278+0000","last_became_active":"2024-03-21T12:58:36.724067+0000","last_became_peered":"2024-03-21T12:58:36.724067+0000","last_unstale":"2024-03-21T13:06:01.274278+0000","last_undegraded":"2024-03-21T13:06:01.274278+0000","last_fullsized":"2024-03-21T13:06:01.274278+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:46.263644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:05:46.263644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:51.823127+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.000743678,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'13","reported_seq":211,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275939+0000","last_change":"2024-03-21T13:05:15.119776+0000","last_active":"2024-03-21T13:06:01.275939+0000","last_peered":"2024-03-21T13:06:01.275939+0000","last_clean":"2024-03-21T13:06:01.275939+0000","last_became_active":"2024-03-21T13:02:26.234230+0000","last_became_peered":"2024-03-21T13:02:26.234230+0000","last_unstale":"2024-03-21T13:06:01.275939+0000","last_undegraded":"2024-03-21T13:06:01.275939+0000","last_fullsized":"2024-03-21T13:06:01.275939+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'13","last_scrub_stamp":"2024-03-21T13:05:15.119643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:15.119643+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:20.975727+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":286,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274287+0000","last_change":"2024-03-21T13:05:23.243712+0000","last_active":"2024-03-21T13:06:01.274287+0000","last_peered":"2024-03-21T13:06:01.274287+0000","last_clean":"2024-03-21T13:06:01.274287+0000","last_became_active":"2024-03-21T13:02:26.228128+0000","last_became_peered":"2024-03-21T13:02:26.228128+0000","last_unstale":"2024-03-21T13:06:01.274287+0000","last_undegraded":"2024-03-21T13:06:01.274287+0000","last_fullsized":"2024-03-21T13:06:01.274287+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:23.243468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.243468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:45.611291+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"189'6814","reported_seq":6866,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:04.740759+0000","last_change":"2024-03-21T13:04:46.260725+0000","last_active":"2024-03-21T13:06:04.740759+0000","last_peered":"2024-03-21T13:06:04.740759+0000","last_clean":"2024-03-21T13:06:04.740759+0000","last_became_active":"2024-03-21T13:04:46.260364+0000","last_became_peered":"2024-03-21T13:04:46.260364+0000","last_unstale":"2024-03-21T13:06:04.740759+0000","last_undegraded":"2024-03-21T13:06:04.740759+0000","last_fullsized":"2024-03-21T13:06:04.740759+0000","mapping_epoch":168,"log_start":"177'2700","ondisk_log_start":"177'2700","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:07.138475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446562304,"num_objects":6814,"num_object_clones":0,"num_object_copies":20442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6814,"num_write_kb":436096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":151,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273671+0000","last_change":"2024-03-21T13:04:27.549179+0000","last_active":"2024-03-21T13:06:01.273671+0000","last_peered":"2024-03-21T13:06:01.273671+0000","last_clean":"2024-03-21T13:06:01.273671+0000","last_became_active":"2024-03-21T13:02:26.235640+0000","last_became_peered":"2024-03-21T13:02:26.235640+0000","last_unstale":"2024-03-21T13:06:01.273671+0000","last_undegraded":"2024-03-21T13:06:01.273671+0000","last_fullsized":"2024-03-21T13:06:01.273671+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:27.549086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:27.549086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":534,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275282+0000","last_change":"2024-03-21T13:04:11.818847+0000","last_active":"2024-03-21T13:06:01.275282+0000","last_peered":"2024-03-21T13:06:01.275282+0000","last_clean":"2024-03-21T13:06:01.275282+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:06:01.275282+0000","last_undegraded":"2024-03-21T13:06:01.275282+0000","last_fullsized":"2024-03-21T13:06:01.275282+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:04:11.818749+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:00:16.180126+0000","last_clean_scrub_stamp":"2024-03-21T13:04:11.818749+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:03.988672+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":277,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272715+0000","last_change":"2024-03-21T13:02:26.242969+0000","last_active":"2024-03-21T13:06:01.272715+0000","last_peered":"2024-03-21T13:06:01.272715+0000","last_clean":"2024-03-21T13:06:01.272715+0000","last_became_active":"2024-03-21T13:02:26.242456+0000","last_became_peered":"2024-03-21T13:02:26.242456+0000","last_unstale":"2024-03-21T13:06:01.272715+0000","last_undegraded":"2024-03-21T13:06:01.272715+0000","last_fullsized":"2024-03-21T13:06:01.272715+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:38.007486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.007486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"183'24","reported_seq":403,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274169+0000","last_change":"2024-03-21T13:05:42.033620+0000","last_active":"2024-03-21T13:06:01.274169+0000","last_peered":"2024-03-21T13:06:01.274169+0000","last_clean":"2024-03-21T13:06:01.274169+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:06:01.274169+0000","last_undegraded":"2024-03-21T13:06:01.274169+0000","last_fullsized":"2024-03-21T13:06:01.274169+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'24","last_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.033266+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:08.844091+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":283,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275703+0000","last_change":"2024-03-21T13:05:20.061939+0000","last_active":"2024-03-21T13:06:01.275703+0000","last_peered":"2024-03-21T13:06:01.275703+0000","last_clean":"2024-03-21T13:06:01.275703+0000","last_became_active":"2024-03-21T13:02:26.235140+0000","last_became_peered":"2024-03-21T13:02:26.235140+0000","last_unstale":"2024-03-21T13:06:01.275703+0000","last_undegraded":"2024-03-21T13:06:01.275703+0000","last_fullsized":"2024-03-21T13:06:01.275703+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:20.061709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.061709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:27.413347+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"189'6630","reported_seq":6682,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:04.907151+0000","last_change":"2024-03-21T13:04:46.262137+0000","last_active":"2024-03-21T13:06:04.907151+0000","last_peered":"2024-03-21T13:06:04.907151+0000","last_clean":"2024-03-21T13:06:04.907151+0000","last_became_active":"2024-03-21T13:04:46.260660+0000","last_became_peered":"2024-03-21T13:04:46.260660+0000","last_unstale":"2024-03-21T13:06:04.907151+0000","last_undegraded":"2024-03-21T13:06:04.907151+0000","last_fullsized":"2024-03-21T13:06:04.907151+0000","mapping_epoch":168,"log_start":"177'2000","ondisk_log_start":"177'2000","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:09.301129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434503680,"num_objects":6630,"num_object_clones":0,"num_object_copies":19890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6630,"num_write_kb":424320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":153,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272798+0000","last_change":"2024-03-21T13:05:29.457708+0000","last_active":"2024-03-21T13:06:01.272798+0000","last_peered":"2024-03-21T13:06:01.272798+0000","last_clean":"2024-03-21T13:06:01.272798+0000","last_became_active":"2024-03-21T13:02:26.242708+0000","last_became_peered":"2024-03-21T13:02:26.242708+0000","last_unstale":"2024-03-21T13:06:01.272798+0000","last_undegraded":"2024-03-21T13:06:01.272798+0000","last_fullsized":"2024-03-21T13:06:01.272798+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.457597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.457597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:31.514232+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":178,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.269060+0000","last_change":"2024-03-21T13:05:11.035789+0000","last_active":"2024-03-21T13:06:01.269060+0000","last_peered":"2024-03-21T13:06:01.269060+0000","last_clean":"2024-03-21T13:06:01.269060+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:06:01.269060+0000","last_undegraded":"2024-03-21T13:06:01.269060+0000","last_fullsized":"2024-03-21T13:06:01.269060+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:11.035680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:05:11.035680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:40.978396+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":737,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:03.656260+0000","last_change":"2024-03-21T13:03:45.889240+0000","last_active":"2024-03-21T13:06:03.656260+0000","last_peered":"2024-03-21T13:06:03.656260+0000","last_clean":"2024-03-21T13:06:03.656260+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:06:03.656260+0000","last_undegraded":"2024-03-21T13:06:03.656260+0000","last_fullsized":"2024-03-21T13:06:03.656260+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:03:45.889083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:03:45.889083+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":438,"num_read_kb":433,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":328,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.279548+0000","last_change":"2024-03-21T13:05:11.320104+0000","last_active":"2024-03-21T13:06:01.279548+0000","last_peered":"2024-03-21T13:06:01.279548+0000","last_clean":"2024-03-21T13:06:01.279548+0000","last_became_active":"2024-03-21T13:02:14.333917+0000","last_became_peered":"2024-03-21T13:02:14.333917+0000","last_unstale":"2024-03-21T13:06:01.279548+0000","last_undegraded":"2024-03-21T13:06:01.279548+0000","last_fullsized":"2024-03-21T13:06:01.279548+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:05:11.319970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:11.319970+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:28.136189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":150,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.276318+0000","last_change":"2024-03-21T13:05:13.085740+0000","last_active":"2024-03-21T13:06:01.276318+0000","last_peered":"2024-03-21T13:06:01.276318+0000","last_clean":"2024-03-21T13:06:01.276318+0000","last_became_active":"2024-03-21T13:02:26.232625+0000","last_became_peered":"2024-03-21T13:02:26.232625+0000","last_unstale":"2024-03-21T13:06:01.276318+0000","last_undegraded":"2024-03-21T13:06:01.276318+0000","last_fullsized":"2024-03-21T13:06:01.276318+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.085625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.085625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:22.265367+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"189'6631","reported_seq":6683,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:05.280539+0000","last_change":"2024-03-21T13:04:46.262924+0000","last_active":"2024-03-21T13:06:05.280539+0000","last_peered":"2024-03-21T13:06:05.280539+0000","last_clean":"2024-03-21T13:06:05.280539+0000","last_became_active":"2024-03-21T13:04:46.259233+0000","last_became_peered":"2024-03-21T13:04:46.259233+0000","last_unstale":"2024-03-21T13:06:05.280539+0000","last_undegraded":"2024-03-21T13:06:05.280539+0000","last_fullsized":"2024-03-21T13:06:05.280539+0000","mapping_epoch":168,"log_start":"178'3000","ondisk_log_start":"178'3000","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3631,"log_dups_size":0,"ondisk_log_size":3631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:08.549536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434569216,"num_objects":6631,"num_object_clones":0,"num_object_copies":19893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6631,"num_write_kb":424384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":163,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273345+0000","last_change":"2024-03-21T13:05:35.000806+0000","last_active":"2024-03-21T13:06:01.273345+0000","last_peered":"2024-03-21T13:06:01.273345+0000","last_clean":"2024-03-21T13:06:01.273345+0000","last_became_active":"2024-03-21T13:02:14.335897+0000","last_became_peered":"2024-03-21T13:02:14.335897+0000","last_unstale":"2024-03-21T13:06:01.273345+0000","last_undegraded":"2024-03-21T13:06:01.273345+0000","last_fullsized":"2024-03-21T13:06:01.273345+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:35.000684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:35.000684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:36.510760+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":169,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273631+0000","last_change":"2024-03-21T13:05:07.282376+0000","last_active":"2024-03-21T13:06:01.273631+0000","last_peered":"2024-03-21T13:06:01.273631+0000","last_clean":"2024-03-21T13:06:01.273631+0000","last_became_active":"2024-03-21T13:02:26.237683+0000","last_became_peered":"2024-03-21T13:02:26.237683+0000","last_unstale":"2024-03-21T13:06:01.273631+0000","last_undegraded":"2024-03-21T13:06:01.273631+0000","last_fullsized":"2024-03-21T13:06:01.273631+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.282145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.282145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:07.856644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"71'19","reported_seq":224,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275555+0000","last_change":"2024-03-21T13:05:28.062353+0000","last_active":"2024-03-21T13:06:01.275555+0000","last_peered":"2024-03-21T13:06:01.275555+0000","last_clean":"2024-03-21T13:06:01.275555+0000","last_became_active":"2024-03-21T13:02:26.239140+0000","last_became_peered":"2024-03-21T13:02:26.239140+0000","last_unstale":"2024-03-21T13:06:01.275555+0000","last_undegraded":"2024-03-21T13:06:01.275555+0000","last_fullsized":"2024-03-21T13:06:01.275555+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:05:28.062237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.062237+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:46.857796+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":285,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272318+0000","last_change":"2024-03-21T13:05:26.932396+0000","last_active":"2024-03-21T13:06:01.272318+0000","last_peered":"2024-03-21T13:06:01.272318+0000","last_clean":"2024-03-21T13:06:01.272318+0000","last_became_active":"2024-03-21T13:02:26.252372+0000","last_became_peered":"2024-03-21T13:02:26.252372+0000","last_unstale":"2024-03-21T13:06:01.272318+0000","last_undegraded":"2024-03-21T13:06:01.272318+0000","last_fullsized":"2024-03-21T13:06:01.272318+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:26.932188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.932188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:43.483509+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"189'6682","reported_seq":6734,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:04.917122+0000","last_change":"2024-03-21T13:04:46.254274+0000","last_active":"2024-03-21T13:06:04.917122+0000","last_peered":"2024-03-21T13:06:04.917122+0000","last_clean":"2024-03-21T13:06:04.917122+0000","last_became_active":"2024-03-21T13:04:46.253974+0000","last_became_peered":"2024-03-21T13:04:46.253974+0000","last_unstale":"2024-03-21T13:06:04.917122+0000","last_undegraded":"2024-03-21T13:06:04.917122+0000","last_fullsized":"2024-03-21T13:06:04.917122+0000","mapping_epoch":168,"log_start":"177'2500","ondisk_log_start":"177'2500","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:05.355266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437911552,"num_objects":6682,"num_object_clones":0,"num_object_copies":20046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":427648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":170,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274918+0000","last_change":"2024-03-21T13:01:48.209228+0000","last_active":"2024-03-21T13:06:01.274918+0000","last_peered":"2024-03-21T13:06:01.274918+0000","last_clean":"2024-03-21T13:06:01.274918+0000","last_became_active":"2024-03-21T13:01:48.208899+0000","last_became_peered":"2024-03-21T13:01:48.208899+0000","last_unstale":"2024-03-21T13:06:01.274918+0000","last_undegraded":"2024-03-21T13:06:01.274918+0000","last_fullsized":"2024-03-21T13:06:01.274918+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:31.121913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.121913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":151,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272571+0000","last_change":"2024-03-21T13:05:13.328899+0000","last_active":"2024-03-21T13:06:01.272571+0000","last_peered":"2024-03-21T13:06:01.272571+0000","last_clean":"2024-03-21T13:06:01.272571+0000","last_became_active":"2024-03-21T13:02:26.243417+0000","last_became_peered":"2024-03-21T13:02:26.243417+0000","last_unstale":"2024-03-21T13:06:01.272571+0000","last_undegraded":"2024-03-21T13:06:01.272571+0000","last_fullsized":"2024-03-21T13:06:01.272571+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.324701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.324701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:41.004405+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"86'15","reported_seq":217,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.270345+0000","last_change":"2024-03-21T13:05:42.308704+0000","last_active":"2024-03-21T13:06:01.270345+0000","last_peered":"2024-03-21T13:06:01.270345+0000","last_clean":"2024-03-21T13:06:01.270345+0000","last_became_active":"2024-03-21T13:02:26.232580+0000","last_became_peered":"2024-03-21T13:02:26.232580+0000","last_unstale":"2024-03-21T13:06:01.270345+0000","last_undegraded":"2024-03-21T13:06:01.270345+0000","last_fullsized":"2024-03-21T13:06:01.270345+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-03-21T13:05:42.308556+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.308556+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:56.807686+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"125'12","reported_seq":267,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272949+0000","last_change":"2024-03-21T13:05:13.903736+0000","last_active":"2024-03-21T13:06:01.272949+0000","last_peered":"2024-03-21T13:06:01.272949+0000","last_clean":"2024-03-21T13:06:01.272949+0000","last_became_active":"2024-03-21T13:02:26.253720+0000","last_became_peered":"2024-03-21T13:02:26.253720+0000","last_unstale":"2024-03-21T13:06:01.272949+0000","last_undegraded":"2024-03-21T13:06:01.272949+0000","last_fullsized":"2024-03-21T13:06:01.272949+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'12","last_scrub_stamp":"2024-03-21T13:05:13.903589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.903589+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:22.554595+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"189'6724","reported_seq":6776,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:04.692403+0000","last_change":"2024-03-21T13:04:46.259414+0000","last_active":"2024-03-21T13:06:04.692403+0000","last_peered":"2024-03-21T13:06:04.692403+0000","last_clean":"2024-03-21T13:06:04.692403+0000","last_became_active":"2024-03-21T13:04:46.259106+0000","last_became_peered":"2024-03-21T13:04:46.259106+0000","last_unstale":"2024-03-21T13:06:04.692403+0000","last_undegraded":"2024-03-21T13:06:04.692403+0000","last_fullsized":"2024-03-21T13:06:04.692403+0000","mapping_epoch":168,"log_start":"177'2600","ondisk_log_start":"177'2600","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4124,"log_dups_size":0,"ondisk_log_size":4124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:05.318642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440664064,"num_objects":6724,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":430336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":273,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275569+0000","last_change":"2024-03-21T13:05:41.951788+0000","last_active":"2024-03-21T13:06:01.275569+0000","last_peered":"2024-03-21T13:06:01.275569+0000","last_clean":"2024-03-21T13:06:01.275569+0000","last_became_active":"2024-03-21T12:58:43.353049+0000","last_became_peered":"2024-03-21T12:58:43.353049+0000","last_unstale":"2024-03-21T13:06:01.275569+0000","last_undegraded":"2024-03-21T13:06:01.275569+0000","last_fullsized":"2024-03-21T13:06:01.275569+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.951655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.951655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:09.559926+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":305,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.279641+0000","last_change":"2024-03-21T13:05:23.570424+0000","last_active":"2024-03-21T13:06:01.279641+0000","last_peered":"2024-03-21T13:06:01.279641+0000","last_clean":"2024-03-21T13:06:01.279641+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:06:01.279641+0000","last_undegraded":"2024-03-21T13:06:01.279641+0000","last_fullsized":"2024-03-21T13:06:01.279641+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:05:23.569936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.569936+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:40.445343+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"87'21","reported_seq":243,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272324+0000","last_change":"2024-03-21T13:05:23.917855+0000","last_active":"2024-03-21T13:06:01.272324+0000","last_peered":"2024-03-21T13:06:01.272324+0000","last_clean":"2024-03-21T13:06:01.272324+0000","last_became_active":"2024-03-21T13:02:26.227165+0000","last_became_peered":"2024-03-21T13:02:26.227165+0000","last_unstale":"2024-03-21T13:06:01.272324+0000","last_undegraded":"2024-03-21T13:06:01.272324+0000","last_fullsized":"2024-03-21T13:06:01.272324+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'21","last_scrub_stamp":"2024-03-21T13:05:23.917691+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.917691+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:32.622590+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"119'9","reported_seq":372,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275522+0000","last_change":"2024-03-21T13:02:14.375956+0000","last_active":"2024-03-21T13:06:01.275522+0000","last_peered":"2024-03-21T13:06:01.275522+0000","last_clean":"2024-03-21T13:06:01.275522+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:06:01.275522+0000","last_undegraded":"2024-03-21T13:06:01.275522+0000","last_fullsized":"2024-03-21T13:06:01.275522+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:25.700549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:25.700549+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:07.522783+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"189'6610","reported_seq":6662,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:04.902134+0000","last_change":"2024-03-21T13:04:46.260954+0000","last_active":"2024-03-21T13:06:04.902134+0000","last_peered":"2024-03-21T13:06:04.902134+0000","last_clean":"2024-03-21T13:06:04.902134+0000","last_became_active":"2024-03-21T13:04:46.260642+0000","last_became_peered":"2024-03-21T13:04:46.260642+0000","last_unstale":"2024-03-21T13:06:04.902134+0000","last_undegraded":"2024-03-21T13:06:04.902134+0000","last_fullsized":"2024-03-21T13:06:04.902134+0000","mapping_epoch":168,"log_start":"177'2400","ondisk_log_start":"177'2400","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4210,"log_dups_size":0,"ondisk_log_size":4210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:15.059160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433192960,"num_objects":6610,"num_object_clones":0,"num_object_copies":19830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6610,"num_write_kb":423040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":192,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273021+0000","last_change":"2024-03-21T13:05:18.416381+0000","last_active":"2024-03-21T13:06:01.273021+0000","last_peered":"2024-03-21T13:06:01.273021+0000","last_clean":"2024-03-21T13:06:01.273021+0000","last_became_active":"2024-03-21T13:01:27.382663+0000","last_became_peered":"2024-03-21T13:01:27.382663+0000","last_unstale":"2024-03-21T13:06:01.273021+0000","last_undegraded":"2024-03-21T13:06:01.273021+0000","last_fullsized":"2024-03-21T13:06:01.273021+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.416262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.416262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:22.282577+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273553+0000","last_change":"2024-03-21T13:04:20.538204+0000","last_active":"2024-03-21T13:06:01.273553+0000","last_peered":"2024-03-21T13:06:01.273553+0000","last_clean":"2024-03-21T13:06:01.273553+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:06:01.273553+0000","last_undegraded":"2024-03-21T13:06:01.273553+0000","last_fullsized":"2024-03-21T13:06:01.273553+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:20.538115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:04:20.538115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"71'18","reported_seq":307,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275289+0000","last_change":"2024-03-21T13:05:12.322522+0000","last_active":"2024-03-21T13:06:01.275289+0000","last_peered":"2024-03-21T13:06:01.275289+0000","last_clean":"2024-03-21T13:06:01.275289+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:06:01.275289+0000","last_undegraded":"2024-03-21T13:06:01.275289+0000","last_fullsized":"2024-03-21T13:06:01.275289+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T13:05:12.322284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:12.322284+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:38.174015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.268809+0000","last_change":"2024-03-21T13:05:07.086071+0000","last_active":"2024-03-21T13:06:01.268809+0000","last_peered":"2024-03-21T13:06:01.268809+0000","last_clean":"2024-03-21T13:06:01.268809+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:06:01.268809+0000","last_undegraded":"2024-03-21T13:06:01.268809+0000","last_fullsized":"2024-03-21T13:06:01.268809+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.085992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.085992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:34.586538+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"189'6629","reported_seq":6681,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:04.730323+0000","last_change":"2024-03-21T13:04:46.260554+0000","last_active":"2024-03-21T13:06:04.730323+0000","last_peered":"2024-03-21T13:06:04.730323+0000","last_clean":"2024-03-21T13:06:04.730323+0000","last_became_active":"2024-03-21T13:04:46.260083+0000","last_became_peered":"2024-03-21T13:04:46.260083+0000","last_unstale":"2024-03-21T13:06:04.730323+0000","last_undegraded":"2024-03-21T13:06:04.730323+0000","last_fullsized":"2024-03-21T13:06:04.730323+0000","mapping_epoch":168,"log_start":"177'2500","ondisk_log_start":"177'2500","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:06.060796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434438144,"num_objects":6629,"num_object_clones":0,"num_object_copies":19887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":424256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273686+0000","last_change":"2024-03-21T13:05:47.329221+0000","last_active":"2024-03-21T13:06:01.273686+0000","last_peered":"2024-03-21T13:06:01.273686+0000","last_clean":"2024-03-21T13:06:01.273686+0000","last_became_active":"2024-03-21T13:02:26.236501+0000","last_became_peered":"2024-03-21T13:02:26.236501+0000","last_unstale":"2024-03-21T13:06:01.273686+0000","last_undegraded":"2024-03-21T13:06:01.273686+0000","last_fullsized":"2024-03-21T13:06:01.273686+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.328925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.328925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:00.534421+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"125'10","reported_seq":232,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.281323+0000","last_change":"2024-03-21T13:05:20.446633+0000","last_active":"2024-03-21T13:06:01.281323+0000","last_peered":"2024-03-21T13:06:01.281323+0000","last_clean":"2024-03-21T13:06:01.281323+0000","last_became_active":"2024-03-21T13:02:26.231372+0000","last_became_peered":"2024-03-21T13:02:26.231372+0000","last_unstale":"2024-03-21T13:06:01.281323+0000","last_undegraded":"2024-03-21T13:06:01.281323+0000","last_fullsized":"2024-03-21T13:06:01.281323+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:05:20.446177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.446177+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:22.239776+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":285,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.275204+0000","last_change":"2024-03-21T13:04:19.520964+0000","last_active":"2024-03-21T13:06:01.275204+0000","last_peered":"2024-03-21T13:06:01.275204+0000","last_clean":"2024-03-21T13:06:01.275204+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:06:01.275204+0000","last_undegraded":"2024-03-21T13:06:01.275204+0000","last_fullsized":"2024-03-21T13:06:01.275204+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:19.520824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:04:19.520824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":215,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273582+0000","last_change":"2024-03-21T13:02:14.377274+0000","last_active":"2024-03-21T13:06:01.273582+0000","last_peered":"2024-03-21T13:06:01.273582+0000","last_clean":"2024-03-21T13:06:01.273582+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:06:01.273582+0000","last_undegraded":"2024-03-21T13:06:01.273582+0000","last_fullsized":"2024-03-21T13:06:01.273582+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:01:46.167736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.167736+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"189'6661","reported_seq":6713,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:04.709895+0000","last_change":"2024-03-21T13:04:46.261080+0000","last_active":"2024-03-21T13:06:04.709895+0000","last_peered":"2024-03-21T13:06:04.709895+0000","last_clean":"2024-03-21T13:06:04.709895+0000","last_became_active":"2024-03-21T13:04:46.260735+0000","last_became_peered":"2024-03-21T13:04:46.260735+0000","last_unstale":"2024-03-21T13:06:04.709895+0000","last_undegraded":"2024-03-21T13:06:04.709895+0000","last_fullsized":"2024-03-21T13:06:04.709895+0000","mapping_epoch":168,"log_start":"178'2700","ondisk_log_start":"178'2700","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3961,"log_dups_size":0,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:08.414156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436535296,"num_objects":6661,"num_object_clones":0,"num_object_copies":19983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6661,"num_write_kb":426304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":278,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272688+0000","last_change":"2024-03-21T13:05:39.518051+0000","last_active":"2024-03-21T13:06:01.272688+0000","last_peered":"2024-03-21T13:06:01.272688+0000","last_clean":"2024-03-21T13:06:01.272688+0000","last_became_active":"2024-03-21T13:02:26.243985+0000","last_became_peered":"2024-03-21T13:02:26.243985+0000","last_unstale":"2024-03-21T13:06:01.272688+0000","last_undegraded":"2024-03-21T13:06:01.272688+0000","last_fullsized":"2024-03-21T13:06:01.272688+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:39.517591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.517591+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:46.995132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":159,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.270539+0000","last_change":"2024-03-21T13:05:38.439542+0000","last_active":"2024-03-21T13:06:01.270539+0000","last_peered":"2024-03-21T13:06:01.270539+0000","last_clean":"2024-03-21T13:06:01.270539+0000","last_became_active":"2024-03-21T13:02:26.229296+0000","last_became_peered":"2024-03-21T13:02:26.229296+0000","last_unstale":"2024-03-21T13:06:01.270539+0000","last_undegraded":"2024-03-21T13:06:01.270539+0000","last_fullsized":"2024-03-21T13:06:01.270539+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:38.439435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.439435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:43.541522+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":287,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.268910+0000","last_change":"2024-03-21T13:05:14.021031+0000","last_active":"2024-03-21T13:06:01.268910+0000","last_peered":"2024-03-21T13:06:01.268910+0000","last_clean":"2024-03-21T13:06:01.268910+0000","last_became_active":"2024-03-21T12:58:37.730858+0000","last_became_peered":"2024-03-21T12:58:37.730858+0000","last_unstale":"2024-03-21T13:06:01.268910+0000","last_undegraded":"2024-03-21T13:06:01.268910+0000","last_fullsized":"2024-03-21T13:06:01.268910+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:14.020911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:14.020911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:16.781006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'11","reported_seq":320,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.269368+0000","last_change":"2024-03-21T13:05:26.195643+0000","last_active":"2024-03-21T13:06:01.269368+0000","last_peered":"2024-03-21T13:06:01.269368+0000","last_clean":"2024-03-21T13:06:01.269368+0000","last_became_active":"2024-03-21T13:02:14.348499+0000","last_became_peered":"2024-03-21T13:02:14.348499+0000","last_unstale":"2024-03-21T13:06:01.269368+0000","last_undegraded":"2024-03-21T13:06:01.269368+0000","last_fullsized":"2024-03-21T13:06:01.269368+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:26.195507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.195507+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:29.731844+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"189'6756","reported_seq":6808,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:04.907410+0000","last_change":"2024-03-21T13:04:46.260817+0000","last_active":"2024-03-21T13:06:04.907410+0000","last_peered":"2024-03-21T13:06:04.907410+0000","last_clean":"2024-03-21T13:06:04.907410+0000","last_became_active":"2024-03-21T13:04:46.260506+0000","last_became_peered":"2024-03-21T13:04:46.260506+0000","last_unstale":"2024-03-21T13:06:04.907410+0000","last_undegraded":"2024-03-21T13:06:04.907410+0000","last_fullsized":"2024-03-21T13:06:04.907410+0000","mapping_epoch":168,"log_start":"177'2200","ondisk_log_start":"177'2200","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:07.244954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442761216,"num_objects":6756,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6756,"num_write_kb":432384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":276,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274296+0000","last_change":"2024-03-21T13:05:36.362310+0000","last_active":"2024-03-21T13:06:01.274296+0000","last_peered":"2024-03-21T13:06:01.274296+0000","last_clean":"2024-03-21T13:06:01.274296+0000","last_became_active":"2024-03-21T12:58:43.183380+0000","last_became_peered":"2024-03-21T12:58:43.183380+0000","last_unstale":"2024-03-21T13:06:01.274296+0000","last_undegraded":"2024-03-21T13:06:01.274296+0000","last_fullsized":"2024-03-21T13:06:01.274296+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:36.362197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:05:36.362197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:06.202139+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":171,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.271408+0000","last_change":"2024-03-21T13:05:33.327069+0000","last_active":"2024-03-21T13:06:01.271408+0000","last_peered":"2024-03-21T13:06:01.271408+0000","last_clean":"2024-03-21T13:06:01.271408+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:06:01.271408+0000","last_undegraded":"2024-03-21T13:06:01.271408+0000","last_fullsized":"2024-03-21T13:06:01.271408+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:33.326990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:33.326990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:37.358494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":293,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272355+0000","last_change":"2024-03-21T13:04:32.808501+0000","last_active":"2024-03-21T13:06:01.272355+0000","last_peered":"2024-03-21T13:06:01.272355+0000","last_clean":"2024-03-21T13:06:01.272355+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:06:01.272355+0000","last_undegraded":"2024-03-21T13:06:01.272355+0000","last_fullsized":"2024-03-21T13:06:01.272355+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:32.808390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:04:32.808390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:06.060698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.273788+0000","last_change":"2024-03-21T13:05:38.216400+0000","last_active":"2024-03-21T13:06:01.273788+0000","last_peered":"2024-03-21T13:06:01.273788+0000","last_clean":"2024-03-21T13:06:01.273788+0000","last_became_active":"2024-03-21T13:02:14.334882+0000","last_became_peered":"2024-03-21T13:02:14.334882+0000","last_unstale":"2024-03-21T13:06:01.273788+0000","last_undegraded":"2024-03-21T13:06:01.273788+0000","last_fullsized":"2024-03-21T13:06:01.273788+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-21T13:05:38.215693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.215693+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:06.349642+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"189'6648","reported_seq":6700,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:05.264454+0000","last_change":"2024-03-21T13:04:46.258753+0000","last_active":"2024-03-21T13:06:05.264454+0000","last_peered":"2024-03-21T13:06:05.264454+0000","last_clean":"2024-03-21T13:06:05.264454+0000","last_became_active":"2024-03-21T13:04:46.258434+0000","last_became_peered":"2024-03-21T13:04:46.258434+0000","last_unstale":"2024-03-21T13:06:05.264454+0000","last_undegraded":"2024-03-21T13:06:05.264454+0000","last_fullsized":"2024-03-21T13:06:05.264454+0000","mapping_epoch":168,"log_start":"178'3000","ondisk_log_start":"178'3000","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3648,"log_dups_size":0,"ondisk_log_size":3648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:09.622644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435683328,"num_objects":6648,"num_object_clones":0,"num_object_copies":19944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6648,"num_write_kb":425472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":171,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274187+0000","last_change":"2024-03-21T13:05:34.291498+0000","last_active":"2024-03-21T13:06:01.274187+0000","last_peered":"2024-03-21T13:06:01.274187+0000","last_clean":"2024-03-21T13:06:01.274187+0000","last_became_active":"2024-03-21T13:02:26.227779+0000","last_became_peered":"2024-03-21T13:02:26.227779+0000","last_unstale":"2024-03-21T13:06:01.274187+0000","last_undegraded":"2024-03-21T13:06:01.274187+0000","last_fullsized":"2024-03-21T13:06:01.274187+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.291369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.291369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:48.086995+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"125'10","reported_seq":389,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.274110+0000","last_change":"2024-03-21T13:05:30.300832+0000","last_active":"2024-03-21T13:06:01.274110+0000","last_peered":"2024-03-21T13:06:01.274110+0000","last_clean":"2024-03-21T13:06:01.274110+0000","last_became_active":"2024-03-21T13:02:26.235954+0000","last_became_peered":"2024-03-21T13:02:26.235954+0000","last_unstale":"2024-03-21T13:06:01.274110+0000","last_undegraded":"2024-03-21T13:06:01.274110+0000","last_fullsized":"2024-03-21T13:06:01.274110+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:05:30.300605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:30.300605+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:41.344464+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272008+0000","last_change":"2024-03-21T13:05:38.492137+0000","last_active":"2024-03-21T13:06:01.272008+0000","last_peered":"2024-03-21T13:06:01.272008+0000","last_clean":"2024-03-21T13:06:01.272008+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:06:01.272008+0000","last_undegraded":"2024-03-21T13:06:01.272008+0000","last_fullsized":"2024-03-21T13:06:01.272008+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:38.492015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.492015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:05.896092+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":226,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-21T13:06:01.272893+0000","last_change":"2024-03-21T13:05:22.406498+0000","last_active":"2024-03-21T13:06:01.272893+0000","last_peered":"2024-03-21T13:06:01.272893+0000","last_clean":"2024-03-21T13:06:01.272893+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:06:01.272893+0000","last_undegraded":"2024-03-21T13:06:01.272893+0000","last_fullsized":"2024-03-21T13:06:01.272893+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:22.406367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.406367+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:36.553896+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6994984960,"num_objects":106735,"num_object_clones":0,"num_object_copies":320205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106735,"num_write_kb":6831040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20978073600,"data_stored":20978073600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64535,"ondisk_log_size":64535,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":438,"num_read_kb":433,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":684,"num_read_kb":441,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":3658,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":432,"ondisk_log_size":432,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168670,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2332304,"kb_used_data":2116488,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":91412848,"statfs":{"total":95995035648,"available":93606756352,"internally_reserved":0,"allocated":2167283712,"data_stored":2166626065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":220985802},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463012,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3643608,"kb_used_data":3387536,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":90101544,"statfs":{"total":95995035648,"available":92263981056,"internally_reserved":0,"allocated":3468836864,"data_stored":3468171311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":262207942},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574972,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738498,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3246032,"kb_used_data":2989960,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":90499120,"statfs":{"total":95995035648,"available":92671098880,"internally_reserved":0,"allocated":3061719040,"data_stored":3061060878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411448,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4116412,"kb_used_data":3806196,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":89628740,"statfs":{"total":95995035648,"available":91779829760,"internally_reserved":0,"allocated":3897544704,"data_stored":3896906158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182515,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5051820,"kb_used_data":4669412,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":88693332,"statfs":{"total":95995035648,"available":90821971968,"internally_reserved":0,"allocated":4781477888,"data_stored":4780791665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":391576007},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018989,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3666100,"kb_used_data":3382252,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":90079052,"statfs":{"total":95995035648,"available":92240949248,"internally_reserved":0,"allocated":3463426048,"data_stored":3462795879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724648,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3077505024,"data_stored":3077505024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3925803008,"data_stored":3925803008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4821221376,"data_stored":4821221376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3477929984,"data_stored":3477929984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3492544512,"data_stored":3492544512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2183069696,"data_stored":2183069696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:06:14.609 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:06:14.672 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:14 smithi204 ceph-mon[33153]: pgmap v322: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-21T13:06:14.702 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:14.750 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:14.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:14.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:14.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:14.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:14.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:14.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:15.021 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:15.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:15.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 77 16 119668 119652 97.1026 75.25 0.00856699 0.0102935 2024-03-21T13:06:15.122 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:06:15.123 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:15.197 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:15.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:15.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:15.253 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:15.299 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:15.361 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:15.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:15.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:15.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:15.506 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:15.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:15.582 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:15 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:06:15.582 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:15 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:06:15.583 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:15 smithi163 ceph-mon[29621]: osdmap e192: 8 total, 8 up, 6 in 2024-03-21T13:06:15.583 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:15 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:15.583 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:15 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:06:15.583 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:15 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:06:15.583 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:15 smithi163 ceph-mon[40675]: osdmap e192: 8 total, 8 up, 6 in 2024-03-21T13:06:15.583 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:15 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:15.608 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:15.643 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:15.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:15.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:15.745 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:15.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:15 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:06:15.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:15 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:06:15.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:15 smithi204 ceph-mon[33153]: osdmap e192: 8 total, 8 up, 6 in 2024-03-21T13:06:15.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:15 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:15.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:15.791 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:15.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:15.853 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:06:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:06:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:06:15.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:15.904 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:06:15.904 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:06:15.930 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:15.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:16.032 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:16.084 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 78 16 120661 120645 96.6534 62.0625 0.00729833 0.0103419 2024-03-21T13:06:16.093 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:16.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:16.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:16.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:16.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:16.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:16.369 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:16 smithi163 ceph-mon[29621]: pgmap v324: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.7 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-21T13:06:16.370 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:16 smithi163 ceph-mon[29621]: osdmap e193: 8 total, 8 up, 6 in 2024-03-21T13:06:16.377 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:16.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:16.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:16.521 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:16.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:16.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:16.609 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:16 smithi204 ceph-mon[33153]: pgmap v324: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.7 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-21T13:06:16.609 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:16 smithi204 ceph-mon[33153]: osdmap e193: 8 total, 8 up, 6 in 2024-03-21T13:06:16.623 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:16.656 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-21T13:06:16.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:16.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:16.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:16.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:16 smithi163 ceph-mon[40675]: pgmap v324: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.7 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-21T13:06:16.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:16 smithi163 ceph-mon[40675]: osdmap e193: 8 total, 8 up, 6 in 2024-03-21T13:06:16.765 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:16.805 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:16.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:16.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:16.907 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:16.952 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:16.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:17.041 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:06:17.053 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:17.086 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 79 16 122035 122019 96.5154 85.875 0.0174883 0.0103565 2024-03-21T13:06:17.091 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:17.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:17.193 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:17.242 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:17.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:17.344 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:17.368 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:06:17.376 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:17.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:17.412 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:06:17.412 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:17 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:06:17.412 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:17 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:06:17.412 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:17 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:06:17.412 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:17 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:06:17.413 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:17 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:06:17.413 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:17 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/731579678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:17.413 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:17 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:06:17.413 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:17 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:06:17.413 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:17 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:06:17.413 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:17 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:06:17.413 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:17 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:06:17.413 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:17 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/731579678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:17.444 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:06:17.468 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:06:17.478 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:17.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:17 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:06:17.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:17 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:06:17.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:17 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:06:17.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:17 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:06:17.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:17 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:06:17.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:17 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/731579678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:17.508 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:06:17.520 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:17.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:17.546 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:06:17.589 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:06:17.622 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:17.627 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:06:17.633 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:06:17.633 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:06:17.633 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:06:17.633 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:06:17.633 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:06:17.633 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:06:17.634 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:06:17.634 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:06:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:17.663 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:17.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:17.764 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:17.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:17.804 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:17.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:17.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:17.943 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:17.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:18.046 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:18.088 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:06:18.087826+0000 min lat: 0.00120788 max lat: 0.163103 avg lat: 0.0103875 2024-03-21T13:06:18.088 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:06:18.088 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 80 16 123205 123189 96.2217 73.125 0.015745 0.0103875 2024-03-21T13:06:18.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:18.093 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:18.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:18.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:18.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:18.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:18.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:18.331 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:06:18.347 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:06:18.389 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:18.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:18.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:18.490 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:18.538 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:18.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:18.597 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:18 smithi163 ceph-mon[29621]: pgmap v326: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-21T13:06:18.598 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:18 smithi163 ceph-mon[40675]: pgmap v326: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-21T13:06:18.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:18.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:18.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:18.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:18 smithi204 ceph-mon[33153]: pgmap v326: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-21T13:06:18.773 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:18.818 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:18.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:18.920 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:18.952 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:18.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:19.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:19.087 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 81 16 124546 124530 96.0684 83.8125 0.0122709 0.0104049 2024-03-21T13:06:19.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:19.088 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:19.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:19.189 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:19.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:19.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:19.331 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:19.369 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:19.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:19.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:19.472 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-03-21T13:06:16.759867+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7831820712,"num_objects":119693,"num_object_clones":0,"num_object_copies":359079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119693,"num_whiteouts":0,"num_read":1368,"num_read_kb":1214,"num_write":120110,"num_write_kb":7650141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1775783,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65150,"ondisk_log_size":65150,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":24522900,"kb_used_data":22811480,"kb_used_omap":9,"kb_used_meta":1711350,"kb_avail":537948012,"statfs":{"total":575970213888,"available":550858764288,"internally_reserved":0,"allocated":23358955520,"data_stored":23354939120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":1752423078},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":401080320,"num_objects":6120,"num_object_clones":0,"num_object_copies":18360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6120,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6120,"num_write_kb":391680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001764"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.361422+0000","last_change":"2024-03-21T13:05:20.083213+0000","last_active":"2024-03-21T13:06:14.361422+0000","last_peered":"2024-03-21T13:06:14.361422+0000","last_clean":"2024-03-21T13:06:14.361422+0000","last_became_active":"2024-03-21T13:02:26.223292+0000","last_became_peered":"2024-03-21T13:02:26.223292+0000","last_unstale":"2024-03-21T13:06:14.361422+0000","last_undegraded":"2024-03-21T13:06:14.361422+0000","last_fullsized":"2024-03-21T13:06:14.361422+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:20.083111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.083111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:37.095715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":159,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.352019+0000","last_change":"2024-03-21T13:03:40.962728+0000","last_active":"2024-03-21T13:06:14.352019+0000","last_peered":"2024-03-21T13:06:14.352019+0000","last_clean":"2024-03-21T13:06:14.352019+0000","last_became_active":"2024-03-21T13:02:26.233709+0000","last_became_peered":"2024-03-21T13:02:26.233709+0000","last_unstale":"2024-03-21T13:06:14.352019+0000","last_undegraded":"2024-03-21T13:06:14.352019+0000","last_fullsized":"2024-03-21T13:06:14.352019+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:40.962617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:03:40.962617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.906008+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":167,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.356495+0000","last_change":"2024-03-21T13:05:49.260935+0000","last_active":"2024-03-21T13:06:14.356495+0000","last_peered":"2024-03-21T13:06:14.356495+0000","last_clean":"2024-03-21T13:06:14.356495+0000","last_became_active":"2024-03-21T13:02:26.237501+0000","last_became_peered":"2024-03-21T13:02:26.237501+0000","last_unstale":"2024-03-21T13:06:14.356495+0000","last_undegraded":"2024-03-21T13:06:14.356495+0000","last_fullsized":"2024-03-21T13:06:14.356495+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:49.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.260693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:01.973518+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"89'13","reported_seq":341,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.357559+0000","last_change":"2024-03-21T13:05:17.080137+0000","last_active":"2024-03-21T13:06:14.357559+0000","last_peered":"2024-03-21T13:06:14.357559+0000","last_clean":"2024-03-21T13:06:14.357559+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:06:14.357559+0000","last_undegraded":"2024-03-21T13:06:14.357559+0000","last_fullsized":"2024-03-21T13:06:14.357559+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-03-21T13:05:17.079993+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:05:17.079993+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:25.868022+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362704+0000","last_change":"2024-03-21T13:05:07.334438+0000","last_active":"2024-03-21T13:06:14.362704+0000","last_peered":"2024-03-21T13:06:14.362704+0000","last_clean":"2024-03-21T13:06:14.362704+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:06:14.362704+0000","last_undegraded":"2024-03-21T13:06:14.362704+0000","last_fullsized":"2024-03-21T13:06:14.362704+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.330296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.330296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:22.315386+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":211,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.349428+0000","last_change":"2024-03-21T13:05:50.991814+0000","last_active":"2024-03-21T13:06:14.349428+0000","last_peered":"2024-03-21T13:06:14.349428+0000","last_clean":"2024-03-21T13:06:14.349428+0000","last_became_active":"2024-03-21T13:02:14.330467+0000","last_became_peered":"2024-03-21T13:02:14.330467+0000","last_unstale":"2024-03-21T13:06:14.349428+0000","last_undegraded":"2024-03-21T13:06:14.349428+0000","last_fullsized":"2024-03-21T13:06:14.349428+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:50.991735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.991735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:06.675583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362154+0000","last_change":"2024-03-21T13:01:38.766164+0000","last_active":"2024-03-21T13:06:14.362154+0000","last_peered":"2024-03-21T13:06:14.362154+0000","last_clean":"2024-03-21T13:06:14.362154+0000","last_became_active":"2024-03-21T12:58:37.728383+0000","last_became_peered":"2024-03-21T12:58:37.728383+0000","last_unstale":"2024-03-21T13:06:14.362154+0000","last_undegraded":"2024-03-21T13:06:14.362154+0000","last_fullsized":"2024-03-21T13:06:14.362154+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_clean_scrub_stamp":"2024-03-21T13:01:38.765896+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554424+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"71'15","reported_seq":306,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.370949+0000","last_change":"2024-03-21T13:05:26.476181+0000","last_active":"2024-03-21T13:06:14.370949+0000","last_peered":"2024-03-21T13:06:14.370949+0000","last_clean":"2024-03-21T13:06:14.370949+0000","last_became_active":"2024-03-21T12:58:39.750396+0000","last_became_peered":"2024-03-21T12:58:39.750396+0000","last_unstale":"2024-03-21T13:06:14.370949+0000","last_undegraded":"2024-03-21T13:06:14.370949+0000","last_fullsized":"2024-03-21T13:06:14.370949+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:26.476043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.476043+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:46.996849+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":171,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.351318+0000","last_change":"2024-03-21T13:04:32.834762+0000","last_active":"2024-03-21T13:06:14.351318+0000","last_peered":"2024-03-21T13:06:14.351318+0000","last_clean":"2024-03-21T13:06:14.351318+0000","last_became_active":"2024-03-21T13:02:14.334221+0000","last_became_peered":"2024-03-21T13:02:14.334221+0000","last_unstale":"2024-03-21T13:06:14.351318+0000","last_undegraded":"2024-03-21T13:06:14.351318+0000","last_fullsized":"2024-03-21T13:06:14.351318+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:32.834646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:32.834646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:14.896410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":165,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.353354+0000","last_change":"2024-03-21T13:05:12.353129+0000","last_active":"2024-03-21T13:06:14.353354+0000","last_peered":"2024-03-21T13:06:14.353354+0000","last_clean":"2024-03-21T13:06:14.353354+0000","last_became_active":"2024-03-21T13:02:26.236707+0000","last_became_peered":"2024-03-21T13:02:26.236707+0000","last_unstale":"2024-03-21T13:06:14.353354+0000","last_undegraded":"2024-03-21T13:06:14.353354+0000","last_fullsized":"2024-03-21T13:06:14.353354+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:12.353011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:05:12.353011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:41.590667+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":157,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.351963+0000","last_change":"2024-03-21T13:05:45.989199+0000","last_active":"2024-03-21T13:06:14.351963+0000","last_peered":"2024-03-21T13:06:14.351963+0000","last_clean":"2024-03-21T13:06:14.351963+0000","last_became_active":"2024-03-21T13:02:26.245402+0000","last_became_peered":"2024-03-21T13:02:26.245402+0000","last_unstale":"2024-03-21T13:06:14.351963+0000","last_undegraded":"2024-03-21T13:06:14.351963+0000","last_fullsized":"2024-03-21T13:06:14.351963+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:45.989114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:45.989114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:01.207660+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":274,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.369920+0000","last_change":"2024-03-21T13:05:51.580203+0000","last_active":"2024-03-21T13:06:14.369920+0000","last_peered":"2024-03-21T13:06:14.369920+0000","last_clean":"2024-03-21T13:06:14.369920+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:06:14.369920+0000","last_undegraded":"2024-03-21T13:06:14.369920+0000","last_fullsized":"2024-03-21T13:06:14.369920+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-21T13:05:51.580074+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:05:51.580074+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:09.467583+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":269,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.370515+0000","last_change":"2024-03-21T13:02:26.229621+0000","last_active":"2024-03-21T13:06:14.370515+0000","last_peered":"2024-03-21T13:06:14.370515+0000","last_clean":"2024-03-21T13:06:14.370515+0000","last_became_active":"2024-03-21T13:02:26.229397+0000","last_became_peered":"2024-03-21T13:02:26.229397+0000","last_unstale":"2024-03-21T13:06:14.370515+0000","last_undegraded":"2024-03-21T13:06:14.370515+0000","last_fullsized":"2024-03-21T13:06:14.370515+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:32.971665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:32.971665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":289,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.352208+0000","last_change":"2024-03-21T13:04:30.831274+0000","last_active":"2024-03-21T13:06:14.352208+0000","last_peered":"2024-03-21T13:06:14.352208+0000","last_clean":"2024-03-21T13:06:14.352208+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:06:14.352208+0000","last_undegraded":"2024-03-21T13:06:14.352208+0000","last_fullsized":"2024-03-21T13:06:14.352208+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:30.831143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:04:30.831143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.906159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.358016+0000","last_change":"2024-03-21T13:05:41.337232+0000","last_active":"2024-03-21T13:06:14.358016+0000","last_peered":"2024-03-21T13:06:14.358016+0000","last_clean":"2024-03-21T13:06:14.358016+0000","last_became_active":"2024-03-21T13:02:26.230057+0000","last_became_peered":"2024-03-21T13:02:26.230057+0000","last_unstale":"2024-03-21T13:06:14.358016+0000","last_undegraded":"2024-03-21T13:06:14.358016+0000","last_fullsized":"2024-03-21T13:06:14.358016+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.337144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.337144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:52.877759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":276,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362235+0000","last_change":"2024-03-21T13:05:16.384862+0000","last_active":"2024-03-21T13:06:14.362235+0000","last_peered":"2024-03-21T13:06:14.362235+0000","last_clean":"2024-03-21T13:06:14.362235+0000","last_became_active":"2024-03-21T13:02:14.349072+0000","last_became_peered":"2024-03-21T13:02:14.349072+0000","last_unstale":"2024-03-21T13:06:14.362235+0000","last_undegraded":"2024-03-21T13:06:14.362235+0000","last_fullsized":"2024-03-21T13:06:14.362235+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T13:05:16.384676+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:05:16.384676+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:29.918580+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":287,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.358412+0000","last_change":"2024-03-21T13:04:30.145245+0000","last_active":"2024-03-21T13:06:14.358412+0000","last_peered":"2024-03-21T13:06:14.358412+0000","last_clean":"2024-03-21T13:06:14.358412+0000","last_became_active":"2024-03-21T13:02:26.226675+0000","last_became_peered":"2024-03-21T13:02:26.226675+0000","last_unstale":"2024-03-21T13:06:14.358412+0000","last_undegraded":"2024-03-21T13:06:14.358412+0000","last_fullsized":"2024-03-21T13:06:14.358412+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:30.145090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:30.145090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:15.408841+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":183,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.358850+0000","last_change":"2024-03-21T13:05:37.359133+0000","last_active":"2024-03-21T13:06:14.358850+0000","last_peered":"2024-03-21T13:06:14.358850+0000","last_clean":"2024-03-21T13:06:14.358850+0000","last_became_active":"2024-03-21T13:02:26.231879+0000","last_became_peered":"2024-03-21T13:02:26.231879+0000","last_unstale":"2024-03-21T13:06:14.358850+0000","last_undegraded":"2024-03-21T13:06:14.358850+0000","last_fullsized":"2024-03-21T13:06:14.358850+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:37.358974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:05:37.358974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:56.298751+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":175,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.352479+0000","last_change":"2024-03-21T13:01:48.210250+0000","last_active":"2024-03-21T13:06:14.352479+0000","last_peered":"2024-03-21T13:06:14.352479+0000","last_clean":"2024-03-21T13:06:14.352479+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:06:14.352479+0000","last_undegraded":"2024-03-21T13:06:14.352479+0000","last_fullsized":"2024-03-21T13:06:14.352479+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:44.005379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:44.005379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.905910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":337,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.354630+0000","last_change":"2024-03-21T13:05:20.158435+0000","last_active":"2024-03-21T13:06:14.354630+0000","last_peered":"2024-03-21T13:06:14.354630+0000","last_clean":"2024-03-21T13:06:14.354630+0000","last_became_active":"2024-03-21T13:02:14.350563+0000","last_became_peered":"2024-03-21T13:02:14.350563+0000","last_unstale":"2024-03-21T13:06:14.354630+0000","last_undegraded":"2024-03-21T13:06:14.354630+0000","last_fullsized":"2024-03-21T13:06:14.354630+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:20.158184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.158184+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:44.893552+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":271,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362897+0000","last_change":"2024-03-21T13:02:26.243438+0000","last_active":"2024-03-21T13:06:14.362897+0000","last_peered":"2024-03-21T13:06:14.362897+0000","last_clean":"2024-03-21T13:06:14.362897+0000","last_became_active":"2024-03-21T13:02:26.242482+0000","last_became_peered":"2024-03-21T13:02:26.242482+0000","last_unstale":"2024-03-21T13:06:14.362897+0000","last_undegraded":"2024-03-21T13:06:14.362897+0000","last_fullsized":"2024-03-21T13:06:14.362897+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:35.036335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.036335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":197,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362894+0000","last_change":"2024-03-21T13:04:29.464933+0000","last_active":"2024-03-21T13:06:14.362894+0000","last_peered":"2024-03-21T13:06:14.362894+0000","last_clean":"2024-03-21T13:06:14.362894+0000","last_became_active":"2024-03-21T13:01:27.366846+0000","last_became_peered":"2024-03-21T13:01:27.366846+0000","last_unstale":"2024-03-21T13:06:14.362894+0000","last_undegraded":"2024-03-21T13:06:14.362894+0000","last_fullsized":"2024-03-21T13:06:14.362894+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:29.464856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:04:29.464856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":307,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.361518+0000","last_change":"2024-03-21T13:05:39.354730+0000","last_active":"2024-03-21T13:06:14.361518+0000","last_peered":"2024-03-21T13:06:14.361518+0000","last_clean":"2024-03-21T13:06:14.361518+0000","last_became_active":"2024-03-21T13:02:14.329156+0000","last_became_peered":"2024-03-21T13:02:14.329156+0000","last_unstale":"2024-03-21T13:06:14.361518+0000","last_undegraded":"2024-03-21T13:06:14.361518+0000","last_fullsized":"2024-03-21T13:06:14.361518+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.354575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.354575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:04.801630+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'12","reported_seq":332,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.361810+0000","last_change":"2024-03-21T13:02:14.362313+0000","last_active":"2024-03-21T13:06:14.361810+0000","last_peered":"2024-03-21T13:06:14.361810+0000","last_clean":"2024-03-21T13:06:14.361810+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:06:14.361810+0000","last_undegraded":"2024-03-21T13:06:14.361810+0000","last_fullsized":"2024-03-21T13:06:14.361810+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:01:39.755345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.755345+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554477+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":112,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.354529+0000","last_change":"2024-03-21T13:05:50.295455+0000","last_active":"2024-03-21T13:06:14.354529+0000","last_peered":"2024-03-21T13:06:14.354529+0000","last_clean":"2024-03-21T13:06:14.354529+0000","last_became_active":"2024-03-21T13:04:20.359494+0000","last_became_peered":"2024-03-21T13:04:20.359494+0000","last_unstale":"2024-03-21T13:06:14.354529+0000","last_undegraded":"2024-03-21T13:06:14.354529+0000","last_fullsized":"2024-03-21T13:06:14.354529+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:50.295057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.295057+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:18.189590+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":388,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.350810+0000","last_change":"2024-03-21T13:05:38.022572+0000","last_active":"2024-03-21T13:06:14.350810+0000","last_peered":"2024-03-21T13:06:14.350810+0000","last_clean":"2024-03-21T13:06:14.350810+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:06:14.350810+0000","last_undegraded":"2024-03-21T13:06:14.350810+0000","last_fullsized":"2024-03-21T13:06:14.350810+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:05:38.022439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.022439+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:06.821939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":183,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.370738+0000","last_change":"2024-03-21T13:04:20.361810+0000","last_active":"2024-03-21T13:06:14.370738+0000","last_peered":"2024-03-21T13:06:14.370738+0000","last_clean":"2024-03-21T13:06:14.370738+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:06:14.370738+0000","last_undegraded":"2024-03-21T13:06:14.370738+0000","last_fullsized":"2024-03-21T13:06:14.370738+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:01:37.244557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:37.244557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"71'10","reported_seq":218,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.353745+0000","last_change":"2024-03-21T13:05:21.117502+0000","last_active":"2024-03-21T13:06:14.353745+0000","last_peered":"2024-03-21T13:06:14.353745+0000","last_clean":"2024-03-21T13:06:14.353745+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:06:14.353745+0000","last_undegraded":"2024-03-21T13:06:14.353745+0000","last_fullsized":"2024-03-21T13:06:14.353745+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:21.117370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.117370+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:47.662070+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":159,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.358345+0000","last_change":"2024-03-21T13:04:29.180370+0000","last_active":"2024-03-21T13:06:14.358345+0000","last_peered":"2024-03-21T13:06:14.358345+0000","last_clean":"2024-03-21T13:06:14.358345+0000","last_became_active":"2024-03-21T13:02:26.229188+0000","last_became_peered":"2024-03-21T13:02:26.229188+0000","last_unstale":"2024-03-21T13:06:14.358345+0000","last_undegraded":"2024-03-21T13:06:14.358345+0000","last_fullsized":"2024-03-21T13:06:14.358345+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:29.180240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:29.180240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:15.408794+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":184,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.353587+0000","last_change":"2024-03-21T13:05:18.088915+0000","last_active":"2024-03-21T13:06:14.353587+0000","last_peered":"2024-03-21T13:06:14.353587+0000","last_clean":"2024-03-21T13:06:14.353587+0000","last_became_active":"2024-03-21T13:01:48.215160+0000","last_became_peered":"2024-03-21T13:01:48.215160+0000","last_unstale":"2024-03-21T13:06:14.353587+0000","last_undegraded":"2024-03-21T13:06:14.353587+0000","last_fullsized":"2024-03-21T13:06:14.353587+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.088812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.088812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:30.428137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.363967+0000","last_change":"2024-03-21T13:05:09.349604+0000","last_active":"2024-03-21T13:06:14.363967+0000","last_peered":"2024-03-21T13:06:14.363967+0000","last_clean":"2024-03-21T13:06:14.363967+0000","last_became_active":"2024-03-21T13:02:14.341286+0000","last_became_peered":"2024-03-21T13:02:14.341286+0000","last_unstale":"2024-03-21T13:06:14.363967+0000","last_undegraded":"2024-03-21T13:06:14.363967+0000","last_fullsized":"2024-03-21T13:06:14.363967+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:09.348646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:09.348646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:20.112225+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'11","reported_seq":344,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.351529+0000","last_change":"2024-03-21T13:05:49.982963+0000","last_active":"2024-03-21T13:06:14.351529+0000","last_peered":"2024-03-21T13:06:14.351529+0000","last_clean":"2024-03-21T13:06:14.351529+0000","last_became_active":"2024-03-21T13:02:26.254436+0000","last_became_peered":"2024-03-21T13:02:26.254436+0000","last_unstale":"2024-03-21T13:06:14.351529+0000","last_undegraded":"2024-03-21T13:06:14.351529+0000","last_fullsized":"2024-03-21T13:06:14.351529+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:49.982812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.982812+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:14.345457+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":271,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.352252+0000","last_change":"2024-03-21T13:01:35.005181+0000","last_active":"2024-03-21T13:06:14.352252+0000","last_peered":"2024-03-21T13:06:14.352252+0000","last_clean":"2024-03-21T13:06:14.352252+0000","last_became_active":"2024-03-21T12:58:43.183475+0000","last_became_peered":"2024-03-21T12:58:43.183475+0000","last_unstale":"2024-03-21T13:06:14.352252+0000","last_undegraded":"2024-03-21T13:06:14.352252+0000","last_fullsized":"2024-03-21T13:06:14.352252+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:01:35.005026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.905892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.349945+0000","last_change":"2024-03-21T13:05:42.990138+0000","last_active":"2024-03-21T13:06:14.349945+0000","last_peered":"2024-03-21T13:06:14.349945+0000","last_clean":"2024-03-21T13:06:14.349945+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:06:14.349945+0000","last_undegraded":"2024-03-21T13:06:14.349945+0000","last_fullsized":"2024-03-21T13:06:14.349945+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:42.990022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.990022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:49.038464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":159,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.363130+0000","last_change":"2024-03-21T13:05:28.499322+0000","last_active":"2024-03-21T13:06:14.363130+0000","last_peered":"2024-03-21T13:06:14.363130+0000","last_clean":"2024-03-21T13:06:14.363130+0000","last_became_active":"2024-03-21T13:02:26.242973+0000","last_became_peered":"2024-03-21T13:02:26.242973+0000","last_unstale":"2024-03-21T13:06:14.363130+0000","last_undegraded":"2024-03-21T13:06:14.363130+0000","last_fullsized":"2024-03-21T13:06:14.363130+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.499185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.499185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:37.620969+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.364013+0000","last_change":"2024-03-21T13:02:14.369668+0000","last_active":"2024-03-21T13:06:14.364013+0000","last_peered":"2024-03-21T13:06:14.364013+0000","last_clean":"2024-03-21T13:06:14.364013+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:06:14.364013+0000","last_undegraded":"2024-03-21T13:06:14.364013+0000","last_fullsized":"2024-03-21T13:06:14.364013+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-21T13:01:30.807979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:30.807979+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554339+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362851+0000","last_change":"2024-03-21T13:05:19.431337+0000","last_active":"2024-03-21T13:06:14.362851+0000","last_peered":"2024-03-21T13:06:14.362851+0000","last_clean":"2024-03-21T13:06:14.362851+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:06:14.362851+0000","last_undegraded":"2024-03-21T13:06:14.362851+0000","last_fullsized":"2024-03-21T13:06:14.362851+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:19.430947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:19.430947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:23.467977+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":275,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.352287+0000","last_change":"2024-03-21T13:03:43.905627+0000","last_active":"2024-03-21T13:06:14.352287+0000","last_peered":"2024-03-21T13:06:14.352287+0000","last_clean":"2024-03-21T13:06:14.352287+0000","last_became_active":"2024-03-21T12:58:41.164784+0000","last_became_peered":"2024-03-21T12:58:41.164784+0000","last_unstale":"2024-03-21T13:06:14.352287+0000","last_undegraded":"2024-03-21T13:06:14.352287+0000","last_fullsized":"2024-03-21T13:06:14.352287+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:43.905509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:03:43.905509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.906037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.353482+0000","last_change":"2024-03-21T13:02:26.236549+0000","last_active":"2024-03-21T13:06:14.353482+0000","last_peered":"2024-03-21T13:06:14.353482+0000","last_clean":"2024-03-21T13:06:14.353482+0000","last_became_active":"2024-03-21T13:02:26.236235+0000","last_became_peered":"2024-03-21T13:02:26.236235+0000","last_unstale":"2024-03-21T13:06:14.353482+0000","last_undegraded":"2024-03-21T13:06:14.353482+0000","last_fullsized":"2024-03-21T13:06:14.353482+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:02:21.087308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:02:21.087308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.905951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"71'11","reported_seq":99,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.354571+0000","last_change":"2024-03-21T13:05:40.262398+0000","last_active":"2024-03-21T13:06:14.354571+0000","last_peered":"2024-03-21T13:06:14.354571+0000","last_clean":"2024-03-21T13:06:14.354571+0000","last_became_active":"2024-03-21T13:04:20.357539+0000","last_became_peered":"2024-03-21T13:04:20.357539+0000","last_unstale":"2024-03-21T13:06:14.354571+0000","last_undegraded":"2024-03-21T13:06:14.354571+0000","last_fullsized":"2024-03-21T13:06:14.354571+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:40.262277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:40.262277+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:05.063873+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":180,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.349284+0000","last_change":"2024-03-21T13:05:16.850442+0000","last_active":"2024-03-21T13:06:14.349284+0000","last_peered":"2024-03-21T13:06:14.349284+0000","last_clean":"2024-03-21T13:06:14.349284+0000","last_became_active":"2024-03-21T13:02:26.252488+0000","last_became_peered":"2024-03-21T13:02:26.252488+0000","last_unstale":"2024-03-21T13:06:14.349284+0000","last_undegraded":"2024-03-21T13:06:14.349284+0000","last_fullsized":"2024-03-21T13:06:14.349284+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:16.850307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:16.850307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:22.767349+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":289,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362462+0000","last_change":"2024-03-21T13:05:31.496412+0000","last_active":"2024-03-21T13:06:14.362462+0000","last_peered":"2024-03-21T13:06:14.362462+0000","last_clean":"2024-03-21T13:06:14.362462+0000","last_became_active":"2024-03-21T13:02:26.232682+0000","last_became_peered":"2024-03-21T13:02:26.232682+0000","last_unstale":"2024-03-21T13:06:14.362462+0000","last_undegraded":"2024-03-21T13:06:14.362462+0000","last_fullsized":"2024-03-21T13:06:14.362462+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.495639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.495639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:54.630189+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":283,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.353716+0000","last_change":"2024-03-21T13:01:43.982673+0000","last_active":"2024-03-21T13:06:14.353716+0000","last_peered":"2024-03-21T13:06:14.353716+0000","last_clean":"2024-03-21T13:06:14.353716+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:06:14.353716+0000","last_undegraded":"2024-03-21T13:06:14.353716+0000","last_fullsized":"2024-03-21T13:06:14.353716+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:43.982553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:43.982553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.905927+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"86'13","reported_seq":248,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.350142+0000","last_change":"2024-03-21T13:05:34.036587+0000","last_active":"2024-03-21T13:06:14.350142+0000","last_peered":"2024-03-21T13:06:14.350142+0000","last_clean":"2024-03-21T13:06:14.350142+0000","last_became_active":"2024-03-21T13:02:14.330196+0000","last_became_peered":"2024-03-21T13:02:14.330196+0000","last_unstale":"2024-03-21T13:06:14.350142+0000","last_undegraded":"2024-03-21T13:06:14.350142+0000","last_fullsized":"2024-03-21T13:06:14.350142+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:05:34.036430+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.036430+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:42.427384+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":118,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.351524+0000","last_change":"2024-03-21T13:05:29.031235+0000","last_active":"2024-03-21T13:06:14.351524+0000","last_peered":"2024-03-21T13:06:14.351524+0000","last_clean":"2024-03-21T13:06:14.351524+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:06:14.351524+0000","last_undegraded":"2024-03-21T13:06:14.351524+0000","last_fullsized":"2024-03-21T13:06:14.351524+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.031062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.031062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:51.141442+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":151,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.370017+0000","last_change":"2024-03-21T13:02:26.242867+0000","last_active":"2024-03-21T13:06:14.370017+0000","last_peered":"2024-03-21T13:06:14.370017+0000","last_clean":"2024-03-21T13:06:14.370017+0000","last_became_active":"2024-03-21T13:02:26.242491+0000","last_became_peered":"2024-03-21T13:02:26.242491+0000","last_unstale":"2024-03-21T13:06:14.370017+0000","last_undegraded":"2024-03-21T13:06:14.370017+0000","last_fullsized":"2024-03-21T13:06:14.370017+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.029629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.029629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.357792+0000","last_change":"2024-03-21T13:04:32.151296+0000","last_active":"2024-03-21T13:06:14.357792+0000","last_peered":"2024-03-21T13:06:14.357792+0000","last_clean":"2024-03-21T13:06:14.357792+0000","last_became_active":"2024-03-21T12:58:37.731659+0000","last_became_peered":"2024-03-21T12:58:37.731659+0000","last_unstale":"2024-03-21T13:06:14.357792+0000","last_undegraded":"2024-03-21T13:06:14.357792+0000","last_fullsized":"2024-03-21T13:06:14.357792+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:32.151217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:04:32.151217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:15.408927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":344,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.358210+0000","last_change":"2024-03-21T13:05:28.260436+0000","last_active":"2024-03-21T13:06:14.358210+0000","last_peered":"2024-03-21T13:06:14.358210+0000","last_clean":"2024-03-21T13:06:14.358210+0000","last_became_active":"2024-03-21T13:02:26.225627+0000","last_became_peered":"2024-03-21T13:02:26.225627+0000","last_unstale":"2024-03-21T13:06:14.358210+0000","last_undegraded":"2024-03-21T13:06:14.358210+0000","last_fullsized":"2024-03-21T13:06:14.358210+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:28.260333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.260333+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:28.630975+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":183,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.353423+0000","last_change":"2024-03-21T13:04:31.805808+0000","last_active":"2024-03-21T13:06:14.353423+0000","last_peered":"2024-03-21T13:06:14.353423+0000","last_clean":"2024-03-21T13:06:14.353423+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:06:14.353423+0000","last_undegraded":"2024-03-21T13:06:14.353423+0000","last_fullsized":"2024-03-21T13:06:14.353423+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:31.805690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:31.805690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:14.896377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"125'10","reported_seq":378,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.361439+0000","last_change":"2024-03-21T13:02:26.247826+0000","last_active":"2024-03-21T13:06:14.361439+0000","last_peered":"2024-03-21T13:06:14.361439+0000","last_clean":"2024-03-21T13:06:14.361439+0000","last_became_active":"2024-03-21T13:02:26.234494+0000","last_became_peered":"2024-03-21T13:02:26.234494+0000","last_unstale":"2024-03-21T13:06:14.361439+0000","last_undegraded":"2024-03-21T13:06:14.361439+0000","last_fullsized":"2024-03-21T13:06:14.361439+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:23.867564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:23.867564+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554257+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.351959+0000","last_change":"2024-03-21T13:05:31.042470+0000","last_active":"2024-03-21T13:06:14.351959+0000","last_peered":"2024-03-21T13:06:14.351959+0000","last_clean":"2024-03-21T13:06:14.351959+0000","last_became_active":"2024-03-21T12:58:37.731988+0000","last_became_peered":"2024-03-21T12:58:37.731988+0000","last_unstale":"2024-03-21T13:06:14.351959+0000","last_undegraded":"2024-03-21T13:06:14.351959+0000","last_fullsized":"2024-03-21T13:06:14.351959+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.042372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.042372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:38.456469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"71'10","reported_seq":314,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.363079+0000","last_change":"2024-03-21T13:05:23.379150+0000","last_active":"2024-03-21T13:06:14.363079+0000","last_peered":"2024-03-21T13:06:14.363079+0000","last_clean":"2024-03-21T13:06:14.363079+0000","last_became_active":"2024-03-21T13:02:14.353938+0000","last_became_peered":"2024-03-21T13:02:14.353938+0000","last_unstale":"2024-03-21T13:06:14.363079+0000","last_undegraded":"2024-03-21T13:06:14.363079+0000","last_fullsized":"2024-03-21T13:06:14.363079+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:23.379004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.379004+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:45.392478+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":287,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.358667+0000","last_change":"2024-03-21T13:05:21.157416+0000","last_active":"2024-03-21T13:06:14.358667+0000","last_peered":"2024-03-21T13:06:14.358667+0000","last_clean":"2024-03-21T13:06:14.358667+0000","last_became_active":"2024-03-21T13:02:26.233049+0000","last_became_peered":"2024-03-21T13:02:26.233049+0000","last_unstale":"2024-03-21T13:06:14.358667+0000","last_undegraded":"2024-03-21T13:06:14.358667+0000","last_fullsized":"2024-03-21T13:06:14.358667+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:21.157295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.157295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:33.608380+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":369,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.363178+0000","last_change":"2024-03-21T13:01:18.526684+0000","last_active":"2024-03-21T13:06:14.363178+0000","last_peered":"2024-03-21T13:06:14.363178+0000","last_clean":"2024-03-21T13:06:14.363178+0000","last_became_active":"2024-03-21T12:58:41.167826+0000","last_became_peered":"2024-03-21T12:58:41.167826+0000","last_unstale":"2024-03-21T13:06:14.363178+0000","last_undegraded":"2024-03-21T13:06:14.363178+0000","last_fullsized":"2024-03-21T13:06:14.363178+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:18.526428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.526428+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554179+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":183,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.352024+0000","last_change":"2024-03-21T13:04:12.819697+0000","last_active":"2024-03-21T13:06:14.352024+0000","last_peered":"2024-03-21T13:06:14.352024+0000","last_clean":"2024-03-21T13:06:14.352024+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:06:14.352024+0000","last_undegraded":"2024-03-21T13:06:14.352024+0000","last_fullsized":"2024-03-21T13:06:14.352024+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:12.819556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:04:12.819556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.906223+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"71'9","reported_seq":202,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.353945+0000","last_change":"2024-03-21T13:04:13.827838+0000","last_active":"2024-03-21T13:06:14.353945+0000","last_peered":"2024-03-21T13:06:14.353945+0000","last_clean":"2024-03-21T13:06:14.353945+0000","last_became_active":"2024-03-21T13:02:26.239161+0000","last_became_peered":"2024-03-21T13:02:26.239161+0000","last_unstale":"2024-03-21T13:06:14.353945+0000","last_undegraded":"2024-03-21T13:06:14.353945+0000","last_fullsized":"2024-03-21T13:06:14.353945+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:04:13.827615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:13.827615+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.906120+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":290,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.354996+0000","last_change":"2024-03-21T13:05:42.187570+0000","last_active":"2024-03-21T13:06:14.354996+0000","last_peered":"2024-03-21T13:06:14.354996+0000","last_clean":"2024-03-21T13:06:14.354996+0000","last_became_active":"2024-03-21T12:58:43.182126+0000","last_became_peered":"2024-03-21T12:58:43.182126+0000","last_unstale":"2024-03-21T13:06:14.354996+0000","last_undegraded":"2024-03-21T13:06:14.354996+0000","last_fullsized":"2024-03-21T13:06:14.354996+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:42.187461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.187461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:07.563556+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":283,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.364283+0000","last_change":"2024-03-21T13:05:17.347764+0000","last_active":"2024-03-21T13:06:14.364283+0000","last_peered":"2024-03-21T13:06:14.364283+0000","last_clean":"2024-03-21T13:06:14.364283+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:06:14.364283+0000","last_undegraded":"2024-03-21T13:06:14.364283+0000","last_fullsized":"2024-03-21T13:06:14.364283+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:17.347604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:17.347604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:21.115130+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.352179+0000","last_change":"2024-03-21T13:05:23.049951+0000","last_active":"2024-03-21T13:06:14.352179+0000","last_peered":"2024-03-21T13:06:14.352179+0000","last_clean":"2024-03-21T13:06:14.352179+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:06:14.352179+0000","last_undegraded":"2024-03-21T13:06:14.352179+0000","last_fullsized":"2024-03-21T13:06:14.352179+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:05:23.049811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.049811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:26.798813+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'9","reported_seq":210,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362047+0000","last_change":"2024-03-21T13:01:48.210983+0000","last_active":"2024-03-21T13:06:14.362047+0000","last_peered":"2024-03-21T13:06:14.362047+0000","last_clean":"2024-03-21T13:06:14.362047+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:06:14.362047+0000","last_undegraded":"2024-03-21T13:06:14.362047+0000","last_fullsized":"2024-03-21T13:06:14.362047+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:01:40.964615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.964615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":271,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.353295+0000","last_change":"2024-03-21T13:03:41.952266+0000","last_active":"2024-03-21T13:06:14.353295+0000","last_peered":"2024-03-21T13:06:14.353295+0000","last_clean":"2024-03-21T13:06:14.353295+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:06:14.353295+0000","last_undegraded":"2024-03-21T13:06:14.353295+0000","last_fullsized":"2024-03-21T13:06:14.353295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:41.952189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:03:41.952189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.905979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362310+0000","last_change":"2024-03-21T13:05:25.466812+0000","last_active":"2024-03-21T13:06:14.362310+0000","last_peered":"2024-03-21T13:06:14.362310+0000","last_clean":"2024-03-21T13:06:14.362310+0000","last_became_active":"2024-03-21T13:02:26.242502+0000","last_became_peered":"2024-03-21T13:02:26.242502+0000","last_unstale":"2024-03-21T13:06:14.362310+0000","last_undegraded":"2024-03-21T13:06:14.362310+0000","last_fullsized":"2024-03-21T13:06:14.362310+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:25.466695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:25.466695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:39.968442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":299,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.349401+0000","last_change":"2024-03-21T13:05:41.004652+0000","last_active":"2024-03-21T13:06:14.349401+0000","last_peered":"2024-03-21T13:06:14.349401+0000","last_clean":"2024-03-21T13:06:14.349401+0000","last_became_active":"2024-03-21T13:02:26.252406+0000","last_became_peered":"2024-03-21T13:02:26.252406+0000","last_unstale":"2024-03-21T13:06:14.349401+0000","last_undegraded":"2024-03-21T13:06:14.349401+0000","last_fullsized":"2024-03-21T13:06:14.349401+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.004412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.004412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:02.118773+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'6","reported_seq":304,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.363360+0000","last_change":"2024-03-21T13:04:22.563945+0000","last_active":"2024-03-21T13:06:14.363360+0000","last_peered":"2024-03-21T13:06:14.363360+0000","last_clean":"2024-03-21T13:06:14.363360+0000","last_became_active":"2024-03-21T13:02:14.352117+0000","last_became_peered":"2024-03-21T13:02:14.352117+0000","last_unstale":"2024-03-21T13:06:14.363360+0000","last_undegraded":"2024-03-21T13:06:14.363360+0000","last_fullsized":"2024-03-21T13:06:14.363360+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T13:04:22.563861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:22.563861+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"192'7482","reported_seq":7540,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.685817+0000","last_change":"2024-03-21T13:04:46.254330+0000","last_active":"2024-03-21T13:06:14.685817+0000","last_peered":"2024-03-21T13:06:14.685817+0000","last_clean":"2024-03-21T13:06:14.685817+0000","last_became_active":"2024-03-21T13:04:46.252954+0000","last_became_peered":"2024-03-21T13:04:46.252954+0000","last_unstale":"2024-03-21T13:06:14.685817+0000","last_undegraded":"2024-03-21T13:06:14.685817+0000","last_fullsized":"2024-03-21T13:06:14.685817+0000","mapping_epoch":168,"log_start":"180'3500","ondisk_log_start":"180'3500","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3982,"log_dups_size":0,"ondisk_log_size":3982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:15.413520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490340352,"num_objects":7482,"num_object_clones":0,"num_object_copies":22446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7482,"num_write_kb":478848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":157,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.349601+0000","last_change":"2024-03-21T13:05:47.973945+0000","last_active":"2024-03-21T13:06:14.349601+0000","last_peered":"2024-03-21T13:06:14.349601+0000","last_clean":"2024-03-21T13:06:14.349601+0000","last_became_active":"2024-03-21T13:02:26.252904+0000","last_became_peered":"2024-03-21T13:02:26.252904+0000","last_unstale":"2024-03-21T13:06:14.349601+0000","last_undegraded":"2024-03-21T13:06:14.349601+0000","last_fullsized":"2024-03-21T13:06:14.349601+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.973796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.973796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:13.525530+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":315,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.352923+0000","last_change":"2024-03-21T13:05:44.816505+0000","last_active":"2024-03-21T13:06:14.352923+0000","last_peered":"2024-03-21T13:06:14.352923+0000","last_clean":"2024-03-21T13:06:14.352923+0000","last_became_active":"2024-03-21T13:04:20.359078+0000","last_became_peered":"2024-03-21T13:04:20.359078+0000","last_unstale":"2024-03-21T13:06:14.352923+0000","last_undegraded":"2024-03-21T13:06:14.352923+0000","last_fullsized":"2024-03-21T13:06:14.352923+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:05:44.816353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:44.816353+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:45.988130+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'15","reported_seq":217,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.351738+0000","last_change":"2024-03-21T13:04:25.787896+0000","last_active":"2024-03-21T13:06:14.351738+0000","last_peered":"2024-03-21T13:06:14.351738+0000","last_clean":"2024-03-21T13:06:14.351738+0000","last_became_active":"2024-03-21T13:02:26.237800+0000","last_became_peered":"2024-03-21T13:02:26.237800+0000","last_unstale":"2024-03-21T13:06:14.351738+0000","last_undegraded":"2024-03-21T13:06:14.351738+0000","last_fullsized":"2024-03-21T13:06:14.351738+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:04:25.787703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:25.787703+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.906195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":376,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.352101+0000","last_change":"2024-03-21T13:04:16.810359+0000","last_active":"2024-03-21T13:06:14.352101+0000","last_peered":"2024-03-21T13:06:14.352101+0000","last_clean":"2024-03-21T13:06:14.352101+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:06:14.352101+0000","last_undegraded":"2024-03-21T13:06:14.352101+0000","last_fullsized":"2024-03-21T13:06:14.352101+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:04:16.810269+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:04:16.810269+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.906241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"192'7379","reported_seq":7437,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.741216+0000","last_change":"2024-03-21T13:04:46.259252+0000","last_active":"2024-03-21T13:06:14.741216+0000","last_peered":"2024-03-21T13:06:14.741216+0000","last_clean":"2024-03-21T13:06:14.741216+0000","last_became_active":"2024-03-21T13:04:46.258930+0000","last_became_peered":"2024-03-21T13:04:46.258930+0000","last_unstale":"2024-03-21T13:06:14.741216+0000","last_undegraded":"2024-03-21T13:06:14.741216+0000","last_fullsized":"2024-03-21T13:06:14.741216+0000","mapping_epoch":168,"log_start":"179'3200","ondisk_log_start":"179'3200","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4179,"log_dups_size":0,"ondisk_log_size":4179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.118409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483590144,"num_objects":7379,"num_object_clones":0,"num_object_copies":22137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7379,"num_write_kb":472256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":279,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362759+0000","last_change":"2024-03-21T13:05:43.596914+0000","last_active":"2024-03-21T13:06:14.362759+0000","last_peered":"2024-03-21T13:06:14.362759+0000","last_clean":"2024-03-21T13:06:14.362759+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:06:14.362759+0000","last_undegraded":"2024-03-21T13:06:14.362759+0000","last_fullsized":"2024-03-21T13:06:14.362759+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.596691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:11.157166+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.353588+0000","last_change":"2024-03-21T13:05:09.138059+0000","last_active":"2024-03-21T13:06:14.353588+0000","last_peered":"2024-03-21T13:06:14.353588+0000","last_clean":"2024-03-21T13:06:14.353588+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:06:14.353588+0000","last_undegraded":"2024-03-21T13:06:14.353588+0000","last_fullsized":"2024-03-21T13:06:14.353588+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:09.137868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:09.137868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:30.933204+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"71'12","reported_seq":215,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.370832+0000","last_change":"2024-03-21T13:02:14.383361+0000","last_active":"2024-03-21T13:06:14.370832+0000","last_peered":"2024-03-21T13:06:14.370832+0000","last_clean":"2024-03-21T13:06:14.370832+0000","last_became_active":"2024-03-21T13:02:14.347102+0000","last_became_peered":"2024-03-21T13:02:14.347102+0000","last_unstale":"2024-03-21T13:06:14.370832+0000","last_undegraded":"2024-03-21T13:06:14.370832+0000","last_fullsized":"2024-03-21T13:06:14.370832+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:00:32.864608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.864608+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.354193+0000","last_change":"2024-03-21T13:05:21.171124+0000","last_active":"2024-03-21T13:06:14.354193+0000","last_peered":"2024-03-21T13:06:14.354193+0000","last_clean":"2024-03-21T13:06:14.354193+0000","last_became_active":"2024-03-21T13:02:26.235083+0000","last_became_peered":"2024-03-21T13:02:26.235083+0000","last_unstale":"2024-03-21T13:06:14.354193+0000","last_undegraded":"2024-03-21T13:06:14.354193+0000","last_fullsized":"2024-03-21T13:06:14.354193+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:21.170999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.170999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:38.625718+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"192'7500","reported_seq":7558,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.730608+0000","last_change":"2024-03-21T13:04:46.257959+0000","last_active":"2024-03-21T13:06:14.730608+0000","last_peered":"2024-03-21T13:06:14.730608+0000","last_clean":"2024-03-21T13:06:14.730608+0000","last_became_active":"2024-03-21T13:04:46.257488+0000","last_became_peered":"2024-03-21T13:04:46.257488+0000","last_unstale":"2024-03-21T13:06:14.730608+0000","last_undegraded":"2024-03-21T13:06:14.730608+0000","last_fullsized":"2024-03-21T13:06:14.730608+0000","mapping_epoch":168,"log_start":"179'3300","ondisk_log_start":"179'3300","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.118390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491520000,"num_objects":7500,"num_object_clones":0,"num_object_copies":22500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7500,"num_write_kb":480000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":279,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.351416+0000","last_change":"2024-03-21T13:04:34.848696+0000","last_active":"2024-03-21T13:06:14.351416+0000","last_peered":"2024-03-21T13:06:14.351416+0000","last_clean":"2024-03-21T13:06:14.351416+0000","last_became_active":"2024-03-21T12:58:43.179007+0000","last_became_peered":"2024-03-21T12:58:43.179007+0000","last_unstale":"2024-03-21T13:06:14.351416+0000","last_undegraded":"2024-03-21T13:06:14.351416+0000","last_fullsized":"2024-03-21T13:06:14.351416+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:34.848616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:34.848616+0000","last_clean_scrub_stamp":"2024-03-21T13:04:34.848616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:14.896345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":292,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.356322+0000","last_change":"2024-03-21T13:05:28.216804+0000","last_active":"2024-03-21T13:06:14.356322+0000","last_peered":"2024-03-21T13:06:14.356322+0000","last_clean":"2024-03-21T13:06:14.356322+0000","last_became_active":"2024-03-21T13:02:26.238116+0000","last_became_peered":"2024-03-21T13:02:26.238116+0000","last_unstale":"2024-03-21T13:06:14.356322+0000","last_undegraded":"2024-03-21T13:06:14.356322+0000","last_fullsized":"2024-03-21T13:06:14.356322+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.216547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.216547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:43.002064+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.353948+0000","last_change":"2024-03-21T13:05:22.078652+0000","last_active":"2024-03-21T13:06:14.353948+0000","last_peered":"2024-03-21T13:06:14.353948+0000","last_clean":"2024-03-21T13:06:14.353948+0000","last_became_active":"2024-03-21T13:02:26.236780+0000","last_became_peered":"2024-03-21T13:02:26.236780+0000","last_unstale":"2024-03-21T13:06:14.353948+0000","last_undegraded":"2024-03-21T13:06:14.353948+0000","last_fullsized":"2024-03-21T13:06:14.353948+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:22.078546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.078546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:48.094404+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"71'19","reported_seq":372,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.349803+0000","last_change":"2024-03-21T13:05:29.992376+0000","last_active":"2024-03-21T13:06:14.349803+0000","last_peered":"2024-03-21T13:06:14.349803+0000","last_clean":"2024-03-21T13:06:14.349803+0000","last_became_active":"2024-03-21T13:02:26.240350+0000","last_became_peered":"2024-03-21T13:02:26.240350+0000","last_unstale":"2024-03-21T13:06:14.349803+0000","last_undegraded":"2024-03-21T13:06:14.349803+0000","last_fullsized":"2024-03-21T13:06:14.349803+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:05:29.992207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.992207+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:55.257135+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"192'7356","reported_seq":7413,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:15.339770+0000","last_change":"2024-03-21T13:04:46.263341+0000","last_active":"2024-03-21T13:06:15.339770+0000","last_peered":"2024-03-21T13:06:15.339770+0000","last_clean":"2024-03-21T13:06:15.339770+0000","last_became_active":"2024-03-21T13:04:46.259554+0000","last_became_peered":"2024-03-21T13:04:46.259554+0000","last_unstale":"2024-03-21T13:06:15.339770+0000","last_undegraded":"2024-03-21T13:06:15.339770+0000","last_fullsized":"2024-03-21T13:06:15.339770+0000","mapping_epoch":168,"log_start":"181'3700","ondisk_log_start":"181'3700","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3656,"log_dups_size":0,"ondisk_log_size":3656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.555779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482082816,"num_objects":7356,"num_object_clones":0,"num_object_copies":22068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7356,"num_write_kb":470784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":184,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.351680+0000","last_change":"2024-03-21T13:05:39.926952+0000","last_active":"2024-03-21T13:06:14.351680+0000","last_peered":"2024-03-21T13:06:14.351680+0000","last_clean":"2024-03-21T13:06:14.351680+0000","last_became_active":"2024-03-21T13:01:48.213995+0000","last_became_peered":"2024-03-21T13:01:48.213995+0000","last_unstale":"2024-03-21T13:06:14.351680+0000","last_undegraded":"2024-03-21T13:06:14.351680+0000","last_fullsized":"2024-03-21T13:06:14.351680+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.926828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.926828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:57.400154+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.369235+0000","last_change":"2024-03-21T13:05:37.225792+0000","last_active":"2024-03-21T13:06:14.369235+0000","last_peered":"2024-03-21T13:06:14.369235+0000","last_clean":"2024-03-21T13:06:14.369235+0000","last_became_active":"2024-03-21T12:58:41.163843+0000","last_became_peered":"2024-03-21T12:58:41.163843+0000","last_unstale":"2024-03-21T13:06:14.369235+0000","last_undegraded":"2024-03-21T13:06:14.369235+0000","last_fullsized":"2024-03-21T13:06:14.369235+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:37.225558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:37.225558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:44.100517+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":149,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362985+0000","last_change":"2024-03-21T13:02:26.244355+0000","last_active":"2024-03-21T13:06:14.362985+0000","last_peered":"2024-03-21T13:06:14.362985+0000","last_clean":"2024-03-21T13:06:14.362985+0000","last_became_active":"2024-03-21T13:02:26.243974+0000","last_became_peered":"2024-03-21T13:02:26.243974+0000","last_unstale":"2024-03-21T13:06:14.362985+0000","last_undegraded":"2024-03-21T13:06:14.362985+0000","last_fullsized":"2024-03-21T13:06:14.362985+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:33.998382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:33.998382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"86'13","reported_seq":247,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.357919+0000","last_change":"2024-03-21T13:05:08.081782+0000","last_active":"2024-03-21T13:06:14.357919+0000","last_peered":"2024-03-21T13:06:14.357919+0000","last_clean":"2024-03-21T13:06:14.357919+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:06:14.357919+0000","last_undegraded":"2024-03-21T13:06:14.357919+0000","last_fullsized":"2024-03-21T13:06:14.357919+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:05:08.081600+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:05:08.081600+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:19.313431+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.8","version":"192'7533","reported_seq":7591,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:15.311987+0000","last_change":"2024-03-21T13:04:46.257489+0000","last_active":"2024-03-21T13:06:15.311987+0000","last_peered":"2024-03-21T13:06:15.311987+0000","last_clean":"2024-03-21T13:06:15.311987+0000","last_became_active":"2024-03-21T13:04:46.257160+0000","last_became_peered":"2024-03-21T13:04:46.257160+0000","last_unstale":"2024-03-21T13:06:15.311987+0000","last_undegraded":"2024-03-21T13:06:15.311987+0000","last_fullsized":"2024-03-21T13:06:15.311987+0000","mapping_epoch":168,"log_start":"181'3900","ondisk_log_start":"181'3900","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3633,"log_dups_size":0,"ondisk_log_size":3633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493682688,"num_objects":7533,"num_object_clones":0,"num_object_copies":22599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7533,"num_write_kb":482112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":265,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362395+0000","last_change":"2024-03-21T13:01:46.738822+0000","last_active":"2024-03-21T13:06:14.362395+0000","last_peered":"2024-03-21T13:06:14.362395+0000","last_clean":"2024-03-21T13:06:14.362395+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:06:14.362395+0000","last_undegraded":"2024-03-21T13:06:14.362395+0000","last_fullsized":"2024-03-21T13:06:14.362395+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.738659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554529+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":188,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.349703+0000","last_change":"2024-03-21T13:05:43.961181+0000","last_active":"2024-03-21T13:06:14.349703+0000","last_peered":"2024-03-21T13:06:14.349703+0000","last_clean":"2024-03-21T13:06:14.349703+0000","last_became_active":"2024-03-21T13:02:26.243692+0000","last_became_peered":"2024-03-21T13:02:26.243692+0000","last_unstale":"2024-03-21T13:06:14.349703+0000","last_undegraded":"2024-03-21T13:06:14.349703+0000","last_fullsized":"2024-03-21T13:06:14.349703+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.961057+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:04.863341+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":301,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.358016+0000","last_change":"2024-03-21T13:05:48.266178+0000","last_active":"2024-03-21T13:06:14.358016+0000","last_peered":"2024-03-21T13:06:14.358016+0000","last_clean":"2024-03-21T13:06:14.358016+0000","last_became_active":"2024-03-21T12:58:37.729454+0000","last_became_peered":"2024-03-21T12:58:37.729454+0000","last_unstale":"2024-03-21T13:06:14.358016+0000","last_undegraded":"2024-03-21T13:06:14.358016+0000","last_fullsized":"2024-03-21T13:06:14.358016+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:48.265982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:48.265982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:50.943082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"119'29","reported_seq":309,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.354863+0000","last_change":"2024-03-21T13:05:22.202039+0000","last_active":"2024-03-21T13:06:14.354863+0000","last_peered":"2024-03-21T13:06:14.354863+0000","last_clean":"2024-03-21T13:06:14.354863+0000","last_became_active":"2024-03-21T13:02:14.351114+0000","last_became_peered":"2024-03-21T13:02:14.351114+0000","last_unstale":"2024-03-21T13:06:14.354863+0000","last_undegraded":"2024-03-21T13:06:14.354863+0000","last_fullsized":"2024-03-21T13:06:14.354863+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'29","last_scrub_stamp":"2024-03-21T13:05:22.201879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.201879+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:32.239279+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":179,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.358433+0000","last_change":"2024-03-21T13:05:34.312717+0000","last_active":"2024-03-21T13:06:14.358433+0000","last_peered":"2024-03-21T13:06:14.358433+0000","last_clean":"2024-03-21T13:06:14.358433+0000","last_became_active":"2024-03-21T13:02:14.336177+0000","last_became_peered":"2024-03-21T13:02:14.336177+0000","last_unstale":"2024-03-21T13:06:14.358433+0000","last_undegraded":"2024-03-21T13:06:14.358433+0000","last_fullsized":"2024-03-21T13:06:14.358433+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.312583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.312583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:40.632357+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.b","version":"192'7447","reported_seq":7505,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:15.299501+0000","last_change":"2024-03-21T13:04:46.259472+0000","last_active":"2024-03-21T13:06:15.299501+0000","last_peered":"2024-03-21T13:06:15.299501+0000","last_clean":"2024-03-21T13:06:15.299501+0000","last_became_active":"2024-03-21T13:04:46.259164+0000","last_became_peered":"2024-03-21T13:04:46.259164+0000","last_unstale":"2024-03-21T13:06:15.299501+0000","last_undegraded":"2024-03-21T13:06:15.299501+0000","last_fullsized":"2024-03-21T13:06:15.299501+0000","mapping_epoch":168,"log_start":"181'3800","ondisk_log_start":"181'3800","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3647,"log_dups_size":0,"ondisk_log_size":3647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488046592,"num_objects":7447,"num_object_clones":0,"num_object_copies":22341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7447,"num_write_kb":476608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":279,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.354066+0000","last_change":"2024-03-21T13:05:08.172825+0000","last_active":"2024-03-21T13:06:14.354066+0000","last_peered":"2024-03-21T13:06:14.354066+0000","last_clean":"2024-03-21T13:06:14.354066+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:06:14.354066+0000","last_undegraded":"2024-03-21T13:06:14.354066+0000","last_fullsized":"2024-03-21T13:06:14.354066+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:08.172705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:08.172705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.906315+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":178,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.354392+0000","last_change":"2024-03-21T13:05:39.247223+0000","last_active":"2024-03-21T13:06:14.354392+0000","last_peered":"2024-03-21T13:06:14.354392+0000","last_clean":"2024-03-21T13:06:14.354392+0000","last_became_active":"2024-03-21T13:02:14.328020+0000","last_became_peered":"2024-03-21T13:02:14.328020+0000","last_unstale":"2024-03-21T13:06:14.354392+0000","last_undegraded":"2024-03-21T13:06:14.354392+0000","last_fullsized":"2024-03-21T13:06:14.354392+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.247107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.247107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:49.706890+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":311,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.354826+0000","last_change":"2024-03-21T13:05:46.263820+0000","last_active":"2024-03-21T13:06:14.354826+0000","last_peered":"2024-03-21T13:06:14.354826+0000","last_clean":"2024-03-21T13:06:14.354826+0000","last_became_active":"2024-03-21T12:58:36.724067+0000","last_became_peered":"2024-03-21T12:58:36.724067+0000","last_unstale":"2024-03-21T13:06:14.354826+0000","last_undegraded":"2024-03-21T13:06:14.354826+0000","last_fullsized":"2024-03-21T13:06:14.354826+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:46.263644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:05:46.263644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:51.823127+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.000743678,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'13","reported_seq":217,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.352578+0000","last_change":"2024-03-21T13:05:15.119776+0000","last_active":"2024-03-21T13:06:14.352578+0000","last_peered":"2024-03-21T13:06:14.352578+0000","last_clean":"2024-03-21T13:06:14.352578+0000","last_became_active":"2024-03-21T13:02:26.234230+0000","last_became_peered":"2024-03-21T13:02:26.234230+0000","last_unstale":"2024-03-21T13:06:14.352578+0000","last_undegraded":"2024-03-21T13:06:14.352578+0000","last_fullsized":"2024-03-21T13:06:14.352578+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'13","last_scrub_stamp":"2024-03-21T13:05:15.119643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:15.119643+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:20.975727+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":292,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.358069+0000","last_change":"2024-03-21T13:05:23.243712+0000","last_active":"2024-03-21T13:06:14.358069+0000","last_peered":"2024-03-21T13:06:14.358069+0000","last_clean":"2024-03-21T13:06:14.358069+0000","last_became_active":"2024-03-21T13:02:26.228128+0000","last_became_peered":"2024-03-21T13:02:26.228128+0000","last_unstale":"2024-03-21T13:06:14.358069+0000","last_undegraded":"2024-03-21T13:06:14.358069+0000","last_fullsized":"2024-03-21T13:06:14.358069+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:23.243468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.243468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:45.611291+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"192'7617","reported_seq":7675,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.703030+0000","last_change":"2024-03-21T13:04:46.260725+0000","last_active":"2024-03-21T13:06:14.703030+0000","last_peered":"2024-03-21T13:06:14.703030+0000","last_clean":"2024-03-21T13:06:14.703030+0000","last_became_active":"2024-03-21T13:04:46.260364+0000","last_became_peered":"2024-03-21T13:04:46.260364+0000","last_unstale":"2024-03-21T13:06:14.703030+0000","last_undegraded":"2024-03-21T13:06:14.703030+0000","last_fullsized":"2024-03-21T13:06:14.703030+0000","mapping_epoch":168,"log_start":"179'3500","ondisk_log_start":"179'3500","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4117,"log_dups_size":0,"ondisk_log_size":4117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.118444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499187712,"num_objects":7617,"num_object_clones":0,"num_object_copies":22851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7617,"num_write_kb":487488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":157,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.363908+0000","last_change":"2024-03-21T13:04:27.549179+0000","last_active":"2024-03-21T13:06:14.363908+0000","last_peered":"2024-03-21T13:06:14.363908+0000","last_clean":"2024-03-21T13:06:14.363908+0000","last_became_active":"2024-03-21T13:02:26.235640+0000","last_became_peered":"2024-03-21T13:02:26.235640+0000","last_unstale":"2024-03-21T13:06:14.363908+0000","last_undegraded":"2024-03-21T13:06:14.363908+0000","last_fullsized":"2024-03-21T13:06:14.363908+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:27.549086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:27.549086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554660+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":540,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.351592+0000","last_change":"2024-03-21T13:04:11.818847+0000","last_active":"2024-03-21T13:06:14.351592+0000","last_peered":"2024-03-21T13:06:14.351592+0000","last_clean":"2024-03-21T13:06:14.351592+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:06:14.351592+0000","last_undegraded":"2024-03-21T13:06:14.351592+0000","last_fullsized":"2024-03-21T13:06:14.351592+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:04:11.818749+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:00:16.180126+0000","last_clean_scrub_stamp":"2024-03-21T13:04:11.818749+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.906085+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":283,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.369773+0000","last_change":"2024-03-21T13:02:26.242969+0000","last_active":"2024-03-21T13:06:14.369773+0000","last_peered":"2024-03-21T13:06:14.369773+0000","last_clean":"2024-03-21T13:06:14.369773+0000","last_became_active":"2024-03-21T13:02:26.242456+0000","last_became_peered":"2024-03-21T13:02:26.242456+0000","last_unstale":"2024-03-21T13:06:14.369773+0000","last_undegraded":"2024-03-21T13:06:14.369773+0000","last_fullsized":"2024-03-21T13:06:14.369773+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:38.007486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.007486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"183'24","reported_seq":411,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.352085+0000","last_change":"2024-03-21T13:05:42.033620+0000","last_active":"2024-03-21T13:06:14.352085+0000","last_peered":"2024-03-21T13:06:14.352085+0000","last_clean":"2024-03-21T13:06:14.352085+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:06:14.352085+0000","last_undegraded":"2024-03-21T13:06:14.352085+0000","last_fullsized":"2024-03-21T13:06:14.352085+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'24","last_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.033266+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:08.844091+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":289,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.353075+0000","last_change":"2024-03-21T13:05:20.061939+0000","last_active":"2024-03-21T13:06:14.353075+0000","last_peered":"2024-03-21T13:06:14.353075+0000","last_clean":"2024-03-21T13:06:14.353075+0000","last_became_active":"2024-03-21T13:02:26.235140+0000","last_became_peered":"2024-03-21T13:02:26.235140+0000","last_unstale":"2024-03-21T13:06:14.353075+0000","last_undegraded":"2024-03-21T13:06:14.353075+0000","last_fullsized":"2024-03-21T13:06:14.353075+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:20.061709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.061709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:27.413347+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"192'7472","reported_seq":7530,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.925913+0000","last_change":"2024-03-21T13:04:46.262137+0000","last_active":"2024-03-21T13:06:14.925913+0000","last_peered":"2024-03-21T13:06:14.925913+0000","last_clean":"2024-03-21T13:06:14.925913+0000","last_became_active":"2024-03-21T13:04:46.260660+0000","last_became_peered":"2024-03-21T13:04:46.260660+0000","last_unstale":"2024-03-21T13:06:14.925913+0000","last_undegraded":"2024-03-21T13:06:14.925913+0000","last_fullsized":"2024-03-21T13:06:14.925913+0000","mapping_epoch":168,"log_start":"178'2900","ondisk_log_start":"178'2900","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4572,"log_dups_size":0,"ondisk_log_size":4572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:16.255288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489684992,"num_objects":7472,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7472,"num_write_kb":478208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":159,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362664+0000","last_change":"2024-03-21T13:05:29.457708+0000","last_active":"2024-03-21T13:06:14.362664+0000","last_peered":"2024-03-21T13:06:14.362664+0000","last_clean":"2024-03-21T13:06:14.362664+0000","last_became_active":"2024-03-21T13:02:26.242708+0000","last_became_peered":"2024-03-21T13:02:26.242708+0000","last_unstale":"2024-03-21T13:06:14.362664+0000","last_undegraded":"2024-03-21T13:06:14.362664+0000","last_fullsized":"2024-03-21T13:06:14.362664+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.457597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.457597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:31.514232+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":184,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.357952+0000","last_change":"2024-03-21T13:05:11.035789+0000","last_active":"2024-03-21T13:06:14.357952+0000","last_peered":"2024-03-21T13:06:14.357952+0000","last_clean":"2024-03-21T13:06:14.357952+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:06:14.357952+0000","last_undegraded":"2024-03-21T13:06:14.357952+0000","last_fullsized":"2024-03-21T13:06:14.357952+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:11.035680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:05:11.035680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:40.978396+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":753,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.351543+0000","last_change":"2024-03-21T13:03:45.889240+0000","last_active":"2024-03-21T13:06:14.351543+0000","last_peered":"2024-03-21T13:06:14.351543+0000","last_clean":"2024-03-21T13:06:14.351543+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:06:14.351543+0000","last_undegraded":"2024-03-21T13:06:14.351543+0000","last_fullsized":"2024-03-21T13:06:14.351543+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:03:45.889083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:03:45.889083+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.906065+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":448,"num_read_kb":443,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":334,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.353966+0000","last_change":"2024-03-21T13:05:11.320104+0000","last_active":"2024-03-21T13:06:14.353966+0000","last_peered":"2024-03-21T13:06:14.353966+0000","last_clean":"2024-03-21T13:06:14.353966+0000","last_became_active":"2024-03-21T13:02:14.333917+0000","last_became_peered":"2024-03-21T13:02:14.333917+0000","last_unstale":"2024-03-21T13:06:14.353966+0000","last_undegraded":"2024-03-21T13:06:14.353966+0000","last_fullsized":"2024-03-21T13:06:14.353966+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:05:11.319970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:11.319970+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:28.136189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":156,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.353032+0000","last_change":"2024-03-21T13:05:13.085740+0000","last_active":"2024-03-21T13:06:14.353032+0000","last_peered":"2024-03-21T13:06:14.353032+0000","last_clean":"2024-03-21T13:06:14.353032+0000","last_became_active":"2024-03-21T13:02:26.232625+0000","last_became_peered":"2024-03-21T13:02:26.232625+0000","last_unstale":"2024-03-21T13:06:14.353032+0000","last_undegraded":"2024-03-21T13:06:14.353032+0000","last_fullsized":"2024-03-21T13:06:14.353032+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.085625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.085625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:22.265367+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"192'7386","reported_seq":7444,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:15.285771+0000","last_change":"2024-03-21T13:04:46.262924+0000","last_active":"2024-03-21T13:06:15.285771+0000","last_peered":"2024-03-21T13:06:15.285771+0000","last_clean":"2024-03-21T13:06:15.285771+0000","last_became_active":"2024-03-21T13:04:46.259233+0000","last_became_peered":"2024-03-21T13:04:46.259233+0000","last_unstale":"2024-03-21T13:06:15.285771+0000","last_undegraded":"2024-03-21T13:06:15.285771+0000","last_fullsized":"2024-03-21T13:06:15.285771+0000","mapping_epoch":168,"log_start":"181'3800","ondisk_log_start":"181'3800","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3586,"log_dups_size":0,"ondisk_log_size":3586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.556038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484048896,"num_objects":7386,"num_object_clones":0,"num_object_copies":22158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7386,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":169,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.349848+0000","last_change":"2024-03-21T13:05:35.000806+0000","last_active":"2024-03-21T13:06:14.349848+0000","last_peered":"2024-03-21T13:06:14.349848+0000","last_clean":"2024-03-21T13:06:14.349848+0000","last_became_active":"2024-03-21T13:02:14.335897+0000","last_became_peered":"2024-03-21T13:02:14.335897+0000","last_unstale":"2024-03-21T13:06:14.349848+0000","last_undegraded":"2024-03-21T13:06:14.349848+0000","last_fullsized":"2024-03-21T13:06:14.349848+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:35.000684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:35.000684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:36.510760+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":175,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.354724+0000","last_change":"2024-03-21T13:05:07.282376+0000","last_active":"2024-03-21T13:06:14.354724+0000","last_peered":"2024-03-21T13:06:14.354724+0000","last_clean":"2024-03-21T13:06:14.354724+0000","last_became_active":"2024-03-21T13:02:26.237683+0000","last_became_peered":"2024-03-21T13:02:26.237683+0000","last_unstale":"2024-03-21T13:06:14.354724+0000","last_undegraded":"2024-03-21T13:06:14.354724+0000","last_fullsized":"2024-03-21T13:06:14.354724+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.282145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.282145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"71'19","reported_seq":230,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.352572+0000","last_change":"2024-03-21T13:05:28.062353+0000","last_active":"2024-03-21T13:06:14.352572+0000","last_peered":"2024-03-21T13:06:14.352572+0000","last_clean":"2024-03-21T13:06:14.352572+0000","last_became_active":"2024-03-21T13:02:26.239140+0000","last_became_peered":"2024-03-21T13:02:26.239140+0000","last_unstale":"2024-03-21T13:06:14.352572+0000","last_undegraded":"2024-03-21T13:06:14.352572+0000","last_fullsized":"2024-03-21T13:06:14.352572+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:05:28.062237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.062237+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:46.857796+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.349914+0000","last_change":"2024-03-21T13:05:26.932396+0000","last_active":"2024-03-21T13:06:14.349914+0000","last_peered":"2024-03-21T13:06:14.349914+0000","last_clean":"2024-03-21T13:06:14.349914+0000","last_became_active":"2024-03-21T13:02:26.252372+0000","last_became_peered":"2024-03-21T13:02:26.252372+0000","last_unstale":"2024-03-21T13:06:14.349914+0000","last_undegraded":"2024-03-21T13:06:14.349914+0000","last_fullsized":"2024-03-21T13:06:14.349914+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:26.932188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.932188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:43.483509+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"192'7509","reported_seq":7567,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.929993+0000","last_change":"2024-03-21T13:04:46.254274+0000","last_active":"2024-03-21T13:06:14.929993+0000","last_peered":"2024-03-21T13:06:14.929993+0000","last_clean":"2024-03-21T13:06:14.929993+0000","last_became_active":"2024-03-21T13:04:46.253974+0000","last_became_peered":"2024-03-21T13:04:46.253974+0000","last_unstale":"2024-03-21T13:06:14.929993+0000","last_undegraded":"2024-03-21T13:06:14.929993+0000","last_fullsized":"2024-03-21T13:06:14.929993+0000","mapping_epoch":168,"log_start":"179'3300","ondisk_log_start":"179'3300","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.496067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492109824,"num_objects":7509,"num_object_clones":0,"num_object_copies":22527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7509,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":176,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.363908+0000","last_change":"2024-03-21T13:01:48.209228+0000","last_active":"2024-03-21T13:06:14.363908+0000","last_peered":"2024-03-21T13:06:14.363908+0000","last_clean":"2024-03-21T13:06:14.363908+0000","last_became_active":"2024-03-21T13:01:48.208899+0000","last_became_peered":"2024-03-21T13:01:48.208899+0000","last_unstale":"2024-03-21T13:06:14.363908+0000","last_undegraded":"2024-03-21T13:06:14.363908+0000","last_fullsized":"2024-03-21T13:06:14.363908+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:31.121913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.121913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":157,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.361583+0000","last_change":"2024-03-21T13:05:13.328899+0000","last_active":"2024-03-21T13:06:14.361583+0000","last_peered":"2024-03-21T13:06:14.361583+0000","last_clean":"2024-03-21T13:06:14.361583+0000","last_became_active":"2024-03-21T13:02:26.243417+0000","last_became_peered":"2024-03-21T13:02:26.243417+0000","last_unstale":"2024-03-21T13:06:14.361583+0000","last_undegraded":"2024-03-21T13:06:14.361583+0000","last_fullsized":"2024-03-21T13:06:14.361583+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.324701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.324701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:41.004405+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"86'15","reported_seq":223,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.361553+0000","last_change":"2024-03-21T13:05:42.308704+0000","last_active":"2024-03-21T13:06:14.361553+0000","last_peered":"2024-03-21T13:06:14.361553+0000","last_clean":"2024-03-21T13:06:14.361553+0000","last_became_active":"2024-03-21T13:02:26.232580+0000","last_became_peered":"2024-03-21T13:02:26.232580+0000","last_unstale":"2024-03-21T13:06:14.361553+0000","last_undegraded":"2024-03-21T13:06:14.361553+0000","last_fullsized":"2024-03-21T13:06:14.361553+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-03-21T13:05:42.308556+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.308556+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:56.807686+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"125'12","reported_seq":275,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.350068+0000","last_change":"2024-03-21T13:05:13.903736+0000","last_active":"2024-03-21T13:06:14.350068+0000","last_peered":"2024-03-21T13:06:14.350068+0000","last_clean":"2024-03-21T13:06:14.350068+0000","last_became_active":"2024-03-21T13:02:26.253720+0000","last_became_peered":"2024-03-21T13:02:26.253720+0000","last_unstale":"2024-03-21T13:06:14.350068+0000","last_undegraded":"2024-03-21T13:06:14.350068+0000","last_fullsized":"2024-03-21T13:06:14.350068+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'12","last_scrub_stamp":"2024-03-21T13:05:13.903589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.903589+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:22.554595+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"192'7555","reported_seq":7613,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.715922+0000","last_change":"2024-03-21T13:04:46.259414+0000","last_active":"2024-03-21T13:06:14.715922+0000","last_peered":"2024-03-21T13:06:14.715922+0000","last_clean":"2024-03-21T13:06:14.715922+0000","last_became_active":"2024-03-21T13:04:46.259106+0000","last_became_peered":"2024-03-21T13:04:46.259106+0000","last_unstale":"2024-03-21T13:06:14.715922+0000","last_undegraded":"2024-03-21T13:06:14.715922+0000","last_fullsized":"2024-03-21T13:06:14.715922+0000","mapping_epoch":168,"log_start":"179'3400","ondisk_log_start":"179'3400","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4155,"log_dups_size":0,"ondisk_log_size":4155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.906272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495124480,"num_objects":7555,"num_object_clones":0,"num_object_copies":22665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7555,"num_write_kb":483520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":279,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.352610+0000","last_change":"2024-03-21T13:05:41.951788+0000","last_active":"2024-03-21T13:06:14.352610+0000","last_peered":"2024-03-21T13:06:14.352610+0000","last_clean":"2024-03-21T13:06:14.352610+0000","last_became_active":"2024-03-21T12:58:43.353049+0000","last_became_peered":"2024-03-21T12:58:43.353049+0000","last_unstale":"2024-03-21T13:06:14.352610+0000","last_undegraded":"2024-03-21T13:06:14.352610+0000","last_fullsized":"2024-03-21T13:06:14.352610+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.951655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.951655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:09.559926+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":311,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.354067+0000","last_change":"2024-03-21T13:05:23.570424+0000","last_active":"2024-03-21T13:06:14.354067+0000","last_peered":"2024-03-21T13:06:14.354067+0000","last_clean":"2024-03-21T13:06:14.354067+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:06:14.354067+0000","last_undegraded":"2024-03-21T13:06:14.354067+0000","last_fullsized":"2024-03-21T13:06:14.354067+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:05:23.569936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.569936+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:40.445343+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"87'21","reported_seq":249,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.349623+0000","last_change":"2024-03-21T13:05:23.917855+0000","last_active":"2024-03-21T13:06:14.349623+0000","last_peered":"2024-03-21T13:06:14.349623+0000","last_clean":"2024-03-21T13:06:14.349623+0000","last_became_active":"2024-03-21T13:02:26.227165+0000","last_became_peered":"2024-03-21T13:02:26.227165+0000","last_unstale":"2024-03-21T13:06:14.349623+0000","last_undegraded":"2024-03-21T13:06:14.349623+0000","last_fullsized":"2024-03-21T13:06:14.349623+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'21","last_scrub_stamp":"2024-03-21T13:05:23.917691+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.917691+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:32.622590+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"119'9","reported_seq":380,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.364525+0000","last_change":"2024-03-21T13:02:14.375956+0000","last_active":"2024-03-21T13:06:14.364525+0000","last_peered":"2024-03-21T13:06:14.364525+0000","last_clean":"2024-03-21T13:06:14.364525+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:06:14.364525+0000","last_undegraded":"2024-03-21T13:06:14.364525+0000","last_fullsized":"2024-03-21T13:06:14.364525+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:25.700549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:25.700549+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554286+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"192'7417","reported_seq":7475,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.888609+0000","last_change":"2024-03-21T13:04:46.260954+0000","last_active":"2024-03-21T13:06:14.888609+0000","last_peered":"2024-03-21T13:06:14.888609+0000","last_clean":"2024-03-21T13:06:14.888609+0000","last_became_active":"2024-03-21T13:04:46.260642+0000","last_became_peered":"2024-03-21T13:04:46.260642+0000","last_unstale":"2024-03-21T13:06:14.888609+0000","last_undegraded":"2024-03-21T13:06:14.888609+0000","last_fullsized":"2024-03-21T13:06:14.888609+0000","mapping_epoch":168,"log_start":"179'3200","ondisk_log_start":"179'3200","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:15.059160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486080512,"num_objects":7417,"num_object_clones":0,"num_object_copies":22251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7417,"num_write_kb":474688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":198,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.370112+0000","last_change":"2024-03-21T13:05:18.416381+0000","last_active":"2024-03-21T13:06:14.370112+0000","last_peered":"2024-03-21T13:06:14.370112+0000","last_clean":"2024-03-21T13:06:14.370112+0000","last_became_active":"2024-03-21T13:01:27.382663+0000","last_became_peered":"2024-03-21T13:01:27.382663+0000","last_unstale":"2024-03-21T13:06:14.370112+0000","last_undegraded":"2024-03-21T13:06:14.370112+0000","last_fullsized":"2024-03-21T13:06:14.370112+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.416262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.416262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:22.282577+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.363217+0000","last_change":"2024-03-21T13:04:20.538204+0000","last_active":"2024-03-21T13:06:14.363217+0000","last_peered":"2024-03-21T13:06:14.363217+0000","last_clean":"2024-03-21T13:06:14.363217+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:06:14.363217+0000","last_undegraded":"2024-03-21T13:06:14.363217+0000","last_fullsized":"2024-03-21T13:06:14.363217+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:20.538115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:04:20.538115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"71'18","reported_seq":313,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.364388+0000","last_change":"2024-03-21T13:05:12.322522+0000","last_active":"2024-03-21T13:06:14.364388+0000","last_peered":"2024-03-21T13:06:14.364388+0000","last_clean":"2024-03-21T13:06:14.364388+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:06:14.364388+0000","last_undegraded":"2024-03-21T13:06:14.364388+0000","last_fullsized":"2024-03-21T13:06:14.364388+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T13:05:12.322284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:12.322284+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:38.174015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":277,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.357698+0000","last_change":"2024-03-21T13:05:07.086071+0000","last_active":"2024-03-21T13:06:14.357698+0000","last_peered":"2024-03-21T13:06:14.357698+0000","last_clean":"2024-03-21T13:06:14.357698+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:06:14.357698+0000","last_undegraded":"2024-03-21T13:06:14.357698+0000","last_fullsized":"2024-03-21T13:06:14.357698+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.085992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.085992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:34.586538+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"192'7440","reported_seq":7498,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.705624+0000","last_change":"2024-03-21T13:04:46.260554+0000","last_active":"2024-03-21T13:06:14.705624+0000","last_peered":"2024-03-21T13:06:14.705624+0000","last_clean":"2024-03-21T13:06:14.705624+0000","last_became_active":"2024-03-21T13:04:46.260083+0000","last_became_peered":"2024-03-21T13:04:46.260083+0000","last_unstale":"2024-03-21T13:06:14.705624+0000","last_undegraded":"2024-03-21T13:06:14.705624+0000","last_fullsized":"2024-03-21T13:06:14.705624+0000","mapping_epoch":168,"log_start":"179'3300","ondisk_log_start":"179'3300","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.118426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487587840,"num_objects":7440,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7440,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.355617+0000","last_change":"2024-03-21T13:05:47.329221+0000","last_active":"2024-03-21T13:06:14.355617+0000","last_peered":"2024-03-21T13:06:14.355617+0000","last_clean":"2024-03-21T13:06:14.355617+0000","last_became_active":"2024-03-21T13:02:26.236501+0000","last_became_peered":"2024-03-21T13:02:26.236501+0000","last_unstale":"2024-03-21T13:06:14.355617+0000","last_undegraded":"2024-03-21T13:06:14.355617+0000","last_fullsized":"2024-03-21T13:06:14.355617+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.328925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.328925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:00.534421+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"125'10","reported_seq":240,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.354259+0000","last_change":"2024-03-21T13:05:20.446633+0000","last_active":"2024-03-21T13:06:14.354259+0000","last_peered":"2024-03-21T13:06:14.354259+0000","last_clean":"2024-03-21T13:06:14.354259+0000","last_became_active":"2024-03-21T13:02:26.231372+0000","last_became_peered":"2024-03-21T13:02:26.231372+0000","last_unstale":"2024-03-21T13:06:14.354259+0000","last_undegraded":"2024-03-21T13:06:14.354259+0000","last_fullsized":"2024-03-21T13:06:14.354259+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:05:20.446177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.446177+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:22.239776+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.364121+0000","last_change":"2024-03-21T13:04:19.520964+0000","last_active":"2024-03-21T13:06:14.364121+0000","last_peered":"2024-03-21T13:06:14.364121+0000","last_clean":"2024-03-21T13:06:14.364121+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:06:14.364121+0000","last_undegraded":"2024-03-21T13:06:14.364121+0000","last_fullsized":"2024-03-21T13:06:14.364121+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:19.520824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:04:19.520824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":221,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.363320+0000","last_change":"2024-03-21T13:02:14.377274+0000","last_active":"2024-03-21T13:06:14.363320+0000","last_peered":"2024-03-21T13:06:14.363320+0000","last_clean":"2024-03-21T13:06:14.363320+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:06:14.363320+0000","last_undegraded":"2024-03-21T13:06:14.363320+0000","last_fullsized":"2024-03-21T13:06:14.363320+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:01:46.167736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.167736+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.554367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"192'7426","reported_seq":7484,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.729196+0000","last_change":"2024-03-21T13:04:46.261080+0000","last_active":"2024-03-21T13:06:14.729196+0000","last_peered":"2024-03-21T13:06:14.729196+0000","last_clean":"2024-03-21T13:06:14.729196+0000","last_became_active":"2024-03-21T13:04:46.260735+0000","last_became_peered":"2024-03-21T13:04:46.260735+0000","last_unstale":"2024-03-21T13:06:14.729196+0000","last_undegraded":"2024-03-21T13:06:14.729196+0000","last_fullsized":"2024-03-21T13:06:14.729196+0000","mapping_epoch":168,"log_start":"179'3400","ondisk_log_start":"179'3400","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:15.413472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486670336,"num_objects":7426,"num_object_clones":0,"num_object_copies":22278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7426,"num_write_kb":475264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362813+0000","last_change":"2024-03-21T13:05:39.518051+0000","last_active":"2024-03-21T13:06:14.362813+0000","last_peered":"2024-03-21T13:06:14.362813+0000","last_clean":"2024-03-21T13:06:14.362813+0000","last_became_active":"2024-03-21T13:02:26.243985+0000","last_became_peered":"2024-03-21T13:02:26.243985+0000","last_unstale":"2024-03-21T13:06:14.362813+0000","last_undegraded":"2024-03-21T13:06:14.362813+0000","last_fullsized":"2024-03-21T13:06:14.362813+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:39.517591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.517591+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:46.995132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":165,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.358584+0000","last_change":"2024-03-21T13:05:38.439542+0000","last_active":"2024-03-21T13:06:14.358584+0000","last_peered":"2024-03-21T13:06:14.358584+0000","last_clean":"2024-03-21T13:06:14.358584+0000","last_became_active":"2024-03-21T13:02:26.229296+0000","last_became_peered":"2024-03-21T13:02:26.229296+0000","last_unstale":"2024-03-21T13:06:14.358584+0000","last_undegraded":"2024-03-21T13:06:14.358584+0000","last_fullsized":"2024-03-21T13:06:14.358584+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:38.439435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.439435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:43.541522+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.357789+0000","last_change":"2024-03-21T13:05:14.021031+0000","last_active":"2024-03-21T13:06:14.357789+0000","last_peered":"2024-03-21T13:06:14.357789+0000","last_clean":"2024-03-21T13:06:14.357789+0000","last_became_active":"2024-03-21T12:58:37.730858+0000","last_became_peered":"2024-03-21T12:58:37.730858+0000","last_unstale":"2024-03-21T13:06:14.357789+0000","last_undegraded":"2024-03-21T13:06:14.357789+0000","last_fullsized":"2024-03-21T13:06:14.357789+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:14.020911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:14.020911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:16.781006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'11","reported_seq":326,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.358370+0000","last_change":"2024-03-21T13:05:26.195643+0000","last_active":"2024-03-21T13:06:14.358370+0000","last_peered":"2024-03-21T13:06:14.358370+0000","last_clean":"2024-03-21T13:06:14.358370+0000","last_became_active":"2024-03-21T13:02:14.348499+0000","last_became_peered":"2024-03-21T13:02:14.348499+0000","last_unstale":"2024-03-21T13:06:14.358370+0000","last_undegraded":"2024-03-21T13:06:14.358370+0000","last_fullsized":"2024-03-21T13:06:14.358370+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:26.195507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.195507+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:29.731844+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"192'7567","reported_seq":7625,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.944154+0000","last_change":"2024-03-21T13:04:46.260817+0000","last_active":"2024-03-21T13:06:14.944154+0000","last_peered":"2024-03-21T13:06:14.944154+0000","last_clean":"2024-03-21T13:06:14.944154+0000","last_became_active":"2024-03-21T13:04:46.260506+0000","last_became_peered":"2024-03-21T13:04:46.260506+0000","last_unstale":"2024-03-21T13:06:14.944154+0000","last_undegraded":"2024-03-21T13:06:14.944154+0000","last_fullsized":"2024-03-21T13:06:14.944154+0000","mapping_epoch":168,"log_start":"178'3000","ondisk_log_start":"178'3000","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495910912,"num_objects":7567,"num_object_clones":0,"num_object_copies":22701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7567,"num_write_kb":484288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.356397+0000","last_change":"2024-03-21T13:05:36.362310+0000","last_active":"2024-03-21T13:06:14.356397+0000","last_peered":"2024-03-21T13:06:14.356397+0000","last_clean":"2024-03-21T13:06:14.356397+0000","last_became_active":"2024-03-21T12:58:43.183380+0000","last_became_peered":"2024-03-21T12:58:43.183380+0000","last_unstale":"2024-03-21T13:06:14.356397+0000","last_undegraded":"2024-03-21T13:06:14.356397+0000","last_fullsized":"2024-03-21T13:06:14.356397+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:36.362197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:05:36.362197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:06.202139+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":177,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.357909+0000","last_change":"2024-03-21T13:05:33.327069+0000","last_active":"2024-03-21T13:06:14.357909+0000","last_peered":"2024-03-21T13:06:14.357909+0000","last_clean":"2024-03-21T13:06:14.357909+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:06:14.357909+0000","last_undegraded":"2024-03-21T13:06:14.357909+0000","last_fullsized":"2024-03-21T13:06:14.357909+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:33.326990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:33.326990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:37.358494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":299,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.349997+0000","last_change":"2024-03-21T13:04:32.808501+0000","last_active":"2024-03-21T13:06:14.349997+0000","last_peered":"2024-03-21T13:06:14.349997+0000","last_clean":"2024-03-21T13:06:14.349997+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:06:14.349997+0000","last_undegraded":"2024-03-21T13:06:14.349997+0000","last_fullsized":"2024-03-21T13:06:14.349997+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:32.808390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:04:32.808390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.118363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":277,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.355846+0000","last_change":"2024-03-21T13:05:38.216400+0000","last_active":"2024-03-21T13:06:14.355846+0000","last_peered":"2024-03-21T13:06:14.355846+0000","last_clean":"2024-03-21T13:06:14.355846+0000","last_became_active":"2024-03-21T13:02:14.334882+0000","last_became_peered":"2024-03-21T13:02:14.334882+0000","last_unstale":"2024-03-21T13:06:14.355846+0000","last_undegraded":"2024-03-21T13:06:14.355846+0000","last_fullsized":"2024-03-21T13:06:14.355846+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-21T13:05:38.215693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.215693+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:06.349642+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"192'7409","reported_seq":7467,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:15.286116+0000","last_change":"2024-03-21T13:04:46.258753+0000","last_active":"2024-03-21T13:06:15.286116+0000","last_peered":"2024-03-21T13:06:15.286116+0000","last_clean":"2024-03-21T13:06:15.286116+0000","last_became_active":"2024-03-21T13:04:46.258434+0000","last_became_peered":"2024-03-21T13:04:46.258434+0000","last_unstale":"2024-03-21T13:06:15.286116+0000","last_undegraded":"2024-03-21T13:06:15.286116+0000","last_fullsized":"2024-03-21T13:06:15.286116+0000","mapping_epoch":168,"log_start":"181'3800","ondisk_log_start":"181'3800","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:18.556090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485556224,"num_objects":7409,"num_object_clones":0,"num_object_copies":22227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7409,"num_write_kb":474176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":177,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.357968+0000","last_change":"2024-03-21T13:05:34.291498+0000","last_active":"2024-03-21T13:06:14.357968+0000","last_peered":"2024-03-21T13:06:14.357968+0000","last_clean":"2024-03-21T13:06:14.357968+0000","last_became_active":"2024-03-21T13:02:26.227779+0000","last_became_peered":"2024-03-21T13:02:26.227779+0000","last_unstale":"2024-03-21T13:06:14.357968+0000","last_undegraded":"2024-03-21T13:06:14.357968+0000","last_fullsized":"2024-03-21T13:06:14.357968+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.291369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.291369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:48.086995+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"125'10","reported_seq":397,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.356009+0000","last_change":"2024-03-21T13:05:30.300832+0000","last_active":"2024-03-21T13:06:14.356009+0000","last_peered":"2024-03-21T13:06:14.356009+0000","last_clean":"2024-03-21T13:06:14.356009+0000","last_became_active":"2024-03-21T13:02:26.235954+0000","last_became_peered":"2024-03-21T13:02:26.235954+0000","last_unstale":"2024-03-21T13:06:14.356009+0000","last_undegraded":"2024-03-21T13:06:14.356009+0000","last_fullsized":"2024-03-21T13:06:14.356009+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:05:30.300605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:30.300605+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:41.344464+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362475+0000","last_change":"2024-03-21T13:05:38.492137+0000","last_active":"2024-03-21T13:06:14.362475+0000","last_peered":"2024-03-21T13:06:14.362475+0000","last_clean":"2024-03-21T13:06:14.362475+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:06:14.362475+0000","last_undegraded":"2024-03-21T13:06:14.362475+0000","last_fullsized":"2024-03-21T13:06:14.362475+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:38.492015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.492015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:05.896092+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":232,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-21T13:06:14.362914+0000","last_change":"2024-03-21T13:05:22.406498+0000","last_active":"2024-03-21T13:06:14.362914+0000","last_peered":"2024-03-21T13:06:14.362914+0000","last_clean":"2024-03-21T13:06:14.362914+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:06:14.362914+0000","last_undegraded":"2024-03-21T13:06:14.362914+0000","last_fullsized":"2024-03-21T13:06:14.362914+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:22.406367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.406367+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:06:36.553896+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":7831224320,"num_objects":119495,"num_object_clones":0,"num_object_copies":358485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119495,"num_write_kb":7647680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23491379200,"data_stored":23491379200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64495,"ondisk_log_size":64495,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":448,"num_read_kb":443,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":684,"num_read_kb":441,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":3658,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":432,"ondisk_log_size":432,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168672,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2585000,"kb_used_data":2369184,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":91160152,"statfs":{"total":95995035648,"available":93347995648,"internally_reserved":0,"allocated":2426044416,"data_stored":2425376079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":220985802},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463014,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4063344,"kb_used_data":3797544,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":89681808,"statfs":{"total":95995035648,"available":91834171392,"internally_reserved":0,"allocated":3888685056,"data_stored":3888008813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":272169414},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574974,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738500,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3614840,"kb_used_data":3353196,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":90130312,"statfs":{"total":95995035648,"available":92293439488,"internally_reserved":0,"allocated":3433672704,"data_stored":3432998532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411450,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4576084,"kb_used_data":4274188,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":89169068,"statfs":{"total":95995035648,"available":91309125632,"internally_reserved":0,"allocated":4376768512,"data_stored":4376119276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182517,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5605012,"kb_used_data":5230920,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":88140140,"statfs":{"total":95995035648,"available":90255503360,"internally_reserved":0,"allocated":5356462080,"data_stored":5355759847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":383056327},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018991,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4078620,"kb_used_data":3786448,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":89666532,"statfs":{"total":95995035648,"available":91818528768,"internally_reserved":0,"allocated":3877322752,"data_stored":3876676573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724650,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3446407168,"data_stored":3446407168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4404019200,"data_stored":4404019200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5393154048,"data_stored":5393154048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3890937856,"data_stored":3890937856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3910926336,"data_stored":3910926336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2445934592,"data_stored":2445934592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:06:19.478 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-21T13:06:19.478 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:06:19.478 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 2, 3, 4, 6, 7] 2024-03-21T13:06:19.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd out 2 2024-03-21T13:06:19.488 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:19.503 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:19.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:19.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:19.605 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:19.657 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:19.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:19 smithi204 ceph-mon[33153]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:06:19.759 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:19.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:19 smithi163 ceph-mon[29621]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:06:19.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:19 smithi163 ceph-mon[40675]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:06:19.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:19.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:19.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:19.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:19.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:19.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:19.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:20.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:20.072 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:20.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:20.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:20.087 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 82 15 125783 125768 95.8404 77.375 0.0137762 0.0104295 2024-03-21T13:06:20.173 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:20.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:20.208 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:20.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:20.282 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:06:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-03-21T13:06:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-03-21T13:06:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:06:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:5937 7.0 7:0cfef427:::benchmark_data_smithi163_80312_object5936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:06:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:00:56.236353+0000", 2024-03-21T13:06:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 324.04262467900003, 2024-03-21T13:06:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.080287529999999996, 2024-03-21T13:06:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5937 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236353+0000", 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236353+0000", 2024-03-21T13:06:20.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236354+0000", 2024-03-21T13:06:20.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3859999999999999e-06 2024-03-21T13:06:20.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236405+0000", 2024-03-21T13:06:20.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1201e-05 2024-03-21T13:06:20.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236424+0000", 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8842e-05 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236432+0000", 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9990000000000008e-06 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236490+0000", 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7497999999999997e-05 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236613+0000", 2024-03-21T13:06:20.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012285099999999999 2024-03-21T13:06:20.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:06:20.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236761+0000", 2024-03-21T13:06:20.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014846799999999999 2024-03-21T13:06:20.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.237277+0000", 2024-03-21T13:06:20.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000516195 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.238004+0000", 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00072655799999999998 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.316548+0000", 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.078544055000000002 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.316590+0000", 2024-03-21T13:06:20.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2398000000000003e-05 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.316640+0000", 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0078999999999999e-05 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:5965 7.0 7:03da52ce:::benchmark_data_smithi163_80312_object5964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:00:56.326347+0000", 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 323.95263106200002, 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.10961245, 2024-03-21T13:06:20.288 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5965 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326347+0000", 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326347+0000", 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326348+0000", 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.451e-06 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326386+0000", 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8075999999999999e-05 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326398+0000", 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2306999999999999e-05 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326408+0000", 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4299999999999995e-06 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326474+0000", 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6591e-05 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326610+0000", 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013582700000000001 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326782+0000", 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017180799999999999 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.327361+0000", 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00057854599999999999 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.328867+0000", 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001506082 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.435844+0000", 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.10697730799999999 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.435914+0000", 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9698999999999998e-05 2024-03-21T13:06:20.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.435959+0000", 2024-03-21T13:06:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5324999999999997e-05 2024-03-21T13:06:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:5979 7.0 7:060be99b:::benchmark_data_smithi163_80312_object5978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:06:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:00:56.330543+0000", 2024-03-21T13:06:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 323.948434969, 2024-03-21T13:06:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.10700628399999999, 2024-03-21T13:06:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5979 2024-03-21T13:06:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330543+0000", 2024-03-21T13:06:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330543+0000", 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330544+0000", 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5209999999999999e-06 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330628+0000", 2024-03-21T13:06:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3535000000000006e-05 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330637+0000", 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7720000000000006e-06 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330644+0000", 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6730000000000003e-06 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330706+0000", 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2160000000000001e-05 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330804+0000", 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7522999999999994e-05 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330931+0000", 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000127584 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.331447+0000", 2024-03-21T13:06:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00051556400000000004 2024-03-21T13:06:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.331956+0000", 2024-03-21T13:06:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000509302 2024-03-21T13:06:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.437417+0000", 2024-03-21T13:06:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.105460939 2024-03-21T13:06:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.437501+0000", 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4078000000000004e-05 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.437549+0000", 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7633000000000003e-05 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:66554 7.0 7:00032620:::benchmark_data_smithi163_80312_object66553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:01:32.675052+0000", 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 287.60392602600001, 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.094370187999999994, 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 66554 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675052+0000", 2024-03-21T13:06:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675052+0000", 2024-03-21T13:06:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675053+0000", 2024-03-21T13:06:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.178e-06 2024-03-21T13:06:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675115+0000", 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2138999999999996e-05 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675125+0000", 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0362e-05 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675131+0000", 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2519999999999996e-06 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675182+0000", 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0618000000000002e-05 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675272+0000", 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0174999999999996e-05 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675363+0000", 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0754999999999994e-05 2024-03-21T13:06:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.676991+0000", 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016278849999999999 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.677307+0000", 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031613000000000002 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.769314+0000", 2024-03-21T13:06:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.092007163000000003 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.769377+0000", 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2644000000000005e-05 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.769422+0000", 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4886999999999999e-05 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:94688 7.0 7:0e49193b:::benchmark_data_smithi163_80312_object94687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:06:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:01:59.258336+0000", 2024-03-21T13:06:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 261.02064195100002, 2024-03-21T13:06:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.099394761999999998, 2024-03-21T13:06:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 94688 2024-03-21T13:06:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258336+0000", 2024-03-21T13:06:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258336+0000", 2024-03-21T13:06:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258337+0000", 2024-03-21T13:06:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2440000000000001e-06 2024-03-21T13:06:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258393+0000", 2024-03-21T13:06:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6415000000000001e-05 2024-03-21T13:06:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258405+0000", 2024-03-21T13:06:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.135e-05 2024-03-21T13:06:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258411+0000", 2024-03-21T13:06:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.993e-06 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258461+0000", 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9886000000000003e-05 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258555+0000", 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4494999999999996e-05 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:06:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258650+0000", 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4661999999999996e-05 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.264002+0000", 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053522109999999999 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.264051+0000", 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8980999999999997e-05 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.357625+0000", 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.093574288000000005 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.357687+0000", 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1737999999999997e-05 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.357730+0000", 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3498999999999998e-05 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:94691 7.0 7:0868e98a:::benchmark_data_smithi163_80312_object94690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:06:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:01:59.260462+0000", 2024-03-21T13:06:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 261.018515476, 2024-03-21T13:06:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.097889528000000003, 2024-03-21T13:06:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 94691 2024-03-21T13:06:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260462+0000", 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260462+0000", 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260463+0000", 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6899999999999996e-07 2024-03-21T13:06:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260528+0000", 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5376000000000006e-05 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260538+0000", 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6600000000000007e-06 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260555+0000", 2024-03-21T13:06:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7503000000000002e-05 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260943+0000", 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038779500000000002 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.261055+0000", 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112186 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.261202+0000", 2024-03-21T13:06:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014699900000000001 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.264043+0000", 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028405769999999999 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.264118+0000", 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4924000000000003e-05 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.358274+0000", 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.09415635 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.358311+0000", 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7169000000000003e-05 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.358352+0000", 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0219999999999998e-05 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:95064 7.0 7:0c9f881e:::benchmark_data_smithi163_80312_object95063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:01:59.727240+0000", 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 260.55173771900002, 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.080850353999999999, 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 95064 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727240+0000", 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727240+0000", 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727241+0000", 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.167e-06 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727287+0000", 2024-03-21T13:06:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5596999999999997e-05 2024-03-21T13:06:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727298+0000", 2024-03-21T13:06:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1144e-05 2024-03-21T13:06:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727304+0000", 2024-03-21T13:06:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3470000000000004e-06 2024-03-21T13:06:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727352+0000", 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8359000000000002e-05 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727438+0000", 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5470000000000007e-05 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727545+0000", 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107428 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.730267+0000", 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027215310000000001 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.731653+0000", 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001386034 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.807949+0000", 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.076295899 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.808042+0000", 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3126000000000003e-05 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.808090+0000", 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8251999999999998e-05 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116056 7.b 7:df017f68:::benchmark_data_smithi163_80312_object116055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.467833+0000", 2024-03-21T13:06:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.81114492699999, 2024-03-21T13:06:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.085221341000000006, 2024-03-21T13:06:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116056 2024-03-21T13:06:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467833+0000", 2024-03-21T13:06:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467833+0000", 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467844+0000", 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1499e-05 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467906+0000", 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1583999999999999e-05 2024-03-21T13:06:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467918+0000", 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2690999999999999e-05 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467926+0000", 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0190000000000006e-06 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467984+0000", 2024-03-21T13:06:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8004000000000002e-05 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.468133+0000", 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014849500000000001 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,6", 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.468321+0000", 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018788600000000001 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.471683+0000", 2024-03-21T13:06:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003362659 2024-03-21T13:06:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.471770+0000", 2024-03-21T13:06:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6512000000000001e-05 2024-03-21T13:06:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.475037+0000", 2024-03-21T13:06:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032672109999999999 2024-03-21T13:06:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.552843+0000", 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.077806288000000001 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.552926+0000", 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2328000000000002e-05 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.553054+0000", 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012816499999999999 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116068 7.0 7:09402460:::benchmark_data_smithi163_80312_object116067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.483157+0000", 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.79582031999999, 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.081778124999999993, 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116068 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483157+0000", 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483157+0000", 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483159+0000", 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.407e-06 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483192+0000", 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3175e-05 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483203+0000", 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1457999999999999e-05 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483211+0000", 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6620000000000005e-06 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483281+0000", 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9734999999999996e-05 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483422+0000", 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014167800000000001 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483585+0000", 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016310599999999999 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.486348+0000", 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027625190000000002 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487663+0000", 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001315309 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.563548+0000", 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.075884518999999998 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.564765+0000", 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001216983 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.564819+0000", 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4404999999999998e-05 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.564935+0000", 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011616900000000001 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116075 7.0 7:08926736:::benchmark_data_smithi163_80312_object116074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.487709+0000", 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.791268866, 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.078736329999999993, 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116075 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487709+0000", 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487709+0000", 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487709+0000", 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6599999999999995e-07 2024-03-21T13:06:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487766+0000", 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.681e-05 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487775+0000", 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5820000000000002e-06 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487782+0000", 2024-03-21T13:06:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2169999999999997e-06 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.490331+0000", 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025494060000000002 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.490445+0000", 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000113776 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.490581+0000", 2024-03-21T13:06:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000136224 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.492567+0000", 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001985503 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.493189+0000", 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00062200299999999999 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.565262+0000", 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072073034999999994 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.566264+0000", 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010021769999999999 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.566331+0000", 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6910999999999997e-05 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.566445+0000", 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011391999999999999 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116240 7.b 7:d015c35c:::benchmark_data_smithi163_80312_object116239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.720013+0000", 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.558964731, 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.130214526, 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116240 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720013+0000", 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720013+0000", 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720014+0000", 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4440000000000001e-06 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720050+0000", 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5490000000000001e-05 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720064+0000", 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3773e-05 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720072+0000", 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2290000000000003e-06 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720132+0000", 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9831999999999999e-05 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720279+0000", 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014782799999999999 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,6", 2024-03-21T13:06:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720455+0000", 2024-03-21T13:06:20.348 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017585499999999999 2024-03-21T13:06:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.723242+0000", 2024-03-21T13:06:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027871419999999998 2024-03-21T13:06:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.723669+0000", 2024-03-21T13:06:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000426933 2024-03-21T13:06:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.723963+0000", 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029338200000000002 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.850012+0000", 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.126049095 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.850083+0000", 2024-03-21T13:06:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1653e-05 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.850227+0000", 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014386999999999999 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116247 7.0 7:0a4d72c9:::benchmark_data_smithi163_80312_object116246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.726834+0000", 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.55214395100001, 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.130439006, 2024-03-21T13:06:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116247 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726834+0000", 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726834+0000", 2024-03-21T13:06:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726835+0000", 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4959999999999999e-06 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726865+0000", 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0080999999999999e-05 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726886+0000", 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1182000000000001e-05 2024-03-21T13:06:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726895+0000", 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4719999999999995e-06 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726971+0000", 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5972999999999999e-05 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.727099+0000", 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000128162 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.727267+0000", 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016778600000000001 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730152+0000", 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028849560000000002 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.732900+0000", 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027485600000000002 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.848690+0000", 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.11578986600000001 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.857079+0000", 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0083888729999999998 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.857149+0000", 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.991e-05 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.857273+0000", 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012368899999999999 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116251 7.0 7:017df485:::benchmark_data_smithi163_80312_object116250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.730529+0000", 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.54844816299999, 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12804560300000001, 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116251 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730529+0000", 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730529+0000", 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730530+0000", 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0809999999999999e-06 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730582+0000", 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1319000000000001e-05 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730592+0000", 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0332e-05 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730598+0000", 2024-03-21T13:06:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3289999999999999e-06 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730645+0000", 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6916000000000003e-05 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730750+0000", 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104949 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730879+0000", 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012888 2024-03-21T13:06:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.732868+0000", 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001989237 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.733022+0000", 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000153108 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.858130+0000", 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.125108673 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.858342+0000", 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000211428 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.858408+0000", 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5899e-05 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.858575+0000", 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016745200000000001 2024-03-21T13:06:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116535 7.b 7:db9c5de4:::benchmark_data_smithi163_80312_object116534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:23.124212+0000", 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.154765481, 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.092375593000000006, 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116535 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124212+0000", 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124212+0000", 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124213+0000", 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3700000000000001e-07 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124242+0000", 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9159e-05 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124253+0000", 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0869e-05 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124259+0000", 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9410000000000001e-06 2024-03-21T13:06:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.127981+0000", 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003721673 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.130206+0000", 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022252589999999998 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,6", 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.130421+0000", 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000214828 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.136255+0000", 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058337229999999999 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.137633+0000", 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013781570000000001 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.138730+0000", 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010970929999999999 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.216445+0000", 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.077715203999999996 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.216500+0000", 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4321999999999999e-05 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.216588+0000", 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7928000000000002e-05 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116566 7.b 7:ddb81152:::benchmark_data_smithi163_80312_object116565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:23.152336+0000", 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.126641232, 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.089229614999999998, 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116566 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152336+0000", 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152336+0000", 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152338+0000", 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.57e-06 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152379+0000", 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1118000000000002e-05 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152399+0000", 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0415e-05 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152423+0000", 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3938999999999999e-05 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152487+0000", 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3590000000000006e-05 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.153143+0000", 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00065597699999999997 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,6", 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.153343+0000", 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020033 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.156773+0000", 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034299539999999998 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.157158+0000", 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000385153 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.157320+0000", 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016143900000000001 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.241425+0000", 2024-03-21T13:06:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.084105503999999998 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.241470+0000", 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5158999999999999e-05 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.241566+0000", 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5466999999999999e-05 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118902 7.0 7:007d3f07:::benchmark_data_smithi163_80312_object1461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.702328+0000", 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 234.57664922999999, 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.511295948, 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118902 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702328+0000", 2024-03-21T13:06:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702328+0000", 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702329+0000", 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1510000000000001e-06 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702334+0000", 2024-03-21T13:06:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0300000000000004e-06 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702343+0000", 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2029999999999998e-06 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702350+0000", 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.182e-06 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702410+0000", 2024-03-21T13:06:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.05e-05 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702498+0000", 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7113000000000004e-05 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702603+0000", 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105567 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.705654+0000", 2024-03-21T13:06:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030509489999999998 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.709103+0000", 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003448914 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.712195+0000", 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030919419999999999 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.750528+0000", 2024-03-21T13:06:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.038333000999999998 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.750541+0000", 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3454e-05 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213611+0000", 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46306928400000003 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213624+0000", 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3658e-05 2024-03-21T13:06:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118942 7.0 7.1f908400 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.753006+0000", 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 234.525971086, 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46065019600000001, 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118942 2024-03-21T13:06:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753006+0000", 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753006+0000", 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753008+0000", 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3999999999999999e-06 2024-03-21T13:06:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753011+0000", 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5899999999999999e-06 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753020+0000", 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.242e-06 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753027+0000", 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3220000000000004e-06 2024-03-21T13:06:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.754026+0000", 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00099910600000000004 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.754036+0000", 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7110000000000007e-06 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213649+0000", 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45961280399999999 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213657+0000", 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0209999999999999e-06 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118952 7.0 7.3905ab10 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.767903+0000", 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 234.51107421200001, 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44580361899999998, 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:06:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118952 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767903+0000", 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767903+0000", 2024-03-21T13:06:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767905+0000", 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5370000000000001e-06 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767914+0000", 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0880000000000001e-06 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767925+0000", 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1231e-05 2024-03-21T13:06:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767934+0000", 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4889999999999995e-06 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.768019+0000", 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4860999999999999e-05 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.768029+0000", 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0558e-05 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213698+0000", 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44566897700000002 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213707+0000", 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8780000000000004e-06 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118953 7.0 7.83d475e0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:06:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.767956+0000", 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 234.51102185799999, 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44578293800000002, 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118953 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767956+0000", 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767956+0000", 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767956+0000", 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7199999999999999e-07 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767959+0000", 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9289999999999998e-06 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767966+0000", 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3280000000000002e-06 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767971+0000", 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6180000000000003e-06 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.768055+0000", 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3486999999999995e-05 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.768062+0000", 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9040000000000003e-06 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213730+0000", 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44566886999999999 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213739+0000", 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2300000000000008e-06 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118970 7.b 7.e8b90beb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.796717+0000", 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 234.48226059800001, 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42020922900000002, 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118970 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796717+0000", 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796717+0000", 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796719+0000", 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0040000000000002e-06 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796728+0000", 2024-03-21T13:06:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9570000000000008e-06 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796748+0000", 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9741e-05 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796758+0000", 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9569999999999997e-06 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796848+0000", 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0754000000000006e-05 2024-03-21T13:06:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796860+0000", 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1265e-05 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.216916+0000", 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42005610100000002 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.216926+0000", 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.045e-05 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:20.398 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:06:20.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:20.444 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:20.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:20.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:20.575 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:20.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:20.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:20.650 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:06:20.677 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:20.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:20 smithi163 ceph-mon[29621]: pgmap v327: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 684 B/s rd, 51 MiB/s wr, 818 op/s 2024-03-21T13:06:20.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:20.696 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:20 smithi163 ceph-mon[40675]: pgmap v327: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 684 B/s rd, 51 MiB/s wr, 818 op/s 2024-03-21T13:06:20.696 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:20.707 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:20.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:20.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:20.729 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:20 smithi204 ceph-mon[33153]: pgmap v327: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 684 B/s rd, 51 MiB/s wr, 818 op/s 2024-03-21T13:06:20.729 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:20.809 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:20.875 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:20.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:20.977 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:21.022 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:21.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:21.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:21.087 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 83 16 127085 127069 95.6653 81.3125 0.00940104 0.0104489 2024-03-21T13:06:21.123 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:21.156 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:21.167 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:21.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:21.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-21T13:06:21.268 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:21.330 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:21.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:21.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:21.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:21.466 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:21.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:21.568 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:21.606 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:21.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:21.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:21.640 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:06:21.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:21.741 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:21.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:21.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:21.843 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:21.883 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:21.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:21.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:06:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:06:21.982 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:21.986 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:22.059 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:22.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:22.087 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 84 16 128401 128385 95.5055 82.25 0.00820302 0.0104664 2024-03-21T13:06:22.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:22.196 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:22.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:22.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:22.330 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:22.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:22.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:22.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:22.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:22.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:22.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:22 smithi204 ceph-mon[33153]: pgmap v328: 148 pgs: 148 active+clean; 7.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-21T13:06:22.561 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:22.604 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:22.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:22.706 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:22.734 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:22.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:22.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:22 smithi163 ceph-mon[29621]: pgmap v328: 148 pgs: 148 active+clean; 7.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-21T13:06:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:22 smithi163 ceph-mon[40675]: pgmap v328: 148 pgs: 148 active+clean; 7.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-21T13:06:22.835 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:22.866 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:22.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:22.968 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:22.992 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:22.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:23.087 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 85 16 129528 129512 95.2105 70.4375 0.0122739 0.0104985 2024-03-21T13:06:23.094 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:23.132 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:23.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:23.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:23.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:23.265 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:23.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:23.366 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:23.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:23.413 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:23.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:23.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:23 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:23.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:23 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:23.514 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:23.554 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:23.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:23.607 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:06:23.607 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:06:23.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:23.607 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:130230 8.3 8:c350b323:::benchmark_data_smithi163_107079_object130229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-03-21T13:06:23.607 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:06:23.602575+0000", 2024-03-21T13:06:23.607 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0041283049999999996, 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041520869999999996, 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 130230 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:06:23.602575+0000", 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:06:23.602575+0000", 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:06:23.602576+0000", 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9899999999999999e-07 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:06:23.602621+0000", 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4631e-05 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:23.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:23.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:23.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:06:23.602630+0000", 2024-03-21T13:06:23.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9989999999999997e-06 2024-03-21T13:06:23.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:23.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:23.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:23.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:06:23.602635+0000", 2024-03-21T13:06:23.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2549999999999997e-06 2024-03-21T13:06:23.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:23.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:23.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:23.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:06:23.602686+0000", 2024-03-21T13:06:23.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1431999999999998e-05 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:06:23.602835+0000", 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000148908 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:06:23.603006+0000", 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017034099999999999 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:06:23.605171+0000", 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021651180000000002 2024-03-21T13:06:23.611 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:23.612 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:23.612 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:23.612 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:23.612 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-03-21T13:06:23.612 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 1 2024-03-21T13:06:23.612 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:06:23.655 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:23.691 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:23.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:23.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:23.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:23 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:23.793 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:23.832 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:23.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:23.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:23.933 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:23.973 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:23.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:23.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:24.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:24.087 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 86 16 130869 130853 95.0779 83.8125 0.00872185 0.0105133 2024-03-21T13:06:24.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:24.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:24.218 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:24.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:24.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:24.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:24.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:24.366 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-21T13:06:24.386 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:24.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:24.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:24.520 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:24.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:24.622 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:24.654 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:24.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:24.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:24 smithi204 ceph-mon[33153]: pgmap v329: 148 pgs: 148 active+clean; 7.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-21T13:06:24.754 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:06:24.756 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:24 smithi163 ceph-mon[29621]: pgmap v329: 148 pgs: 148 active+clean; 7.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-21T13:06:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:24 smithi163 ceph-mon[40675]: pgmap v329: 148 pgs: 148 active+clean; 7.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-21T13:06:24.804 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:24.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:24.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:24.906 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:24.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:24.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:25.045 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:25.083 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:25.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:25.087 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 87 16 132259 132243 94.9836 86.875 0.00587016 0.0105239 2024-03-21T13:06:25.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:25.226 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:25.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:25.327 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:25.371 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:25.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:25.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:25.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:25.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:25 smithi163 ceph-mon[29621]: pgmap v330: 148 pgs: 148 active+clean; 7.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s 2024-03-21T13:06:25.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:25 smithi163 ceph-mon[40675]: pgmap v330: 148 pgs: 148 active+clean; 7.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s 2024-03-21T13:06:25.505 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:25.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:25.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:25.607 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:25.657 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:25.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:25.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:25.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:25 smithi204 ceph-mon[33153]: pgmap v330: 148 pgs: 148 active+clean; 7.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s 2024-03-21T13:06:25.759 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:25.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:06:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:06:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:06:25.791 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:25.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:25.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:25.893 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:25.924 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:25.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:26.026 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:26.061 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:26.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:26.088 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 88 16 133450 133434 94.75 74.4375 0.00827743 0.01055 2024-03-21T13:06:26.162 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:26.203 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:26.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:26.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:26.305 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:26.345 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:26.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:26.371 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:06:26.371 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:06:26.371 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:06:26.372 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:06:26.372 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:06:26.403 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:06:26.443 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:06:26.446 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:26.454 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:06:26.486 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:06:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:26.492 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:26.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:26.503 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:06:26.503 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:06:26.504 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:06:26.504 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:06:26.504 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:06:26.504 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:06:26.504 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:06:26.504 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:06:26.521 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:26 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2187375761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:06:26.521 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:26 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:06:26.521 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:26 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1835545822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:26.593 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:26.632 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:26.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:26.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:26 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2187375761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:06:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:26 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:06:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:26 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1835545822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:26 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2187375761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:06:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:26 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:06:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:26 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1835545822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:26.767 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:26.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:26.869 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:26.910 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:26.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:26.958 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:06:26.958 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:06:27.012 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:27.041 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:27.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:27.088 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 89 16 134609 134593 94.4992 72.4375 0.0145889 0.0105776 2024-03-21T13:06:27.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:27.182 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:27.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:27.284 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:27.326 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:27.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:27.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:27.428 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:27.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:27.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:27.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:27.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:27.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:27.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:27.690 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:27 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:06:27.690 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:27 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:06:27.690 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:27 smithi204 ceph-mon[33153]: osdmap e194: 8 total, 8 up, 6 in 2024-03-21T13:06:27.690 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:27 smithi204 ceph-mon[33153]: 6.1d scrub starts 2024-03-21T13:06:27.690 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:27 smithi204 ceph-mon[33153]: 6.1d scrub ok 2024-03-21T13:06:27.690 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:27 smithi204 ceph-mon[33153]: pgmap v332: 148 pgs: 148 active+clean; 8.1 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-21T13:06:27.699 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:27.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:27.735 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:27.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:27.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-21T13:06:27.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:27 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:06:27.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:27 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:06:27.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:27 smithi163 ceph-mon[29621]: osdmap e194: 8 total, 8 up, 6 in 2024-03-21T13:06:27.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:27 smithi163 ceph-mon[29621]: 6.1d scrub starts 2024-03-21T13:06:27.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:27 smithi163 ceph-mon[29621]: 6.1d scrub ok 2024-03-21T13:06:27.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:27 smithi163 ceph-mon[29621]: pgmap v332: 148 pgs: 148 active+clean; 8.1 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-21T13:06:27.753 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:27 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:06:27.753 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:27 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:06:27.753 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:27 smithi163 ceph-mon[40675]: osdmap e194: 8 total, 8 up, 6 in 2024-03-21T13:06:27.753 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:27 smithi163 ceph-mon[40675]: 6.1d scrub starts 2024-03-21T13:06:27.753 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:27 smithi163 ceph-mon[40675]: 6.1d scrub ok 2024-03-21T13:06:27.753 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:27 smithi163 ceph-mon[40675]: pgmap v332: 148 pgs: 148 active+clean; 8.1 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-21T13:06:27.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:27.869 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:27.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:27.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:27.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:28.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:28.012 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:28.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:28.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:28 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a[29617]: 2024-03-21T13:06:27.999+0000 7f8e1b000700 -1 mon.a@0(leader).osd e194 definitely_dead 0 2024-03-21T13:06:28.088 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Total time run: 90.0047 2024-03-21T13:06:28.089 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Total writes made: 135870 2024-03-21T13:06:28.089 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Write size: 65536 2024-03-21T13:06:28.089 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Object size: 65536 2024-03-21T13:06:28.089 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Bandwidth (MB/sec): 94.3493 2024-03-21T13:06:28.089 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev Bandwidth: 29.8631 2024-03-21T13:06:28.090 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max bandwidth (MB/sec): 213.688 2024-03-21T13:06:28.090 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min bandwidth (MB/sec): 62.0625 2024-03-21T13:06:28.090 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Average IOPS: 1509 2024-03-21T13:06:28.090 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev IOPS: 477.809 2024-03-21T13:06:28.090 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max IOPS: 3419 2024-03-21T13:06:28.090 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min IOPS: 993 2024-03-21T13:06:28.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Average Latency(s): 0.0105951 2024-03-21T13:06:28.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev Latency(s): 0.00797854 2024-03-21T13:06:28.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max latency(s): 0.163103 2024-03-21T13:06:28.091 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min latency(s): 0.00120788 2024-03-21T13:06:28.093 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Cleaning up (deleting benchmark objects) 2024-03-21T13:06:28.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:28.142 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:06:28.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:28.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:28.251 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:28.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:28.312 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:28.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:28.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:28.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:28.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:28.486 INFO:teuthology.orchestra.run.smithi163.stderr:marked out osd.2. 2024-03-21T13:06:28.543 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:28 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3151525070' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-21T13:06:28.543 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:28 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-21T13:06:28.543 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:28 smithi204 ceph-mon[33153]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-21T13:06:28.551 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:28.576 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:28.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:28.679 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:28.703 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:28.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:28.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:28 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3151525070' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-21T13:06:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:28 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-21T13:06:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:28 smithi163 ceph-mon[29621]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-21T13:06:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:28 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3151525070' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-21T13:06:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:28 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-21T13:06:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:28 smithi163 ceph-mon[40675]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-21T13:06:28.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:28.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:28.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:28.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:28.931 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:28.955 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:28.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:29.057 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:29.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:29.084 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:29.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:29.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:29.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:29.210 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:29.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:29.311 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:29.336 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:29.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:29.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:29.438 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:29.470 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:29.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:29.573 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:29.600 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:29.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:29.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:29.665 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:29 smithi204 ceph-mon[33153]: 8.e scrub starts 2024-03-21T13:06:29.666 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:29 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-21T13:06:29.666 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:29 smithi204 ceph-mon[33153]: osdmap e195: 8 total, 8 up, 5 in 2024-03-21T13:06:29.666 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:29 smithi204 ceph-mon[33153]: 8.8 scrub starts 2024-03-21T13:06:29.666 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:29 smithi204 ceph-mon[33153]: pgmap v334: 148 pgs: 148 active+clean; 8.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 639 B/s rd, 50 MiB/s wr, 806 op/s 2024-03-21T13:06:29.666 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:29 smithi204 ceph-mon[33153]: 3.1b scrub starts 2024-03-21T13:06:29.666 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:29 smithi204 ceph-mon[33153]: 3.1b scrub ok 2024-03-21T13:06:29.666 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:29 smithi204 ceph-mon[33153]: osdmap e196: 8 total, 8 up, 5 in 2024-03-21T13:06:29.703 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:29.728 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:29.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[29621]: 8.e scrub starts 2024-03-21T13:06:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-21T13:06:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[29621]: osdmap e195: 8 total, 8 up, 5 in 2024-03-21T13:06:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[29621]: 8.8 scrub starts 2024-03-21T13:06:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[29621]: pgmap v334: 148 pgs: 148 active+clean; 8.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 639 B/s rd, 50 MiB/s wr, 806 op/s 2024-03-21T13:06:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[29621]: 3.1b scrub starts 2024-03-21T13:06:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[29621]: 3.1b scrub ok 2024-03-21T13:06:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[29621]: osdmap e196: 8 total, 8 up, 5 in 2024-03-21T13:06:29.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[40675]: 8.e scrub starts 2024-03-21T13:06:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-21T13:06:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[40675]: osdmap e195: 8 total, 8 up, 5 in 2024-03-21T13:06:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[40675]: 8.8 scrub starts 2024-03-21T13:06:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[40675]: pgmap v334: 148 pgs: 148 active+clean; 8.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 639 B/s rd, 50 MiB/s wr, 806 op/s 2024-03-21T13:06:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[40675]: 3.1b scrub starts 2024-03-21T13:06:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[40675]: 3.1b scrub ok 2024-03-21T13:06:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:29 smithi163 ceph-mon[40675]: osdmap e196: 8 total, 8 up, 5 in 2024-03-21T13:06:29.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:29.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:29.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:29.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:29.939 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:06:29.939 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:06:29.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:29.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:29.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:30.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:30.109 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:30.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:30.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:30.211 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:30.234 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:30.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:30.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:30.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:30.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:30.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:30.495 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:30.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:30.596 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:30.597 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:06:30.635 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:30.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:30.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:30.641 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:30 smithi204 ceph-mon[33153]: 8.c scrub starts 2024-03-21T13:06:30.641 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:30 smithi204 ceph-mon[33153]: 8.8 scrub ok 2024-03-21T13:06:30.641 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:30 smithi204 ceph-mon[33153]: 6.12 scrub starts 2024-03-21T13:06:30.641 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:30 smithi204 ceph-mon[33153]: 6.12 scrub ok 2024-03-21T13:06:30.641 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:30 smithi204 ceph-mon[33153]: osdmap e197: 8 total, 8 up, 5 in 2024-03-21T13:06:30.690 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-21T13:06:30.737 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:30.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:30 smithi163 ceph-mon[29621]: 8.c scrub starts 2024-03-21T13:06:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:30 smithi163 ceph-mon[29621]: 8.8 scrub ok 2024-03-21T13:06:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:30 smithi163 ceph-mon[29621]: 6.12 scrub starts 2024-03-21T13:06:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:30 smithi163 ceph-mon[29621]: 6.12 scrub ok 2024-03-21T13:06:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:30 smithi163 ceph-mon[29621]: osdmap e197: 8 total, 8 up, 5 in 2024-03-21T13:06:30.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:30 smithi163 ceph-mon[40675]: 8.c scrub starts 2024-03-21T13:06:30.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:30 smithi163 ceph-mon[40675]: 8.8 scrub ok 2024-03-21T13:06:30.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:30 smithi163 ceph-mon[40675]: 6.12 scrub starts 2024-03-21T13:06:30.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:30 smithi163 ceph-mon[40675]: 6.12 scrub ok 2024-03-21T13:06:30.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:30 smithi163 ceph-mon[40675]: osdmap e197: 8 total, 8 up, 5 in 2024-03-21T13:06:30.773 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:30.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:30.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:30.875 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:30.921 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:30.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:31.022 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:31.036 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:31.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:31.073 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:31.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:31.101 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:06:31.174 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:31.219 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:31.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:31.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:31.363 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:31.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:31.465 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:31.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:31.504 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:31.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:31.605 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:31.643 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:31.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:31.744 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 3.a scrub starts 2024-03-21T13:06:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 3.a scrub ok 2024-03-21T13:06:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 8.c scrub ok 2024-03-21T13:06:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 6.1e scrub starts 2024-03-21T13:06:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 6.1e scrub ok 2024-03-21T13:06:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 8.1 starting backfill to osd.6 from (0'0,0'0] MAX to 194'8595 2024-03-21T13:06:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 8.a starting backfill to osd.6 from (0'0,0'0] MAX to 194'8594 2024-03-21T13:06:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 8.b starting backfill to osd.7 from (0'0,0'0] MAX to 194'8490 2024-03-21T13:06:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 8.f starting backfill to osd.6 from (0'0,0'0] MAX to 194'8411 2024-03-21T13:06:31.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 8.7 starting backfill to osd.4 from (0'0,0'0] MAX to 194'8409 2024-03-21T13:06:31.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 8.d starting backfill to osd.4 from (0'0,0'0] MAX to 194'8639 2024-03-21T13:06:31.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 8.4 starting backfill to osd.3 from (0'0,0'0] MAX to 194'8544 2024-03-21T13:06:31.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 194'8580 2024-03-21T13:06:31.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: 8.3 starting backfill to osd.4 from (0'0,0'0] MAX to 194'8513 2024-03-21T13:06:31.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[29621]: pgmap v337: 148 pgs: 1 active+clean+scrubbing, 9 remapped+peering, 138 active+clean; 8.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 1.00k op/s; 325 B/s, 12 objects/s recovering 2024-03-21T13:06:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 3.a scrub starts 2024-03-21T13:06:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 3.a scrub ok 2024-03-21T13:06:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 8.c scrub ok 2024-03-21T13:06:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 6.1e scrub starts 2024-03-21T13:06:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 6.1e scrub ok 2024-03-21T13:06:31.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 8.1 starting backfill to osd.6 from (0'0,0'0] MAX to 194'8595 2024-03-21T13:06:31.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 8.a starting backfill to osd.6 from (0'0,0'0] MAX to 194'8594 2024-03-21T13:06:31.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 8.b starting backfill to osd.7 from (0'0,0'0] MAX to 194'8490 2024-03-21T13:06:31.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 8.f starting backfill to osd.6 from (0'0,0'0] MAX to 194'8411 2024-03-21T13:06:31.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 8.7 starting backfill to osd.4 from (0'0,0'0] MAX to 194'8409 2024-03-21T13:06:31.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 8.d starting backfill to osd.4 from (0'0,0'0] MAX to 194'8639 2024-03-21T13:06:31.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 8.4 starting backfill to osd.3 from (0'0,0'0] MAX to 194'8544 2024-03-21T13:06:31.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 194'8580 2024-03-21T13:06:31.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: 8.3 starting backfill to osd.4 from (0'0,0'0] MAX to 194'8513 2024-03-21T13:06:31.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:31 smithi163 ceph-mon[40675]: pgmap v337: 148 pgs: 1 active+clean+scrubbing, 9 remapped+peering, 138 active+clean; 8.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 1.00k op/s; 325 B/s, 12 objects/s recovering 2024-03-21T13:06:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:31.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:31.886 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:31.924 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:31.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:31.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:32.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:06:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 3.a scrub starts 2024-03-21T13:06:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 3.a scrub ok 2024-03-21T13:06:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 8.c scrub ok 2024-03-21T13:06:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 6.1e scrub starts 2024-03-21T13:06:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 6.1e scrub ok 2024-03-21T13:06:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 8.1 starting backfill to osd.6 from (0'0,0'0] MAX to 194'8595 2024-03-21T13:06:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 8.a starting backfill to osd.6 from (0'0,0'0] MAX to 194'8594 2024-03-21T13:06:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 8.b starting backfill to osd.7 from (0'0,0'0] MAX to 194'8490 2024-03-21T13:06:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 8.f starting backfill to osd.6 from (0'0,0'0] MAX to 194'8411 2024-03-21T13:06:32.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 8.7 starting backfill to osd.4 from (0'0,0'0] MAX to 194'8409 2024-03-21T13:06:32.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 8.d starting backfill to osd.4 from (0'0,0'0] MAX to 194'8639 2024-03-21T13:06:32.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 8.4 starting backfill to osd.3 from (0'0,0'0] MAX to 194'8544 2024-03-21T13:06:32.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 194'8580 2024-03-21T13:06:32.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: 8.3 starting backfill to osd.4 from (0'0,0'0] MAX to 194'8513 2024-03-21T13:06:32.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:31 smithi204 ceph-mon[33153]: pgmap v337: 148 pgs: 1 active+clean+scrubbing, 9 remapped+peering, 138 active+clean; 8.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 1.00k op/s; 325 B/s, 12 objects/s recovering 2024-03-21T13:06:32.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:32.049 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:32.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:32.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:32.151 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:32.192 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:32.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:32.241 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:06:32.293 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:32.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:32.337 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:32.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:32.438 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:32.491 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:32.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:32.593 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:32.644 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:32.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:32.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:32.747 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:32.779 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:32.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:32.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:32.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:32 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:32.881 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:32.882 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:32 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:32.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:32 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:32.918 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:32.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:33.019 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:33.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:33.079 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:33.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:33.082 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:33.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:33.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:33.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:33.409 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:33.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:33.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:33.545 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:06:33.548 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:33.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:33.580 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:33 smithi204 ceph-mon[33153]: pgmap v338: 148 pgs: 1 active+clean+scrubbing, 9 remapped+peering, 138 active+clean; 8.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 853 B/s rd, 39 MiB/s wr, 723 op/s; 234 B/s, 9 objects/s recovering 2024-03-21T13:06:33.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:33 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2115054649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:06:33.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:33 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:06:33.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:33.688 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:33.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:33.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:33.790 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:33.809 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:06:33.810 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:06:33.832 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:33.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:33.894 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:33 smithi163 ceph-mon[29621]: pgmap v338: 148 pgs: 1 active+clean+scrubbing, 9 remapped+peering, 138 active+clean; 8.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 853 B/s rd, 39 MiB/s wr, 723 op/s; 234 B/s, 9 objects/s recovering 2024-03-21T13:06:33.894 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:33 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2115054649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:06:33.894 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:33 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:06:33.894 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:33 smithi163 ceph-mon[40675]: pgmap v338: 148 pgs: 1 active+clean+scrubbing, 9 remapped+peering, 138 active+clean; 8.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 853 B/s rd, 39 MiB/s wr, 723 op/s; 234 B/s, 9 objects/s recovering 2024-03-21T13:06:33.894 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:33 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2115054649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:06:33.894 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:33 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:06:33.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:33.974 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:33.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:34.076 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:34.116 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:34.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:34.217 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:34.253 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:34.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:34.354 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:34.386 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:06:34.387 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:06:34.387 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-21T13:06:34.387 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.578715 2024-03-21T13:06:34.387 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 4 0.5787153536246727 2024-03-21T13:06:34.390 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:34.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:34.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:34.492 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:34.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:34.535 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:34.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:34.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-21T13:06:34.636 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:34.675 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:34.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:34.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:34 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:06:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:34 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:06:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:34 smithi204 ceph-mon[33153]: 4.6 scrub starts 2024-03-21T13:06:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:34 smithi204 ceph-mon[33153]: 4.6 scrub ok 2024-03-21T13:06:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:34 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:06:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:34 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:06:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:34 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:06:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:34 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:06:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:34 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:06:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:34 smithi204 ceph-mon[33153]: osdmap e198: 8 total, 8 up, 5 in 2024-03-21T13:06:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:34 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:06:34.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:34 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:34.778 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:34.817 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:34.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:34.919 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:06:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:06:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[29621]: 4.6 scrub starts 2024-03-21T13:06:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[29621]: 4.6 scrub ok 2024-03-21T13:06:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:06:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:06:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:06:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:06:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:06:34.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[29621]: osdmap e198: 8 total, 8 up, 5 in 2024-03-21T13:06:34.948 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:06:34.948 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:06:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:06:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[40675]: 4.6 scrub starts 2024-03-21T13:06:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[40675]: 4.6 scrub ok 2024-03-21T13:06:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:06:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:06:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:06:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:06:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:06:34.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[40675]: osdmap e198: 8 total, 8 up, 5 in 2024-03-21T13:06:34.949 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:06:34.949 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:34 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:34.977 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:34.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:35.027 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:06:35.078 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:35.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:35.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:35.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:35.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:35.255 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:35.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:35.356 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:35.379 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:35.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:35.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:35.518 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:35.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:35.612 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:35.620 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:35.656 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:35.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:35.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:35.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:35.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:35.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:35.837 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:35 smithi163 ceph-mon[29621]: 5.e scrub starts 2024-03-21T13:06:35.837 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:35 smithi163 ceph-mon[29621]: 5.e scrub ok 2024-03-21T13:06:35.837 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:35 smithi163 ceph-mon[29621]: pgmap v340: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 8 remapped+peering, 138 active+clean; 8.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.7 KiB/s rd, 39 MiB/s wr, 785 op/s; 7594/405366 objects misplaced (1.873%); 6.3 MiB/s, 110 objects/s recovering 2024-03-21T13:06:35.838 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:35 smithi163 ceph-mon[40675]: 5.e scrub starts 2024-03-21T13:06:35.838 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:35 smithi163 ceph-mon[40675]: 5.e scrub ok 2024-03-21T13:06:35.838 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:35 smithi163 ceph-mon[40675]: pgmap v340: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 8 remapped+peering, 138 active+clean; 8.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.7 KiB/s rd, 39 MiB/s wr, 785 op/s; 7594/405366 objects misplaced (1.873%); 6.3 MiB/s, 110 objects/s recovering 2024-03-21T13:06:35.838 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:06:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:06:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:06:35.866 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:35 smithi204 ceph-mon[33153]: 5.e scrub starts 2024-03-21T13:06:35.866 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:35 smithi204 ceph-mon[33153]: 5.e scrub ok 2024-03-21T13:06:35.867 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:35 smithi204 ceph-mon[33153]: pgmap v340: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 8 remapped+peering, 138 active+clean; 8.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.7 KiB/s rd, 39 MiB/s wr, 785 op/s; 7594/405366 objects misplaced (1.873%); 6.3 MiB/s, 110 objects/s recovering 2024-03-21T13:06:35.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:35.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:35.940 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:35.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:36.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:36.080 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:36.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:36.181 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:36.216 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:36.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:36.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:36.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:36.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:36.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:36.451 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:36.489 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:36.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:36.590 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:36.630 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:36.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:36.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:36.755 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:36.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:36.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:36.766 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:06:36.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:36.896 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:36.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:36.998 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:37.028 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:37.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:37.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:37.130 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:37.167 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:37.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:37.269 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:37.312 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:37.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:37.407 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:06:37.407 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:06:37.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:37.445 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:37.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:37.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:37 smithi204 ceph-mon[33153]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:06:37.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:37.587 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:37.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:37.596 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:37 smithi163 ceph-mon[29621]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:06:37.596 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:37 smithi163 ceph-mon[40675]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:06:37.688 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:37.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:37.730 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:37.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:37.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:37.873 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:37.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:37.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:37.975 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:38.012 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:38.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:38.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:38.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:38.123 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-21T13:06:38.152 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:38.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:38.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:38.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:38.307 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:38.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:38.409 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:38.445 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:38.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:38.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:38 smithi204 ceph-mon[33153]: pgmap v341: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 7.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.4 KiB/s rd, 32 MiB/s wr, 1.36k op/s; 69632/389817 objects misplaced (17.863%); 29 MiB/s, 468 objects/s recovering 2024-03-21T13:06:38.546 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:06:38.547 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:38.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:38.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:38.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:38.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:38.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:38.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:38.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:38.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:38 smithi163 ceph-mon[29621]: pgmap v341: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 7.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.4 KiB/s rd, 32 MiB/s wr, 1.36k op/s; 69632/389817 objects misplaced (17.863%); 29 MiB/s, 468 objects/s recovering 2024-03-21T13:06:38.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:38 smithi163 ceph-mon[40675]: pgmap v341: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 7.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.4 KiB/s rd, 32 MiB/s wr, 1.36k op/s; 69632/389817 objects misplaced (17.863%); 29 MiB/s, 468 objects/s recovering 2024-03-21T13:06:38.827 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:38.867 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:38.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:38.968 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:38.993 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:38.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:39.093 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:39.129 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:39.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:39.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:39.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:39.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:39.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:39.362 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:39.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:39.399 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:39.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:39.500 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:39.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:39.537 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:39.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:39.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:39.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:39.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:39.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:39.776 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:39.803 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:39.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:39.905 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:39.940 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:39.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:39.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:40.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:40.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:40.079 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:40.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:40.180 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:40.203 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:40.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:40.305 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:40.338 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:40.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:40.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:40.469 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:40.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:40.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:40.571 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:40.594 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:40.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:40.696 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:40.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:40.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:40 smithi204 ceph-mon[33153]: pgmap v342: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 7.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 616 B/s rd, 0 B/s wr, 669 op/s; 69632/389817 objects misplaced (17.863%); 25 MiB/s, 405 objects/s recovering 2024-03-21T13:06:40.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:40 smithi163 ceph-mon[29621]: pgmap v342: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 7.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 616 B/s rd, 0 B/s wr, 669 op/s; 69632/389817 objects misplaced (17.863%); 25 MiB/s, 405 objects/s recovering 2024-03-21T13:06:40.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:40 smithi163 ceph-mon[40675]: pgmap v342: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 7.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 616 B/s rd, 0 B/s wr, 669 op/s; 69632/389817 objects misplaced (17.863%); 25 MiB/s, 405 objects/s recovering 2024-03-21T13:06:40.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:40.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:40.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:40.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:40.977 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:41.011 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:41.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:41.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:41.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:41.150 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:41.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:41.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:41.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:41.289 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:41.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:41.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:41.430 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:41.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:41.531 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:41.556 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:41.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:41.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:41.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:41.696 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:41.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:41.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:06:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:06:41.799 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:41.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:41.834 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:41.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:41.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:41.959 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:41.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:42.060 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:42.097 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:42.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:42.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:42.231 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:42.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:42.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:42.333 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:42.389 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:42.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:42.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:42.491 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:42.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:42 smithi163 ceph-mon[29621]: pgmap v343: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 7.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 62580/372081 objects misplaced (16.819%); 45 MiB/s, 725 objects/s recovering 2024-03-21T13:06:42.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:42 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:06:42.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:42 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:06:42.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:42 smithi163 ceph-mon[40675]: pgmap v343: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 7.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 62580/372081 objects misplaced (16.819%); 45 MiB/s, 725 objects/s recovering 2024-03-21T13:06:42.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:42 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:06:42.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:42 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:06:42.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:42.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:42.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:42.628 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:42.683 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:42.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:42 smithi204 ceph-mon[33153]: pgmap v343: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 7.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 62580/372081 objects misplaced (16.819%); 45 MiB/s, 725 objects/s recovering 2024-03-21T13:06:42.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:42 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:06:42.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:42 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:06:42.785 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:42.795 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:06:42.795 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:06:42.795 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:06:42.795 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:06:42.795 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:06:42.819 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:42.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:42.827 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:06:42.893 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:06:42.920 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:42.932 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:06:42.936 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:06:42.936 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:06:42.936 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:06:42.936 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:06:42.936 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:06:42.936 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:06:42.936 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:06:42.936 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:06:42.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:42.973 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:42.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:43.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:43.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:43.107 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:43.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:43.209 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:43.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:43.248 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:43.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:43.349 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:43.383 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:43.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:43.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:43.486 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:43.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:43.520 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:43.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:43.621 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:43.662 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:43.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:43.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:43 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3112319735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:43.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:43.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:43 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3112319735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:43.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:43.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:43 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3112319735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:43.763 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:43.792 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:43.809 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:43.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:43.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:43.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:43.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:43.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:44.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:44.085 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:44.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:44.186 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:44.213 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:44.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:44.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:44.315 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:44.355 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:44.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:44.371 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.4 primary-affinity to 0.578715 (8379262) 2024-03-21T13:06:44.457 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:44.486 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:44.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:44.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:44.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:44.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:44.729 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:44.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:44 smithi204 ceph-mon[33153]: pgmap v344: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 7.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 62580/372081 objects misplaced (16.819%); 45 MiB/s, 725 objects/s recovering 2024-03-21T13:06:44.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:44 smithi204 ceph-mon[33153]: 5.1b scrub starts 2024-03-21T13:06:44.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:44 smithi204 ceph-mon[33153]: 5.1b scrub ok 2024-03-21T13:06:44.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2123234269' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5787153536246727}]: dispatch 2024-03-21T13:06:44.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:44 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5787153536246727}]: dispatch 2024-03-21T13:06:44.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:44 smithi163 ceph-mon[29621]: pgmap v344: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 7.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 62580/372081 objects misplaced (16.819%); 45 MiB/s, 725 objects/s recovering 2024-03-21T13:06:44.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:44 smithi163 ceph-mon[29621]: 5.1b scrub starts 2024-03-21T13:06:44.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:44 smithi163 ceph-mon[29621]: 5.1b scrub ok 2024-03-21T13:06:44.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2123234269' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5787153536246727}]: dispatch 2024-03-21T13:06:44.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:44 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5787153536246727}]: dispatch 2024-03-21T13:06:44.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:44 smithi163 ceph-mon[40675]: pgmap v344: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 7.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 62580/372081 objects misplaced (16.819%); 45 MiB/s, 725 objects/s recovering 2024-03-21T13:06:44.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:44 smithi163 ceph-mon[40675]: 5.1b scrub starts 2024-03-21T13:06:44.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:44 smithi163 ceph-mon[40675]: 5.1b scrub ok 2024-03-21T13:06:44.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2123234269' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5787153536246727}]: dispatch 2024-03-21T13:06:44.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:44 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5787153536246727}]: dispatch 2024-03-21T13:06:44.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:44.762 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:44.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:44.863 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:44.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:44.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:44.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:45.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:45.037 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:45.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:45.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:45.182 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:45.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:45.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:45.319 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:45.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:45.421 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:45.457 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:45.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:45.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:45.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:45.598 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:45.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:45.664 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:45 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5787153536246727}]': finished 2024-03-21T13:06:45.664 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:45 smithi204 ceph-mon[33153]: osdmap e199: 8 total, 8 up, 5 in 2024-03-21T13:06:45.699 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:45.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:45.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:45.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:45.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:45 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5787153536246727}]': finished 2024-03-21T13:06:45.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:45 smithi163 ceph-mon[29621]: osdmap e199: 8 total, 8 up, 5 in 2024-03-21T13:06:45.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:06:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:06:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:06:45.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:45 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5787153536246727}]': finished 2024-03-21T13:06:45.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:45 smithi163 ceph-mon[40675]: osdmap e199: 8 total, 8 up, 5 in 2024-03-21T13:06:45.840 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:45.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:45.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:45.983 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:46.016 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:46.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:46.118 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:46.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:46.156 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:46.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:46.187 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:06:46.187 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-03-21T13:06:46.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-03-21T13:06:46.187 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:06:46.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.187 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:126317 7.6 7:6d89952b:::benchmark_data_smithi163_80312_object8876:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-21T13:06:46.187 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:33.942283+0000", 2024-03-21T13:06:46.187 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 252.23906655600001, 2024-03-21T13:06:46.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.14407416100000001, 2024-03-21T13:06:46.187 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.187 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.188 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.188 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.188 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.188 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 126317 2024-03-21T13:06:46.188 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.188 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.188 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.188 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.942283+0000", 2024-03-21T13:06:46.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.188 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.188 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.942283+0000", 2024-03-21T13:06:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.942284+0000", 2024-03-21T13:06:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5120000000000001e-06 2024-03-21T13:06:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.942289+0000", 2024-03-21T13:06:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9010000000000002e-06 2024-03-21T13:06:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.942300+0000", 2024-03-21T13:06:46.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0611999999999999e-05 2024-03-21T13:06:46.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.942308+0000", 2024-03-21T13:06:46.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5900000000000002e-06 2024-03-21T13:06:46.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.942357+0000", 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9873000000000002e-05 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.942470+0000", 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112304 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.942575+0000", 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105178 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.944677+0000", 2024-03-21T13:06:46.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021018510000000001 2024-03-21T13:06:46.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.944911+0000", 2024-03-21T13:06:46.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023414400000000001 2024-03-21T13:06:46.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:34.086255+0000", 2024-03-21T13:06:46.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.141344145 2024-03-21T13:06:46.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.193 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.193 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.193 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:34.086312+0000", 2024-03-21T13:06:46.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6462999999999998e-05 2024-03-21T13:06:46.193 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.193 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.193 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.193 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:34.086357+0000", 2024-03-21T13:06:46.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5587999999999998e-05 2024-03-21T13:06:46.193 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.193 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.193 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.193 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.194 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.194 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:126319 7.5 7:a59aa12b:::benchmark_data_smithi163_80312_object8878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-21T13:06:46.194 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:33.944925+0000", 2024-03-21T13:06:46.194 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 252.23642478599999, 2024-03-21T13:06:46.194 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.120827952, 2024-03-21T13:06:46.194 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.194 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.194 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.194 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.194 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.194 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 126319 2024-03-21T13:06:46.194 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.194 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.944925+0000", 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.944925+0000", 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.944925+0000", 2024-03-21T13:06:46.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.46e-07 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.944927+0000", 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1830000000000001e-06 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.944931+0000", 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1799999999999998e-06 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.944935+0000", 2024-03-21T13:06:46.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5470000000000001e-06 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.947001+0000", 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020660729999999999 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.947099+0000", 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7879000000000003e-05 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-03-21T13:06:46.197 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.947175+0000", 2024-03-21T13:06:46.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5632999999999999e-05 2024-03-21T13:06:46.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:33.948591+0000", 2024-03-21T13:06:46.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001416388 2024-03-21T13:06:46.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:34.063957+0000", 2024-03-21T13:06:46.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.11536619200000001 2024-03-21T13:06:46.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:34.065689+0000", 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017313879999999999 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:34.065724+0000", 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5478000000000002e-05 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:34.065753+0000", 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8665000000000001e-05 2024-03-21T13:06:46.199 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.200 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.200 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.200 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.200 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.200 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:127782 7.6 7:6d883e73:::benchmark_data_smithi163_80312_object10341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-21T13:06:46.200 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:35.555036+0000", 2024-03-21T13:06:46.200 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 250.62631378699999, 2024-03-21T13:06:46.200 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.119824075, 2024-03-21T13:06:46.200 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.200 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.200 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.200 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.200 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.201 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 127782 2024-03-21T13:06:46.201 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.201 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.201 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.201 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.201 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.555036+0000", 2024-03-21T13:06:46.201 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.201 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.201 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.201 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.201 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.555036+0000", 2024-03-21T13:06:46.201 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.201 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.202 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.202 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.555037+0000", 2024-03-21T13:06:46.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6800000000000009e-07 2024-03-21T13:06:46.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.202 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.202 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.555040+0000", 2024-03-21T13:06:46.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2090000000000001e-06 2024-03-21T13:06:46.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.202 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.202 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.555047+0000", 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5340000000000002e-06 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.555053+0000", 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4190000000000001e-06 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.555093+0000", 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9938999999999998e-05 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.555174+0000", 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1698000000000003e-05 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.555246+0000", 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1561000000000001e-05 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.557153+0000", 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019065130000000001 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.204 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.674013+0000", 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.116860494 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.674754+0000", 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000741109 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.674812+0000", 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7601e-05 2024-03-21T13:06:46.205 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.206 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.206 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.206 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:35.674860+0000", 2024-03-21T13:06:46.206 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8029999999999999e-05 2024-03-21T13:06:46.206 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.206 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.206 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.206 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.206 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.206 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153170 7.9 7:9d09162f:::benchmark_data_smithi163_80312_object35729:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.206 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.285100+0000", 2024-03-21T13:06:46.206 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.89624959, 2024-03-21T13:06:46.206 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.020565492, 2024-03-21T13:06:46.207 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.207 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:06:46.207 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.207 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.207 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.207 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153170 2024-03-21T13:06:46.207 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.207 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.207 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.207 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.207 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.285100+0000", 2024-03-21T13:06:46.207 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.207 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.208 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.208 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.208 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.285100+0000", 2024-03-21T13:06:46.208 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.208 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.208 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.208 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.208 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.285101+0000", 2024-03-21T13:06:46.208 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0299999999999998e-07 2024-03-21T13:06:46.208 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.208 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.208 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.208 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.285104+0000", 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7419999999999999e-06 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.285114+0000", 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4970000000000007e-06 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.285121+0000", 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6919999999999996e-06 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.209 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.210 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.285179+0000", 2024-03-21T13:06:46.210 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8681000000000003e-05 2024-03-21T13:06:46.210 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.210 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.210 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.210 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.285188+0000", 2024-03-21T13:06:46.210 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.524e-06 2024-03-21T13:06:46.210 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.210 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.210 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.210 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.305579+0000", 2024-03-21T13:06:46.211 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.020391627 2024-03-21T13:06:46.211 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.211 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.211 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.211 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.305665+0000", 2024-03-21T13:06:46.211 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6026000000000006e-05 2024-03-21T13:06:46.211 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.211 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.211 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.211 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.211 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.211 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153216 7.9 7:91f73df4:::benchmark_data_smithi163_80312_object35775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.211 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.315950+0000", 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.86539981000001, 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99694698100000001, 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153216 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.315950+0000", 2024-03-21T13:06:46.212 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.315950+0000", 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.315951+0000", 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.212e-06 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.213 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.315956+0000", 2024-03-21T13:06:46.214 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.203e-06 2024-03-21T13:06:46.214 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.214 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.214 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.214 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.315968+0000", 2024-03-21T13:06:46.214 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1418e-05 2024-03-21T13:06:46.214 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.214 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.214 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.214 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.315976+0000", 2024-03-21T13:06:46.214 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0909999999999993e-06 2024-03-21T13:06:46.214 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.215 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.215 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.215 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.316055+0000", 2024-03-21T13:06:46.215 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9188000000000001e-05 2024-03-21T13:06:46.215 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.215 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.215 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.215 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.316064+0000", 2024-03-21T13:06:46.215 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0610000000000002e-06 2024-03-21T13:06:46.215 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.215 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.216 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.216 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.306065+0000", 2024-03-21T13:06:46.216 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99000070600000001 2024-03-21T13:06:46.216 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.216 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.216 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.217 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.306158+0000", 2024-03-21T13:06:46.217 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3289999999999996e-05 2024-03-21T13:06:46.217 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.217 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.217 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.217 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.306268+0000", 2024-03-21T13:06:46.217 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010986599999999999 2024-03-21T13:06:46.217 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.217 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.217 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.217 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.312642+0000", 2024-03-21T13:06:46.217 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063740059999999998 2024-03-21T13:06:46.217 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.218 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.218 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.218 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.312730+0000", 2024-03-21T13:06:46.218 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8001e-05 2024-03-21T13:06:46.218 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.218 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.218 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.218 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.312786+0000", 2024-03-21T13:06:46.218 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5875999999999998e-05 2024-03-21T13:06:46.218 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.218 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.218 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.218 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.312841+0000", 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5748999999999998e-05 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.312897+0000", 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5313999999999999e-05 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153253 7.9 7:93df882d:::benchmark_data_smithi163_80312_object35812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.349581+0000", 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.83176879499999, 2024-03-21T13:06:46.219 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96820903199999997, 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153253 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349581+0000", 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.220 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349581+0000", 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349582+0000", 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.156e-06 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349587+0000", 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1819999999999997e-06 2024-03-21T13:06:46.221 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.222 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.222 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.222 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349601+0000", 2024-03-21T13:06:46.222 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4228e-05 2024-03-21T13:06:46.222 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.222 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.222 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.222 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349609+0000", 2024-03-21T13:06:46.222 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0620000000000001e-06 2024-03-21T13:06:46.222 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.222 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.222 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.222 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349663+0000", 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3477000000000003e-05 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349672+0000", 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2890000000000003e-06 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.308324+0000", 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95865230999999995 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.223 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.308423+0000", 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9018000000000006e-05 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.308534+0000", 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001104 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.312747+0000", 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042135929999999999 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.224 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.317629+0000", 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0048818189999999999 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.317683+0000", 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3535999999999997e-05 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.317744+0000", 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1001000000000001e-05 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.317790+0000", 2024-03-21T13:06:46.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5961000000000003e-05 2024-03-21T13:06:46.226 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.226 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.226 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.226 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.226 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153254 7.9 7:940fbc80:::benchmark_data_smithi163_80312_object35813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.226 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.349645+0000", 2024-03-21T13:06:46.226 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.83170473800001, 2024-03-21T13:06:46.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96835058100000004, 2024-03-21T13:06:46.226 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.226 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153254 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349645+0000", 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349645+0000", 2024-03-21T13:06:46.227 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349645+0000", 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0100000000000005e-07 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349648+0000", 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8090000000000001e-06 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349655+0000", 2024-03-21T13:06:46.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2289999999999997e-06 2024-03-21T13:06:46.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.229 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.229 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349662+0000", 2024-03-21T13:06:46.229 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8480000000000003e-06 2024-03-21T13:06:46.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.229 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.229 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349696+0000", 2024-03-21T13:06:46.229 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3794999999999997e-05 2024-03-21T13:06:46.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.229 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.349703+0000", 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6769999999999998e-06 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.310620+0000", 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96091728499999995 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.310753+0000", 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013303100000000001 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.310879+0000", 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012600099999999999 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.317643+0000", 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067640499999999998 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.317848+0000", 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020458600000000001 2024-03-21T13:06:46.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.317902+0000", 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3968000000000002e-05 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.317953+0000", 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1257000000000001e-05 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.317995+0000", 2024-03-21T13:06:46.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2543999999999998e-05 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153264 7.9 7:91e7ae46:::benchmark_data_smithi163_80312_object35823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.358854+0000", 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.82249523600001, 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96400667200000001, 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.233 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153264 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.358854+0000", 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.358854+0000", 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.358864+0000", 2024-03-21T13:06:46.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2890000000000003e-06 2024-03-21T13:06:46.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.358981+0000", 2024-03-21T13:06:46.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117081 2024-03-21T13:06:46.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.358989+0000", 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4440000000000008e-06 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.358996+0000", 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3690000000000001e-06 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.359052+0000", 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5075999999999999e-05 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.359060+0000", 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3729999999999993e-06 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.312955+0000", 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95389520400000005 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.313095+0000", 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013971900000000001 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.313214+0000", 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119104 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.317804+0000", 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0045904090000000002 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.318053+0000", 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024894000000000001 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.322757+0000", 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047037789999999999 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.322811+0000", 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3850000000000001e-05 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.322861+0000", 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0034999999999998e-05 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.239 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153280 7.9 7:9cb9d23b:::benchmark_data_smithi163_80312_object35839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.369231+0000", 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.81211853600001, 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95379065100000004, 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153280 2024-03-21T13:06:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.369231+0000", 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.369231+0000", 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.369232+0000", 2024-03-21T13:06:46.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3850000000000001e-06 2024-03-21T13:06:46.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.242 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.242 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.369239+0000", 2024-03-21T13:06:46.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3389999999999998e-06 2024-03-21T13:06:46.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.242 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.242 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.369249+0000", 2024-03-21T13:06:46.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0190999999999999e-05 2024-03-21T13:06:46.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.242 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.369256+0000", 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5349999999999999e-06 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.369312+0000", 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5313999999999999e-05 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.369321+0000", 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9160000000000007e-06 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.315257+0000", 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94593656400000004 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.315522+0000", 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026526700000000002 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.315630+0000", 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010784499999999999 2024-03-21T13:06:46.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.317861+0000", 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022309819999999998 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.322911+0000", 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0050495779999999999 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.322948+0000", 2024-03-21T13:06:46.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6674000000000002e-05 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.322988+0000", 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0769000000000002e-05 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.323022+0000", 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3291999999999999e-05 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.247 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153292 7.9 7:91b13366:::benchmark_data_smithi163_80312_object35851:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.247 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.379400+0000", 2024-03-21T13:06:46.247 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.801949544, 2024-03-21T13:06:46.247 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.950570472, 2024-03-21T13:06:46.247 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.247 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.247 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.247 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.247 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.247 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153292 2024-03-21T13:06:46.247 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.247 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.248 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.248 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379400+0000", 2024-03-21T13:06:46.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.248 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.248 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379400+0000", 2024-03-21T13:06:46.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.248 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379402+0000", 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5910000000000001e-06 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379407+0000", 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7939999999999999e-06 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379421+0000", 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3370000000000001e-05 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379430+0000", 2024-03-21T13:06:46.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4110000000000002e-06 2024-03-21T13:06:46.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379495+0000", 2024-03-21T13:06:46.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4728999999999996e-05 2024-03-21T13:06:46.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379505+0000", 2024-03-21T13:06:46.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.005e-05 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.318098+0000", 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.93859305599999998 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.318213+0000", 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011506999999999999 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.318334+0000", 2024-03-21T13:06:46.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000120748 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.323062+0000", 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047280239999999999 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.329752+0000", 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066898690000000002 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.329861+0000", 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108902 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.329923+0000", 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2063000000000005e-05 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.329970+0000", 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7794999999999999e-05 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.253 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153293 7.9 7:9e9f3f8f:::benchmark_data_smithi163_80312_object35852:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.379454+0000", 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.80189502299999, 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95073481500000001, 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153293 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.254 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379454+0000", 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379454+0000", 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379455+0000", 2024-03-21T13:06:46.255 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-03-21T13:06:46.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.256 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379458+0000", 2024-03-21T13:06:46.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2540000000000001e-06 2024-03-21T13:06:46.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.256 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379466+0000", 2024-03-21T13:06:46.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0600000000000008e-06 2024-03-21T13:06:46.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379473+0000", 2024-03-21T13:06:46.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6549999999999998e-06 2024-03-21T13:06:46.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.257 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.257 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:46.258 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379543+0000", 2024-03-21T13:06:46.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9978e-05 2024-03-21T13:06:46.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.259 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.259 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.379550+0000", 2024-03-21T13:06:46.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8499999999999996e-06 2024-03-21T13:06:46.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.259 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.259 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.320467+0000", 2024-03-21T13:06:46.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94091704200000004 2024-03-21T13:06:46.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.260 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.260 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.320598+0000", 2024-03-21T13:06:46.260 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000131514 2024-03-21T13:06:46.260 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.260 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.260 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.260 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.320724+0000", 2024-03-21T13:06:46.260 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125375 2024-03-21T13:06:46.260 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.260 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.260 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.260 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.329758+0000", 2024-03-21T13:06:46.260 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0090339210000000003 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.330030+0000", 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027241400000000001 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.330079+0000", 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9354000000000002e-05 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.330150+0000", 2024-03-21T13:06:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0446999999999998e-05 2024-03-21T13:06:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.330189+0000", 2024-03-21T13:06:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9360000000000003e-05 2024-03-21T13:06:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153300 7.9 7:9f3c01be:::benchmark_data_smithi163_80312_object35859:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.384813+0000", 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.79653650399999, 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95477722600000003, 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153300 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.384813+0000", 2024-03-21T13:06:46.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.384813+0000", 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.384814+0000", 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.257e-06 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.384820+0000", 2024-03-21T13:06:46.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5520000000000004e-06 2024-03-21T13:06:46.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.384834+0000", 2024-03-21T13:06:46.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4076e-05 2024-03-21T13:06:46.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.384842+0000", 2024-03-21T13:06:46.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1419999999999993e-06 2024-03-21T13:06:46.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.384904+0000", 2024-03-21T13:06:46.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1502999999999997e-05 2024-03-21T13:06:46.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.384913+0000", 2024-03-21T13:06:46.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6649999999999998e-06 2024-03-21T13:06:46.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.323097+0000", 2024-03-21T13:06:46.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.93818375499999995 2024-03-21T13:06:46.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.323214+0000", 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116606 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.323351+0000", 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013725099999999999 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.329767+0000", 2024-03-21T13:06:46.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064157720000000001 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.330233+0000", 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000466506 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.339499+0000", 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0092658930000000007 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.339550+0000", 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0587999999999997e-05 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.339590+0000", 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.066e-05 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153305 7.9 7:937cc94c:::benchmark_data_smithi163_80312_object35864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.390034+0000", 2024-03-21T13:06:46.269 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.791315236, 2024-03-21T13:06:46.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95400041999999996, 2024-03-21T13:06:46.270 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.270 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.270 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.270 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.270 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.270 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153305 2024-03-21T13:06:46.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.270 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.390034+0000", 2024-03-21T13:06:46.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.390034+0000", 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.390035+0000", 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0809999999999999e-06 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.390040+0000", 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6500000000000004e-06 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.390053+0000", 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2849e-05 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.390061+0000", 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3389999999999993e-06 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.272 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.390113+0000", 2024-03-21T13:06:46.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1888999999999997e-05 2024-03-21T13:06:46.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.390121+0000", 2024-03-21T13:06:46.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.481e-06 2024-03-21T13:06:46.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.325365+0000", 2024-03-21T13:06:46.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.93524459500000001 2024-03-21T13:06:46.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.325469+0000", 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103793 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.325577+0000", 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108121 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.330245+0000", 2024-03-21T13:06:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0046681680000000003 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.330282+0000", 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6962000000000003e-05 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.343932+0000", 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013649809000000001 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.343985+0000", 2024-03-21T13:06:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2642999999999997e-05 2024-03-21T13:06:46.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.344035+0000", 2024-03-21T13:06:46.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0040000000000002e-05 2024-03-21T13:06:46.276 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.276 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.276 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.276 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153329 7.9 7:90a03416:::benchmark_data_smithi163_80312_object35888:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.276 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.412748+0000", 2024-03-21T13:06:46.276 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.768601633, 2024-03-21T13:06:46.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.932701746, 2024-03-21T13:06:46.277 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.277 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.277 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.277 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.277 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.277 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153329 2024-03-21T13:06:46.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.277 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.412748+0000", 2024-03-21T13:06:46.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.412748+0000", 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.412750+0000", 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.782e-06 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.412756+0000", 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0889999999999996e-06 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.412769+0000", 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2829e-05 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.412777+0000", 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1750000000000005e-06 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.412835+0000", 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8489000000000002e-05 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.412845+0000", 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0236e-05 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.327509+0000", 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.91466382400000001 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.327609+0000", 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9226000000000002e-05 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.327715+0000", 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106167 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.334941+0000", 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072261790000000001 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.334999+0000", 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7868999999999998e-05 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.345358+0000", 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010358792 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.345411+0000", 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3152000000000002e-05 2024-03-21T13:06:46.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.345450+0000", 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8936999999999997e-05 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153344 7.9 7:9de1b397:::benchmark_data_smithi163_80312_object35903:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.430852+0000", 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.75049754400001, 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.91571856399999996, 2024-03-21T13:06:46.283 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153344 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.430852+0000", 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.430852+0000", 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.430860+0000", 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8010000000000004e-06 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.430866+0000", 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6309999999999997e-06 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.430887+0000", 2024-03-21T13:06:46.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0886999999999998e-05 2024-03-21T13:06:46.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.430896+0000", 2024-03-21T13:06:46.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.721e-06 2024-03-21T13:06:46.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.430956+0000", 2024-03-21T13:06:46.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0134999999999999e-05 2024-03-21T13:06:46.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.430966+0000", 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6719999999999999e-06 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.330315+0000", 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.89934900500000003 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.330417+0000", 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102592 2024-03-21T13:06:46.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.330525+0000", 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010749199999999999 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.335010+0000", 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044849349999999998 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.339445+0000", 2024-03-21T13:06:46.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044350819999999999 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.346472+0000", 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0070270929999999999 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.346528+0000", 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5989999999999998e-05 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.346571+0000", 2024-03-21T13:06:46.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2527999999999997e-05 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153371 7.5 7:ab790543:::benchmark_data_smithi163_80312_object35930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.462706+0000", 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.71864342200001, 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13309364500000001, 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.290 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.291 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153371 2024-03-21T13:06:46.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.291 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.462706+0000", 2024-03-21T13:06:46.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.462706+0000", 2024-03-21T13:06:46.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.462708+0000", 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.632e-06 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.462715+0000", 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4479999999999997e-06 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.462731+0000", 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5503999999999999e-05 2024-03-21T13:06:46.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.462740+0000", 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5170000000000005e-06 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.462827+0000", 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6432000000000002e-05 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.462980+0000", 2024-03-21T13:06:46.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000153275 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.463330+0000", 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00034988800000000002 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.465441+0000", 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002111183 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.465611+0000", 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000169767 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.595701+0000", 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.130090597 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.595755+0000", 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3338000000000003e-05 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.595800+0000", 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5064000000000001e-05 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153387 7.9 7:92f59d76:::benchmark_data_smithi163_80312_object35946:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.634206+0000", 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.54714348600001, 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.71251094999999998, 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.296 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153387 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.634206+0000", 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.634206+0000", 2024-03-21T13:06:46.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.634208+0000", 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0490000000000002e-06 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.634215+0000", 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.905e-06 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.634231+0000", 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6188000000000002e-05 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.634240+0000", 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2029999999999998e-06 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.634307+0000", 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6858000000000003e-05 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.634319+0000", 2024-03-21T13:06:46.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1527e-05 2024-03-21T13:06:46.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.332722+0000", 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69840297500000004 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.332827+0000", 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105571 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.332944+0000", 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117106 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.339397+0000", 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006452859 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.339635+0000", 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000237555 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.346623+0000", 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069884489999999999 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.346678+0000", 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4979999999999999e-05 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.346717+0000", 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8724999999999999e-05 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.303 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153390 7.9 7:991a4e76:::benchmark_data_smithi163_80312_object35949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.304 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.636139+0000", 2024-03-21T13:06:46.304 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.54521029200001, 2024-03-21T13:06:46.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.71069978099999997, 2024-03-21T13:06:46.304 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.304 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.304 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.304 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.304 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.304 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153390 2024-03-21T13:06:46.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.304 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.636139+0000", 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.636139+0000", 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.636140+0000", 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.085e-06 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.636145+0000", 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.079e-06 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.636157+0000", 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1688000000000001e-05 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.636166+0000", 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8599999999999999e-06 2024-03-21T13:06:46.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.636227+0000", 2024-03-21T13:06:46.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1143999999999996e-05 2024-03-21T13:06:46.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.636238+0000", 2024-03-21T13:06:46.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1116e-05 2024-03-21T13:06:46.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.335036+0000", 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69879817600000005 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.335136+0000", 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9419000000000005e-05 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.335247+0000", 2024-03-21T13:06:46.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011116300000000001 2024-03-21T13:06:46.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.339403+0000", 2024-03-21T13:06:46.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041558360000000004 2024-03-21T13:06:46.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.343190+0000", 2024-03-21T13:06:46.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037873080000000001 2024-03-21T13:06:46.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.346753+0000", 2024-03-21T13:06:46.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035630280000000002 2024-03-21T13:06:46.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.346799+0000", 2024-03-21T13:06:46.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5482999999999998e-05 2024-03-21T13:06:46.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.346839+0000", 2024-03-21T13:06:46.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0395999999999997e-05 2024-03-21T13:06:46.310 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153395 7.9 7:9da658f1:::benchmark_data_smithi163_80312_object35954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.646978+0000", 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.53437196199999, 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69999115499999998, 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153395 2024-03-21T13:06:46.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.646978+0000", 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.646978+0000", 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.646979+0000", 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.702e-06 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.646986+0000", 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7089999999999996e-06 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.647003+0000", 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7368e-05 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.647013+0000", 2024-03-21T13:06:46.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8819999999999996e-06 2024-03-21T13:06:46.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.647089+0000", 2024-03-21T13:06:46.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5801000000000002e-05 2024-03-21T13:06:46.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.647104+0000", 2024-03-21T13:06:46.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.509e-05 2024-03-21T13:06:46.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.337313+0000", 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69020935699999997 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.337409+0000", 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5947999999999997e-05 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.337511+0000", 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102086 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.339658+0000", 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002146289 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.343230+0000", 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035723230000000001 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.346890+0000", 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036598009999999999 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.346934+0000", 2024-03-21T13:06:46.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4369999999999997e-05 2024-03-21T13:06:46.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.346969+0000", 2024-03-21T13:06:46.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4428999999999999e-05 2024-03-21T13:06:46.317 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.317 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.317 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:153397 7.9 7:904102cb:::benchmark_data_smithi163_80312_object35956:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:03:00.651909+0000", 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.52944083099999, 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69751050599999997, 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 153397 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.651909+0000", 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.651909+0000", 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.651910+0000", 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2449999999999999e-06 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.651916+0000", 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5289999999999997e-06 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.651934+0000", 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7116e-05 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.651942+0000", 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7360000000000007e-06 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.652001+0000", 2024-03-21T13:06:46.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8721000000000002e-05 2024-03-21T13:06:46.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:06:46.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:00.652013+0000", 2024-03-21T13:06:46.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1652e-05 2024-03-21T13:06:46.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:06:46.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.339699+0000", 2024-03-21T13:06:46.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.68768674200000002 2024-03-21T13:06:46.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:06:46.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.339812+0000", 2024-03-21T13:06:46.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112126 2024-03-21T13:06:46.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:06:46.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.339930+0000", 2024-03-21T13:06:46.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011882 2024-03-21T13:06:46.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.341821+0000", 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001890636 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.345295+0000", 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003474043 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.349339+0000", 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040434770000000002 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.349382+0000", 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3390999999999998e-05 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:03:01.349419+0000", 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7271999999999998e-05 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:06:46.324 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:06:46.333 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:46.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:46.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:46.436 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:46.461 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:06:46.471 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:46.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:46.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:46.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:46 smithi204 ceph-mon[33153]: pgmap v346: 148 pgs: 3 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 136 active+clean; 7.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 921 B/s rd, 0 B/s wr, 1.14k op/s; 61397/371145 objects misplaced (16.543%); 47 MiB/s, 760 objects/s recovering 2024-03-21T13:06:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:46 smithi204 ceph-mon[33153]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-21T13:06:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:46 smithi204 ceph-mon[33153]: osdmap e200: 8 total, 8 up, 5 in 2024-03-21T13:06:46.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:46 smithi163 ceph-mon[40675]: pgmap v346: 148 pgs: 3 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 136 active+clean; 7.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 921 B/s rd, 0 B/s wr, 1.14k op/s; 61397/371145 objects misplaced (16.543%); 47 MiB/s, 760 objects/s recovering 2024-03-21T13:06:46.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:46 smithi163 ceph-mon[40675]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-21T13:06:46.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:46 smithi163 ceph-mon[40675]: osdmap e200: 8 total, 8 up, 5 in 2024-03-21T13:06:46.508 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:46 smithi163 ceph-mon[29621]: pgmap v346: 148 pgs: 3 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 136 active+clean; 7.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 921 B/s rd, 0 B/s wr, 1.14k op/s; 61397/371145 objects misplaced (16.543%); 47 MiB/s, 760 objects/s recovering 2024-03-21T13:06:46.508 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:46 smithi163 ceph-mon[29621]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-21T13:06:46.508 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:46 smithi163 ceph-mon[29621]: osdmap e200: 8 total, 8 up, 5 in 2024-03-21T13:06:46.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:46.613 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:46.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:46.714 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:46.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:46.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:46.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:46.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:46.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:46.989 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:47.029 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:47.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:47.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:47.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:47.166 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:47.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:47.271 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:47.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:47.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:47.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:47.332 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-21T13:06:47.421 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:47.456 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:47.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:47.558 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:47.595 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:47.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:47.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:47.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:47.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:47.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:47.838 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:47.875 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:47.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:47.976 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:48.015 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:48.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:48.061 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:48.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:48.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:48.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:48.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:48.252 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:48.296 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:48.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:48.398 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:48.472 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:48.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:48.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:48 smithi163 ceph-mon[29621]: pgmap v348: 148 pgs: 6 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 133 active+clean; 7.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 56343/357498 objects misplaced (15.760%); 60 MiB/s, 960 objects/s recovering 2024-03-21T13:06:48.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:48 smithi163 ceph-mon[40675]: pgmap v348: 148 pgs: 6 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 133 active+clean; 7.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 56343/357498 objects misplaced (15.760%); 60 MiB/s, 960 objects/s recovering 2024-03-21T13:06:48.574 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:48.597 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:48.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:48.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:48.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:48.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:48.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:48.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:48 smithi204 ceph-mon[33153]: pgmap v348: 148 pgs: 6 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 133 active+clean; 7.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 56343/357498 objects misplaced (15.760%); 60 MiB/s, 960 objects/s recovering 2024-03-21T13:06:48.838 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:48.881 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:48.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:48.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:48.984 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:49.024 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:49.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:49.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:49.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:49.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:49.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:49.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:49.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:49.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:49.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:49.466 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:49.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:49.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:49.569 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:49.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:49.610 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:49.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:49.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:49.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:49.750 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:49.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:49.851 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:49.889 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:49.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:49.990 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:50.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:50.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:50.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:50.134 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:50.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:50.171 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:50.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:50.272 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:50.310 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:50.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:50.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:50.412 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:50.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:50.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:50.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:50.558 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:50 smithi163 ceph-mon[29621]: pgmap v349: 148 pgs: 6 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 133 active+clean; 7.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 383 B/s rd, 0 B/s wr, 607 op/s; 56343/357498 objects misplaced (15.760%); 30 MiB/s, 474 objects/s recovering 2024-03-21T13:06:50.558 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:06:50.558 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:50.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:50 smithi163 ceph-mon[40675]: pgmap v349: 148 pgs: 6 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 133 active+clean; 7.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 383 B/s rd, 0 B/s wr, 607 op/s; 56343/357498 objects misplaced (15.760%); 30 MiB/s, 474 objects/s recovering 2024-03-21T13:06:50.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:06:50.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:50.588 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:50.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:50.690 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:50.693 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:50.746 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:50.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:50.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:50 smithi204 ceph-mon[33153]: pgmap v349: 148 pgs: 6 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 133 active+clean; 7.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 383 B/s rd, 0 B/s wr, 607 op/s; 56343/357498 objects misplaced (15.760%); 30 MiB/s, 474 objects/s recovering 2024-03-21T13:06:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:06:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:50.847 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:50.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:50.895 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:50.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:50.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:51.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:51.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:51.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:51.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:51.159 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:51.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:51.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:51.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:51.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:51.300 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:51.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:51.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:51.433 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:06:51.439 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:51.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:51.541 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:51.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:51.581 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:51.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:51.662 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:06:51.662 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:06:51.662 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-21T13:06:51.663 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-21T13:06:51.672 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:51 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1274684400' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:06:51.672 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:51 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:06:51.672 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:51 smithi163 ceph-mon[29621]: Health check update: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-21T13:06:51.672 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:51 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1274684400' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:06:51.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:51 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:06:51.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:51 smithi163 ceph-mon[40675]: Health check update: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-21T13:06:51.682 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:51.727 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:51.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:51.735 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:06:51.735 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:06:51.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:51 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1274684400' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:06:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:51 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:06:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:51 smithi204 ceph-mon[33153]: Health check update: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-21T13:06:51.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:06:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:06:51.754 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:06:51.797 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:06:51.829 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:51.842 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:06:51.865 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:51.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:51.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:51.881 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:06:51.928 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:06:51.967 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:51.973 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:06:51.978 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:06:51.978 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:06:51.978 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:06:51.978 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:06:51.978 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:06:51.978 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:06:51.978 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:06:51.979 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:06:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:52.018 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:52.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:52.119 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:52.160 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:52.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:52.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:52.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:52.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:52.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:52.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:52.445 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:52.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:52.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:52.586 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:52.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:52.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:52.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:52.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:52 smithi204 ceph-mon[33153]: pgmap v350: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.42k op/s; 49134/337941 objects misplaced (14.539%); 62 MiB/s, 997 objects/s recovering 2024-03-21T13:06:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:52 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:06:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:52 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:06:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:52 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:06:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:52 smithi204 ceph-mon[33153]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-21T13:06:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:52 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:06:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:52 smithi204 ceph-mon[33153]: osdmap e201: 8 total, 8 up, 5 in 2024-03-21T13:06:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:52 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/409558258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:52.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[29621]: pgmap v350: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.42k op/s; 49134/337941 objects misplaced (14.539%); 62 MiB/s, 997 objects/s recovering 2024-03-21T13:06:52.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:06:52.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:06:52.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:06:52.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[29621]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-21T13:06:52.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:06:52.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[29621]: osdmap e201: 8 total, 8 up, 5 in 2024-03-21T13:06:52.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/409558258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:52.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[40675]: pgmap v350: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.42k op/s; 49134/337941 objects misplaced (14.539%); 62 MiB/s, 997 objects/s recovering 2024-03-21T13:06:52.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:06:52.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:06:52.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:06:52.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[40675]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-21T13:06:52.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:06:52.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[40675]: osdmap e201: 8 total, 8 up, 5 in 2024-03-21T13:06:52.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:52 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/409558258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:06:52.827 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:52.863 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:52.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:52.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:52.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:52.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:53.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:53.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:53.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:53.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:53.140 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:53.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:53.241 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:53.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:53.280 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:53.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:53.381 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:53.421 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:53.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:53.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:53.522 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:53.556 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:53.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:53.658 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:53.707 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:53.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:53.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:53.808 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:53.846 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:53.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:53.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:53.948 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:53.985 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:53.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:53.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:54.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:54.107 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:54.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:54.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:54.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:54.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:54.263 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:54.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:54.364 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:54.405 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:54.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:54.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:54.542 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:54.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:54.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:54.679 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:54.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:54.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:54.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:54 smithi163 ceph-mon[29621]: pgmap v352: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 511 B/s rd, 0 B/s wr, 1.38k op/s; 49134/337941 objects misplaced (14.539%); 55 MiB/s, 877 objects/s recovering 2024-03-21T13:06:54.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:54 smithi163 ceph-mon[40675]: pgmap v352: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 511 B/s rd, 0 B/s wr, 1.38k op/s; 49134/337941 objects misplaced (14.539%); 55 MiB/s, 877 objects/s recovering 2024-03-21T13:06:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:54 smithi204 ceph-mon[33153]: pgmap v352: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 511 B/s rd, 0 B/s wr, 1.38k op/s; 49134/337941 objects misplaced (14.539%); 55 MiB/s, 877 objects/s recovering 2024-03-21T13:06:54.781 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:54.817 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:54.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:54.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:54.932 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:06:54.953 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:54.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:55.055 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:55.089 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:55.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:55.191 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:06:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:55.243 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:55.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:06:55.344 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:55.382 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:55.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:55.477 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:06:55.477 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:06:55.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:55.519 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:55.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:55.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:55.621 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:55.659 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:55.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:55.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:55.797 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:55.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:55.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:55.899 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:55.937 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:55.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:56.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:06:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:06:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:06:56.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:56.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:56.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:56.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:56.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:06:56.180 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:56.248 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:56.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:56.350 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:56.386 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:56.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:56.488 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:56.527 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:56.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:56.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:56.629 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:56.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:56.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:56.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:56 smithi204 ceph-mon[33153]: pgmap v353: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 545 B/s rd, 0 B/s wr, 922 op/s; 49134/337941 objects misplaced (14.539%); 37 MiB/s, 598 objects/s recovering 2024-03-21T13:06:56.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:56 smithi163 ceph-mon[29621]: pgmap v353: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 545 B/s rd, 0 B/s wr, 922 op/s; 49134/337941 objects misplaced (14.539%); 37 MiB/s, 598 objects/s recovering 2024-03-21T13:06:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:56 smithi163 ceph-mon[40675]: pgmap v353: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 545 B/s rd, 0 B/s wr, 922 op/s; 49134/337941 objects misplaced (14.539%); 37 MiB/s, 598 objects/s recovering 2024-03-21T13:06:56.770 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:56.805 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:56.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:56.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:56.907 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:56.912 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-21T13:06:56.941 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:56.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:56.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:56.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:57.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:57.080 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:57.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:57.182 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:57.229 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:57.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:57.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:57.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:57.365 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:57.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:57.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:57.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:57.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:57.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:57.639 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:57.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:57.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:57.741 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:06:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:57 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:06:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:57 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:06:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:57 smithi204 ceph-mon[33153]: osdmap e202: 8 total, 8 up, 5 in 2024-03-21T13:06:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:57 smithi204 ceph-mon[33153]: pgmap v355: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 42338/318150 objects misplaced (13.308%); 64 MiB/s, 1.02k objects/s recovering 2024-03-21T13:06:57.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:57 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:06:57.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:57 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:06:57.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:57 smithi163 ceph-mon[40675]: osdmap e202: 8 total, 8 up, 5 in 2024-03-21T13:06:57.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:57 smithi163 ceph-mon[40675]: pgmap v355: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 42338/318150 objects misplaced (13.308%); 64 MiB/s, 1.02k objects/s recovering 2024-03-21T13:06:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:57 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:06:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:57 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:06:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:57 smithi163 ceph-mon[29621]: osdmap e202: 8 total, 8 up, 5 in 2024-03-21T13:06:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:57 smithi163 ceph-mon[29621]: pgmap v355: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 42338/318150 objects misplaced (13.308%); 64 MiB/s, 1.02k objects/s recovering 2024-03-21T13:06:57.776 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:57.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:06:57.878 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:57.921 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:57.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:58.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:58.057 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:58.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:58.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:58.147 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:06:58.147 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:06:58.147 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 202, 2024-03-21T13:06:58.147 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "d4bf79c0-e781-11ee-95cd-87774f69a715", 2024-03-21T13:06:58.147 INFO:teuthology.orchestra.run.smithi163.stdout: "created": "2024-03-21T12:52:56.031419+0000", 2024-03-21T13:06:58.147 INFO:teuthology.orchestra.run.smithi163.stdout: "modified": "2024-03-21T13:06:56.424940+0000", 2024-03-21T13:06:58.147 INFO:teuthology.orchestra.run.smithi163.stdout: "last_up_change": "2024-03-21T12:58:23.775302+0000", 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: "last_in_change": "2024-03-21T13:06:28.461702+0000", 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_num": 5801984, 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_set": [ 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: "noscrub", 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: "pglog_hardlimit", 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: "purged_snapdirs", 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_deletes", 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: "sortbitwise" 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_version": 18, 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: "full_ratio": 0.94999998807907104, 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-21T13:06:58.148 INFO:teuthology.orchestra.run.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-21T13:06:58.149 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_snapshot": "", 2024-03-21T13:06:58.149 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_max": 8, 2024-03-21T13:06:58.149 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd": 8, 2024-03-21T13:06:58.149 INFO:teuthology.orchestra.run.smithi163.stdout: "require_min_compat_client": "luminous", 2024-03-21T13:06:58.149 INFO:teuthology.orchestra.run.smithi163.stdout: "min_compat_client": "luminous", 2024-03-21T13:06:58.149 INFO:teuthology.orchestra.run.smithi163.stdout: "require_osd_release": "squid", 2024-03-21T13:06:58.149 INFO:teuthology.orchestra.run.smithi163.stdout: "allow_crimson": false, 2024-03-21T13:06:58.149 INFO:teuthology.orchestra.run.smithi163.stdout: "pools": [ 2024-03-21T13:06:58.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.149 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 1, 2024-03-21T13:06:58.149 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".mgr", 2024-03-21T13:06:58.149 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:56:20.557490+0000", 2024-03-21T13:06:58.150 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:06:58.150 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:06:58.150 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:06:58.150 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:06:58.150 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:06:58.150 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:06:58.150 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:06:58.150 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:06:58.150 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:06:58.150 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:06:58.150 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:06:58.150 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:06:58.150 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 1, 2024-03-21T13:06:58.151 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 1, 2024-03-21T13:06:58.151 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 1, 2024-03-21T13:06:58.151 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 1, 2024-03-21T13:06:58.151 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 1, 2024-03-21T13:06:58.151 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:06:58.151 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:06:58.151 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:06:58.151 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:06:58.151 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:06:58.151 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:06:58.151 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:06:58.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.151 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "21", 2024-03-21T13:06:58.152 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:06:58.152 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:06:58.152 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:06:58.152 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:06:58.152 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:06:58.152 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:06:58.152 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:06:58.152 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:06:58.152 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:06:58.152 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:06:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:06:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:06:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:06:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:06:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:06:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:06:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:06:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:06:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:06:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:06:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:06:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:06:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:06:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:06:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:06:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:06:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:06:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:06:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:06:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:06:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:06:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:06:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:06:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:06:58.155 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:06:58.155 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:06:58.155 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:06:58.155 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-03-21T13:06:58.155 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_max": 32, 2024-03-21T13:06:58.155 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_min": 1 2024-03-21T13:06:58.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.155 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:06:58.155 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": {} 2024-03-21T13:06:58.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.155 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:06:58.155 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:06:58.156 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 7.8899998664855957, 2024-03-21T13:06:58.156 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 7.8899998664855957, 2024-03-21T13:06:58.156 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.37999999523162842, 2024-03-21T13:06:58.156 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 3, 2024-03-21T13:06:58.156 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 3, 2024-03-21T13:06:58.156 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:06:58.156 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:06:58.156 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:06:58.156 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.156 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 2, 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "datapool", 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:35.713383+0000", 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:06:58.157 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:06:58.158 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:06:58.158 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:06:58.158 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 3, 2024-03-21T13:06:58.158 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 3, 2024-03-21T13:06:58.158 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 3, 2024-03-21T13:06:58.158 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 3, 2024-03-21T13:06:58.158 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 3, 2024-03-21T13:06:58.158 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:06:58.158 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:06:58.158 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:06:58.158 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:06:58.158 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:06:58.158 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "67", 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 2, 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 67, 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:06:58.159 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:06:58.160 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:06:58.160 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:06:58.160 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:06:58.160 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:06:58.160 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:06:58.160 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:06:58.160 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:06:58.160 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:06:58.160 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:06:58.160 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:06:58.161 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:06:58.162 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:06:58.162 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:06:58.162 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:06:58.162 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:06:58.162 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:06:58.162 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd": {} 2024-03-21T13:06:58.162 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.162 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:06:58.162 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:06:58.162 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.6600000858306885, 2024-03-21T13:06:58.162 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.6600000858306885, 2024-03-21T13:06:58.162 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.5, 2024-03-21T13:06:58.162 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.3300000429153442, 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.3300000429153442, 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 3, 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".rgw.root", 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:36.436384+0000", 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:06:58.163 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:06:58.164 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:06:58.165 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:06:58.165 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:06:58.165 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:06:58.165 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:06:58.165 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:06:58.165 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:06:58.165 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.165 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "65", 2024-03-21T13:06:58.165 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:06:58.165 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:06:58.165 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:06:58.165 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:06:58.165 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:06:58.166 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:06:58.166 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:06:58.166 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:06:58.166 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:06:58.166 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:06:58.166 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:06:58.166 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:06:58.166 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:06:58.166 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:06:58.166 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:06:58.166 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:06:58.167 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:06:58.168 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:06:58.168 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:06:58.168 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:06:58.168 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:06:58.168 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:06:58.168 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:06:58.168 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:06:58.168 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:06:58.168 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:06:58.168 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:06:58.168 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:06:58.168 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.168 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.4800000190734863, 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.4800000190734863, 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.62999999523162842, 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.559999942779541, 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.559999942779541, 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 4, 2024-03-21T13:06:58.169 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.log", 2024-03-21T13:06:58.170 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:38.049304+0000", 2024-03-21T13:06:58.170 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:06:58.170 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:06:58.170 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:06:58.170 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:06:58.170 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:06:58.170 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:06:58.170 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:06:58.170 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:06:58.170 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:06:58.170 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:06:58.170 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:06:58.170 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.171 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "67", 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:06:58.172 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:06:58.173 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:06:58.173 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:06:58.173 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:06:58.173 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:06:58.173 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:06:58.173 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:06:58.173 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:06:58.173 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:06:58.173 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:06:58.173 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:06:58.173 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:06:58.173 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:06:58.173 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.174 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:06:58.174 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:06:58.174 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:06:58.174 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:06:58.174 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:06:58.174 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:06:58.174 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:06:58.174 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:06:58.174 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:06:58.174 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:06:58.174 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:06:58.174 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:06:58.174 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 3.7100000381469727, 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 3.7100000381469727, 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.62999999523162842, 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 2.3399999141693115, 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 2.3399999141693115, 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.175 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.176 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.176 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 5, 2024-03-21T13:06:58.176 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.control", 2024-03-21T13:06:58.176 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:39.990512+0000", 2024-03-21T13:06:58.176 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:06:58.176 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:06:58.176 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:06:58.176 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:06:58.176 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:06:58.176 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:06:58.176 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:06:58.176 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:06:58.176 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:06:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:06:58.178 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:06:58.178 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.178 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "69", 2024-03-21T13:06:58.178 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:06:58.178 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:06:58.178 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:06:58.178 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:06:58.178 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:06:58.178 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:06:58.178 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:06:58.178 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:06:58.179 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:06:58.180 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:06:58.181 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:06:58.181 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:06:58.181 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:06:58.181 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:06:58.181 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:06:58.181 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.181 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:06:58.181 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:06:58.181 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.4800000190734863, 2024-03-21T13:06:58.181 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.4800000190734863, 2024-03-21T13:06:58.181 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.62999999523162842, 2024-03-21T13:06:58.181 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.559999942779541, 2024-03-21T13:06:58.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:58.183 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.559999942779541, 2024-03-21T13:06:58.183 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:06:58.183 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:06:58.183 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:06:58.183 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.183 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.183 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 6, 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.meta", 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:41.463686+0000", 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:06:58.184 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:06:58.185 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:06:58.185 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:06:58.185 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:06:58.185 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:06:58.185 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:06:58.185 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:06:58.185 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:06:58.185 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:06:58.185 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:06:58.186 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:06:58.186 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:06:58.186 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:06:58.186 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:06:58.186 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.186 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "71", 2024-03-21T13:06:58.186 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:06:58.186 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:06:58.186 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:06:58.186 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:06:58.186 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:06:58.186 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:06:58.186 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:06:58.187 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:06:58.188 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:06:58.188 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:06:58.188 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:06:58.188 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:06:58.188 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:06:58.188 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.188 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:06:58.188 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:06:58.188 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:06:58.188 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:06:58.188 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:06:58.188 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:06:58.188 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_bias": 4 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 3.2200000286102295, 2024-03-21T13:06:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 3.2200000286102295, 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.62999999523162842, 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 2.0299999713897705, 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 2.0299999713897705, 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 8, 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "unique_pool_1", 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T13:04:45.973511+0000", 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:06:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:06:58.191 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:06:58.191 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:06:58.191 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:06:58.191 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:06:58.191 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:06:58.191 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:06:58.191 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:06:58.191 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:06:58.191 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:06:58.191 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:06:58.191 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 16, 2024-03-21T13:06:58.191 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 16, 2024-03-21T13:06:58.191 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 16, 2024-03-21T13:06:58.192 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 16, 2024-03-21T13:06:58.192 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 16, 2024-03-21T13:06:58.192 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:06:58.192 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:06:58.192 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:06:58.192 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:06:58.192 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:06:58.192 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:06:58.192 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:06:58.192 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "173", 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:06:58.193 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:06:58.194 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:06:58.194 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:06:58.194 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:06:58.194 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:06:58.194 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:06:58.194 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:06:58.194 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:06:58.194 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:06:58.194 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:06:58.194 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:06:58.194 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:06:58.194 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:06:58.195 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:06:58.196 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:06:58.196 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-03-21T13:06:58.196 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.196 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:06:58.196 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:06:58.196 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 3.4800000190734863, 2024-03-21T13:06:58.196 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 3.4800000190734863, 2024-03-21T13:06:58.196 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.62999999523162842, 2024-03-21T13:06:58.196 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 2.190000057220459, 2024-03-21T13:06:58.196 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 2.190000057220459, 2024-03-21T13:06:58.196 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:06:58.196 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:06:58.196 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f", 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0.0852813720703125, 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:06:58.197 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 8, 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 196, 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6801", 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:06:58.198 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6802", 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6804", 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:06:58.199 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:06:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6803", 2024-03-21T13:06:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:06:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6801/3086410981", 2024-03-21T13:06:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6802/3086410981", 2024-03-21T13:06:58.201 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6804/3086410981", 2024-03-21T13:06:58.201 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6803/3086410981", 2024-03-21T13:06:58.201 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:06:58.201 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:06:58.201 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:06:58.201 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.201 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.201 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-03-21T13:06:58.201 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "36e05f89-c238-41f7-be65-1553b529e8ce", 2024-03-21T13:06:58.201 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:06:58.201 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 0, 2024-03-21T13:06:58.201 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 0, 2024-03-21T13:06:58.202 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:06:58.202 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:06:58.202 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:06:58.202 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 13, 2024-03-21T13:06:58.202 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 110, 2024-03-21T13:06:58.202 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:06:58.202 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:06:58.202 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:06:58.202 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.203 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.206 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.206 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6805", 2024-03-21T13:06:58.206 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:06:58.206 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.206 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.206 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.206 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:06:58.206 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.206 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.206 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6806", 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6808", 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6807", 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6805/3666062784", 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6806/3666062784", 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6808/3666062784", 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6807/3666062784", 2024-03-21T13:06:58.208 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:06:58.209 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:06:58.209 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:06:58.209 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.209 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.209 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-03-21T13:06:58.209 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f", 2024-03-21T13:06:58.209 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:06:58.209 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 0, 2024-03-21T13:06:58.209 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 0, 2024-03-21T13:06:58.209 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:06:58.209 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:06:58.209 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:06:58.210 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 18, 2024-03-21T13:06:58.210 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 168, 2024-03-21T13:06:58.210 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:06:58.210 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:06:58.210 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:06:58.210 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.210 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.210 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.210 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6809", 2024-03-21T13:06:58.210 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:06:58.210 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.210 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6810", 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.211 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6812", 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6811", 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.213 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6809/4009213507", 2024-03-21T13:06:58.213 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6810/4009213507", 2024-03-21T13:06:58.213 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6812/4009213507", 2024-03-21T13:06:58.213 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6811/4009213507", 2024-03-21T13:06:58.213 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:06:58.213 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:06:58.213 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:06:58.213 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.213 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-03-21T13:06:58.213 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d", 2024-03-21T13:06:58.213 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:06:58.214 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:06:58.214 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:06:58.214 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:06:58.214 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:06:58.214 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:06:58.214 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 25, 2024-03-21T13:06:58.214 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 196, 2024-03-21T13:06:58.214 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:06:58.214 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:06:58.214 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:06:58.214 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.214 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.215 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6813", 2024-03-21T13:06:58.215 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:06:58.215 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.215 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.215 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:06:58.215 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.215 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.215 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6814", 2024-03-21T13:06:58.215 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:06:58.215 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.215 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6816", 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.216 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6815", 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6813/3981878189", 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6814/3981878189", 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6816/3981878189", 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6815/3981878189", 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.218 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-03-21T13:06:58.218 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471", 2024-03-21T13:06:58.218 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:06:58.218 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:06:58.218 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:06:58.218 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0.578704833984375, 2024-03-21T13:06:58.218 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:06:58.218 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:06:58.218 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 30, 2024-03-21T13:06:58.218 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 196, 2024-03-21T13:06:58.218 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:06:58.218 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:06:58.219 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:06:58.219 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.219 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.219 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6800", 2024-03-21T13:06:58.219 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:06:58.219 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.219 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.219 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:06:58.219 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.219 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6801", 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6803", 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6802", 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6800/719276054", 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6801/719276054", 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6803/719276054", 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6802/719276054", 2024-03-21T13:06:58.221 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1", 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 0, 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 0, 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:06:58.222 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 39, 2024-03-21T13:06:58.223 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 123, 2024-03-21T13:06:58.223 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:06:58.223 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:06:58.223 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:06:58.223 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.223 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.223 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6804", 2024-03-21T13:06:58.224 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:06:58.225 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.225 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.225 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:06:58.225 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.225 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.225 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6805", 2024-03-21T13:06:58.225 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:06:58.225 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.225 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.225 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:06:58.226 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.226 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.226 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6807", 2024-03-21T13:06:58.226 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:06:58.226 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.226 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.226 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:06:58.226 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.226 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.226 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6806", 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6804/1346947668", 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6805/1346947668", 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6807/1346947668", 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6806/1346947668", 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b", 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 47, 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 199, 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:06:58.228 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.229 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.229 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6808", 2024-03-21T13:06:58.229 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:06:58.229 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.229 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.229 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:06:58.229 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.229 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.229 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6809", 2024-03-21T13:06:58.229 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6811", 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:06:58.230 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.231 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.231 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6810", 2024-03-21T13:06:58.231 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:06:58.231 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.231 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.231 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6808/200077988", 2024-03-21T13:06:58.231 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6809/200077988", 2024-03-21T13:06:58.231 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6811/200077988", 2024-03-21T13:06:58.231 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6810/200077988", 2024-03-21T13:06:58.231 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:06:58.231 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49", 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 56, 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 199, 2024-03-21T13:06:58.232 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6812", 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.233 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6813", 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6815", 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6814", 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6812/315664888", 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6813/315664888", 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6815/315664888", 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6814/315664888", 2024-03-21T13:06:58.235 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_xinfo": [ 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:06:58.236 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:06:15.867503+0000", 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 65536, 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:06:14.315506+0000", 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.237 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-03-21T13:06:58.238 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:06:58.238 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:06:58.238 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:06:58.238 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:06:58.238 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 65536, 2024-03-21T13:06:58.238 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:06:33.507624+0000", 2024-03-21T13:06:58.238 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:06:58.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.238 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-03-21T13:06:58.238 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:06:58.238 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:06:58.238 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:06:10.599314+0000", 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:06:00.427280+0000", 2024-03-21T13:06:58.239 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 65536, 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:06:32.719778+0000", 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-03-21T13:06:58.240 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:06:58.241 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:06:58.241 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:06:58.241 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:06:58.241 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:06:58.241 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:06:51.232427+0000", 2024-03-21T13:06:58.241 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:06:58.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.241 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-03-21T13:06:58.241 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:06:58.241 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:06:58.241 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:06:58.241 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:06:58.242 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:06:58.242 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:06:56.228460+0000", 2024-03-21T13:06:58.242 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:06:58.242 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.242 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:06:58.242 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap": [ 2024-03-21T13:06:58.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.242 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "6.11", 2024-03-21T13:06:58.242 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:06:58.242 INFO:teuthology.orchestra.run.smithi163.stdout: 7, 2024-03-21T13:06:58.242 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:06:58.242 INFO:teuthology.orchestra.run.smithi163.stdout: 4 2024-03-21T13:06:58.242 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.243 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.243 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:06:58.243 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_items": [ 2024-03-21T13:06:58.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.243 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "3.19", 2024-03-21T13:06:58.243 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:06:58.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.243 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 4, 2024-03-21T13:06:58.243 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 7 2024-03-21T13:06:58.243 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.243 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.244 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "6.1c", 2024-03-21T13:06:58.244 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:06:58.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.244 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 7, 2024-03-21T13:06:58.244 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 0 2024-03-21T13:06:58.244 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.244 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.244 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.244 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:06:58.244 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_primaries": [], 2024-03-21T13:06:58.244 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_temp": [ 2024-03-21T13:06:58.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.245 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "8.1", 2024-03-21T13:06:58.245 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:06:58.245 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:06:58.245 INFO:teuthology.orchestra.run.smithi163.stdout: 0, 2024-03-21T13:06:58.245 INFO:teuthology.orchestra.run.smithi163.stdout: 2 2024-03-21T13:06:58.245 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.245 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "8.3", 2024-03-21T13:06:58.245 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:06:58.245 INFO:teuthology.orchestra.run.smithi163.stdout: 6, 2024-03-21T13:06:58.245 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:06:58.245 INFO:teuthology.orchestra.run.smithi163.stdout: 2 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "8.4", 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: 4, 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: 6, 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: 2 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "8.7", 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:06:58.246 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: 6, 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: 2 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "8.a", 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: 4, 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: 2 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "8.b", 2024-03-21T13:06:58.247 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: 4, 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: 2 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "8.d", 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: 7, 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: 6, 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: 2 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "8.e", 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: 0, 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: 4, 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: 2 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "8.f", 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: 4, 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: 2 2024-03-21T13:06:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_temp": [], 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: "blocklist": { 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3453855427": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/2571244897": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/3073378854": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1334591766": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/116934309": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3890775846": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1783921524": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2553440211": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1812694077": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:06:58.250 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1070540283": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/2354157786": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1591817392": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/4152198951": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/1798115633": "2024-03-22T12:54:00.224135+0000" 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: "range_blocklist": {}, 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profiles": { 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: "default": { 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van" 2024-03-21T13:06:58.251 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps_queue": [], 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout: "new_removed_snaps": [], 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout: "new_purged_snaps": [], 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_node_flags": {}, 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout: "device_class_flags": {}, 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode": { 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_enabled": false, 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_bucket_count": 0, 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout: "degraded_stretch_mode": 0, 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout: "recovering_stretch_mode": 0, 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_bucket": 0 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:06:58.252 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:06:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:58.253 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:58.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:58.354 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:58.395 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:58.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:58.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:58.535 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:58.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:58.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:58.637 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:06:58.676 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:58.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:58.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:06:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:58 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/311340533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:06:58.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:58 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/311340533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:06:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:58 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/311340533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:06:58.778 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:06:58.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:58.816 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:58.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:06:58.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:58.953 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:58.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:59.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:06:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:59.087 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:59.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:06:59.188 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:59.227 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:59.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:59.329 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:59.364 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:59.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:59.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:59.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:59.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:59.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:59.604 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:06:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:59.648 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:06:59.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:06:59.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:06:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:06:59 smithi204 ceph-mon[33153]: pgmap v356: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 825 op/s; 42338/318150 objects misplaced (13.308%); 31 MiB/s, 500 objects/s recovering 2024-03-21T13:06:59.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:06:59 smithi163 ceph-mon[29621]: pgmap v356: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 825 op/s; 42338/318150 objects misplaced (13.308%); 31 MiB/s, 500 objects/s recovering 2024-03-21T13:06:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:06:59 smithi163 ceph-mon[40675]: pgmap v356: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 825 op/s; 42338/318150 objects misplaced (13.308%); 31 MiB/s, 500 objects/s recovering 2024-03-21T13:06:59.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:59.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:59.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:06:59.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:06:59.929 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:06:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:06:59.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:06:59.979 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:00.022 INFO:tasks.thrashosds.thrasher:j is {'epoch': 202, 'fsid': 'd4bf79c0-e781-11ee-95cd-87774f69a715', 'created': '2024-03-21T12:52:56.031419+0000', 'modified': '2024-03-21T13:06:56.424940+0000', 'last_up_change': '2024-03-21T12:58:23.775302+0000', 'last_in_change': '2024-03-21T13:06:28.461702+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T12:56:20.557490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-21T12:58:35.713383+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6600000858306885, 'score_stable': 2.6600000858306885, 'optimal_score': 0.5, 'raw_score_acting': 1.3300000429153442, 'raw_score_stable': 1.3300000429153442, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-21T12:58:36.436384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4800000190734863, 'score_stable': 2.4800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-21T12:58:38.049304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.7100000381469727, 'score_stable': 3.7100000381469727, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.3399999141693115, 'raw_score_stable': 2.3399999141693115, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-21T12:58:39.990512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4800000190734863, 'score_stable': 2.4800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-21T12:58:41.463686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2200000286102295, 'score_stable': 3.2200000286102295, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.0299999713897705, 'raw_score_stable': 2.0299999713897705, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-21T13:04:45.973511+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '173', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.4800000190734863, 'score_stable': 3.4800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'c34a3b25-9e02-46e6-8f9a-d0af2615258f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.0852813720703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 196, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6801', 'nonce': 3086410981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6802', 'nonce': 3086410981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6804', 'nonce': 3086410981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6803', 'nonce': 3086410981}]}, 'public_addr': '172.21.15.163:6801/3086410981', 'cluster_addr': '172.21.15.163:6802/3086410981', 'heartbeat_back_addr': '172.21.15.163:6804/3086410981', 'heartbeat_front_addr': '172.21.15.163:6803/3086410981', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '36e05f89-c238-41f7-be65-1553b529e8ce', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6805', 'nonce': 3666062784}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6806', 'nonce': 3666062784}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6808', 'nonce': 3666062784}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6807', 'nonce': 3666062784}]}, 'public_addr': '172.21.15.163:6805/3666062784', 'cluster_addr': '172.21.15.163:6806/3666062784', 'heartbeat_back_addr': '172.21.15.163:6808/3666062784', 'heartbeat_front_addr': '172.21.15.163:6807/3666062784', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '968ec78d-25b8-4905-b328-d490c38ca85f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 168, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6809', 'nonce': 4009213507}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6810', 'nonce': 4009213507}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6812', 'nonce': 4009213507}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6811', 'nonce': 4009213507}]}, 'public_addr': '172.21.15.163:6809/4009213507', 'cluster_addr': '172.21.15.163:6810/4009213507', 'heartbeat_back_addr': '172.21.15.163:6812/4009213507', 'heartbeat_front_addr': '172.21.15.163:6811/4009213507', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '67dfa4f6-1709-48f6-971d-c6287233492d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 196, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6813', 'nonce': 3981878189}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6814', 'nonce': 3981878189}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6816', 'nonce': 3981878189}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6815', 'nonce': 3981878189}]}, 'public_addr': '172.21.15.163:6813/3981878189', 'cluster_addr': '172.21.15.163:6814/3981878189', 'heartbeat_back_addr': '172.21.15.163:6816/3981878189', 'heartbeat_front_addr': '172.21.15.163:6815/3981878189', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7fd311a9-f3c2-456d-8ab1-341cf5b33471', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.578704833984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 196, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 719276054}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6801', 'nonce': 719276054}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6803', 'nonce': 719276054}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 719276054}]}, 'public_addr': '172.21.15.204:6800/719276054', 'cluster_addr': '172.21.15.204:6801/719276054', 'heartbeat_back_addr': '172.21.15.204:6803/719276054', 'heartbeat_front_addr': '172.21.15.204:6802/719276054', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a87759c8-251c-415d-9387-a49f7d2f36d1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 123, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 1346947668}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6805', 'nonce': 1346947668}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6807', 'nonce': 1346947668}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 1346947668}]}, 'public_addr': '172.21.15.204:6804/1346947668', 'cluster_addr': '172.21.15.204:6805/1346947668', 'heartbeat_back_addr': '172.21.15.204:6807/1346947668', 'heartbeat_front_addr': '172.21.15.204:6806/1346947668', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5eb135cd-f677-4d0c-b23a-738f31615b6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 199, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 200077988}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6809', 'nonce': 200077988}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6811', 'nonce': 200077988}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 200077988}]}, 'public_addr': '172.21.15.204:6808/200077988', 'cluster_addr': '172.21.15.204:6809/200077988', 'heartbeat_back_addr': '172.21.15.204:6811/200077988', 'heartbeat_front_addr': '172.21.15.204:6810/200077988', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4f528e10-2c93-4bc2-a54c-06ca4132ed49', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 199, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 315664888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6813', 'nonce': 315664888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6815', 'nonce': 315664888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 315664888}]}, 'public_addr': '172.21.15.204:6812/315664888', 'cluster_addr': '172.21.15.204:6813/315664888', 'heartbeat_back_addr': '172.21.15.204:6815/315664888', 'heartbeat_front_addr': '172.21.15.204:6814/315664888', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:06:15.867503+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-21T13:06:14.315506+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-21T13:06:33.507624+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:06:10.599314+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:06:00.427280+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-21T13:06:32.719778+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:06:51.232427+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:06:56.228460+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '6.11', 'osds': [7, 3, 4]}], 'pg_upmap_items': [{'pgid': '3.19', 'mappings': [{'from': 4, 'to': 7}]}, {'pgid': '6.1c', 'mappings': [{'from': 7, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.1', 'osds': [3, 0, 2]}, {'pgid': '8.3', 'osds': [6, 3, 2]}, {'pgid': '8.4', 'osds': [4, 6, 2]}, {'pgid': '8.7', 'osds': [3, 6, 2]}, {'pgid': '8.a', 'osds': [3, 4, 2]}, {'pgid': '8.b', 'osds': [3, 4, 2]}, {'pgid': '8.d', 'osds': [7, 6, 2]}, {'pgid': '8.e', 'osds': [0, 4, 2]}, {'pgid': '8.f', 'osds': [3, 4, 2]}], 'primary_temp': [], 'blocklist': {'172.21.15.163:0/3453855427': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/2571244897': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:6800/3073378854': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/1334591766': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/116934309': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:0/3890775846': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/1783921524': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:6800/2553440211': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1812694077': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1070540283': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/2354157786': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1591817392': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/4152198951': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:6800/1798115633': '2024-03-22T12:54:00.224135+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-21T13:07:00.022 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:07:00.030 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:00.050 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:00.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:00.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:00.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:00.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:00.201 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:00.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:00.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:00.339 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:00.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:00.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:00.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:00.485 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:00.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:00.586 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:00.623 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:00.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:00.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:00.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:00.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:00.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:00.902 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:00.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:00.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:01.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:01.044 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:01.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:01.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:01.182 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:01.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:01.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:01.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:01.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:01.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:01.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:01.423 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:01.453 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:01.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:01.555 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:01.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:01.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:01.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:01.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:01.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:01.830 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:01.832 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:01.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:01.877 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:01.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:01.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:01 smithi204 ceph-mon[33153]: pgmap v357: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.42k op/s; 35656/298134 objects misplaced (11.960%); 55 MiB/s, 877 objects/s recovering 2024-03-21T13:07:02.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:07:01 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:07:02.017 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:02.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:02.118 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:01 smithi163 ceph-mon[29621]: pgmap v357: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.42k op/s; 35656/298134 objects misplaced (11.960%); 55 MiB/s, 877 objects/s recovering 2024-03-21T13:07:02.118 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:01 smithi163 ceph-mon[40675]: pgmap v357: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.42k op/s; 35656/298134 objects misplaced (11.960%); 55 MiB/s, 877 objects/s recovering 2024-03-21T13:07:02.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:02.155 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:02.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:02.257 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:02.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:02.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:02.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:02.432 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:02.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:02.443 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:07:02.485 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:07:02.519 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:07:02.533 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:02.572 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:02.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:02.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:02.580 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:07:02.619 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:07:02.661 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:07:02.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:02.699 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:07:02.712 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:02.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:02.747 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:07:02.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:07:02.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:07:02.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:07:02.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:07:02.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:07:02.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:07:02.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:07:02.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:07:02.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:02.852 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:02.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:02.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:02.953 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:02.992 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:02.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:03.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:02 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:03.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:02 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3336335418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:07:03.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:02 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:03.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:02 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3336335418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:07:03.094 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:03.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:03.130 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:03.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:03.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:03.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:02 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:03.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:02 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3336335418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:07:03.259 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:03.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:03.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:03.400 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:03.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:03.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:03.501 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:03.535 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:03.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:03.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:03.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:03.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:03.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:03.773 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:03.808 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:03.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:03.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:03.938 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:03.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:03.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:04.040 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:04.078 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:04.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:04.104 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:03 smithi163 ceph-mon[29621]: pgmap v358: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s; 35656/298134 objects misplaced (11.960%); 51 MiB/s, 820 objects/s recovering 2024-03-21T13:07:04.104 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:03 smithi163 ceph-mon[40675]: pgmap v358: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s; 35656/298134 objects misplaced (11.960%); 51 MiB/s, 820 objects/s recovering 2024-03-21T13:07:04.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:04.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:04.216 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:04.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:04.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:03 smithi204 ceph-mon[33153]: pgmap v358: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s; 35656/298134 objects misplaced (11.960%); 51 MiB/s, 820 objects/s recovering 2024-03-21T13:07:04.317 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:04.354 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:04.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:04.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:04.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:04.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:04.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:04.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:04.625 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:04.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:04.727 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:04.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:04.764 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:04.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:04.865 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:04.902 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:04.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:04.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:05.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:05.043 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:05.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:05.145 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:05.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:05.169 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:05.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:05.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:05.308 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:05.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:05.409 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:05.438 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:05.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:05.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:05.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:05.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:05.680 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:05.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:05.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:05.808 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:05.845 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:05.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:05.946 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:05.972 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:05.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:05.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:07:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:05 smithi204 ceph-mon[33153]: pgmap v359: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s; 35656/298134 objects misplaced (11.960%); 51 MiB/s, 820 objects/s recovering 2024-03-21T13:07:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:07:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:05 smithi163 ceph-mon[29621]: pgmap v359: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s; 35656/298134 objects misplaced (11.960%); 51 MiB/s, 820 objects/s recovering 2024-03-21T13:07:06.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:07:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:07:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:07:06.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:07:06.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:06.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:05 smithi163 ceph-mon[40675]: pgmap v359: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 6 active+remapped+backfilling, 139 active+clean; 6.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s; 35656/298134 objects misplaced (11.960%); 51 MiB/s, 820 objects/s recovering 2024-03-21T13:07:06.074 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:06.110 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:06.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:06.212 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:06.250 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:06.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:06.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:06.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:06.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:06.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:06.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:06.518 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:06.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:06.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:06.620 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:06.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:06.657 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:06.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:06.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:06.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:06.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:06.900 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:06.937 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:06.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:07.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:07.082 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:07.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:07.183 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:07.207 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:07.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:07.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:07.308 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:07.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:07.350 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:07.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:07.451 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:07.487 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:07.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:07.589 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:07.619 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:07.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:07.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:07.721 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:07.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:07.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:07 smithi204 ceph-mon[33153]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:07:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:07 smithi204 ceph-mon[33153]: osdmap e203: 8 total, 8 up, 5 in 2024-03-21T13:07:07.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:07 smithi163 ceph-mon[29621]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:07:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:07 smithi163 ceph-mon[29621]: osdmap e203: 8 total, 8 up, 5 in 2024-03-21T13:07:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:07 smithi163 ceph-mon[40675]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:07:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:07 smithi163 ceph-mon[40675]: osdmap e203: 8 total, 8 up, 5 in 2024-03-21T13:07:07.854 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:07.884 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:07.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:07.986 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:08.016 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:08.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:08.118 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:08.147 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:08.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:08.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:08.249 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:08.277 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:08.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:08.379 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:08.416 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:08.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:08.517 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:08.541 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:08.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:08.643 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:08.679 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:08.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:08.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:08 smithi204 ceph-mon[33153]: pgmap v360: 148 pgs: 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 139 active+clean; 5.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 989 B/s rd, 0 B/s wr, 1.22k op/s; 28999/280371 objects misplaced (10.343%); 54 MiB/s, 859 objects/s recovering 2024-03-21T13:07:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:08 smithi204 ceph-mon[33153]: osdmap e204: 8 total, 8 up, 5 in 2024-03-21T13:07:08.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:08 smithi163 ceph-mon[29621]: pgmap v360: 148 pgs: 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 139 active+clean; 5.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 989 B/s rd, 0 B/s wr, 1.22k op/s; 28999/280371 objects misplaced (10.343%); 54 MiB/s, 859 objects/s recovering 2024-03-21T13:07:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:08 smithi163 ceph-mon[29621]: osdmap e204: 8 total, 8 up, 5 in 2024-03-21T13:07:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:08 smithi163 ceph-mon[40675]: pgmap v360: 148 pgs: 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 139 active+clean; 5.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 989 B/s rd, 0 B/s wr, 1.22k op/s; 28999/280371 objects misplaced (10.343%); 54 MiB/s, 859 objects/s recovering 2024-03-21T13:07:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:08 smithi163 ceph-mon[40675]: osdmap e204: 8 total, 8 up, 5 in 2024-03-21T13:07:08.781 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:08.811 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:08.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:08.912 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:08.941 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:08.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:08.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:09.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:09.072 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:09.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:09.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:09.174 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:09.203 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:09.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:09.305 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:09.336 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:09.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:09.438 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:09.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:09.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:09.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:09.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:09.603 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:09.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:09.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:09.739 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:09.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:09.841 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:09.881 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:09.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:09.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:10.006 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:10.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:10.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:10.108 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:10.145 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:10.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:10.247 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:10.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:10.272 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:10.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:10.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:10.409 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:10.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:10.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:10.547 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:10.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:10.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:10.673 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:10.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:10.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:10 smithi204 ceph-mon[33153]: pgmap v363: 148 pgs: 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 139 active+clean; 5.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 740 op/s; 28999/280371 objects misplaced (10.343%); 37 MiB/s, 587 objects/s recovering 2024-03-21T13:07:10.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:10 smithi204 ceph-mon[33153]: osdmap e205: 8 total, 8 up, 5 in 2024-03-21T13:07:10.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:10 smithi163 ceph-mon[29621]: pgmap v363: 148 pgs: 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 139 active+clean; 5.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 740 op/s; 28999/280371 objects misplaced (10.343%); 37 MiB/s, 587 objects/s recovering 2024-03-21T13:07:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:10 smithi163 ceph-mon[29621]: osdmap e205: 8 total, 8 up, 5 in 2024-03-21T13:07:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:10 smithi163 ceph-mon[40675]: pgmap v363: 148 pgs: 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 139 active+clean; 5.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 740 op/s; 28999/280371 objects misplaced (10.343%); 37 MiB/s, 587 objects/s recovering 2024-03-21T13:07:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:10 smithi163 ceph-mon[40675]: osdmap e205: 8 total, 8 up, 5 in 2024-03-21T13:07:10.774 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:10.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:10.813 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:10.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:10.914 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:10.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:10.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:11.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:11.080 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:11.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:11.182 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:11.218 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:11.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:11.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:11.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:11.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:11.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:11.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:11.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:11.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:11.594 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:11.631 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:11.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:11.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:11.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:11.756 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:11.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:11.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:11.898 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:11.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:11.999 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:12.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:07:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:07:12.039 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:12.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:12.141 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:12.175 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:12.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:12.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:12.278 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:12.317 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:12.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:12.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:12.419 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:12.450 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:12.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:12.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:12.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:12.593 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:12.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:12.695 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:12.723 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:12.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:12.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:12 smithi204 ceph-mon[33153]: pgmap v365: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.57k op/s; 21705/269844 objects misplaced (8.044%); 114 MiB/s, 1.83k objects/s recovering 2024-03-21T13:07:12.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:12 smithi163 ceph-mon[29621]: pgmap v365: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.57k op/s; 21705/269844 objects misplaced (8.044%); 114 MiB/s, 1.83k objects/s recovering 2024-03-21T13:07:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:12 smithi163 ceph-mon[40675]: pgmap v365: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.57k op/s; 21705/269844 objects misplaced (8.044%); 114 MiB/s, 1.83k objects/s recovering 2024-03-21T13:07:12.825 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:12.863 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:12.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:12.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:12.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:12.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:13.098 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:13.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:13.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:13.239 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:13.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:13.265 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:13.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:13.366 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:13.409 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:13.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:13.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:13.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:13 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:13.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:13 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:13.511 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:13.548 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:13.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:13.649 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:13.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:13.689 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:13.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:13.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:13 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:13.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:13.830 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:13.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:13.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:13.966 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:13.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:14.068 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:14.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:14.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:14.207 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:14.246 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:14.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:14.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:14.377 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:14 smithi163 ceph-mon[29621]: pgmap v366: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 585 op/s; 21705/269844 objects misplaced (8.044%); 65 MiB/s, 1.04k objects/s recovering 2024-03-21T13:07:14.378 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:14 smithi163 ceph-mon[40675]: pgmap v366: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 585 op/s; 21705/269844 objects misplaced (8.044%); 65 MiB/s, 1.04k objects/s recovering 2024-03-21T13:07:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:14.382 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:14.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:14.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:14.513 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:14.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:14.614 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:14.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:14.657 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:14.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:14 smithi204 ceph-mon[33153]: pgmap v366: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 585 op/s; 21705/269844 objects misplaced (8.044%); 65 MiB/s, 1.04k objects/s recovering 2024-03-21T13:07:14.758 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:14.788 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:14.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:14.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:14.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:14.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:14.932 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:14.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:15.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:15.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:15.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:15.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:15.160 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:15.203 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:15.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:15.304 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:15.327 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:15.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:15.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:15.463 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:15.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:15.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:15.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:15.595 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:15.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:15.696 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:15.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:15.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:15.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:15.868 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:15.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:15.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:15.970 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:16.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:16.009 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:16.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:16.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:07:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:07:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:07:16.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:16.142 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:16.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:16.244 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:16.286 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:16.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:16.388 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:16.428 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:16.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:16 smithi163 ceph-mon[29621]: pgmap v367: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 687 B/s rd, 0 B/s wr, 471 op/s; 21705/269844 objects misplaced (8.044%); 53 MiB/s, 842 objects/s recovering 2024-03-21T13:07:16.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:16 smithi163 ceph-mon[40675]: pgmap v367: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 687 B/s rd, 0 B/s wr, 471 op/s; 21705/269844 objects misplaced (8.044%); 53 MiB/s, 842 objects/s recovering 2024-03-21T13:07:16.530 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:16.556 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:16.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:16.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:16.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:16.701 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:16.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:16 smithi204 ceph-mon[33153]: pgmap v367: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 687 B/s rd, 0 B/s wr, 471 op/s; 21705/269844 objects misplaced (8.044%); 53 MiB/s, 842 objects/s recovering 2024-03-21T13:07:16.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:16.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:16.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:16.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:16.929 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:16.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:16.973 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:16.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:17.074 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:17.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:17.120 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:17.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:17.221 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:17.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:17.266 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:17.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:17.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:17.406 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:17.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:17.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:17.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:17.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:17.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:17.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:17.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:17.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:17.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:17 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:17 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:07:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:17 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:07:17.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:17 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:17.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:17 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:07:17.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:17 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:07:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:17 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:17 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:07:17.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:17 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:07:17.772 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:17.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:17.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:17.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:17.945 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:17.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:17.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:18.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:18.086 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:18.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:18.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:18.232 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:18.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:18.333 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:18.394 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:18.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:18.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:18.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:18.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:18 smithi163 ceph-mon[29621]: pgmap v368: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 17847/252816 objects misplaced (7.059%); 66 MiB/s, 1.05k objects/s recovering 2024-03-21T13:07:18.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:18 smithi163 ceph-mon[29621]: osdmap e206: 8 total, 8 up, 5 in 2024-03-21T13:07:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:18 smithi163 ceph-mon[40675]: pgmap v368: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 17847/252816 objects misplaced (7.059%); 66 MiB/s, 1.05k objects/s recovering 2024-03-21T13:07:18.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:18 smithi163 ceph-mon[40675]: osdmap e206: 8 total, 8 up, 5 in 2024-03-21T13:07:18.528 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:18.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:18.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:18.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:18.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:18.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:18 smithi204 ceph-mon[33153]: pgmap v368: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 17847/252816 objects misplaced (7.059%); 66 MiB/s, 1.05k objects/s recovering 2024-03-21T13:07:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:18 smithi204 ceph-mon[33153]: osdmap e206: 8 total, 8 up, 5 in 2024-03-21T13:07:18.768 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:18.802 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:18.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:18.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:18.933 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:18.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:19.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:19.064 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:19.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:19.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:19.194 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:19.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:19.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:19.296 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:19.329 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:19.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:19.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:19.431 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:19.467 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:19.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:19.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:19.569 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:19.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:19.612 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:19.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:19.713 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:19.751 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:19.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:19 smithi204 ceph-mon[33153]: osdmap e207: 8 total, 8 up, 5 in 2024-03-21T13:07:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:07:19.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:19 smithi163 ceph-mon[40675]: osdmap e207: 8 total, 8 up, 5 in 2024-03-21T13:07:19.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:19.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:19.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:07:19.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:19 smithi163 ceph-mon[29621]: osdmap e207: 8 total, 8 up, 5 in 2024-03-21T13:07:19.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:19.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:19.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:07:19.854 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:19.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:19.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:20.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:20.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:20.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:20.134 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:20.173 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:20.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:20.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:20.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:20.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:20.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:20.412 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:20.448 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:20.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:20.549 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:20.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:20.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:20.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:20.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:20 smithi204 ceph-mon[33153]: pgmap v371: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 709 op/s; 17847/252816 objects misplaced (7.059%); 20 MiB/s, 324 objects/s recovering 2024-03-21T13:07:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:20 smithi204 ceph-mon[33153]: osdmap e208: 8 total, 8 up, 5 in 2024-03-21T13:07:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:07:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:20.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:20 smithi163 ceph-mon[29621]: pgmap v371: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 709 op/s; 17847/252816 objects misplaced (7.059%); 20 MiB/s, 324 objects/s recovering 2024-03-21T13:07:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:20 smithi163 ceph-mon[29621]: osdmap e208: 8 total, 8 up, 5 in 2024-03-21T13:07:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:07:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:20 smithi163 ceph-mon[40675]: pgmap v371: 148 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 141 active+clean; 5.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 709 op/s; 17847/252816 objects misplaced (7.059%); 20 MiB/s, 324 objects/s recovering 2024-03-21T13:07:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:20 smithi163 ceph-mon[40675]: osdmap e208: 8 total, 8 up, 5 in 2024-03-21T13:07:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:07:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:20.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:20.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:20.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:20.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:20.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:20.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:21.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:21.093 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:07:21.101 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:21.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:21.195 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:21.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:21.296 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:21.349 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:21.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:21.451 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:21.498 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:21.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:21.571 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:07:21.571 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:07:21.571 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-03-21T13:07:21.571 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-03-21T13:07:21.571 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:07:21.600 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:21.633 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:21.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:21.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:21.781 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:21.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:21.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:21.883 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:21.923 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:21.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:22.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:07:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:07:22.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:22.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:22.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:22.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:22.200 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:22.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:22.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:22.302 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:22.342 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:22.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:22.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:22.444 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:22.485 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:22.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:22.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:22.586 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:22.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:22.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:22.671 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:07:22.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:22 smithi163 ceph-mon[29621]: pgmap v373: 148 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.78k op/s; 12631/237717 objects misplaced (5.313%); 71 MiB/s, 1.14k objects/s recovering 2024-03-21T13:07:22.672 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:22 smithi163 ceph-mon[40675]: pgmap v373: 148 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.78k op/s; 12631/237717 objects misplaced (5.313%); 71 MiB/s, 1.14k objects/s recovering 2024-03-21T13:07:22.683 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:07:22.722 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:22.754 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:22.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:22.755 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:22 smithi204 ceph-mon[33153]: pgmap v373: 148 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.78k op/s; 12631/237717 objects misplaced (5.313%); 71 MiB/s, 1.14k objects/s recovering 2024-03-21T13:07:22.802 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-21T13:07:22.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:22.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:22.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:22.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:22.986 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:23.016 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:23.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:23.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:23.148 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:23.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:23.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:23.251 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:23.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:23.275 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:23.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:23.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:23.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:23.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:23.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:23.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:23.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:23.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:23.641 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:23.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:23.681 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:23.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:23.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:23 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:23.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:23 smithi204 ceph-mon[33153]: osdmap e209: 8 total, 8 up, 5 in 2024-03-21T13:07:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:23 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:23 smithi163 ceph-mon[29621]: osdmap e209: 8 total, 8 up, 5 in 2024-03-21T13:07:23.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:23 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:23.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:23 smithi163 ceph-mon[40675]: osdmap e209: 8 total, 8 up, 5 in 2024-03-21T13:07:23.782 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:23.820 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:23.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:23.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:23.922 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:23.962 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:23.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:24.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:24.096 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:24.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:24.198 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:24.235 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:24.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:24.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:24.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:24.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:24.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:24.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:24.474 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:24.509 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:24.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:24.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:24.611 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:24.639 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:24.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:24.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:24.741 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:24 smithi204 ceph-mon[33153]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:07:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:24 smithi204 ceph-mon[33153]: pgmap v375: 148 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 946 B/s rd, 0 B/s wr, 931 op/s; 12631/237717 objects misplaced (5.313%); 49 MiB/s, 783 objects/s recovering 2024-03-21T13:07:24.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:24 smithi204 ceph-mon[33153]: osdmap e210: 8 total, 8 up, 5 in 2024-03-21T13:07:24.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:24 smithi163 ceph-mon[29621]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:07:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:24 smithi163 ceph-mon[29621]: pgmap v375: 148 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 946 B/s rd, 0 B/s wr, 931 op/s; 12631/237717 objects misplaced (5.313%); 49 MiB/s, 783 objects/s recovering 2024-03-21T13:07:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:24 smithi163 ceph-mon[29621]: osdmap e210: 8 total, 8 up, 5 in 2024-03-21T13:07:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:24 smithi163 ceph-mon[40675]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:07:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:24 smithi163 ceph-mon[40675]: pgmap v375: 148 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 946 B/s rd, 0 B/s wr, 931 op/s; 12631/237717 objects misplaced (5.313%); 49 MiB/s, 783 objects/s recovering 2024-03-21T13:07:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:24 smithi163 ceph-mon[40675]: osdmap e210: 8 total, 8 up, 5 in 2024-03-21T13:07:24.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:24.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:24.872 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:24.899 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:24.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:25.000 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:25.024 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:25.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:25.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:25.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:25.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:25.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:25.253 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:25.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:25.279 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:25.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:25.380 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:25.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:25.406 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:25.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:25.507 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:25.548 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:25.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:25.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:25.650 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:25.683 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:25.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:25.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:25 smithi204 ceph-mon[33153]: osdmap e211: 8 total, 8 up, 5 in 2024-03-21T13:07:25.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:25 smithi163 ceph-mon[29621]: osdmap e211: 8 total, 8 up, 5 in 2024-03-21T13:07:25.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:25 smithi163 ceph-mon[40675]: osdmap e211: 8 total, 8 up, 5 in 2024-03-21T13:07:25.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:07:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:07:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:07:25.785 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:25.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:25.814 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:25.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:25.915 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:25.941 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:25.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:26.043 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:26.070 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:26.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:26.172 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:26.196 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:26.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:26.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:26.324 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:26.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:26.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:26.426 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:26.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:26.469 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:26.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:26.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:26.599 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:26.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:26.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:26.701 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:26.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:26.727 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:26.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:26 smithi204 ceph-mon[33153]: pgmap v378: 148 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 948 B/s rd, 0 B/s wr, 933 op/s; 12631/237717 objects misplaced (5.313%); 49 MiB/s, 785 objects/s recovering 2024-03-21T13:07:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:26 smithi204 ceph-mon[33153]: osdmap e212: 8 total, 8 up, 5 in 2024-03-21T13:07:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:26 smithi204 ceph-mon[33153]: osdmap e213: 8 total, 8 up, 5 in 2024-03-21T13:07:26.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:26 smithi163 ceph-mon[40675]: pgmap v378: 148 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 948 B/s rd, 0 B/s wr, 933 op/s; 12631/237717 objects misplaced (5.313%); 49 MiB/s, 785 objects/s recovering 2024-03-21T13:07:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:26 smithi163 ceph-mon[40675]: osdmap e212: 8 total, 8 up, 5 in 2024-03-21T13:07:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:26 smithi163 ceph-mon[40675]: osdmap e213: 8 total, 8 up, 5 in 2024-03-21T13:07:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:26 smithi163 ceph-mon[29621]: pgmap v378: 148 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 948 B/s rd, 0 B/s wr, 933 op/s; 12631/237717 objects misplaced (5.313%); 49 MiB/s, 785 objects/s recovering 2024-03-21T13:07:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:26 smithi163 ceph-mon[29621]: osdmap e212: 8 total, 8 up, 5 in 2024-03-21T13:07:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:26 smithi163 ceph-mon[29621]: osdmap e213: 8 total, 8 up, 5 in 2024-03-21T13:07:26.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:26.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:26.853 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:26.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:26.954 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:26.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:26.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:26.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:27.080 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:27.105 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:27.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:27.206 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:27.230 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:27.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:27.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:27.335 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:27.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:27.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:27.474 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:27.512 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:27.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:27.614 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:27.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:27.647 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:27.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:27.748 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:27.783 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:27.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:27.885 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:27.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:27.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:28.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:28.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:28.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:28.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:28.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:28.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:28.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:28.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:28.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:28.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:28.421 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:28.467 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-03-21T13:07:22.412072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5180561856,"num_objects":79239,"num_object_clones":0,"num_object_copies":237717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12631,"num_objects_unfound":0,"num_objects_dirty":79239,"num_whiteouts":0,"num_read":1431,"num_read_kb":1277,"num_write":193316,"num_write_kb":8698142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41497,"num_bytes_recovered":2705333713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54327,"ondisk_log_size":54327,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":16425544,"kb_used_data":14446376,"kb_used_omap":7,"kb_used_meta":1978680,"kb_avail":452300216,"statfs":{"total":479975178240,"available":463155421184,"internally_reserved":0,"allocated":14793089024,"data_stored":14791080333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7967,"internal_metadata":2026168545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,3,5,0,0,0,0,0,6,10],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-329842688,"num_objects":-5033,"num_object_clones":0,"num_object_copies":-15099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-5216,"num_objects_unfound":0,"num_objects_dirty":-5033,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5033,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4239,"num_bytes_recovered":277807104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.633559"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":166,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385672+0000","last_change":"2024-03-21T13:06:45.404501+0000","last_active":"2024-03-21T13:07:19.385672+0000","last_peered":"2024-03-21T13:07:19.385672+0000","last_clean":"2024-03-21T13:07:19.385672+0000","last_became_active":"2024-03-21T13:06:45.404217+0000","last_became_peered":"2024-03-21T13:06:45.404217+0000","last_unstale":"2024-03-21T13:07:19.385672+0000","last_undegraded":"2024-03-21T13:07:19.385672+0000","last_fullsized":"2024-03-21T13:07:19.385672+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:20.083111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.083111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":198,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386318+0000","last_change":"2024-03-21T13:06:29.490923+0000","last_active":"2024-03-21T13:07:19.386318+0000","last_peered":"2024-03-21T13:07:19.386318+0000","last_clean":"2024-03-21T13:07:19.386318+0000","last_became_active":"2024-03-21T13:06:29.490568+0000","last_became_peered":"2024-03-21T13:06:29.490568+0000","last_unstale":"2024-03-21T13:07:19.386318+0000","last_undegraded":"2024-03-21T13:07:19.386318+0000","last_fullsized":"2024-03-21T13:07:19.386318+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:40.962617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:03:40.962617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":40,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391063+0000","last_change":"2024-03-21T13:06:29.484964+0000","last_active":"2024-03-21T13:07:19.391063+0000","last_peered":"2024-03-21T13:07:19.391063+0000","last_clean":"2024-03-21T13:07:19.391063+0000","last_became_active":"2024-03-21T13:06:29.484528+0000","last_became_peered":"2024-03-21T13:06:29.484528+0000","last_unstale":"2024-03-21T13:07:19.391063+0000","last_undegraded":"2024-03-21T13:07:19.391063+0000","last_fullsized":"2024-03-21T13:07:19.391063+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:49.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.260693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"89'13","reported_seq":373,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391919+0000","last_change":"2024-03-21T13:05:17.080137+0000","last_active":"2024-03-21T13:07:19.391919+0000","last_peered":"2024-03-21T13:07:19.391919+0000","last_clean":"2024-03-21T13:07:19.391919+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:07:19.391919+0000","last_undegraded":"2024-03-21T13:07:19.391919+0000","last_fullsized":"2024-03-21T13:07:19.391919+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-03-21T13:05:17.079993+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:05:17.079993+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":305,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390981+0000","last_change":"2024-03-21T13:05:07.334438+0000","last_active":"2024-03-21T13:07:19.390981+0000","last_peered":"2024-03-21T13:07:19.390981+0000","last_clean":"2024-03-21T13:07:19.390981+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:07:19.390981+0000","last_undegraded":"2024-03-21T13:07:19.390981+0000","last_fullsized":"2024-03-21T13:07:19.390981+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.330296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.330296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":250,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389207+0000","last_change":"2024-03-21T13:06:29.490705+0000","last_active":"2024-03-21T13:07:19.389207+0000","last_peered":"2024-03-21T13:07:19.389207+0000","last_clean":"2024-03-21T13:07:19.389207+0000","last_became_active":"2024-03-21T13:06:29.490004+0000","last_became_peered":"2024-03-21T13:06:29.490004+0000","last_unstale":"2024-03-21T13:07:19.389207+0000","last_undegraded":"2024-03-21T13:07:19.389207+0000","last_fullsized":"2024-03-21T13:07:19.389207+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:50.991735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.991735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:19.848887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":327,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391660+0000","last_change":"2024-03-21T13:06:29.542506+0000","last_active":"2024-03-21T13:07:19.391660+0000","last_peered":"2024-03-21T13:07:19.391660+0000","last_clean":"2024-03-21T13:07:19.391660+0000","last_became_active":"2024-03-21T13:06:29.511102+0000","last_became_peered":"2024-03-21T13:06:29.511102+0000","last_unstale":"2024-03-21T13:07:19.391660+0000","last_undegraded":"2024-03-21T13:07:19.391660+0000","last_fullsized":"2024-03-21T13:07:19.391660+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_clean_scrub_stamp":"2024-03-21T13:01:38.765896+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"71'15","reported_seq":360,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390617+0000","last_change":"2024-03-21T13:06:29.546224+0000","last_active":"2024-03-21T13:07:19.390617+0000","last_peered":"2024-03-21T13:07:19.390617+0000","last_clean":"2024-03-21T13:07:19.390617+0000","last_became_active":"2024-03-21T13:06:29.509791+0000","last_became_peered":"2024-03-21T13:06:29.509791+0000","last_unstale":"2024-03-21T13:07:19.390617+0000","last_undegraded":"2024-03-21T13:07:19.390617+0000","last_fullsized":"2024-03-21T13:07:19.390617+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:26.476043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.476043+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":210,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385329+0000","last_change":"2024-03-21T13:06:29.489731+0000","last_active":"2024-03-21T13:07:19.385329+0000","last_peered":"2024-03-21T13:07:19.385329+0000","last_clean":"2024-03-21T13:07:19.385329+0000","last_became_active":"2024-03-21T13:06:29.487372+0000","last_became_peered":"2024-03-21T13:06:29.487372+0000","last_unstale":"2024-03-21T13:07:19.385329+0000","last_undegraded":"2024-03-21T13:07:19.385329+0000","last_fullsized":"2024-03-21T13:07:19.385329+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:32.834646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:32.834646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":205,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385996+0000","last_change":"2024-03-21T13:06:43.363491+0000","last_active":"2024-03-21T13:07:19.385996+0000","last_peered":"2024-03-21T13:07:19.385996+0000","last_clean":"2024-03-21T13:07:19.385996+0000","last_became_active":"2024-03-21T13:02:26.236707+0000","last_became_peered":"2024-03-21T13:02:26.236707+0000","last_unstale":"2024-03-21T13:07:19.385996+0000","last_undegraded":"2024-03-21T13:07:19.385996+0000","last_fullsized":"2024-03-21T13:07:19.385996+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:43.363202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:06:43.363202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:08:05.576865+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":189,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390929+0000","last_change":"2024-03-21T13:05:45.989199+0000","last_active":"2024-03-21T13:07:19.390929+0000","last_peered":"2024-03-21T13:07:19.390929+0000","last_clean":"2024-03-21T13:07:19.390929+0000","last_became_active":"2024-03-21T13:02:26.245402+0000","last_became_peered":"2024-03-21T13:02:26.245402+0000","last_unstale":"2024-03-21T13:07:19.390929+0000","last_undegraded":"2024-03-21T13:07:19.390929+0000","last_fullsized":"2024-03-21T13:07:19.390929+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:45.989114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:45.989114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":307,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390416+0000","last_change":"2024-03-21T13:05:51.580203+0000","last_active":"2024-03-21T13:07:19.390416+0000","last_peered":"2024-03-21T13:07:19.390416+0000","last_clean":"2024-03-21T13:07:19.390416+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:07:19.390416+0000","last_undegraded":"2024-03-21T13:07:19.390416+0000","last_fullsized":"2024-03-21T13:07:19.390416+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-21T13:05:51.580074+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:05:51.580074+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":308,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389395+0000","last_change":"2024-03-21T13:06:29.505858+0000","last_active":"2024-03-21T13:07:19.389395+0000","last_peered":"2024-03-21T13:07:19.389395+0000","last_clean":"2024-03-21T13:07:19.389395+0000","last_became_active":"2024-03-21T13:06:29.505469+0000","last_became_peered":"2024-03-21T13:06:29.505469+0000","last_unstale":"2024-03-21T13:07:19.389395+0000","last_undegraded":"2024-03-21T13:07:19.389395+0000","last_fullsized":"2024-03-21T13:07:19.389395+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:32.971665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:32.971665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":321,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385816+0000","last_change":"2024-03-21T13:04:30.831274+0000","last_active":"2024-03-21T13:07:19.385816+0000","last_peered":"2024-03-21T13:07:19.385816+0000","last_clean":"2024-03-21T13:07:19.385816+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:07:19.385816+0000","last_undegraded":"2024-03-21T13:07:19.385816+0000","last_fullsized":"2024-03-21T13:07:19.385816+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:30.831143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:04:30.831143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":329,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392893+0000","last_change":"2024-03-21T13:05:41.337232+0000","last_active":"2024-03-21T13:07:19.392893+0000","last_peered":"2024-03-21T13:07:19.392893+0000","last_clean":"2024-03-21T13:07:19.392893+0000","last_became_active":"2024-03-21T13:02:26.230057+0000","last_became_peered":"2024-03-21T13:02:26.230057+0000","last_unstale":"2024-03-21T13:07:19.392893+0000","last_undegraded":"2024-03-21T13:07:19.392893+0000","last_fullsized":"2024-03-21T13:07:19.392893+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.337144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.337144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":321,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390718+0000","last_change":"2024-03-21T13:06:29.517935+0000","last_active":"2024-03-21T13:07:19.390718+0000","last_peered":"2024-03-21T13:07:19.390718+0000","last_clean":"2024-03-21T13:07:19.390718+0000","last_became_active":"2024-03-21T13:06:29.506772+0000","last_became_peered":"2024-03-21T13:06:29.506772+0000","last_unstale":"2024-03-21T13:07:19.390718+0000","last_undegraded":"2024-03-21T13:07:19.390718+0000","last_fullsized":"2024-03-21T13:07:19.390718+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T13:05:16.384676+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:05:16.384676+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":328,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.393507+0000","last_change":"2024-03-21T13:06:30.400604+0000","last_active":"2024-03-21T13:07:19.393507+0000","last_peered":"2024-03-21T13:07:19.393507+0000","last_clean":"2024-03-21T13:07:19.393507+0000","last_became_active":"2024-03-21T13:02:26.226675+0000","last_became_peered":"2024-03-21T13:02:26.226675+0000","last_unstale":"2024-03-21T13:07:19.393507+0000","last_undegraded":"2024-03-21T13:07:19.393507+0000","last_fullsized":"2024-03-21T13:07:19.393507+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:30.400488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:06:30.400488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:44.632819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":216,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.393320+0000","last_change":"2024-03-21T13:05:37.359133+0000","last_active":"2024-03-21T13:07:19.393320+0000","last_peered":"2024-03-21T13:07:19.393320+0000","last_clean":"2024-03-21T13:07:19.393320+0000","last_became_active":"2024-03-21T13:02:26.231879+0000","last_became_peered":"2024-03-21T13:02:26.231879+0000","last_unstale":"2024-03-21T13:07:19.393320+0000","last_undegraded":"2024-03-21T13:07:19.393320+0000","last_fullsized":"2024-03-21T13:07:19.393320+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:37.358974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:05:37.358974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":216,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386464+0000","last_change":"2024-03-21T13:06:28.985227+0000","last_active":"2024-03-21T13:07:19.386464+0000","last_peered":"2024-03-21T13:07:19.386464+0000","last_clean":"2024-03-21T13:07:19.386464+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:07:19.386464+0000","last_undegraded":"2024-03-21T13:07:19.386464+0000","last_fullsized":"2024-03-21T13:07:19.386464+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:28.985145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:28.985145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:57.861880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":96,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391187+0000","last_change":"2024-03-21T13:06:29.540373+0000","last_active":"2024-03-21T13:07:19.391187+0000","last_peered":"2024-03-21T13:07:19.391187+0000","last_clean":"2024-03-21T13:07:19.391187+0000","last_became_active":"2024-03-21T13:06:29.504235+0000","last_became_peered":"2024-03-21T13:06:29.504235+0000","last_unstale":"2024-03-21T13:07:19.391187+0000","last_undegraded":"2024-03-21T13:07:19.391187+0000","last_fullsized":"2024-03-21T13:07:19.391187+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:20.158184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.158184+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":310,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391820+0000","last_change":"2024-03-21T13:06:29.512640+0000","last_active":"2024-03-21T13:07:19.391820+0000","last_peered":"2024-03-21T13:07:19.391820+0000","last_clean":"2024-03-21T13:07:19.391820+0000","last_became_active":"2024-03-21T13:06:29.512081+0000","last_became_peered":"2024-03-21T13:06:29.512081+0000","last_unstale":"2024-03-21T13:07:19.391820+0000","last_undegraded":"2024-03-21T13:07:19.391820+0000","last_fullsized":"2024-03-21T13:07:19.391820+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:35.036335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:00:35.036335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":236,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390879+0000","last_change":"2024-03-21T13:06:29.500559+0000","last_active":"2024-03-21T13:07:19.390879+0000","last_peered":"2024-03-21T13:07:19.390879+0000","last_clean":"2024-03-21T13:07:19.390879+0000","last_became_active":"2024-03-21T13:06:29.498872+0000","last_became_peered":"2024-03-21T13:06:29.498872+0000","last_unstale":"2024-03-21T13:07:19.390879+0000","last_undegraded":"2024-03-21T13:07:19.390879+0000","last_fullsized":"2024-03-21T13:07:19.390879+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:29.464856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:04:29.464856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":345,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392524+0000","last_change":"2024-03-21T13:06:29.488224+0000","last_active":"2024-03-21T13:07:19.392524+0000","last_peered":"2024-03-21T13:07:19.392524+0000","last_clean":"2024-03-21T13:07:19.392524+0000","last_became_active":"2024-03-21T13:06:29.487644+0000","last_became_peered":"2024-03-21T13:06:29.487644+0000","last_unstale":"2024-03-21T13:07:19.392524+0000","last_undegraded":"2024-03-21T13:07:19.392524+0000","last_fullsized":"2024-03-21T13:07:19.392524+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.354575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.354575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'12","reported_seq":364,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391552+0000","last_change":"2024-03-21T13:02:14.362313+0000","last_active":"2024-03-21T13:07:19.391552+0000","last_peered":"2024-03-21T13:07:19.391552+0000","last_clean":"2024-03-21T13:07:19.391552+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:07:19.391552+0000","last_undegraded":"2024-03-21T13:07:19.391552+0000","last_fullsized":"2024-03-21T13:07:19.391552+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:01:39.755345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.755345+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":171,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390766+0000","last_change":"2024-03-21T13:06:29.537271+0000","last_active":"2024-03-21T13:07:19.390766+0000","last_peered":"2024-03-21T13:07:19.390766+0000","last_clean":"2024-03-21T13:07:19.390766+0000","last_became_active":"2024-03-21T13:06:29.504570+0000","last_became_peered":"2024-03-21T13:06:29.504570+0000","last_unstale":"2024-03-21T13:07:19.390766+0000","last_undegraded":"2024-03-21T13:07:19.390766+0000","last_fullsized":"2024-03-21T13:07:19.390766+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:50.295057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.295057+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":434,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391584+0000","last_change":"2024-03-21T13:05:38.022572+0000","last_active":"2024-03-21T13:07:19.391584+0000","last_peered":"2024-03-21T13:07:19.391584+0000","last_clean":"2024-03-21T13:07:19.391584+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:07:19.391584+0000","last_undegraded":"2024-03-21T13:07:19.391584+0000","last_fullsized":"2024-03-21T13:07:19.391584+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:05:38.022439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.022439+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":216,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390109+0000","last_change":"2024-03-21T13:04:20.361810+0000","last_active":"2024-03-21T13:07:19.390109+0000","last_peered":"2024-03-21T13:07:19.390109+0000","last_clean":"2024-03-21T13:07:19.390109+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:07:19.390109+0000","last_undegraded":"2024-03-21T13:07:19.390109+0000","last_fullsized":"2024-03-21T13:07:19.390109+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:01:37.244557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:37.244557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"71'10","reported_seq":250,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.387162+0000","last_change":"2024-03-21T13:05:21.117502+0000","last_active":"2024-03-21T13:07:19.387162+0000","last_peered":"2024-03-21T13:07:19.387162+0000","last_clean":"2024-03-21T13:07:19.387162+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:07:19.387162+0000","last_undegraded":"2024-03-21T13:07:19.387162+0000","last_fullsized":"2024-03-21T13:07:19.387162+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:21.117370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.117370+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":205,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392631+0000","last_change":"2024-03-21T13:06:29.499863+0000","last_active":"2024-03-21T13:07:19.392631+0000","last_peered":"2024-03-21T13:07:19.392631+0000","last_clean":"2024-03-21T13:07:19.392631+0000","last_became_active":"2024-03-21T13:06:29.499497+0000","last_became_peered":"2024-03-21T13:06:29.499497+0000","last_unstale":"2024-03-21T13:07:19.392631+0000","last_undegraded":"2024-03-21T13:07:19.392631+0000","last_fullsized":"2024-03-21T13:07:19.392631+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:26.498073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:06:26.498073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:37.222950+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":222,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.387117+0000","last_change":"2024-03-21T13:06:29.501937+0000","last_active":"2024-03-21T13:07:19.387117+0000","last_peered":"2024-03-21T13:07:19.387117+0000","last_clean":"2024-03-21T13:07:19.387117+0000","last_became_active":"2024-03-21T13:06:29.501232+0000","last_became_peered":"2024-03-21T13:06:29.501232+0000","last_unstale":"2024-03-21T13:07:19.387117+0000","last_undegraded":"2024-03-21T13:07:19.387117+0000","last_fullsized":"2024-03-21T13:07:19.387117+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.088812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.088812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":329,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389240+0000","last_change":"2024-03-21T13:06:29.501866+0000","last_active":"2024-03-21T13:07:19.389240+0000","last_peered":"2024-03-21T13:07:19.389240+0000","last_clean":"2024-03-21T13:07:19.389240+0000","last_became_active":"2024-03-21T13:06:29.501471+0000","last_became_peered":"2024-03-21T13:06:29.501471+0000","last_unstale":"2024-03-21T13:07:19.389240+0000","last_undegraded":"2024-03-21T13:07:19.389240+0000","last_fullsized":"2024-03-21T13:07:19.389240+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:09.348646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:09.348646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'11","reported_seq":377,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391585+0000","last_change":"2024-03-21T13:05:49.982963+0000","last_active":"2024-03-21T13:07:19.391585+0000","last_peered":"2024-03-21T13:07:19.391585+0000","last_clean":"2024-03-21T13:07:19.391585+0000","last_became_active":"2024-03-21T13:02:26.254436+0000","last_became_peered":"2024-03-21T13:02:26.254436+0000","last_unstale":"2024-03-21T13:07:19.391585+0000","last_undegraded":"2024-03-21T13:07:19.391585+0000","last_fullsized":"2024-03-21T13:07:19.391585+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:49.982812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.982812+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":317,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385923+0000","last_change":"2024-03-21T13:06:29.980827+0000","last_active":"2024-03-21T13:07:19.385923+0000","last_peered":"2024-03-21T13:07:19.385923+0000","last_clean":"2024-03-21T13:07:19.385923+0000","last_became_active":"2024-03-21T13:06:29.503679+0000","last_became_peered":"2024-03-21T13:06:29.503679+0000","last_unstale":"2024-03-21T13:07:19.385923+0000","last_undegraded":"2024-03-21T13:07:19.385923+0000","last_fullsized":"2024-03-21T13:07:19.385923+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:29.980750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:06:29.980750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:44.698296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":317,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389520+0000","last_change":"2024-03-21T13:05:42.990138+0000","last_active":"2024-03-21T13:07:19.389520+0000","last_peered":"2024-03-21T13:07:19.389520+0000","last_clean":"2024-03-21T13:07:19.389520+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:07:19.389520+0000","last_undegraded":"2024-03-21T13:07:19.389520+0000","last_fullsized":"2024-03-21T13:07:19.389520+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:42.990022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.990022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":192,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392298+0000","last_change":"2024-03-21T13:05:28.499322+0000","last_active":"2024-03-21T13:07:19.392298+0000","last_peered":"2024-03-21T13:07:19.392298+0000","last_clean":"2024-03-21T13:07:19.392298+0000","last_became_active":"2024-03-21T13:02:26.242973+0000","last_became_peered":"2024-03-21T13:02:26.242973+0000","last_unstale":"2024-03-21T13:07:19.392298+0000","last_undegraded":"2024-03-21T13:07:19.392298+0000","last_fullsized":"2024-03-21T13:07:19.392298+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.499185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.499185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":323,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389300+0000","last_change":"2024-03-21T13:02:14.369668+0000","last_active":"2024-03-21T13:07:19.389300+0000","last_peered":"2024-03-21T13:07:19.389300+0000","last_clean":"2024-03-21T13:07:19.389300+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:07:19.389300+0000","last_undegraded":"2024-03-21T13:07:19.389300+0000","last_fullsized":"2024-03-21T13:07:19.389300+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-21T13:01:30.807979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:30.807979+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":305,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391104+0000","last_change":"2024-03-21T13:05:19.431337+0000","last_active":"2024-03-21T13:07:19.391104+0000","last_peered":"2024-03-21T13:07:19.391104+0000","last_clean":"2024-03-21T13:07:19.391104+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:07:19.391104+0000","last_undegraded":"2024-03-21T13:07:19.391104+0000","last_fullsized":"2024-03-21T13:07:19.391104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:19.430947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:19.430947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":307,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386001+0000","last_change":"2024-03-21T13:03:43.905627+0000","last_active":"2024-03-21T13:07:19.386001+0000","last_peered":"2024-03-21T13:07:19.386001+0000","last_clean":"2024-03-21T13:07:19.386001+0000","last_became_active":"2024-03-21T12:58:41.164784+0000","last_became_peered":"2024-03-21T12:58:41.164784+0000","last_unstale":"2024-03-21T13:07:19.386001+0000","last_undegraded":"2024-03-21T13:07:19.386001+0000","last_fullsized":"2024-03-21T13:07:19.386001+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:43.905509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:03:43.905509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":312,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.387076+0000","last_change":"2024-03-21T13:06:29.501880+0000","last_active":"2024-03-21T13:07:19.387076+0000","last_peered":"2024-03-21T13:07:19.387076+0000","last_clean":"2024-03-21T13:07:19.387076+0000","last_became_active":"2024-03-21T13:06:29.501190+0000","last_became_peered":"2024-03-21T13:06:29.501190+0000","last_unstale":"2024-03-21T13:07:19.387076+0000","last_undegraded":"2024-03-21T13:07:19.387076+0000","last_fullsized":"2024-03-21T13:07:19.387076+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:02:21.087308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:02:21.087308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"71'11","reported_seq":47,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390358+0000","last_change":"2024-03-21T13:06:29.532046+0000","last_active":"2024-03-21T13:07:19.390358+0000","last_peered":"2024-03-21T13:07:19.390358+0000","last_clean":"2024-03-21T13:07:19.390358+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:07:19.390358+0000","last_undegraded":"2024-03-21T13:07:19.390358+0000","last_fullsized":"2024-03-21T13:07:19.390358+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:40.262277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:40.262277+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":213,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389017+0000","last_change":"2024-03-21T13:05:16.850442+0000","last_active":"2024-03-21T13:07:19.389017+0000","last_peered":"2024-03-21T13:07:19.389017+0000","last_clean":"2024-03-21T13:07:19.389017+0000","last_became_active":"2024-03-21T13:02:26.252488+0000","last_became_peered":"2024-03-21T13:02:26.252488+0000","last_unstale":"2024-03-21T13:07:19.389017+0000","last_undegraded":"2024-03-21T13:07:19.389017+0000","last_fullsized":"2024-03-21T13:07:19.389017+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:16.850307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:16.850307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":321,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390879+0000","last_change":"2024-03-21T13:05:31.496412+0000","last_active":"2024-03-21T13:07:19.390879+0000","last_peered":"2024-03-21T13:07:19.390879+0000","last_clean":"2024-03-21T13:07:19.390879+0000","last_became_active":"2024-03-21T13:02:26.232682+0000","last_became_peered":"2024-03-21T13:02:26.232682+0000","last_unstale":"2024-03-21T13:07:19.390879+0000","last_undegraded":"2024-03-21T13:07:19.390879+0000","last_fullsized":"2024-03-21T13:07:19.390879+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.495639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.495639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":315,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386659+0000","last_change":"2024-03-21T13:01:43.982673+0000","last_active":"2024-03-21T13:07:19.386659+0000","last_peered":"2024-03-21T13:07:19.386659+0000","last_clean":"2024-03-21T13:07:19.386659+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:07:19.386659+0000","last_undegraded":"2024-03-21T13:07:19.386659+0000","last_fullsized":"2024-03-21T13:07:19.386659+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:43.982553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:43.982553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"86'13","reported_seq":297,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390229+0000","last_change":"2024-03-21T13:06:29.518281+0000","last_active":"2024-03-21T13:07:19.390229+0000","last_peered":"2024-03-21T13:07:19.390229+0000","last_clean":"2024-03-21T13:07:19.390229+0000","last_became_active":"2024-03-21T13:06:29.496686+0000","last_became_peered":"2024-03-21T13:06:29.496686+0000","last_unstale":"2024-03-21T13:07:19.390229+0000","last_undegraded":"2024-03-21T13:07:19.390229+0000","last_fullsized":"2024-03-21T13:07:19.390229+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:05:34.036430+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.036430+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":150,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385432+0000","last_change":"2024-03-21T13:05:29.031235+0000","last_active":"2024-03-21T13:07:19.385432+0000","last_peered":"2024-03-21T13:07:19.385432+0000","last_clean":"2024-03-21T13:07:19.385432+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:07:19.385432+0000","last_undegraded":"2024-03-21T13:07:19.385432+0000","last_fullsized":"2024-03-21T13:07:19.385432+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.031062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.031062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":184,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390544+0000","last_change":"2024-03-21T13:02:26.242867+0000","last_active":"2024-03-21T13:07:19.390544+0000","last_peered":"2024-03-21T13:07:19.390544+0000","last_clean":"2024-03-21T13:07:19.390544+0000","last_became_active":"2024-03-21T13:02:26.242491+0000","last_became_peered":"2024-03-21T13:02:26.242491+0000","last_unstale":"2024-03-21T13:07:19.390544+0000","last_undegraded":"2024-03-21T13:07:19.390544+0000","last_fullsized":"2024-03-21T13:07:19.390544+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.029629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.029629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":28,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386291+0000","last_change":"2024-03-21T13:06:45.406205+0000","last_active":"2024-03-21T13:07:19.386291+0000","last_peered":"2024-03-21T13:07:19.386291+0000","last_clean":"2024-03-21T13:07:19.386291+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:07:19.386291+0000","last_undegraded":"2024-03-21T13:07:19.386291+0000","last_fullsized":"2024-03-21T13:07:19.386291+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:32.151217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:04:32.151217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":394,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.393059+0000","last_change":"2024-03-21T13:06:29.506628+0000","last_active":"2024-03-21T13:07:19.393059+0000","last_peered":"2024-03-21T13:07:19.393059+0000","last_clean":"2024-03-21T13:07:19.393059+0000","last_became_active":"2024-03-21T13:06:29.487725+0000","last_became_peered":"2024-03-21T13:06:29.487725+0000","last_unstale":"2024-03-21T13:07:19.393059+0000","last_undegraded":"2024-03-21T13:07:19.393059+0000","last_fullsized":"2024-03-21T13:07:19.393059+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:28.260333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.260333+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":216,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386979+0000","last_change":"2024-03-21T13:04:31.805808+0000","last_active":"2024-03-21T13:07:19.386979+0000","last_peered":"2024-03-21T13:07:19.386979+0000","last_clean":"2024-03-21T13:07:19.386979+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:07:19.386979+0000","last_undegraded":"2024-03-21T13:07:19.386979+0000","last_fullsized":"2024-03-21T13:07:19.386979+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:31.805690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:31.805690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"125'10","reported_seq":424,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390434+0000","last_change":"2024-03-21T13:02:26.247826+0000","last_active":"2024-03-21T13:07:19.390434+0000","last_peered":"2024-03-21T13:07:19.390434+0000","last_clean":"2024-03-21T13:07:19.390434+0000","last_became_active":"2024-03-21T13:02:26.234494+0000","last_became_peered":"2024-03-21T13:02:26.234494+0000","last_unstale":"2024-03-21T13:07:19.390434+0000","last_undegraded":"2024-03-21T13:07:19.390434+0000","last_fullsized":"2024-03-21T13:07:19.390434+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:23.867564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:23.867564+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":329,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386252+0000","last_change":"2024-03-21T13:06:29.489915+0000","last_active":"2024-03-21T13:07:19.386252+0000","last_peered":"2024-03-21T13:07:19.386252+0000","last_clean":"2024-03-21T13:07:19.386252+0000","last_became_active":"2024-03-21T13:06:29.489392+0000","last_became_peered":"2024-03-21T13:06:29.489392+0000","last_unstale":"2024-03-21T13:07:19.386252+0000","last_undegraded":"2024-03-21T13:07:19.386252+0000","last_fullsized":"2024-03-21T13:07:19.386252+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.042372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.042372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"71'10","reported_seq":364,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391192+0000","last_change":"2024-03-21T13:06:29.530165+0000","last_active":"2024-03-21T13:07:19.391192+0000","last_peered":"2024-03-21T13:07:19.391192+0000","last_clean":"2024-03-21T13:07:19.391192+0000","last_became_active":"2024-03-21T13:06:29.506163+0000","last_became_peered":"2024-03-21T13:06:29.506163+0000","last_unstale":"2024-03-21T13:07:19.391192+0000","last_undegraded":"2024-03-21T13:07:19.391192+0000","last_fullsized":"2024-03-21T13:07:19.391192+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:23.379004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.379004+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":320,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386312+0000","last_change":"2024-03-21T13:06:45.402756+0000","last_active":"2024-03-21T13:07:19.386312+0000","last_peered":"2024-03-21T13:07:19.386312+0000","last_clean":"2024-03-21T13:07:19.386312+0000","last_became_active":"2024-03-21T13:06:45.402482+0000","last_became_peered":"2024-03-21T13:06:45.402482+0000","last_unstale":"2024-03-21T13:07:19.386312+0000","last_undegraded":"2024-03-21T13:07:19.386312+0000","last_fullsized":"2024-03-21T13:07:19.386312+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:21.157295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.157295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":425,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391285+0000","last_change":"2024-03-21T13:06:29.533651+0000","last_active":"2024-03-21T13:07:19.391285+0000","last_peered":"2024-03-21T13:07:19.391285+0000","last_clean":"2024-03-21T13:07:19.391285+0000","last_became_active":"2024-03-21T13:06:29.507429+0000","last_became_peered":"2024-03-21T13:06:29.507429+0000","last_unstale":"2024-03-21T13:07:19.391285+0000","last_undegraded":"2024-03-21T13:07:19.391285+0000","last_fullsized":"2024-03-21T13:07:19.391285+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:18.526428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.526428+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":216,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385714+0000","last_change":"2024-03-21T13:04:12.819697+0000","last_active":"2024-03-21T13:07:19.385714+0000","last_peered":"2024-03-21T13:07:19.385714+0000","last_clean":"2024-03-21T13:07:19.385714+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:07:19.385714+0000","last_undegraded":"2024-03-21T13:07:19.385714+0000","last_fullsized":"2024-03-21T13:07:19.385714+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:12.819556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:04:12.819556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"71'9","reported_seq":235,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386935+0000","last_change":"2024-03-21T13:04:13.827838+0000","last_active":"2024-03-21T13:07:19.386935+0000","last_peered":"2024-03-21T13:07:19.386935+0000","last_clean":"2024-03-21T13:07:19.386935+0000","last_became_active":"2024-03-21T13:02:26.239161+0000","last_became_peered":"2024-03-21T13:02:26.239161+0000","last_unstale":"2024-03-21T13:07:19.386935+0000","last_undegraded":"2024-03-21T13:07:19.386935+0000","last_fullsized":"2024-03-21T13:07:19.386935+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:04:13.827615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:13.827615+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":36,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.393420+0000","last_change":"2024-03-21T13:06:29.489166+0000","last_active":"2024-03-21T13:07:19.393420+0000","last_peered":"2024-03-21T13:07:19.393420+0000","last_clean":"2024-03-21T13:07:19.393420+0000","last_became_active":"2024-03-21T13:06:29.488783+0000","last_became_peered":"2024-03-21T13:06:29.488783+0000","last_unstale":"2024-03-21T13:07:19.393420+0000","last_undegraded":"2024-03-21T13:07:19.393420+0000","last_fullsized":"2024-03-21T13:07:19.393420+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:42.187461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.187461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":315,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.393125+0000","last_change":"2024-03-21T13:05:17.347764+0000","last_active":"2024-03-21T13:07:19.393125+0000","last_peered":"2024-03-21T13:07:19.393125+0000","last_clean":"2024-03-21T13:07:19.393125+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:07:19.393125+0000","last_undegraded":"2024-03-21T13:07:19.393125+0000","last_fullsized":"2024-03-21T13:07:19.393125+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:17.347604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:17.347604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":329,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385654+0000","last_change":"2024-03-21T13:05:23.049951+0000","last_active":"2024-03-21T13:07:19.385654+0000","last_peered":"2024-03-21T13:07:19.385654+0000","last_clean":"2024-03-21T13:07:19.385654+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:07:19.385654+0000","last_undegraded":"2024-03-21T13:07:19.385654+0000","last_fullsized":"2024-03-21T13:07:19.385654+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:05:23.049811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.049811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'9","reported_seq":243,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391558+0000","last_change":"2024-03-21T13:01:48.210983+0000","last_active":"2024-03-21T13:07:19.391558+0000","last_peered":"2024-03-21T13:07:19.391558+0000","last_clean":"2024-03-21T13:07:19.391558+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:07:19.391558+0000","last_undegraded":"2024-03-21T13:07:19.391558+0000","last_fullsized":"2024-03-21T13:07:19.391558+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:01:40.964615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.964615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":303,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386055+0000","last_change":"2024-03-21T13:03:41.952266+0000","last_active":"2024-03-21T13:07:19.386055+0000","last_peered":"2024-03-21T13:07:19.386055+0000","last_clean":"2024-03-21T13:07:19.386055+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:07:19.386055+0000","last_undegraded":"2024-03-21T13:07:19.386055+0000","last_fullsized":"2024-03-21T13:07:19.386055+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:41.952189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:03:41.952189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":306,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391774+0000","last_change":"2024-03-21T13:05:25.466812+0000","last_active":"2024-03-21T13:07:19.391774+0000","last_peered":"2024-03-21T13:07:19.391774+0000","last_clean":"2024-03-21T13:07:19.391774+0000","last_became_active":"2024-03-21T13:02:26.242502+0000","last_became_peered":"2024-03-21T13:02:26.242502+0000","last_unstale":"2024-03-21T13:07:19.391774+0000","last_undegraded":"2024-03-21T13:07:19.391774+0000","last_fullsized":"2024-03-21T13:07:19.391774+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:25.466695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:25.466695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":331,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389146+0000","last_change":"2024-03-21T13:05:41.004652+0000","last_active":"2024-03-21T13:07:19.389146+0000","last_peered":"2024-03-21T13:07:19.389146+0000","last_clean":"2024-03-21T13:07:19.389146+0000","last_became_active":"2024-03-21T13:02:26.252406+0000","last_became_peered":"2024-03-21T13:02:26.252406+0000","last_unstale":"2024-03-21T13:07:19.389146+0000","last_undegraded":"2024-03-21T13:07:19.389146+0000","last_fullsized":"2024-03-21T13:07:19.389146+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.004412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.004412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'6","reported_seq":350,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392539+0000","last_change":"2024-03-21T13:06:29.546967+0000","last_active":"2024-03-21T13:07:19.392539+0000","last_peered":"2024-03-21T13:07:19.392539+0000","last_clean":"2024-03-21T13:07:19.392539+0000","last_became_active":"2024-03-21T13:06:29.513331+0000","last_became_peered":"2024-03-21T13:06:29.513331+0000","last_unstale":"2024-03-21T13:07:19.392539+0000","last_undegraded":"2024-03-21T13:07:19.392539+0000","last_fullsized":"2024-03-21T13:07:19.392539+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T13:04:22.563861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:22.563861+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"208'12061","reported_seq":24416,"reported_epoch":208,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:07:19.873924+0000","last_change":"2024-03-21T13:06:30.554953+0000","last_active":"2024-03-21T13:07:19.873924+0000","last_peered":"2024-03-21T13:07:19.873924+0000","last_clean":"2024-03-21T13:06:28.440945+0000","last_became_active":"2024-03-21T13:06:30.531682+0000","last_became_peered":"2024-03-21T13:06:30.531682+0000","last_unstale":"2024-03-21T13:07:19.873924+0000","last_undegraded":"2024-03-21T13:07:19.873924+0000","last_fullsized":"2024-03-21T13:07:19.873924+0000","mapping_epoch":196,"log_start":"197'8748","ondisk_log_start":"197'8748","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3313,"log_dups_size":0,"ondisk_log_size":3313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325779456,"num_objects":4971,"num_object_clones":0,"num_object_copies":14913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":223,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12061,"num_write_kb":545024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6129,"num_bytes_recovered":401670144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,2],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"2,4,6","objects":4971}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":196,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389597+0000","last_change":"2024-03-21T13:06:29.509970+0000","last_active":"2024-03-21T13:07:19.389597+0000","last_peered":"2024-03-21T13:07:19.389597+0000","last_clean":"2024-03-21T13:07:19.389597+0000","last_became_active":"2024-03-21T13:06:29.509616+0000","last_became_peered":"2024-03-21T13:06:29.509616+0000","last_unstale":"2024-03-21T13:07:19.389597+0000","last_undegraded":"2024-03-21T13:07:19.389597+0000","last_fullsized":"2024-03-21T13:07:19.389597+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.973796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.973796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":359,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386532+0000","last_change":"2024-03-21T13:06:29.498660+0000","last_active":"2024-03-21T13:07:19.386532+0000","last_peered":"2024-03-21T13:07:19.386532+0000","last_clean":"2024-03-21T13:07:19.386532+0000","last_became_active":"2024-03-21T13:06:29.487439+0000","last_became_peered":"2024-03-21T13:06:29.487439+0000","last_unstale":"2024-03-21T13:07:19.386532+0000","last_undegraded":"2024-03-21T13:07:19.386532+0000","last_fullsized":"2024-03-21T13:07:19.386532+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:05:44.816353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:44.816353+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'15","reported_seq":250,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385833+0000","last_change":"2024-03-21T13:04:25.787896+0000","last_active":"2024-03-21T13:07:19.385833+0000","last_peered":"2024-03-21T13:07:19.385833+0000","last_clean":"2024-03-21T13:07:19.385833+0000","last_became_active":"2024-03-21T13:02:26.237800+0000","last_became_peered":"2024-03-21T13:02:26.237800+0000","last_unstale":"2024-03-21T13:07:19.385833+0000","last_undegraded":"2024-03-21T13:07:19.385833+0000","last_fullsized":"2024-03-21T13:07:19.385833+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:04:25.787703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:25.787703+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":421,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386378+0000","last_change":"2024-03-21T13:04:16.810359+0000","last_active":"2024-03-21T13:07:19.386378+0000","last_peered":"2024-03-21T13:07:19.386378+0000","last_clean":"2024-03-21T13:07:19.386378+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:07:19.386378+0000","last_undegraded":"2024-03-21T13:07:19.386378+0000","last_fullsized":"2024-03-21T13:07:19.386378+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:04:16.810269+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:04:16.810269+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"208'11877","reported_seq":11968,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.797020+0000","last_change":"2024-03-21T13:04:46.259252+0000","last_active":"2024-03-21T13:07:19.797020+0000","last_peered":"2024-03-21T13:07:19.797020+0000","last_clean":"2024-03-21T13:07:19.797020+0000","last_became_active":"2024-03-21T13:04:46.258930+0000","last_became_peered":"2024-03-21T13:04:46.258930+0000","last_unstale":"2024-03-21T13:07:19.797020+0000","last_undegraded":"2024-03-21T13:07:19.797020+0000","last_fullsized":"2024-03-21T13:07:19.797020+0000","mapping_epoch":168,"log_start":"194'8343","ondisk_log_start":"194'8343","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3534,"log_dups_size":0,"ondisk_log_size":3534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323026944,"num_objects":4929,"num_object_clones":0,"num_object_copies":14787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":537792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":311,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391055+0000","last_change":"2024-03-21T13:05:43.596914+0000","last_active":"2024-03-21T13:07:19.391055+0000","last_peered":"2024-03-21T13:07:19.391055+0000","last_clean":"2024-03-21T13:07:19.391055+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:07:19.391055+0000","last_undegraded":"2024-03-21T13:07:19.391055+0000","last_fullsized":"2024-03-21T13:07:19.391055+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.596691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":329,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.387061+0000","last_change":"2024-03-21T13:05:09.138059+0000","last_active":"2024-03-21T13:07:19.387061+0000","last_peered":"2024-03-21T13:07:19.387061+0000","last_clean":"2024-03-21T13:07:19.387061+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:07:19.387061+0000","last_undegraded":"2024-03-21T13:07:19.387061+0000","last_fullsized":"2024-03-21T13:07:19.387061+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:09.137868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:09.137868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"71'12","reported_seq":248,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390262+0000","last_change":"2024-03-21T13:02:14.383361+0000","last_active":"2024-03-21T13:07:19.390262+0000","last_peered":"2024-03-21T13:07:19.390262+0000","last_clean":"2024-03-21T13:07:19.390262+0000","last_became_active":"2024-03-21T13:02:14.347102+0000","last_became_peered":"2024-03-21T13:02:14.347102+0000","last_unstale":"2024-03-21T13:07:19.390262+0000","last_undegraded":"2024-03-21T13:07:19.390262+0000","last_fullsized":"2024-03-21T13:07:19.390262+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:00:32.864608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.864608+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":179,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391692+0000","last_change":"2024-03-21T13:06:29.487148+0000","last_active":"2024-03-21T13:07:19.391692+0000","last_peered":"2024-03-21T13:07:19.391692+0000","last_clean":"2024-03-21T13:07:19.391692+0000","last_became_active":"2024-03-21T13:06:29.486743+0000","last_became_peered":"2024-03-21T13:06:29.486743+0000","last_unstale":"2024-03-21T13:07:19.391692+0000","last_undegraded":"2024-03-21T13:07:19.391692+0000","last_fullsized":"2024-03-21T13:07:19.391692+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:21.170999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.170999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"208'12039","reported_seq":12127,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.813473+0000","last_change":"2024-03-21T13:04:46.257959+0000","last_active":"2024-03-21T13:07:19.813473+0000","last_peered":"2024-03-21T13:07:19.813473+0000","last_clean":"2024-03-21T13:07:19.813473+0000","last_became_active":"2024-03-21T13:04:46.257488+0000","last_became_peered":"2024-03-21T13:04:46.257488+0000","last_unstale":"2024-03-21T13:07:19.813473+0000","last_undegraded":"2024-03-21T13:07:19.813473+0000","last_fullsized":"2024-03-21T13:07:19.813473+0000","mapping_epoch":168,"log_start":"194'8497","ondisk_log_start":"194'8497","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3542,"log_dups_size":0,"ondisk_log_size":3542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330235904,"num_objects":5039,"num_object_clones":0,"num_object_copies":15117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12039,"num_write_kb":546496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":317,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385337+0000","last_change":"2024-03-21T13:06:29.504532+0000","last_active":"2024-03-21T13:07:19.385337+0000","last_peered":"2024-03-21T13:07:19.385337+0000","last_clean":"2024-03-21T13:07:19.385337+0000","last_became_active":"2024-03-21T13:06:29.503980+0000","last_became_peered":"2024-03-21T13:06:29.503980+0000","last_unstale":"2024-03-21T13:07:19.385337+0000","last_undegraded":"2024-03-21T13:07:19.385337+0000","last_fullsized":"2024-03-21T13:07:19.385337+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:34.848616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:34.848616+0000","last_clean_scrub_stamp":"2024-03-21T13:04:34.848616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":173,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385684+0000","last_change":"2024-03-21T13:06:29.503568+0000","last_active":"2024-03-21T13:07:19.385684+0000","last_peered":"2024-03-21T13:07:19.385684+0000","last_clean":"2024-03-21T13:07:19.385684+0000","last_became_active":"2024-03-21T13:06:29.503252+0000","last_became_peered":"2024-03-21T13:06:29.503252+0000","last_unstale":"2024-03-21T13:07:19.385684+0000","last_undegraded":"2024-03-21T13:07:19.385684+0000","last_fullsized":"2024-03-21T13:07:19.385684+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.216547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.216547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":329,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386823+0000","last_change":"2024-03-21T13:05:22.078652+0000","last_active":"2024-03-21T13:07:19.386823+0000","last_peered":"2024-03-21T13:07:19.386823+0000","last_clean":"2024-03-21T13:07:19.386823+0000","last_became_active":"2024-03-21T13:02:26.236780+0000","last_became_peered":"2024-03-21T13:02:26.236780+0000","last_unstale":"2024-03-21T13:07:19.386823+0000","last_undegraded":"2024-03-21T13:07:19.386823+0000","last_fullsized":"2024-03-21T13:07:19.386823+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:22.078546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.078546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"71'19","reported_seq":405,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389932+0000","last_change":"2024-03-21T13:05:29.992376+0000","last_active":"2024-03-21T13:07:19.389932+0000","last_peered":"2024-03-21T13:07:19.389932+0000","last_clean":"2024-03-21T13:07:19.389932+0000","last_became_active":"2024-03-21T13:02:26.240350+0000","last_became_peered":"2024-03-21T13:02:26.240350+0000","last_unstale":"2024-03-21T13:07:19.389932+0000","last_undegraded":"2024-03-21T13:07:19.389932+0000","last_fullsized":"2024-03-21T13:07:19.389932+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:05:29.992207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.992207+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"208'11911","reported_seq":22452,"reported_epoch":208,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:07:20.704199+0000","last_change":"2024-03-21T13:06:30.579343+0000","last_active":"2024-03-21T13:07:20.704199+0000","last_peered":"2024-03-21T13:07:20.704199+0000","last_clean":"2024-03-21T13:06:28.473967+0000","last_became_active":"2024-03-21T13:06:30.570964+0000","last_became_peered":"2024-03-21T13:06:30.570964+0000","last_unstale":"2024-03-21T13:07:20.704199+0000","last_undegraded":"2024-03-21T13:07:20.704199+0000","last_fullsized":"2024-03-21T13:07:20.704199+0000","mapping_epoch":196,"log_start":"198'8879","ondisk_log_start":"198'8879","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3032,"log_dups_size":0,"ondisk_log_size":3032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316342272,"num_objects":4827,"num_object_clones":0,"num_object_copies":14481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":733,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11911,"num_write_kb":535616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5223,"num_bytes_recovered":342294528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,2],"avail_no_missing":["3","2","6"],"object_location_counts":[{"shards":"2,3,6","objects":4827}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":222,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385548+0000","last_change":"2024-03-21T13:06:29.501377+0000","last_active":"2024-03-21T13:07:19.385548+0000","last_peered":"2024-03-21T13:07:19.385548+0000","last_clean":"2024-03-21T13:07:19.385548+0000","last_became_active":"2024-03-21T13:06:29.500961+0000","last_became_peered":"2024-03-21T13:06:29.500961+0000","last_unstale":"2024-03-21T13:07:19.385548+0000","last_undegraded":"2024-03-21T13:07:19.385548+0000","last_fullsized":"2024-03-21T13:07:19.385548+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.926828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.926828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":36,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392639+0000","last_change":"2024-03-21T13:06:29.503278+0000","last_active":"2024-03-21T13:07:19.392639+0000","last_peered":"2024-03-21T13:07:19.392639+0000","last_clean":"2024-03-21T13:07:19.392639+0000","last_became_active":"2024-03-21T13:06:29.502745+0000","last_became_peered":"2024-03-21T13:06:29.502745+0000","last_unstale":"2024-03-21T13:07:19.392639+0000","last_undegraded":"2024-03-21T13:07:19.392639+0000","last_fullsized":"2024-03-21T13:07:19.392639+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:37.225558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:37.225558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":182,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391103+0000","last_change":"2024-03-21T13:02:26.244355+0000","last_active":"2024-03-21T13:07:19.391103+0000","last_peered":"2024-03-21T13:07:19.391103+0000","last_clean":"2024-03-21T13:07:19.391103+0000","last_became_active":"2024-03-21T13:02:26.243974+0000","last_became_peered":"2024-03-21T13:02:26.243974+0000","last_unstale":"2024-03-21T13:07:19.391103+0000","last_undegraded":"2024-03-21T13:07:19.391103+0000","last_fullsized":"2024-03-21T13:07:19.391103+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:33.998382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:33.998382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"86'13","reported_seq":280,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392772+0000","last_change":"2024-03-21T13:05:08.081782+0000","last_active":"2024-03-21T13:07:19.392772+0000","last_peered":"2024-03-21T13:07:19.392772+0000","last_clean":"2024-03-21T13:07:19.392772+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:07:19.392772+0000","last_undegraded":"2024-03-21T13:07:19.392772+0000","last_fullsized":"2024-03-21T13:07:19.392772+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:05:08.081600+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:05:08.081600+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.8","version":"208'12130","reported_seq":12313,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:20.682879+0000","last_change":"2024-03-21T13:06:29.535965+0000","last_active":"2024-03-21T13:07:20.682879+0000","last_peered":"2024-03-21T13:07:20.682879+0000","last_clean":"2024-03-21T13:07:20.682879+0000","last_became_active":"2024-03-21T13:04:46.257160+0000","last_became_peered":"2024-03-21T13:04:46.257160+0000","last_unstale":"2024-03-21T13:07:20.682879+0000","last_undegraded":"2024-03-21T13:07:20.682879+0000","last_fullsized":"2024-03-21T13:07:20.682879+0000","mapping_epoch":168,"log_start":"198'9023","ondisk_log_start":"198'9023","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'8553","last_scrub_stamp":"2024-03-21T13:06:29.535762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:06:29.535762+0000","objects_scrubbed":8459,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:44.691312+0000","scrub_duration":868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319946752,"num_objects":4882,"num_object_clones":0,"num_object_copies":14646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12130,"num_write_kb":544384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":297,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391753+0000","last_change":"2024-03-21T13:01:46.738822+0000","last_active":"2024-03-21T13:07:19.391753+0000","last_peered":"2024-03-21T13:07:19.391753+0000","last_clean":"2024-03-21T13:07:19.391753+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:07:19.391753+0000","last_undegraded":"2024-03-21T13:07:19.391753+0000","last_fullsized":"2024-03-21T13:07:19.391753+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.738659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":221,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390331+0000","last_change":"2024-03-21T13:05:43.961181+0000","last_active":"2024-03-21T13:07:19.390331+0000","last_peered":"2024-03-21T13:07:19.390331+0000","last_clean":"2024-03-21T13:07:19.390331+0000","last_became_active":"2024-03-21T13:02:26.243692+0000","last_became_peered":"2024-03-21T13:02:26.243692+0000","last_unstale":"2024-03-21T13:07:19.390331+0000","last_undegraded":"2024-03-21T13:07:19.390331+0000","last_fullsized":"2024-03-21T13:07:19.390331+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.961057+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":28,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389653+0000","last_change":"2024-03-21T13:06:45.410372+0000","last_active":"2024-03-21T13:07:19.389653+0000","last_peered":"2024-03-21T13:07:19.389653+0000","last_clean":"2024-03-21T13:07:19.389653+0000","last_became_active":"2024-03-21T13:06:45.410017+0000","last_became_peered":"2024-03-21T13:06:45.410017+0000","last_unstale":"2024-03-21T13:07:19.389653+0000","last_undegraded":"2024-03-21T13:07:19.389653+0000","last_fullsized":"2024-03-21T13:07:19.389653+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:48.265982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:48.265982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"196'30","reported_seq":138,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391346+0000","last_change":"2024-03-21T13:06:29.547664+0000","last_active":"2024-03-21T13:07:19.391346+0000","last_peered":"2024-03-21T13:07:19.391346+0000","last_clean":"2024-03-21T13:07:19.391346+0000","last_became_active":"2024-03-21T13:06:29.508620+0000","last_became_peered":"2024-03-21T13:06:29.508620+0000","last_unstale":"2024-03-21T13:07:19.391346+0000","last_undegraded":"2024-03-21T13:07:19.391346+0000","last_fullsized":"2024-03-21T13:07:19.391346+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'29","last_scrub_stamp":"2024-03-21T13:05:22.201879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.201879+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":32,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390768+0000","last_change":"2024-03-21T13:06:45.407111+0000","last_active":"2024-03-21T13:07:19.390768+0000","last_peered":"2024-03-21T13:07:19.390768+0000","last_clean":"2024-03-21T13:07:19.390768+0000","last_became_active":"2024-03-21T13:06:45.406734+0000","last_became_peered":"2024-03-21T13:06:45.406734+0000","last_unstale":"2024-03-21T13:07:19.390768+0000","last_undegraded":"2024-03-21T13:07:19.390768+0000","last_fullsized":"2024-03-21T13:07:19.390768+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.312583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.312583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.b","version":"208'11990","reported_seq":25181,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:20.682919+0000","last_change":"2024-03-21T13:07:19.414805+0000","last_active":"2024-03-21T13:07:20.682919+0000","last_peered":"2024-03-21T13:07:20.682919+0000","last_clean":"2024-03-21T13:07:20.682919+0000","last_became_active":"2024-03-21T13:07:19.393088+0000","last_became_peered":"2024-03-21T13:07:19.393088+0000","last_unstale":"2024-03-21T13:07:20.682919+0000","last_undegraded":"2024-03-21T13:07:20.682919+0000","last_fullsized":"2024-03-21T13:07:20.682919+0000","mapping_epoch":207,"log_start":"198'8860","ondisk_log_start":"198'8860","created":168,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3130,"log_dups_size":0,"ondisk_log_size":3130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322568192,"num_objects":4922,"num_object_clones":0,"num_object_copies":14766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11990,"num_write_kb":541184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6543,"num_bytes_recovered":428802048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":312,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386922+0000","last_change":"2024-03-21T13:05:08.172825+0000","last_active":"2024-03-21T13:07:19.386922+0000","last_peered":"2024-03-21T13:07:19.386922+0000","last_clean":"2024-03-21T13:07:19.386922+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:07:19.386922+0000","last_undegraded":"2024-03-21T13:07:19.386922+0000","last_fullsized":"2024-03-21T13:07:19.386922+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:08.172705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:08.172705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":38,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389386+0000","last_change":"2024-03-21T13:06:29.510026+0000","last_active":"2024-03-21T13:07:19.389386+0000","last_peered":"2024-03-21T13:07:19.389386+0000","last_clean":"2024-03-21T13:07:19.389386+0000","last_became_active":"2024-03-21T13:06:29.509674+0000","last_became_peered":"2024-03-21T13:06:29.509674+0000","last_unstale":"2024-03-21T13:07:19.389386+0000","last_undegraded":"2024-03-21T13:07:19.389386+0000","last_fullsized":"2024-03-21T13:07:19.389386+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.247107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.247107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":36,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389627+0000","last_change":"2024-03-21T13:06:29.510171+0000","last_active":"2024-03-21T13:07:19.389627+0000","last_peered":"2024-03-21T13:07:19.389627+0000","last_clean":"2024-03-21T13:07:19.389627+0000","last_became_active":"2024-03-21T13:06:29.509315+0000","last_became_peered":"2024-03-21T13:06:29.509315+0000","last_unstale":"2024-03-21T13:07:19.389627+0000","last_undegraded":"2024-03-21T13:07:19.389627+0000","last_fullsized":"2024-03-21T13:07:19.389627+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:46.263644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:05:46.263644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"71'13","reported_seq":250,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386381+0000","last_change":"2024-03-21T13:05:15.119776+0000","last_active":"2024-03-21T13:07:19.386381+0000","last_peered":"2024-03-21T13:07:19.386381+0000","last_clean":"2024-03-21T13:07:19.386381+0000","last_became_active":"2024-03-21T13:02:26.234230+0000","last_became_peered":"2024-03-21T13:02:26.234230+0000","last_unstale":"2024-03-21T13:07:19.386381+0000","last_undegraded":"2024-03-21T13:07:19.386381+0000","last_fullsized":"2024-03-21T13:07:19.386381+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'13","last_scrub_stamp":"2024-03-21T13:05:15.119643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:15.119643+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":173,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.387170+0000","last_change":"2024-03-21T13:06:29.502194+0000","last_active":"2024-03-21T13:07:19.387170+0000","last_peered":"2024-03-21T13:07:19.387170+0000","last_clean":"2024-03-21T13:07:19.387170+0000","last_became_active":"2024-03-21T13:06:29.501625+0000","last_became_peered":"2024-03-21T13:06:29.501625+0000","last_unstale":"2024-03-21T13:07:19.387170+0000","last_undegraded":"2024-03-21T13:07:19.387170+0000","last_fullsized":"2024-03-21T13:07:19.387170+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:23.243468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.243468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.9","version":"208'12264","reported_seq":12355,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.797184+0000","last_change":"2024-03-21T13:04:46.260725+0000","last_active":"2024-03-21T13:07:19.797184+0000","last_peered":"2024-03-21T13:07:19.797184+0000","last_clean":"2024-03-21T13:07:19.797184+0000","last_became_active":"2024-03-21T13:04:46.260364+0000","last_became_peered":"2024-03-21T13:04:46.260364+0000","last_unstale":"2024-03-21T13:07:19.797184+0000","last_undegraded":"2024-03-21T13:07:19.797184+0000","last_fullsized":"2024-03-21T13:07:19.797184+0000","mapping_epoch":168,"log_start":"197'8702","ondisk_log_start":"197'8702","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3562,"log_dups_size":0,"ondisk_log_size":3562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331218944,"num_objects":5054,"num_object_clones":0,"num_object_copies":15162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12264,"num_write_kb":554176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":190,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.393010+0000","last_change":"2024-03-21T13:04:27.549179+0000","last_active":"2024-03-21T13:07:19.393010+0000","last_peered":"2024-03-21T13:07:19.393010+0000","last_clean":"2024-03-21T13:07:19.393010+0000","last_became_active":"2024-03-21T13:02:26.235640+0000","last_became_peered":"2024-03-21T13:02:26.235640+0000","last_unstale":"2024-03-21T13:07:19.393010+0000","last_undegraded":"2024-03-21T13:07:19.393010+0000","last_fullsized":"2024-03-21T13:07:19.393010+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:27.549086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:27.549086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":573,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385448+0000","last_change":"2024-03-21T13:04:11.818847+0000","last_active":"2024-03-21T13:07:19.385448+0000","last_peered":"2024-03-21T13:07:19.385448+0000","last_clean":"2024-03-21T13:07:19.385448+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:07:19.385448+0000","last_undegraded":"2024-03-21T13:07:19.385448+0000","last_fullsized":"2024-03-21T13:07:19.385448+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:04:11.818749+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:00:16.180126+0000","last_clean_scrub_stamp":"2024-03-21T13:04:11.818749+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":316,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390185+0000","last_change":"2024-03-21T13:02:26.242969+0000","last_active":"2024-03-21T13:07:19.390185+0000","last_peered":"2024-03-21T13:07:19.390185+0000","last_clean":"2024-03-21T13:07:19.390185+0000","last_became_active":"2024-03-21T13:02:26.242456+0000","last_became_peered":"2024-03-21T13:02:26.242456+0000","last_unstale":"2024-03-21T13:07:19.390185+0000","last_undegraded":"2024-03-21T13:07:19.390185+0000","last_fullsized":"2024-03-21T13:07:19.390185+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:38.007486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.007486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"183'24","reported_seq":457,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391330+0000","last_change":"2024-03-21T13:05:42.033620+0000","last_active":"2024-03-21T13:07:19.391330+0000","last_peered":"2024-03-21T13:07:19.391330+0000","last_clean":"2024-03-21T13:07:19.391330+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:07:19.391330+0000","last_undegraded":"2024-03-21T13:07:19.391330+0000","last_fullsized":"2024-03-21T13:07:19.391330+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'24","last_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.033266+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":328,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386592+0000","last_change":"2024-03-21T13:06:29.504771+0000","last_active":"2024-03-21T13:07:19.386592+0000","last_peered":"2024-03-21T13:07:19.386592+0000","last_clean":"2024-03-21T13:07:19.386592+0000","last_became_active":"2024-03-21T13:06:29.504250+0000","last_became_peered":"2024-03-21T13:06:29.504250+0000","last_unstale":"2024-03-21T13:07:19.386592+0000","last_undegraded":"2024-03-21T13:07:19.386592+0000","last_fullsized":"2024-03-21T13:07:19.386592+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:20.061709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.061709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"208'12153","reported_seq":12247,"reported_epoch":208,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-21T13:07:20.689825+0000","last_change":"2024-03-21T13:06:30.630989+0000","last_active":"2024-03-21T13:07:20.689825+0000","last_peered":"2024-03-21T13:07:20.689825+0000","last_clean":"2024-03-21T13:06:28.451225+0000","last_became_active":"2024-03-21T13:06:30.598008+0000","last_became_peered":"2024-03-21T13:06:30.598008+0000","last_unstale":"2024-03-21T13:07:20.689825+0000","last_undegraded":"2024-03-21T13:07:20.689825+0000","last_fullsized":"2024-03-21T13:07:20.689825+0000","mapping_epoch":196,"log_start":"198'9064","ondisk_log_start":"198'9064","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3089,"log_dups_size":0,"ondisk_log_size":3089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324075520,"num_objects":4945,"num_object_clones":0,"num_object_copies":14835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4945,"num_objects_unfound":0,"num_objects_dirty":4945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12153,"num_write_kb":547136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":4945}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":192,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390779+0000","last_change":"2024-03-21T13:05:29.457708+0000","last_active":"2024-03-21T13:07:19.390779+0000","last_peered":"2024-03-21T13:07:19.390779+0000","last_clean":"2024-03-21T13:07:19.390779+0000","last_became_active":"2024-03-21T13:02:26.242708+0000","last_became_peered":"2024-03-21T13:02:26.242708+0000","last_unstale":"2024-03-21T13:07:19.390779+0000","last_undegraded":"2024-03-21T13:07:19.390779+0000","last_fullsized":"2024-03-21T13:07:19.390779+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.457597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.457597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":216,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392289+0000","last_change":"2024-03-21T13:05:11.035789+0000","last_active":"2024-03-21T13:07:19.392289+0000","last_peered":"2024-03-21T13:07:19.392289+0000","last_clean":"2024-03-21T13:07:19.392289+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:07:19.392289+0000","last_undegraded":"2024-03-21T13:07:19.392289+0000","last_fullsized":"2024-03-21T13:07:19.392289+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:11.035680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:05:11.035680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":850,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.385443+0000","last_change":"2024-03-21T13:03:45.889240+0000","last_active":"2024-03-21T13:07:19.385443+0000","last_peered":"2024-03-21T13:07:19.385443+0000","last_clean":"2024-03-21T13:07:19.385443+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:07:19.385443+0000","last_undegraded":"2024-03-21T13:07:19.385443+0000","last_fullsized":"2024-03-21T13:07:19.385443+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:03:45.889083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:03:45.889083+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":512,"num_read_kb":507,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":395,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386621+0000","last_change":"2024-03-21T13:06:33.328433+0000","last_active":"2024-03-21T13:07:19.386621+0000","last_peered":"2024-03-21T13:07:19.386621+0000","last_clean":"2024-03-21T13:07:19.386621+0000","last_became_active":"2024-03-21T13:06:29.526702+0000","last_became_peered":"2024-03-21T13:06:29.526702+0000","last_unstale":"2024-03-21T13:07:19.386621+0000","last_undegraded":"2024-03-21T13:07:19.386621+0000","last_fullsized":"2024-03-21T13:07:19.386621+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:06:33.328260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:06:33.328260+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:36.122344+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":189,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386630+0000","last_change":"2024-03-21T13:05:13.085740+0000","last_active":"2024-03-21T13:07:19.386630+0000","last_peered":"2024-03-21T13:07:19.386630+0000","last_clean":"2024-03-21T13:07:19.386630+0000","last_became_active":"2024-03-21T13:02:26.232625+0000","last_became_peered":"2024-03-21T13:02:26.232625+0000","last_unstale":"2024-03-21T13:07:19.386630+0000","last_undegraded":"2024-03-21T13:07:19.386630+0000","last_fullsized":"2024-03-21T13:07:19.386630+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.085625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.085625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"208'11908","reported_seq":22783,"reported_epoch":208,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:07:20.700546+0000","last_change":"2024-03-21T13:06:30.605819+0000","last_active":"2024-03-21T13:07:20.700546+0000","last_peered":"2024-03-21T13:07:20.700546+0000","last_clean":"2024-03-21T13:06:28.443810+0000","last_became_active":"2024-03-21T13:06:30.588924+0000","last_became_peered":"2024-03-21T13:06:30.588924+0000","last_unstale":"2024-03-21T13:07:20.700546+0000","last_undegraded":"2024-03-21T13:07:20.700546+0000","last_fullsized":"2024-03-21T13:07:20.700546+0000","mapping_epoch":196,"log_start":"198'8781","ondisk_log_start":"198'8781","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318242816,"num_objects":4856,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":610,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":536448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5389,"num_bytes_recovered":353173504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":4856}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":208,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389449+0000","last_change":"2024-03-21T13:06:29.489827+0000","last_active":"2024-03-21T13:07:19.389449+0000","last_peered":"2024-03-21T13:07:19.389449+0000","last_clean":"2024-03-21T13:07:19.389449+0000","last_became_active":"2024-03-21T13:06:29.489464+0000","last_became_peered":"2024-03-21T13:06:29.489464+0000","last_unstale":"2024-03-21T13:07:19.389449+0000","last_undegraded":"2024-03-21T13:07:19.389449+0000","last_fullsized":"2024-03-21T13:07:19.389449+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:35.000684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:35.000684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":57,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390304+0000","last_change":"2024-03-21T13:06:29.484504+0000","last_active":"2024-03-21T13:07:19.390304+0000","last_peered":"2024-03-21T13:07:19.390304+0000","last_clean":"2024-03-21T13:07:19.390304+0000","last_became_active":"2024-03-21T13:06:29.484174+0000","last_became_peered":"2024-03-21T13:06:29.484174+0000","last_unstale":"2024-03-21T13:07:19.390304+0000","last_undegraded":"2024-03-21T13:07:19.390304+0000","last_fullsized":"2024-03-21T13:07:19.390304+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.282145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.282145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"71'19","reported_seq":287,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386556+0000","last_change":"2024-03-21T13:06:29.508182+0000","last_active":"2024-03-21T13:07:19.386556+0000","last_peered":"2024-03-21T13:07:19.386556+0000","last_clean":"2024-03-21T13:07:19.386556+0000","last_became_active":"2024-03-21T13:06:29.492612+0000","last_became_peered":"2024-03-21T13:06:29.492612+0000","last_unstale":"2024-03-21T13:07:19.386556+0000","last_undegraded":"2024-03-21T13:07:19.386556+0000","last_fullsized":"2024-03-21T13:07:19.386556+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:05:28.062237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.062237+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":330,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389986+0000","last_change":"2024-03-21T13:06:29.497191+0000","last_active":"2024-03-21T13:07:19.389986+0000","last_peered":"2024-03-21T13:07:19.389986+0000","last_clean":"2024-03-21T13:07:19.389986+0000","last_became_active":"2024-03-21T13:06:29.496679+0000","last_became_peered":"2024-03-21T13:06:29.496679+0000","last_unstale":"2024-03-21T13:07:19.389986+0000","last_undegraded":"2024-03-21T13:07:19.389986+0000","last_fullsized":"2024-03-21T13:07:19.389986+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:26.932188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.932188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"208'12134","reported_seq":19227,"reported_epoch":208,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:07:20.023069+0000","last_change":"2024-03-21T13:07:00.587293+0000","last_active":"2024-03-21T13:07:20.023069+0000","last_peered":"2024-03-21T13:07:20.023069+0000","last_clean":"2024-03-21T13:06:28.457861+0000","last_became_active":"2024-03-21T13:06:30.561284+0000","last_became_peered":"2024-03-21T13:06:30.561284+0000","last_unstale":"2024-03-21T13:07:20.023069+0000","last_undegraded":"2024-03-21T13:07:20.023069+0000","last_fullsized":"2024-03-21T13:07:20.023069+0000","mapping_epoch":196,"log_start":"193'8283","ondisk_log_start":"193'8283","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":299,"log_size":3851,"log_dups_size":0,"ondisk_log_size":3851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324141056,"num_objects":4946,"num_object_clones":0,"num_object_copies":14838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1779,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12134,"num_write_kb":546560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3493,"num_bytes_recovered":228851712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,2],"avail_no_missing":["0","2","4"],"object_location_counts":[{"shards":"0,2,4","objects":4946}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":214,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389112+0000","last_change":"2024-03-21T13:06:29.502310+0000","last_active":"2024-03-21T13:07:19.389112+0000","last_peered":"2024-03-21T13:07:19.389112+0000","last_clean":"2024-03-21T13:07:19.389112+0000","last_became_active":"2024-03-21T13:06:29.501983+0000","last_became_peered":"2024-03-21T13:06:29.501983+0000","last_unstale":"2024-03-21T13:07:19.389112+0000","last_undegraded":"2024-03-21T13:07:19.389112+0000","last_fullsized":"2024-03-21T13:07:19.389112+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:31.121913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.121913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":196,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.391269+0000","last_change":"2024-03-21T13:06:29.511420+0000","last_active":"2024-03-21T13:07:19.391269+0000","last_peered":"2024-03-21T13:07:19.391269+0000","last_clean":"2024-03-21T13:07:19.391269+0000","last_became_active":"2024-03-21T13:06:29.510833+0000","last_became_peered":"2024-03-21T13:06:29.510833+0000","last_unstale":"2024-03-21T13:07:19.391269+0000","last_undegraded":"2024-03-21T13:07:19.391269+0000","last_fullsized":"2024-03-21T13:07:19.391269+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.324701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.324701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"86'15","reported_seq":274,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392783+0000","last_change":"2024-03-21T13:06:29.513960+0000","last_active":"2024-03-21T13:07:19.392783+0000","last_peered":"2024-03-21T13:07:19.392783+0000","last_clean":"2024-03-21T13:07:19.392783+0000","last_became_active":"2024-03-21T13:06:29.492863+0000","last_became_peered":"2024-03-21T13:06:29.492863+0000","last_unstale":"2024-03-21T13:07:19.392783+0000","last_undegraded":"2024-03-21T13:07:19.392783+0000","last_fullsized":"2024-03-21T13:07:19.392783+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-03-21T13:05:42.308556+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.308556+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"125'12","reported_seq":320,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389959+0000","last_change":"2024-03-21T13:05:13.903736+0000","last_active":"2024-03-21T13:07:19.389959+0000","last_peered":"2024-03-21T13:07:19.389959+0000","last_clean":"2024-03-21T13:07:19.389959+0000","last_became_active":"2024-03-21T13:02:26.253720+0000","last_became_peered":"2024-03-21T13:02:26.253720+0000","last_unstale":"2024-03-21T13:07:19.389959+0000","last_undegraded":"2024-03-21T13:07:19.389959+0000","last_fullsized":"2024-03-21T13:07:19.389959+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'12","last_scrub_stamp":"2024-03-21T13:05:13.903589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.903589+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"208'12181","reported_seq":26215,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.773204+0000","last_change":"2024-03-21T13:07:08.359129+0000","last_active":"2024-03-21T13:07:19.773204+0000","last_peered":"2024-03-21T13:07:19.773204+0000","last_clean":"2024-03-21T13:07:19.773204+0000","last_became_active":"2024-03-21T13:07:08.338910+0000","last_became_peered":"2024-03-21T13:07:08.338910+0000","last_unstale":"2024-03-21T13:07:19.773204+0000","last_undegraded":"2024-03-21T13:07:19.773204+0000","last_fullsized":"2024-03-21T13:07:19.773204+0000","mapping_epoch":203,"log_start":"197'8769","ondisk_log_start":"197'8769","created":168,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3412,"log_dups_size":0,"ondisk_log_size":3412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328466456,"num_objects":5013,"num_object_clones":0,"num_object_copies":15039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12181,"num_write_kb":550145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6964,"num_bytes_recovered":456327192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":318,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386562+0000","last_change":"2024-03-21T13:06:29.489617+0000","last_active":"2024-03-21T13:07:19.386562+0000","last_peered":"2024-03-21T13:07:19.386562+0000","last_clean":"2024-03-21T13:07:19.386562+0000","last_became_active":"2024-03-21T13:06:29.489289+0000","last_became_peered":"2024-03-21T13:06:29.489289+0000","last_unstale":"2024-03-21T13:07:19.386562+0000","last_undegraded":"2024-03-21T13:07:19.386562+0000","last_fullsized":"2024-03-21T13:07:19.386562+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.951655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.951655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":352,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386611+0000","last_change":"2024-03-21T13:06:41.354102+0000","last_active":"2024-03-21T13:07:19.386611+0000","last_peered":"2024-03-21T13:07:19.386611+0000","last_clean":"2024-03-21T13:07:19.386611+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:07:19.386611+0000","last_undegraded":"2024-03-21T13:07:19.386611+0000","last_fullsized":"2024-03-21T13:07:19.386611+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:06:41.353996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:41.353996+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:44.457186+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"87'21","reported_seq":282,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389221+0000","last_change":"2024-03-21T13:05:23.917855+0000","last_active":"2024-03-21T13:07:19.389221+0000","last_peered":"2024-03-21T13:07:19.389221+0000","last_clean":"2024-03-21T13:07:19.389221+0000","last_became_active":"2024-03-21T13:02:26.227165+0000","last_became_peered":"2024-03-21T13:02:26.227165+0000","last_unstale":"2024-03-21T13:07:19.389221+0000","last_undegraded":"2024-03-21T13:07:19.389221+0000","last_fullsized":"2024-03-21T13:07:19.389221+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'21","last_scrub_stamp":"2024-03-21T13:05:23.917691+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.917691+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"119'9","reported_seq":426,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390594+0000","last_change":"2024-03-21T13:02:14.375956+0000","last_active":"2024-03-21T13:07:19.390594+0000","last_peered":"2024-03-21T13:07:19.390594+0000","last_clean":"2024-03-21T13:07:19.390594+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:07:19.390594+0000","last_undegraded":"2024-03-21T13:07:19.390594+0000","last_fullsized":"2024-03-21T13:07:19.390594+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:25.700549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:25.700549+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"208'11919","reported_seq":12101,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:20.011720+0000","last_change":"2024-03-21T13:06:30.386663+0000","last_active":"2024-03-21T13:07:20.011720+0000","last_peered":"2024-03-21T13:07:20.011720+0000","last_clean":"2024-03-21T13:07:20.011720+0000","last_became_active":"2024-03-21T13:04:46.260642+0000","last_became_peered":"2024-03-21T13:04:46.260642+0000","last_unstale":"2024-03-21T13:07:20.011720+0000","last_undegraded":"2024-03-21T13:07:20.011720+0000","last_fullsized":"2024-03-21T13:07:20.011720+0000","mapping_epoch":168,"log_start":"193'8059","ondisk_log_start":"193'8059","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'8456","last_scrub_stamp":"2024-03-21T13:06:30.386488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:06:30.386488+0000","objects_scrubbed":8380,"log_size":3860,"log_dups_size":0,"ondisk_log_size":3860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:54.533306+0000","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322240512,"num_objects":4917,"num_object_clones":0,"num_object_copies":14751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11919,"num_write_kb":538752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":236,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390598+0000","last_change":"2024-03-21T13:06:29.491923+0000","last_active":"2024-03-21T13:07:19.390598+0000","last_peered":"2024-03-21T13:07:19.390598+0000","last_clean":"2024-03-21T13:07:19.390598+0000","last_became_active":"2024-03-21T13:06:29.491598+0000","last_became_peered":"2024-03-21T13:06:29.491598+0000","last_unstale":"2024-03-21T13:07:19.390598+0000","last_undegraded":"2024-03-21T13:07:19.390598+0000","last_fullsized":"2024-03-21T13:07:19.390598+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.416262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.416262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":318,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392384+0000","last_change":"2024-03-21T13:04:20.538204+0000","last_active":"2024-03-21T13:07:19.392384+0000","last_peered":"2024-03-21T13:07:19.392384+0000","last_clean":"2024-03-21T13:07:19.392384+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:07:19.392384+0000","last_undegraded":"2024-03-21T13:07:19.392384+0000","last_fullsized":"2024-03-21T13:07:19.392384+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:20.538115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:04:20.538115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"71'18","reported_seq":346,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389534+0000","last_change":"2024-03-21T13:05:12.322522+0000","last_active":"2024-03-21T13:07:19.389534+0000","last_peered":"2024-03-21T13:07:19.389534+0000","last_clean":"2024-03-21T13:07:19.389534+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:07:19.389534+0000","last_undegraded":"2024-03-21T13:07:19.389534+0000","last_fullsized":"2024-03-21T13:07:19.389534+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T13:05:12.322284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:12.322284+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":310,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392095+0000","last_change":"2024-03-21T13:05:07.086071+0000","last_active":"2024-03-21T13:07:19.392095+0000","last_peered":"2024-03-21T13:07:19.392095+0000","last_clean":"2024-03-21T13:07:19.392095+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:07:19.392095+0000","last_undegraded":"2024-03-21T13:07:19.392095+0000","last_fullsized":"2024-03-21T13:07:19.392095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.085992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.085992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"208'11984","reported_seq":25901,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.807191+0000","last_change":"2024-03-21T13:07:09.355754+0000","last_active":"2024-03-21T13:07:19.807191+0000","last_peered":"2024-03-21T13:07:19.807191+0000","last_clean":"2024-03-21T13:07:19.807191+0000","last_became_active":"2024-03-21T13:07:09.337338+0000","last_became_peered":"2024-03-21T13:07:09.337338+0000","last_unstale":"2024-03-21T13:07:19.807191+0000","last_undegraded":"2024-03-21T13:07:19.807191+0000","last_fullsized":"2024-03-21T13:07:19.807191+0000","mapping_epoch":204,"log_start":"194'8425","ondisk_log_start":"194'8425","created":168,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3559,"log_dups_size":0,"ondisk_log_size":3559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326369280,"num_objects":4980,"num_object_clones":0,"num_object_copies":14940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11984,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6904,"num_bytes_recovered":452395008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":158,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390396+0000","last_change":"2024-03-21T13:06:29.489514+0000","last_active":"2024-03-21T13:07:19.390396+0000","last_peered":"2024-03-21T13:07:19.390396+0000","last_clean":"2024-03-21T13:07:19.390396+0000","last_became_active":"2024-03-21T13:06:29.488826+0000","last_became_peered":"2024-03-21T13:06:29.488826+0000","last_unstale":"2024-03-21T13:07:19.390396+0000","last_undegraded":"2024-03-21T13:07:19.390396+0000","last_fullsized":"2024-03-21T13:07:19.390396+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.328925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.328925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.e","version":"196'11","reported_seq":305,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386789+0000","last_change":"2024-03-21T13:06:34.322811+0000","last_active":"2024-03-21T13:07:19.386789+0000","last_peered":"2024-03-21T13:07:19.386789+0000","last_clean":"2024-03-21T13:07:19.386789+0000","last_became_active":"2024-03-21T13:06:29.505444+0000","last_became_peered":"2024-03-21T13:06:29.505444+0000","last_unstale":"2024-03-21T13:07:19.386789+0000","last_undegraded":"2024-03-21T13:07:19.386789+0000","last_fullsized":"2024-03-21T13:07:19.386789+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'11","last_scrub_stamp":"2024-03-21T13:06:34.322163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:06:34.322163+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:08:02.271008+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":324,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.389500+0000","last_change":"2024-03-21T13:04:19.520964+0000","last_active":"2024-03-21T13:07:19.389500+0000","last_peered":"2024-03-21T13:07:19.389500+0000","last_clean":"2024-03-21T13:07:19.389500+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:07:19.389500+0000","last_undegraded":"2024-03-21T13:07:19.389500+0000","last_fullsized":"2024-03-21T13:07:19.389500+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:19.520824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:04:19.520824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":254,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392441+0000","last_change":"2024-03-21T13:02:14.377274+0000","last_active":"2024-03-21T13:07:19.392441+0000","last_peered":"2024-03-21T13:07:19.392441+0000","last_clean":"2024-03-21T13:07:19.392441+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:07:19.392441+0000","last_undegraded":"2024-03-21T13:07:19.392441+0000","last_fullsized":"2024-03-21T13:07:19.392441+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:01:46.167736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.167736+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"208'12020","reported_seq":12110,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.878853+0000","last_change":"2024-03-21T13:04:46.261080+0000","last_active":"2024-03-21T13:07:19.878853+0000","last_peered":"2024-03-21T13:07:19.878853+0000","last_clean":"2024-03-21T13:07:19.878853+0000","last_became_active":"2024-03-21T13:04:46.260735+0000","last_became_peered":"2024-03-21T13:04:46.260735+0000","last_unstale":"2024-03-21T13:07:19.878853+0000","last_undegraded":"2024-03-21T13:07:19.878853+0000","last_fullsized":"2024-03-21T13:07:19.878853+0000","mapping_epoch":168,"log_start":"197'8714","ondisk_log_start":"197'8714","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3306,"log_dups_size":0,"ondisk_log_size":3306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323878912,"num_objects":4942,"num_object_clones":0,"num_object_copies":14826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12020,"num_write_kb":542784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":316,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390871+0000","last_change":"2024-03-21T13:05:39.518051+0000","last_active":"2024-03-21T13:07:19.390871+0000","last_peered":"2024-03-21T13:07:19.390871+0000","last_clean":"2024-03-21T13:07:19.390871+0000","last_became_active":"2024-03-21T13:02:26.243985+0000","last_became_peered":"2024-03-21T13:02:26.243985+0000","last_unstale":"2024-03-21T13:07:19.390871+0000","last_undegraded":"2024-03-21T13:07:19.390871+0000","last_fullsized":"2024-03-21T13:07:19.390871+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:39.517591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.517591+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":204,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.393240+0000","last_change":"2024-03-21T13:06:29.501672+0000","last_active":"2024-03-21T13:07:19.393240+0000","last_peered":"2024-03-21T13:07:19.393240+0000","last_clean":"2024-03-21T13:07:19.393240+0000","last_became_active":"2024-03-21T13:06:29.501230+0000","last_became_peered":"2024-03-21T13:06:29.501230+0000","last_unstale":"2024-03-21T13:07:19.393240+0000","last_undegraded":"2024-03-21T13:07:19.393240+0000","last_fullsized":"2024-03-21T13:07:19.393240+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:38.439435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.439435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":332,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392164+0000","last_change":"2024-03-21T13:06:29.499616+0000","last_active":"2024-03-21T13:07:19.392164+0000","last_peered":"2024-03-21T13:07:19.392164+0000","last_clean":"2024-03-21T13:07:19.392164+0000","last_became_active":"2024-03-21T13:06:29.499291+0000","last_became_peered":"2024-03-21T13:06:29.499291+0000","last_unstale":"2024-03-21T13:07:19.392164+0000","last_undegraded":"2024-03-21T13:07:19.392164+0000","last_fullsized":"2024-03-21T13:07:19.392164+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:14.020911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:14.020911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'11","reported_seq":66,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390563+0000","last_change":"2024-03-21T13:06:45.412306+0000","last_active":"2024-03-21T13:07:19.390563+0000","last_peered":"2024-03-21T13:07:19.390563+0000","last_clean":"2024-03-21T13:07:19.390563+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:07:19.390563+0000","last_undegraded":"2024-03-21T13:07:19.390563+0000","last_fullsized":"2024-03-21T13:07:19.390563+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:26.195507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.195507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"208'12184","reported_seq":13501,"reported_epoch":208,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:07:20.700257+0000","last_change":"2024-03-21T13:07:17.912438+0000","last_active":"2024-03-21T13:07:20.700257+0000","last_peered":"2024-03-21T13:07:20.700257+0000","last_clean":"2024-03-21T13:06:28.456323+0000","last_became_active":"2024-03-21T13:06:30.609894+0000","last_became_peered":"2024-03-21T13:06:30.609894+0000","last_unstale":"2024-03-21T13:07:20.700257+0000","last_undegraded":"2024-03-21T13:07:20.700257+0000","last_fullsized":"2024-03-21T13:07:20.700257+0000","mapping_epoch":196,"log_start":"198'9065","ondisk_log_start":"198'9065","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3119,"log_dups_size":0,"ondisk_log_size":3119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323354624,"num_objects":4934,"num_object_clones":0,"num_object_copies":14802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4341,"num_objects_unfound":0,"num_objects_dirty":4934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":611,"num_bytes_recovered":40042496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,2],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,2,3","objects":4934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":36,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.392063+0000","last_change":"2024-03-21T13:06:29.508869+0000","last_active":"2024-03-21T13:07:19.392063+0000","last_peered":"2024-03-21T13:07:19.392063+0000","last_clean":"2024-03-21T13:07:19.392063+0000","last_became_active":"2024-03-21T13:06:29.508508+0000","last_became_peered":"2024-03-21T13:06:29.508508+0000","last_unstale":"2024-03-21T13:07:19.392063+0000","last_undegraded":"2024-03-21T13:07:19.392063+0000","last_fullsized":"2024-03-21T13:07:19.392063+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:36.362197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:05:36.362197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":210,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.393206+0000","last_change":"2024-03-21T13:05:33.327069+0000","last_active":"2024-03-21T13:07:19.393206+0000","last_peered":"2024-03-21T13:07:19.393206+0000","last_clean":"2024-03-21T13:07:19.393206+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:07:19.393206+0000","last_undegraded":"2024-03-21T13:07:19.393206+0000","last_fullsized":"2024-03-21T13:07:19.393206+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:33.326990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:33.326990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":340,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390121+0000","last_change":"2024-03-21T13:06:30.274121+0000","last_active":"2024-03-21T13:07:19.390121+0000","last_peered":"2024-03-21T13:07:19.390121+0000","last_clean":"2024-03-21T13:07:19.390121+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:07:19.390121+0000","last_undegraded":"2024-03-21T13:07:19.390121+0000","last_fullsized":"2024-03-21T13:07:19.390121+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:30.274033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:30.274033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:42.802362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":138,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390608+0000","last_change":"2024-03-21T13:06:29.531422+0000","last_active":"2024-03-21T13:07:19.390608+0000","last_peered":"2024-03-21T13:07:19.390608+0000","last_clean":"2024-03-21T13:07:19.390608+0000","last_became_active":"2024-03-21T13:06:29.489192+0000","last_became_peered":"2024-03-21T13:06:29.489192+0000","last_unstale":"2024-03-21T13:07:19.390608+0000","last_undegraded":"2024-03-21T13:07:19.390608+0000","last_fullsized":"2024-03-21T13:07:19.390608+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-21T13:05:38.215693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.215693+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.0","version":"208'11946","reported_seq":12037,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:20.658977+0000","last_change":"2024-03-21T13:04:46.258753+0000","last_active":"2024-03-21T13:07:20.658977+0000","last_peered":"2024-03-21T13:07:20.658977+0000","last_clean":"2024-03-21T13:07:20.658977+0000","last_became_active":"2024-03-21T13:04:46.258434+0000","last_became_peered":"2024-03-21T13:04:46.258434+0000","last_unstale":"2024-03-21T13:07:20.658977+0000","last_undegraded":"2024-03-21T13:07:20.658977+0000","last_fullsized":"2024-03-21T13:07:20.658977+0000","mapping_epoch":168,"log_start":"198'8821","ondisk_log_start":"198'8821","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3125,"log_dups_size":0,"ondisk_log_size":3125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320077824,"num_objects":4884,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11946,"num_write_kb":538560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":59,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386624+0000","last_change":"2024-03-21T13:06:29.497122+0000","last_active":"2024-03-21T13:07:19.386624+0000","last_peered":"2024-03-21T13:07:19.386624+0000","last_clean":"2024-03-21T13:07:19.386624+0000","last_became_active":"2024-03-21T13:06:29.496779+0000","last_became_peered":"2024-03-21T13:06:29.496779+0000","last_unstale":"2024-03-21T13:07:19.386624+0000","last_undegraded":"2024-03-21T13:07:19.386624+0000","last_fullsized":"2024-03-21T13:07:19.386624+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.291369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.291369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"196'11","reported_seq":257,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.386766+0000","last_change":"2024-03-21T13:06:29.522757+0000","last_active":"2024-03-21T13:07:19.386766+0000","last_peered":"2024-03-21T13:07:19.386766+0000","last_clean":"2024-03-21T13:07:19.386766+0000","last_became_active":"2024-03-21T13:06:29.501395+0000","last_became_peered":"2024-03-21T13:06:29.501395+0000","last_unstale":"2024-03-21T13:07:19.386766+0000","last_undegraded":"2024-03-21T13:07:19.386766+0000","last_fullsized":"2024-03-21T13:07:19.386766+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:05:30.300605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:30.300605+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":317,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390867+0000","last_change":"2024-03-21T13:05:38.492137+0000","last_active":"2024-03-21T13:07:19.390867+0000","last_peered":"2024-03-21T13:07:19.390867+0000","last_clean":"2024-03-21T13:07:19.390867+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:07:19.390867+0000","last_undegraded":"2024-03-21T13:07:19.390867+0000","last_fullsized":"2024-03-21T13:07:19.390867+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:38.492015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.492015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":264,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-21T13:07:19.390988+0000","last_change":"2024-03-21T13:05:22.406498+0000","last_active":"2024-03-21T13:07:19.390988+0000","last_peered":"2024-03-21T13:07:19.390988+0000","last_clean":"2024-03-21T13:07:19.390988+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:07:19.390988+0000","last_undegraded":"2024-03-21T13:07:19.390988+0000","last_fullsized":"2024-03-21T13:07:19.390988+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:22.406367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.406367+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5179965464,"num_objects":79041,"num_object_clones":0,"num_object_copies":237123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12631,"num_objects_unfound":0,"num_objects_dirty":79041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192701,"num_write_kb":8695681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41256,"num_bytes_recovered":2703556632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16678858752,"data_stored":16678846536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53668,"ondisk_log_size":53668,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":512,"num_read_kb":507,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":5213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":682,"num_read_kb":439,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4428,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":15782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":433,"ondisk_log_size":433,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168685,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2455336,"kb_used_data":2121352,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91289816,"statfs":{"total":95995035648,"available":93480771584,"internally_reserved":0,"allocated":2172264448,"data_stored":2171847704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":341834186},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463027,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3218112,"kb_used_data":2862068,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":90527040,"statfs":{"total":95995035648,"available":92699688960,"internally_reserved":0,"allocated":2930757632,"data_stored":2930360640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":364575174},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574987,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738513,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2544152,"kb_used_data":2246260,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":91201000,"statfs":{"total":95995035648,"available":93389824000,"internally_reserved":0,"allocated":2300170240,"data_stored":2299779489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411463,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182530,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4281816,"kb_used_data":3754896,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":89463336,"statfs":{"total":95995035648,"available":91610456064,"internally_reserved":0,"allocated":3845013504,"data_stored":3844594692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":539425223},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,0,0,0,6,10],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019004,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3926128,"kb_used_data":3461800,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":89819024,"statfs":{"total":95995035648,"available":91974680576,"internally_reserved":0,"allocated":3544883200,"data_stored":3544497808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724663,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2279145472,"data_stored":2279145472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1947336704,"data_stored":1947336704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3841392640,"data_stored":3841392640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3522039808,"data_stored":3522035736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2921467904,"data_stored":2921463832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2167476224,"data_stored":2167472152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:07:28.474 INFO:tasks.thrashosds.thrasher:Setting 4.13 to [0, 5, 6] 2024-03-21T13:07:28.474 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.13', '0', '5', '6'] 2024-03-21T13:07:28.474 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pg-upmap 4.13 0 5 6 2024-03-21T13:07:28.483 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:28.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:28.502 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:28.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:28.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:28.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:28.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:28.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:28 smithi163 ceph-mon[29621]: pgmap v381: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 144 active+clean; 4.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 8142/222348 objects misplaced (3.662%); 60 MiB/s, 965 objects/s recovering 2024-03-21T13:07:28.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:28 smithi163 ceph-mon[29621]: osdmap e214: 8 total, 8 up, 5 in 2024-03-21T13:07:28.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:28 smithi163 ceph-mon[29621]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:07:28.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:28 smithi163 ceph-mon[40675]: pgmap v381: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 144 active+clean; 4.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 8142/222348 objects misplaced (3.662%); 60 MiB/s, 965 objects/s recovering 2024-03-21T13:07:28.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:28 smithi163 ceph-mon[40675]: osdmap e214: 8 total, 8 up, 5 in 2024-03-21T13:07:28.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:28 smithi163 ceph-mon[40675]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:07:28.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:28 smithi204 ceph-mon[33153]: pgmap v381: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 144 active+clean; 4.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 8142/222348 objects misplaced (3.662%); 60 MiB/s, 965 objects/s recovering 2024-03-21T13:07:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:28 smithi204 ceph-mon[33153]: osdmap e214: 8 total, 8 up, 5 in 2024-03-21T13:07:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:28 smithi204 ceph-mon[33153]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:07:28.773 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:28.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:28.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:28.875 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:28.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:28.911 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:28.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:29.012 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:29.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:29.047 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:29.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:29.148 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:29.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:29.186 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:29.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:29.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:29.323 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:29.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:29.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:29.425 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:29.460 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:29.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:29.540 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:29.561 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:29.604 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:29.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:29.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:29.705 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:29.740 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:29.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:29.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:29.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:29.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:29.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:29.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:30.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:30.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:30.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:30.116 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:30.154 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:30.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:30.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:30.256 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:30.295 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:30.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:30.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:30.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:30.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:30.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:30 smithi163 ceph-mon[29621]: pgmap v383: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 144 active+clean; 4.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.18k op/s; 8142/222348 objects misplaced (3.662%); 55 MiB/s, 887 objects/s recovering 2024-03-21T13:07:30.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:30 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:07:30.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:30 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:07:30.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:30 smithi163 ceph-mon[29621]: osdmap e215: 8 total, 8 up, 5 in 2024-03-21T13:07:30.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:30 smithi163 ceph-mon[40675]: pgmap v383: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 144 active+clean; 4.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.18k op/s; 8142/222348 objects misplaced (3.662%); 55 MiB/s, 887 objects/s recovering 2024-03-21T13:07:30.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:30 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:07:30.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:30 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:07:30.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:30 smithi163 ceph-mon[40675]: osdmap e215: 8 total, 8 up, 5 in 2024-03-21T13:07:30.537 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:30.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:30.580 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:30.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:30.628 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:30.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:30.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:30 smithi204 ceph-mon[33153]: pgmap v383: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 144 active+clean; 4.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.18k op/s; 8142/222348 objects misplaced (3.662%); 55 MiB/s, 887 objects/s recovering 2024-03-21T13:07:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:30 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:07:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:30 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:07:30.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:30 smithi204 ceph-mon[33153]: osdmap e215: 8 total, 8 up, 5 in 2024-03-21T13:07:30.822 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:30.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:30.860 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:30.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:30.961 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:30.996 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:30.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:31.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:31.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:31.136 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:31.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:31.237 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:31.275 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:31.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:31.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:31.340 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:07:31.377 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:31.413 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:31.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:31.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:31.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:31 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1776168470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:07:31.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:31 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:07:31.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:31 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:07:31.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:31 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1776168470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:07:31.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:31 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:07:31.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:31 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:07:31.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:31.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:31.552 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:31.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:31.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:31.687 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:31 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1776168470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:07:31.687 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:31 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:07:31.687 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:31 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:07:31.688 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:31.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:31.790 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:31.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:31.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:31.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:31.966 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:31.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:31.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:32.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:07:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:07:32.069 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:32.106 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:32.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:32.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:32.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:32.247 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:32.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:32.347 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:32.383 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:32.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:32.389 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:32 smithi163 ceph-mon[29621]: pgmap v385: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 957 B/s rd, 0 B/s wr, 1.31k op/s; 5972/207231 objects misplaced (2.882%); 49 MiB/s, 782 objects/s recovering 2024-03-21T13:07:32.389 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:32 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:07:32.389 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:32 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1776168470' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:07:32.390 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:32 smithi163 ceph-mon[29621]: osdmap e216: 8 total, 8 up, 5 in 2024-03-21T13:07:32.390 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:32 smithi163 ceph-mon[40675]: pgmap v385: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 957 B/s rd, 0 B/s wr, 1.31k op/s; 5972/207231 objects misplaced (2.882%); 49 MiB/s, 782 objects/s recovering 2024-03-21T13:07:32.390 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:32 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:07:32.390 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:32 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1776168470' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:07:32.390 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:32 smithi163 ceph-mon[40675]: osdmap e216: 8 total, 8 up, 5 in 2024-03-21T13:07:32.485 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:32.519 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:32.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:32.621 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:32.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:32.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:32 smithi204 ceph-mon[33153]: pgmap v385: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 957 B/s rd, 0 B/s wr, 1.31k op/s; 5972/207231 objects misplaced (2.882%); 49 MiB/s, 782 objects/s recovering 2024-03-21T13:07:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:32 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:07:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:32 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1776168470' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:07:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:32 smithi204 ceph-mon[33153]: osdmap e216: 8 total, 8 up, 5 in 2024-03-21T13:07:32.755 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:32.795 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:32.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:32.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:32.897 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:32.931 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:32.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:33.033 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:33.068 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:33.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:33.170 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:33.206 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:33.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:33.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:33.308 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:33.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:33.350 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:33.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:33.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:33.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:33.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:33.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:33.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:33 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:33.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:33 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:33.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:33.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:33.625 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:33.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:33.726 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:33.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:33 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:33.765 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:33.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:33.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:33.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:33.905 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:33.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:34.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:34.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:34.041 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:34.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:34.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:34.178 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:34.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:34.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:07:34.238 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:07:34.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:34.281 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:07:34.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:34.313 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:34.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:34.321 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:07:34.360 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:07:34.401 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:07:34.414 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:34.440 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:07:34.460 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:34.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:34.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:34.477 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:07:34.513 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:07:34.517 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:07:34.517 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:07:34.517 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:07:34.517 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:07:34.518 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:07:34.518 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:07:34.518 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:07:34.518 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:07:34.536 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:34 smithi163 ceph-mon[29621]: pgmap v387: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 840 op/s; 5972/207231 objects misplaced (2.882%); 18 MiB/s, 281 objects/s recovering 2024-03-21T13:07:34.536 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:34 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:07:34.536 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:34 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:07:34.536 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:34 smithi163 ceph-mon[29621]: osdmap e217: 8 total, 8 up, 5 in 2024-03-21T13:07:34.536 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:34 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1521390342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:07:34.537 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:34 smithi163 ceph-mon[40675]: pgmap v387: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 840 op/s; 5972/207231 objects misplaced (2.882%); 18 MiB/s, 281 objects/s recovering 2024-03-21T13:07:34.537 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:34 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:07:34.537 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:34 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:07:34.537 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:34 smithi163 ceph-mon[40675]: osdmap e217: 8 total, 8 up, 5 in 2024-03-21T13:07:34.537 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:34 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1521390342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:07:34.561 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:34.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:34.612 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:34.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:34.713 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:34.747 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:34.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:34.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:34 smithi204 ceph-mon[33153]: pgmap v387: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 840 op/s; 5972/207231 objects misplaced (2.882%); 18 MiB/s, 281 objects/s recovering 2024-03-21T13:07:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:34 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:07:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:34 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:07:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:34 smithi204 ceph-mon[33153]: osdmap e217: 8 total, 8 up, 5 in 2024-03-21T13:07:34.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:34 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1521390342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:07:34.849 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:34.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:34.886 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:34.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:34.987 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:35.022 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:35.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:35.123 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:35.159 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:35.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:35.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:35.261 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:35.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:35.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:35.402 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:35.440 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:35.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:35.541 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:35.581 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:35.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:35.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:35.654 INFO:teuthology.orchestra.run.smithi163.stderr:set 4.13 pg_upmap mapping to [0,5,6] 2024-03-21T13:07:35.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:35.717 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:35.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:35.818 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:35.862 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:35.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:35.964 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:35.997 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:35.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:35.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:07:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:35 smithi204 ceph-mon[33153]: pgmap v389: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 840 op/s; 5972/207231 objects misplaced (2.882%); 18 MiB/s, 281 objects/s recovering 2024-03-21T13:07:36.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:35 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3224165925' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.13", "id": [0, 5, 6]}]: dispatch 2024-03-21T13:07:36.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:07:36.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:35 smithi163 ceph-mon[29621]: pgmap v389: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 840 op/s; 5972/207231 objects misplaced (2.882%); 18 MiB/s, 281 objects/s recovering 2024-03-21T13:07:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:35 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3224165925' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.13", "id": [0, 5, 6]}]: dispatch 2024-03-21T13:07:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:07:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:35 smithi163 ceph-mon[40675]: pgmap v389: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 840 op/s; 5972/207231 objects misplaced (2.882%); 18 MiB/s, 281 objects/s recovering 2024-03-21T13:07:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:35 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3224165925' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.13", "id": [0, 5, 6]}]: dispatch 2024-03-21T13:07:36.012 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:07:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:07:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:07:36.099 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:36.139 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:36.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:36.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:36.242 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:36.281 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:36.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:36.382 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:36.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:36.423 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:36.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:36.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:36.559 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:36.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:36.661 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:36.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:36.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:36 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3224165925' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.13", "id": [0, 5, 6]}]': finished 2024-03-21T13:07:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:36 smithi163 ceph-mon[29621]: osdmap e218: 8 total, 8 up, 5 in 2024-03-21T13:07:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:36 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3224165925' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.13", "id": [0, 5, 6]}]': finished 2024-03-21T13:07:36.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:36 smithi163 ceph-mon[40675]: osdmap e218: 8 total, 8 up, 5 in 2024-03-21T13:07:36.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:36.806 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:36.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:36.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:36.881 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:07:36.881 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:07:36.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:36.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:36.967 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:36.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:36 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3224165925' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.13", "id": [0, 5, 6]}]': finished 2024-03-21T13:07:37.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:36 smithi204 ceph-mon[33153]: osdmap e218: 8 total, 8 up, 5 in 2024-03-21T13:07:37.068 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:37.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:37.108 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:37.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:37.209 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:37.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:37.245 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:37.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:37.346 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:37.383 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:37.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:37.484 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:37.525 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:37.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:37.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:37.628 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:37.631 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:07:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:37.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:37.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:37.767 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:37.810 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:37.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:37.911 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:37.947 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:37.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:37.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:38.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:37 smithi204 ceph-mon[33153]: pgmap v391: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.39k op/s; 4754/182202 objects misplaced (2.609%); 5.7 MiB/s, 91 objects/s recovering 2024-03-21T13:07:38.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:37 smithi163 ceph-mon[29621]: pgmap v391: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.39k op/s; 4754/182202 objects misplaced (2.609%); 5.7 MiB/s, 91 objects/s recovering 2024-03-21T13:07:38.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:37 smithi163 ceph-mon[40675]: pgmap v391: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.39k op/s; 4754/182202 objects misplaced (2.609%); 5.7 MiB/s, 91 objects/s recovering 2024-03-21T13:07:38.049 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:38.083 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:38.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:38.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:38.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:38.217 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:38.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:38.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:38.319 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:38.352 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:38.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:38.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:38.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-21T13:07:38.454 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:38.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:38.542 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:38.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:38.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:38.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:38.681 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:38.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:38.782 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:38.819 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:38.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:38.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:38.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:38.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:38.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:38.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:39.058 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:39.109 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:39.148 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:39.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:39.252 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:39.293 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:39.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:39.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:39.428 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:39.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:39.530 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:39.565 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:39.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:39.667 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:39.701 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:39.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:39.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:39.803 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:39.839 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:39.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:39.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:39.975 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:39.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:39.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:40.020 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:40.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:40.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:40.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:40.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:40.156 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:39 smithi163 ceph-mon[29621]: pgmap v392: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 686 B/s rd, 0 B/s wr, 1.12k op/s; 4754/182202 objects misplaced (2.609%); 4.6 MiB/s, 73 objects/s recovering 2024-03-21T13:07:40.156 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:39 smithi163 ceph-mon[40675]: pgmap v392: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 686 B/s rd, 0 B/s wr, 1.12k op/s; 4754/182202 objects misplaced (2.609%); 4.6 MiB/s, 73 objects/s recovering 2024-03-21T13:07:40.212 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:40.247 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:40.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:40.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:39 smithi204 ceph-mon[33153]: pgmap v392: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 686 B/s rd, 0 B/s wr, 1.12k op/s; 4754/182202 objects misplaced (2.609%); 4.6 MiB/s, 73 objects/s recovering 2024-03-21T13:07:40.349 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:40.387 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:40.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:40.489 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:40.528 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:40.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:40.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:40.676 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:40.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:40.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:40.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:40.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:40.913 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:40.950 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:40.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:41.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:41.090 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:41.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:41.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:41.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:41.227 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:41.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:41.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:41.363 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:41.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:41.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:41.465 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:41.498 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:41.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:41.514 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:07:41.514 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:07:41.515 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-03-21T13:07:41.515 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-03-21T13:07:41.515 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:07:41.600 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:41.641 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:41.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:41.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:41.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:41.777 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:41.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:41.878 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:41.941 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:41.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:41 smithi204 ceph-mon[33153]: pgmap v393: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2.09k op/s; 3594/157185 objects misplaced (2.286%); 8.6 MiB/s, 137 objects/s recovering 2024-03-21T13:07:42.003 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:07:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:07:42.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:42.081 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:42.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:42.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:41 smithi163 ceph-mon[29621]: pgmap v393: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2.09k op/s; 3594/157185 objects misplaced (2.286%); 8.6 MiB/s, 137 objects/s recovering 2024-03-21T13:07:42.148 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:41 smithi163 ceph-mon[40675]: pgmap v393: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2.09k op/s; 3594/157185 objects misplaced (2.286%); 8.6 MiB/s, 137 objects/s recovering 2024-03-21T13:07:42.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:42.215 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:42.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:42.318 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:42.360 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:42.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:42.462 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:42.499 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:42.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:42.549 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:07:42.570 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:07:42.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:42.612 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:07:42.633 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:42.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:42.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:42.651 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:07:42.694 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:07:42.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:42.738 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:07:42.776 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:42.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:42.779 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:07:42.827 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:07:42.829 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:07:42.829 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:07:42.829 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:07:42.880 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:42.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:42.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:42.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:43.006 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:43.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:42 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:43.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:42 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1695667404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:07:43.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:42 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:43.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:42 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1695667404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:07:43.043 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:43.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:43.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:43.145 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:43.185 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:43.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:43.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:42 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:43.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:42 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1695667404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:07:43.286 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:43.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:43.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:43.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:43.417 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:43.460 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:43.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:43.561 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:43.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:43.597 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:43.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:43.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:43.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:43.736 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:43.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:43.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:43.871 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:43.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:43.879 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:07:43.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:43.997 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:43.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:44.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:44.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:44.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:44.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:44.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:44.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:44.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:44.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:44.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:43 smithi204 ceph-mon[33153]: pgmap v394: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.77k op/s; 3594/157185 objects misplaced (2.286%); 7.3 MiB/s, 116 objects/s recovering 2024-03-21T13:07:44.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:43 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/247510796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:07:44.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:43 smithi163 ceph-mon[29621]: pgmap v394: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.77k op/s; 3594/157185 objects misplaced (2.286%); 7.3 MiB/s, 116 objects/s recovering 2024-03-21T13:07:44.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:43 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/247510796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:07:44.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:43 smithi163 ceph-mon[40675]: pgmap v394: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.77k op/s; 3594/157185 objects misplaced (2.286%); 7.3 MiB/s, 116 objects/s recovering 2024-03-21T13:07:44.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:43 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/247510796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:07:44.351 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:44.375 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:44.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:44.477 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:44.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:44.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:44.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:44.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:44.626 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:44.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:44.727 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:44.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:44.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:44.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:44.852 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:44.861 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:44 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:07:44.882 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:44.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:44.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:44.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/247510796' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:07:44.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:44 smithi163 ceph-mon[29621]: osdmap e219: 8 total, 8 up, 5 in 2024-03-21T13:07:44.914 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:44 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:07:44.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/247510796' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:07:44.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:44 smithi163 ceph-mon[40675]: osdmap e219: 8 total, 8 up, 5 in 2024-03-21T13:07:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:44.959 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:07:44.959 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:07:44.959 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-03-21T13:07:44.959 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:07:44.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:45.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:45.020 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:45.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:45.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:45.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:45.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:45.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:45.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:44 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:07:45.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/247510796' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:07:45.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:44 smithi204 ceph-mon[33153]: osdmap e219: 8 total, 8 up, 5 in 2024-03-21T13:07:45.260 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:45.297 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:45.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:45.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:45.399 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:45.434 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:45.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:45.535 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:45.581 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:45.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:45.682 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:45.714 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:45.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:45.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:45.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:07:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:07:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:07:45.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:45.851 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:45.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:45.953 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:45.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:45.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:46.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:46.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:46.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:46.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:46.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:45 smithi163 ceph-mon[29621]: 6.1f scrub starts 2024-03-21T13:07:46.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:45 smithi163 ceph-mon[29621]: pgmap v396: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 560 B/s rd, 0 B/s wr, 1.31k op/s; 3594/157185 objects misplaced (2.286%); 7.5 MiB/s, 120 objects/s recovering 2024-03-21T13:07:46.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:45 smithi163 ceph-mon[29621]: 6.1f scrub ok 2024-03-21T13:07:46.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:45 smithi163 ceph-mon[29621]: osdmap e220: 8 total, 8 up, 5 in 2024-03-21T13:07:46.180 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:45 smithi163 ceph-mon[40675]: 6.1f scrub starts 2024-03-21T13:07:46.180 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:45 smithi163 ceph-mon[40675]: pgmap v396: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 560 B/s rd, 0 B/s wr, 1.31k op/s; 3594/157185 objects misplaced (2.286%); 7.5 MiB/s, 120 objects/s recovering 2024-03-21T13:07:46.181 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:45 smithi163 ceph-mon[40675]: 6.1f scrub ok 2024-03-21T13:07:46.181 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:45 smithi163 ceph-mon[40675]: osdmap e220: 8 total, 8 up, 5 in 2024-03-21T13:07:46.226 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:46.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:45 smithi204 ceph-mon[33153]: 6.1f scrub starts 2024-03-21T13:07:46.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:45 smithi204 ceph-mon[33153]: pgmap v396: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 560 B/s rd, 0 B/s wr, 1.31k op/s; 3594/157185 objects misplaced (2.286%); 7.5 MiB/s, 120 objects/s recovering 2024-03-21T13:07:46.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:45 smithi204 ceph-mon[33153]: 6.1f scrub ok 2024-03-21T13:07:46.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:45 smithi204 ceph-mon[33153]: osdmap e220: 8 total, 8 up, 5 in 2024-03-21T13:07:46.260 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:46.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:46.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:46.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:46.395 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:46.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:46.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:46.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:46.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:46.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:46.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:46.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:46.645 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:07:46.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:46.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:46.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:46.681 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-21T13:07:46.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:46.807 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:46.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:46.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:46.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:46.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:46.945 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:46.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:47.046 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:47.080 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:47.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:47.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:47.222 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:47.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:47.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:47.324 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:47.360 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:47.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:47.461 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:47.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:47.503 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:47.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:47.604 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:47.637 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:47.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:47.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:47.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:47.775 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:47.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:47.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:47.915 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:47.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:48.016 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:48.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:48.051 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:48.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:48.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:48.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:48.190 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:48.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:48.225 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:07:48.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:48.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:48.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:48.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:48.374 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:48 smithi163 ceph-mon[29621]: pgmap v398: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.8 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.85k op/s; 2911/137757 objects misplaced (2.113%); 6.4 MiB/s, 102 objects/s recovering 2024-03-21T13:07:48.374 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:48 smithi163 ceph-mon[29621]: 6.1e scrub starts 2024-03-21T13:07:48.374 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:48 smithi163 ceph-mon[29621]: 6.1e scrub ok 2024-03-21T13:07:48.374 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:48 smithi163 ceph-mon[40675]: pgmap v398: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.8 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.85k op/s; 2911/137757 objects misplaced (2.113%); 6.4 MiB/s, 102 objects/s recovering 2024-03-21T13:07:48.374 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:48 smithi163 ceph-mon[40675]: 6.1e scrub starts 2024-03-21T13:07:48.374 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:48 smithi163 ceph-mon[40675]: 6.1e scrub ok 2024-03-21T13:07:48.431 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:48.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:48.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:48.565 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:48.600 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:48.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:48.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:48.702 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:48.734 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:48.737 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:48.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:48.742 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:48.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:48.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:48 smithi204 ceph-mon[33153]: pgmap v398: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.8 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.85k op/s; 2911/137757 objects misplaced (2.113%); 6.4 MiB/s, 102 objects/s recovering 2024-03-21T13:07:48.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:48 smithi204 ceph-mon[33153]: 6.1e scrub starts 2024-03-21T13:07:48.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:48 smithi204 ceph-mon[33153]: 6.1e scrub ok 2024-03-21T13:07:48.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:48.878 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:48.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:48.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:48.945 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:07:48.949 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:07:48.980 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:49.004 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:49.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:49.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:49.129 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:49.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:49.157 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:49.232 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:49.264 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:49.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:49.366 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:49.390 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:49.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:49.491 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:49.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:49.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:49.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:49 smithi163 ceph-mon[29621]: 3.15 scrub starts 2024-03-21T13:07:49.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:49 smithi163 ceph-mon[29621]: 3.15 scrub ok 2024-03-21T13:07:49.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:49 smithi163 ceph-mon[29621]: 4.9 scrub starts 2024-03-21T13:07:49.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:49 smithi163 ceph-mon[29621]: 4.9 scrub ok 2024-03-21T13:07:49.600 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:49 smithi163 ceph-mon[40675]: 3.15 scrub starts 2024-03-21T13:07:49.600 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:49 smithi163 ceph-mon[40675]: 3.15 scrub ok 2024-03-21T13:07:49.600 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:49 smithi163 ceph-mon[40675]: 4.9 scrub starts 2024-03-21T13:07:49.600 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:49 smithi163 ceph-mon[40675]: 4.9 scrub ok 2024-03-21T13:07:49.627 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:49.659 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:49.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:49.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:49.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:49 smithi204 ceph-mon[33153]: 3.15 scrub starts 2024-03-21T13:07:49.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:49 smithi204 ceph-mon[33153]: 3.15 scrub ok 2024-03-21T13:07:49.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:49 smithi204 ceph-mon[33153]: 4.9 scrub starts 2024-03-21T13:07:49.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:49 smithi204 ceph-mon[33153]: 4.9 scrub ok 2024-03-21T13:07:49.761 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:49.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:49.794 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:49.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:49.895 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:49.929 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:49.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:49.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:50.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:50.066 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:50.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:50.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:50.168 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:50.202 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:50.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:50.304 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:50.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:50.330 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:50.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:50.431 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:50.468 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:50.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:50.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:50.597 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:50.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:50.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:50.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:50.735 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:50.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:50 smithi204 ceph-mon[33153]: pgmap v399: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.8 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 809 op/s; 2911/137757 objects misplaced (2.113%); 2.1 MiB/s, 33 objects/s recovering 2024-03-21T13:07:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:50 smithi204 ceph-mon[33153]: from='client.15219 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:07:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:50 smithi204 ceph-mon[33153]: 8.2 scrub starts 2024-03-21T13:07:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:50 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:07:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:50 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:07:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:50 smithi204 ceph-mon[33153]: 3.2 scrub starts 2024-03-21T13:07:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:50 smithi204 ceph-mon[33153]: 3.2 scrub ok 2024-03-21T13:07:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[29621]: pgmap v399: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.8 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 809 op/s; 2911/137757 objects misplaced (2.113%); 2.1 MiB/s, 33 objects/s recovering 2024-03-21T13:07:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[29621]: from='client.15219 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:07:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[29621]: 8.2 scrub starts 2024-03-21T13:07:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:07:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:07:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[29621]: 3.2 scrub starts 2024-03-21T13:07:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[29621]: 3.2 scrub ok 2024-03-21T13:07:50.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[40675]: pgmap v399: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.8 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 809 op/s; 2911/137757 objects misplaced (2.113%); 2.1 MiB/s, 33 objects/s recovering 2024-03-21T13:07:50.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[40675]: from='client.15219 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:07:50.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[40675]: 8.2 scrub starts 2024-03-21T13:07:50.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:07:50.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:07:50.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:50.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[40675]: 3.2 scrub starts 2024-03-21T13:07:50.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:50 smithi163 ceph-mon[40675]: 3.2 scrub ok 2024-03-21T13:07:50.837 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:50.870 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:50.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:50.973 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:51.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:51.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:51.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:51.142 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:51.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:51.244 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:51.280 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:51.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:51.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:51.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:51.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:51.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:51.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:51.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:51.546 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:51.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:51.647 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:51.687 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:51.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:51.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:51 smithi204 ceph-mon[33153]: 8.e scrub starts 2024-03-21T13:07:51.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:51 smithi204 ceph-mon[33153]: osdmap e221: 8 total, 8 up, 5 in 2024-03-21T13:07:51.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:51 smithi204 ceph-mon[33153]: 8.2 scrub ok 2024-03-21T13:07:51.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:51 smithi204 ceph-mon[33153]: 3.16 scrub starts 2024-03-21T13:07:51.709 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:51 smithi204 ceph-mon[33153]: 3.16 scrub ok 2024-03-21T13:07:51.709 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:51 smithi204 ceph-mon[33153]: 8.e scrub ok 2024-03-21T13:07:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:51 smithi163 ceph-mon[29621]: 8.e scrub starts 2024-03-21T13:07:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:51 smithi163 ceph-mon[29621]: osdmap e221: 8 total, 8 up, 5 in 2024-03-21T13:07:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:51 smithi163 ceph-mon[29621]: 8.2 scrub ok 2024-03-21T13:07:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:51 smithi163 ceph-mon[29621]: 3.16 scrub starts 2024-03-21T13:07:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:51 smithi163 ceph-mon[29621]: 3.16 scrub ok 2024-03-21T13:07:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:51 smithi163 ceph-mon[29621]: 8.e scrub ok 2024-03-21T13:07:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:51 smithi163 ceph-mon[40675]: 8.e scrub starts 2024-03-21T13:07:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:51 smithi163 ceph-mon[40675]: osdmap e221: 8 total, 8 up, 5 in 2024-03-21T13:07:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:51 smithi163 ceph-mon[40675]: 8.2 scrub ok 2024-03-21T13:07:51.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:51 smithi163 ceph-mon[40675]: 3.16 scrub starts 2024-03-21T13:07:51.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:51 smithi163 ceph-mon[40675]: 3.16 scrub ok 2024-03-21T13:07:51.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:51 smithi163 ceph-mon[40675]: 8.e scrub ok 2024-03-21T13:07:51.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:51.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:51.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:51.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:51.929 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:51.968 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:51.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:52.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:07:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:07:52.069 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:52.094 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:52.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:52.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:52.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:52.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:52.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:52.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:52.368 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:52.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:52.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:52.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:52.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:52.604 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:52.640 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:52.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:52.741 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:52 smithi204 ceph-mon[33153]: pgmap v401: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 145 active+clean; 2.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2.03k op/s; 2451/115116 objects misplaced (2.129%); 2.4 MiB/s, 38 objects/s recovering 2024-03-21T13:07:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:52 smithi204 ceph-mon[33153]: 4.6 scrub starts 2024-03-21T13:07:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:52 smithi204 ceph-mon[33153]: 4.6 scrub ok 2024-03-21T13:07:52.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:52 smithi163 ceph-mon[29621]: pgmap v401: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 145 active+clean; 2.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2.03k op/s; 2451/115116 objects misplaced (2.129%); 2.4 MiB/s, 38 objects/s recovering 2024-03-21T13:07:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:52 smithi163 ceph-mon[29621]: 4.6 scrub starts 2024-03-21T13:07:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:52 smithi163 ceph-mon[29621]: 4.6 scrub ok 2024-03-21T13:07:52.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:52 smithi163 ceph-mon[40675]: pgmap v401: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 145 active+clean; 2.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2.03k op/s; 2451/115116 objects misplaced (2.129%); 2.4 MiB/s, 38 objects/s recovering 2024-03-21T13:07:52.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:52 smithi163 ceph-mon[40675]: 4.6 scrub starts 2024-03-21T13:07:52.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:52 smithi163 ceph-mon[40675]: 4.6 scrub ok 2024-03-21T13:07:52.765 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:52.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:52.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:52.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:52.904 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:52.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:53.005 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:53.036 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:53.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:53.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:53.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:53.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:53.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:53.265 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:53.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:53.300 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:53.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:53.402 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:53.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:53.433 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:53.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:53.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:53.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:53.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:53.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:53.672 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:53.706 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:53.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:53.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:53 smithi204 ceph-mon[33153]: 8.4 scrub starts 2024-03-21T13:07:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:53 smithi204 ceph-mon[33153]: 6.15 scrub starts 2024-03-21T13:07:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:53 smithi204 ceph-mon[33153]: 6.15 scrub ok 2024-03-21T13:07:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:53 smithi204 ceph-mon[33153]: 5.14 scrub starts 2024-03-21T13:07:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:53 smithi204 ceph-mon[33153]: 5.14 scrub ok 2024-03-21T13:07:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:53 smithi204 ceph-mon[33153]: 8.4 scrub ok 2024-03-21T13:07:53.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[29621]: 8.4 scrub starts 2024-03-21T13:07:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[29621]: 6.15 scrub starts 2024-03-21T13:07:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[29621]: 6.15 scrub ok 2024-03-21T13:07:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[29621]: 5.14 scrub starts 2024-03-21T13:07:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[29621]: 5.14 scrub ok 2024-03-21T13:07:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[29621]: 8.4 scrub ok 2024-03-21T13:07:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[40675]: 8.4 scrub starts 2024-03-21T13:07:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[40675]: 6.15 scrub starts 2024-03-21T13:07:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[40675]: 6.15 scrub ok 2024-03-21T13:07:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[40675]: 5.14 scrub starts 2024-03-21T13:07:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[40675]: 5.14 scrub ok 2024-03-21T13:07:53.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:53 smithi163 ceph-mon[40675]: 8.4 scrub ok 2024-03-21T13:07:53.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:53.843 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:53.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:53.944 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:53.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:53.981 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:53.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:54.082 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:54.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:54.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:54.221 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:54.253 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:54.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:54.355 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:54.390 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:54.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:54.492 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:54.527 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:54.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:54.628 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:54.652 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:54.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:54.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:54.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:54 smithi163 ceph-mon[29621]: pgmap v402: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 145 active+clean; 2.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s; 2451/115116 objects misplaced (2.129%); 2.1 MiB/s, 33 objects/s recovering 2024-03-21T13:07:54.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:54 smithi163 ceph-mon[40675]: pgmap v402: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 145 active+clean; 2.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s; 2451/115116 objects misplaced (2.129%); 2.1 MiB/s, 33 objects/s recovering 2024-03-21T13:07:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:54 smithi204 ceph-mon[33153]: pgmap v402: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 145 active+clean; 2.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s; 2451/115116 objects misplaced (2.129%); 2.1 MiB/s, 33 objects/s recovering 2024-03-21T13:07:54.754 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:54.793 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:54.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:54.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:54.896 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:54.932 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:54.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:55.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:55.069 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:55.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:55.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:55.207 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:55.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:55.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:55.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:55.346 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:55.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:55.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:55.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:55.486 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:55.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:55.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:55 smithi163 ceph-mon[29621]: 5.19 scrub starts 2024-03-21T13:07:55.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:55 smithi163 ceph-mon[29621]: 5.19 scrub ok 2024-03-21T13:07:55.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:55 smithi163 ceph-mon[40675]: 5.19 scrub starts 2024-03-21T13:07:55.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:55 smithi163 ceph-mon[40675]: 5.19 scrub ok 2024-03-21T13:07:55.587 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:55.621 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:55.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:55.723 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:55.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:55 smithi204 ceph-mon[33153]: 5.19 scrub starts 2024-03-21T13:07:55.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:55 smithi204 ceph-mon[33153]: 5.19 scrub ok 2024-03-21T13:07:55.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:55.762 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:55.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:55.863 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:55.898 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:55.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:55.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:56.000 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:56.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:07:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:07:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:07:56.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:56.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:56.134 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:56.170 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:56.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:56.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:56.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:56.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:56.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:56.412 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:56.448 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:56.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:56.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:56.550 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:56.578 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:56.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:56.679 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:56.716 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:56.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:56.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:56 smithi204 ceph-mon[33153]: pgmap v403: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 145 active+clean; 2.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 516 B/s rd, 0 B/s wr, 1.21k op/s; 2451/115116 objects misplaced (2.129%); 1.7 MiB/s, 27 objects/s recovering 2024-03-21T13:07:56.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:56 smithi204 ceph-mon[33153]: 3.9 scrub starts 2024-03-21T13:07:56.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:56 smithi204 ceph-mon[33153]: 3.9 scrub ok 2024-03-21T13:07:56.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:56 smithi163 ceph-mon[29621]: pgmap v403: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 145 active+clean; 2.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 516 B/s rd, 0 B/s wr, 1.21k op/s; 2451/115116 objects misplaced (2.129%); 1.7 MiB/s, 27 objects/s recovering 2024-03-21T13:07:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:56 smithi163 ceph-mon[29621]: 3.9 scrub starts 2024-03-21T13:07:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:56 smithi163 ceph-mon[29621]: 3.9 scrub ok 2024-03-21T13:07:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:56 smithi163 ceph-mon[40675]: pgmap v403: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 145 active+clean; 2.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 516 B/s rd, 0 B/s wr, 1.21k op/s; 2451/115116 objects misplaced (2.129%); 1.7 MiB/s, 27 objects/s recovering 2024-03-21T13:07:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:56 smithi163 ceph-mon[40675]: 3.9 scrub starts 2024-03-21T13:07:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:56 smithi163 ceph-mon[40675]: 3.9 scrub ok 2024-03-21T13:07:56.818 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:56.849 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:56.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:56.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:56.986 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:56.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:56.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:57.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:57.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:57.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:57.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:57.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:07:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:57.262 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:57.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:07:57.363 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:57.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:57.397 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:57.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:57.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:57.535 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:57.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:57.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:57.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:57.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:57.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:57.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:57 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:07:57.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:57 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:07:57.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:57 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:07:57.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:57 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:07:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:57 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:07:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:57 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:07:57.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:57.808 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:57.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:57.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:57.948 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:57.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:57.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:58.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:58.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:58.088 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:58.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:58.189 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:58.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:58.226 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:58.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:58.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:07:58.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:58.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:58.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:07:58.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:58.499 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:58.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:58.601 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:58.640 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:58.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:58.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:58.741 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:58 smithi204 ceph-mon[33153]: pgmap v404: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.9 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.44k op/s; 1755/94545 objects misplaced (1.856%); 1.5 MiB/s, 24 objects/s recovering 2024-03-21T13:07:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:58 smithi204 ceph-mon[33153]: 8.c scrub starts 2024-03-21T13:07:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:58 smithi204 ceph-mon[33153]: 8.c scrub ok 2024-03-21T13:07:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:58 smithi204 ceph-mon[33153]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:07:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:58 smithi163 ceph-mon[29621]: pgmap v404: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.9 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.44k op/s; 1755/94545 objects misplaced (1.856%); 1.5 MiB/s, 24 objects/s recovering 2024-03-21T13:07:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:58 smithi163 ceph-mon[29621]: 8.c scrub starts 2024-03-21T13:07:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:58 smithi163 ceph-mon[29621]: 8.c scrub ok 2024-03-21T13:07:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:58 smithi163 ceph-mon[29621]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:07:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:58 smithi163 ceph-mon[40675]: pgmap v404: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.9 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.44k op/s; 1755/94545 objects misplaced (1.856%); 1.5 MiB/s, 24 objects/s recovering 2024-03-21T13:07:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:58 smithi163 ceph-mon[40675]: 8.c scrub starts 2024-03-21T13:07:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:58 smithi163 ceph-mon[40675]: 8.c scrub ok 2024-03-21T13:07:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:58 smithi163 ceph-mon[40675]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:07:58.778 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:58.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:58.880 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:07:58.916 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:58.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:58.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:07:59.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:59.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:59.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:59.155 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:59.192 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:59.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:59.293 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:07:59.325 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:59.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:07:59.427 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:07:59.461 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:59.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:07:59.549 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:59 smithi163 ceph-mon[29621]: 4.b scrub starts 2024-03-21T13:07:59.549 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:07:59 smithi163 ceph-mon[29621]: 4.b scrub ok 2024-03-21T13:07:59.550 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:59 smithi163 ceph-mon[40675]: 4.b scrub starts 2024-03-21T13:07:59.550 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:07:59 smithi163 ceph-mon[40675]: 4.b scrub ok 2024-03-21T13:07:59.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:07:59.590 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-03-21T13:07:48.788968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2996902336,"num_objects":45919,"num_object_clones":0,"num_object_copies":137757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2911,"num_objects_unfound":0,"num_objects_dirty":45919,"num_whiteouts":0,"num_read":1456,"num_read_kb":1302,"num_write":226636,"num_write_kb":8698142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48416,"num_bytes_recovered":3158777297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54247,"ondisk_log_size":54247,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":10833540,"kb_used_data":8715976,"kb_used_omap":7,"kb_used_meta":2117496,"kb_avail":457892220,"statfs":{"total":479975178240,"available":468881633280,"internally_reserved":0,"allocated":8925159424,"data_stored":8923112522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7967,"internal_metadata":2168316129},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,5,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-424411136,"num_objects":-6476,"num_object_clones":0,"num_object_copies":-19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-683,"num_objects_unfound":0,"num_objects_dirty":-6476,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6476,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":17629184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002414"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":188,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883134+0000","last_change":"2024-03-21T13:06:45.404501+0000","last_active":"2024-03-21T13:07:43.883134+0000","last_peered":"2024-03-21T13:07:43.883134+0000","last_clean":"2024-03-21T13:07:43.883134+0000","last_became_active":"2024-03-21T13:06:45.404217+0000","last_became_peered":"2024-03-21T13:06:45.404217+0000","last_unstale":"2024-03-21T13:07:43.883134+0000","last_undegraded":"2024-03-21T13:07:43.883134+0000","last_fullsized":"2024-03-21T13:07:43.883134+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:20.083111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.083111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":220,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882781+0000","last_change":"2024-03-21T13:06:29.490923+0000","last_active":"2024-03-21T13:07:43.882781+0000","last_peered":"2024-03-21T13:07:43.882781+0000","last_clean":"2024-03-21T13:07:43.882781+0000","last_became_active":"2024-03-21T13:06:29.490568+0000","last_became_peered":"2024-03-21T13:06:29.490568+0000","last_unstale":"2024-03-21T13:07:43.882781+0000","last_undegraded":"2024-03-21T13:07:43.882781+0000","last_fullsized":"2024-03-21T13:07:43.882781+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:40.962617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:03:40.962617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":62,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882503+0000","last_change":"2024-03-21T13:06:29.484964+0000","last_active":"2024-03-21T13:07:43.882503+0000","last_peered":"2024-03-21T13:07:43.882503+0000","last_clean":"2024-03-21T13:07:43.882503+0000","last_became_active":"2024-03-21T13:06:29.484528+0000","last_became_peered":"2024-03-21T13:06:29.484528+0000","last_unstale":"2024-03-21T13:07:43.882503+0000","last_undegraded":"2024-03-21T13:07:43.882503+0000","last_fullsized":"2024-03-21T13:07:43.882503+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:49.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.260693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"89'13","reported_seq":397,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882535+0000","last_change":"2024-03-21T13:05:17.080137+0000","last_active":"2024-03-21T13:07:44.882535+0000","last_peered":"2024-03-21T13:07:44.882535+0000","last_clean":"2024-03-21T13:07:44.882535+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:07:44.882535+0000","last_undegraded":"2024-03-21T13:07:44.882535+0000","last_fullsized":"2024-03-21T13:07:44.882535+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-03-21T13:05:17.079993+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:05:17.079993+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116180+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":329,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.881258+0000","last_change":"2024-03-21T13:05:07.334438+0000","last_active":"2024-03-21T13:07:44.881258+0000","last_peered":"2024-03-21T13:07:44.881258+0000","last_clean":"2024-03-21T13:07:44.881258+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:07:44.881258+0000","last_undegraded":"2024-03-21T13:07:44.881258+0000","last_fullsized":"2024-03-21T13:07:44.881258+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.330296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.330296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":272,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882281+0000","last_change":"2024-03-21T13:06:29.490705+0000","last_active":"2024-03-21T13:07:43.882281+0000","last_peered":"2024-03-21T13:07:43.882281+0000","last_clean":"2024-03-21T13:07:43.882281+0000","last_became_active":"2024-03-21T13:06:29.490004+0000","last_became_peered":"2024-03-21T13:06:29.490004+0000","last_unstale":"2024-03-21T13:07:43.882281+0000","last_undegraded":"2024-03-21T13:07:43.882281+0000","last_fullsized":"2024-03-21T13:07:43.882281+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:50.991735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.991735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":351,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884031+0000","last_change":"2024-03-21T13:06:29.542506+0000","last_active":"2024-03-21T13:07:44.884031+0000","last_peered":"2024-03-21T13:07:44.884031+0000","last_clean":"2024-03-21T13:07:44.884031+0000","last_became_active":"2024-03-21T13:06:29.511102+0000","last_became_peered":"2024-03-21T13:06:29.511102+0000","last_unstale":"2024-03-21T13:07:44.884031+0000","last_undegraded":"2024-03-21T13:07:44.884031+0000","last_fullsized":"2024-03-21T13:07:44.884031+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_clean_scrub_stamp":"2024-03-21T13:01:38.765896+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"71'15","reported_seq":384,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.881311+0000","last_change":"2024-03-21T13:06:29.546224+0000","last_active":"2024-03-21T13:07:44.881311+0000","last_peered":"2024-03-21T13:07:44.881311+0000","last_clean":"2024-03-21T13:07:44.881311+0000","last_became_active":"2024-03-21T13:06:29.509791+0000","last_became_peered":"2024-03-21T13:06:29.509791+0000","last_unstale":"2024-03-21T13:07:44.881311+0000","last_undegraded":"2024-03-21T13:07:44.881311+0000","last_fullsized":"2024-03-21T13:07:44.881311+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:26.476043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.476043+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":232,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.881386+0000","last_change":"2024-03-21T13:06:29.489731+0000","last_active":"2024-03-21T13:07:43.881386+0000","last_peered":"2024-03-21T13:07:43.881386+0000","last_clean":"2024-03-21T13:07:43.881386+0000","last_became_active":"2024-03-21T13:06:29.487372+0000","last_became_peered":"2024-03-21T13:06:29.487372+0000","last_unstale":"2024-03-21T13:07:43.881386+0000","last_undegraded":"2024-03-21T13:07:43.881386+0000","last_fullsized":"2024-03-21T13:07:43.881386+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:32.834646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:32.834646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":230,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883016+0000","last_change":"2024-03-21T13:06:43.363491+0000","last_active":"2024-03-21T13:07:44.883016+0000","last_peered":"2024-03-21T13:07:44.883016+0000","last_clean":"2024-03-21T13:07:44.883016+0000","last_became_active":"2024-03-21T13:02:26.236707+0000","last_became_peered":"2024-03-21T13:02:26.236707+0000","last_unstale":"2024-03-21T13:07:44.883016+0000","last_undegraded":"2024-03-21T13:07:44.883016+0000","last_fullsized":"2024-03-21T13:07:44.883016+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:43.363202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:06:43.363202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:08:05.576865+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":212,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.885653+0000","last_change":"2024-03-21T13:05:45.989199+0000","last_active":"2024-03-21T13:07:43.885653+0000","last_peered":"2024-03-21T13:07:43.885653+0000","last_clean":"2024-03-21T13:07:43.885653+0000","last_became_active":"2024-03-21T13:02:26.245402+0000","last_became_peered":"2024-03-21T13:02:26.245402+0000","last_unstale":"2024-03-21T13:07:43.885653+0000","last_undegraded":"2024-03-21T13:07:43.885653+0000","last_fullsized":"2024-03-21T13:07:43.885653+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:45.989114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:45.989114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269663+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":331,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.881178+0000","last_change":"2024-03-21T13:05:51.580203+0000","last_active":"2024-03-21T13:07:44.881178+0000","last_peered":"2024-03-21T13:07:44.881178+0000","last_clean":"2024-03-21T13:07:44.881178+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:07:44.881178+0000","last_undegraded":"2024-03-21T13:07:44.881178+0000","last_fullsized":"2024-03-21T13:07:44.881178+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-21T13:05:51.580074+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:05:51.580074+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":332,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.880425+0000","last_change":"2024-03-21T13:06:29.505858+0000","last_active":"2024-03-21T13:07:44.880425+0000","last_peered":"2024-03-21T13:07:44.880425+0000","last_clean":"2024-03-21T13:07:44.880425+0000","last_became_active":"2024-03-21T13:06:29.505469+0000","last_became_peered":"2024-03-21T13:06:29.505469+0000","last_unstale":"2024-03-21T13:07:44.880425+0000","last_undegraded":"2024-03-21T13:07:44.880425+0000","last_fullsized":"2024-03-21T13:07:44.880425+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:32.971665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:32.971665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":343,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883390+0000","last_change":"2024-03-21T13:04:30.831274+0000","last_active":"2024-03-21T13:07:43.883390+0000","last_peered":"2024-03-21T13:07:43.883390+0000","last_clean":"2024-03-21T13:07:43.883390+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:07:43.883390+0000","last_undegraded":"2024-03-21T13:07:43.883390+0000","last_fullsized":"2024-03-21T13:07:43.883390+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:30.831143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:04:30.831143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":354,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883173+0000","last_change":"2024-03-21T13:05:41.337232+0000","last_active":"2024-03-21T13:07:44.883173+0000","last_peered":"2024-03-21T13:07:44.883173+0000","last_clean":"2024-03-21T13:07:44.883173+0000","last_became_active":"2024-03-21T13:02:26.230057+0000","last_became_peered":"2024-03-21T13:02:26.230057+0000","last_unstale":"2024-03-21T13:07:44.883173+0000","last_undegraded":"2024-03-21T13:07:44.883173+0000","last_fullsized":"2024-03-21T13:07:44.883173+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.337144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.337144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":345,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.880684+0000","last_change":"2024-03-21T13:06:29.517935+0000","last_active":"2024-03-21T13:07:44.880684+0000","last_peered":"2024-03-21T13:07:44.880684+0000","last_clean":"2024-03-21T13:07:44.880684+0000","last_became_active":"2024-03-21T13:06:29.506772+0000","last_became_peered":"2024-03-21T13:06:29.506772+0000","last_unstale":"2024-03-21T13:07:44.880684+0000","last_undegraded":"2024-03-21T13:07:44.880684+0000","last_fullsized":"2024-03-21T13:07:44.880684+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T13:05:16.384676+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:05:16.384676+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":352,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883892+0000","last_change":"2024-03-21T13:06:30.400604+0000","last_active":"2024-03-21T13:07:44.883892+0000","last_peered":"2024-03-21T13:07:44.883892+0000","last_clean":"2024-03-21T13:07:44.883892+0000","last_became_active":"2024-03-21T13:02:26.226675+0000","last_became_peered":"2024-03-21T13:02:26.226675+0000","last_unstale":"2024-03-21T13:07:44.883892+0000","last_undegraded":"2024-03-21T13:07:44.883892+0000","last_fullsized":"2024-03-21T13:07:44.883892+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:30.400488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:06:30.400488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":240,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883812+0000","last_change":"2024-03-21T13:05:37.359133+0000","last_active":"2024-03-21T13:07:44.883812+0000","last_peered":"2024-03-21T13:07:44.883812+0000","last_clean":"2024-03-21T13:07:44.883812+0000","last_became_active":"2024-03-21T13:02:26.231879+0000","last_became_peered":"2024-03-21T13:02:26.231879+0000","last_unstale":"2024-03-21T13:07:44.883812+0000","last_undegraded":"2024-03-21T13:07:44.883812+0000","last_fullsized":"2024-03-21T13:07:44.883812+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:37.358974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:05:37.358974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116422+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":238,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883296+0000","last_change":"2024-03-21T13:06:28.985227+0000","last_active":"2024-03-21T13:07:43.883296+0000","last_peered":"2024-03-21T13:07:43.883296+0000","last_clean":"2024-03-21T13:07:43.883296+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:07:43.883296+0000","last_undegraded":"2024-03-21T13:07:43.883296+0000","last_fullsized":"2024-03-21T13:07:43.883296+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:28.985145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:28.985145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:57.861880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":120,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882958+0000","last_change":"2024-03-21T13:06:29.540373+0000","last_active":"2024-03-21T13:07:44.882958+0000","last_peered":"2024-03-21T13:07:44.882958+0000","last_clean":"2024-03-21T13:07:44.882958+0000","last_became_active":"2024-03-21T13:06:29.504235+0000","last_became_peered":"2024-03-21T13:06:29.504235+0000","last_unstale":"2024-03-21T13:07:44.882958+0000","last_undegraded":"2024-03-21T13:07:44.882958+0000","last_fullsized":"2024-03-21T13:07:44.882958+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:20.158184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.158184+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":342,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883754+0000","last_change":"2024-03-21T13:07:44.816849+0000","last_active":"2024-03-21T13:07:44.883754+0000","last_peered":"2024-03-21T13:07:44.883754+0000","last_clean":"2024-03-21T13:07:44.883754+0000","last_became_active":"2024-03-21T13:06:29.512081+0000","last_became_peered":"2024-03-21T13:06:29.512081+0000","last_unstale":"2024-03-21T13:07:44.883754+0000","last_undegraded":"2024-03-21T13:07:44.883754+0000","last_fullsized":"2024-03-21T13:07:44.883754+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:44.816722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:44.816722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:06.483247+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":260,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882523+0000","last_change":"2024-03-21T13:06:29.500559+0000","last_active":"2024-03-21T13:07:44.882523+0000","last_peered":"2024-03-21T13:07:44.882523+0000","last_clean":"2024-03-21T13:07:44.882523+0000","last_became_active":"2024-03-21T13:06:29.498872+0000","last_became_peered":"2024-03-21T13:06:29.498872+0000","last_unstale":"2024-03-21T13:07:44.882523+0000","last_undegraded":"2024-03-21T13:07:44.882523+0000","last_fullsized":"2024-03-21T13:07:44.882523+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:29.464856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:04:29.464856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":369,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882824+0000","last_change":"2024-03-21T13:06:29.488224+0000","last_active":"2024-03-21T13:07:44.882824+0000","last_peered":"2024-03-21T13:07:44.882824+0000","last_clean":"2024-03-21T13:07:44.882824+0000","last_became_active":"2024-03-21T13:06:29.487644+0000","last_became_peered":"2024-03-21T13:06:29.487644+0000","last_unstale":"2024-03-21T13:07:44.882824+0000","last_undegraded":"2024-03-21T13:07:44.882824+0000","last_fullsized":"2024-03-21T13:07:44.882824+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.354575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.354575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116450+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'12","reported_seq":388,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883250+0000","last_change":"2024-03-21T13:02:14.362313+0000","last_active":"2024-03-21T13:07:44.883250+0000","last_peered":"2024-03-21T13:07:44.883250+0000","last_clean":"2024-03-21T13:07:44.883250+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:07:44.883250+0000","last_undegraded":"2024-03-21T13:07:44.883250+0000","last_fullsized":"2024-03-21T13:07:44.883250+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:01:39.755345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.755345+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":195,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883662+0000","last_change":"2024-03-21T13:06:29.537271+0000","last_active":"2024-03-21T13:07:44.883662+0000","last_peered":"2024-03-21T13:07:44.883662+0000","last_clean":"2024-03-21T13:07:44.883662+0000","last_became_active":"2024-03-21T13:06:29.504570+0000","last_became_peered":"2024-03-21T13:06:29.504570+0000","last_unstale":"2024-03-21T13:07:44.883662+0000","last_undegraded":"2024-03-21T13:07:44.883662+0000","last_fullsized":"2024-03-21T13:07:44.883662+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:50.295057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.295057+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":461,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882701+0000","last_change":"2024-03-21T13:05:38.022572+0000","last_active":"2024-03-21T13:07:43.882701+0000","last_peered":"2024-03-21T13:07:43.882701+0000","last_clean":"2024-03-21T13:07:43.882701+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:07:43.882701+0000","last_undegraded":"2024-03-21T13:07:43.882701+0000","last_fullsized":"2024-03-21T13:07:43.882701+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:05:38.022439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.022439+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":240,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.880841+0000","last_change":"2024-03-21T13:04:20.361810+0000","last_active":"2024-03-21T13:07:44.880841+0000","last_peered":"2024-03-21T13:07:44.880841+0000","last_clean":"2024-03-21T13:07:44.880841+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:07:44.880841+0000","last_undegraded":"2024-03-21T13:07:44.880841+0000","last_fullsized":"2024-03-21T13:07:44.880841+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:01:37.244557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:37.244557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"71'10","reported_seq":272,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.884282+0000","last_change":"2024-03-21T13:05:21.117502+0000","last_active":"2024-03-21T13:07:43.884282+0000","last_peered":"2024-03-21T13:07:43.884282+0000","last_clean":"2024-03-21T13:07:43.884282+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:07:43.884282+0000","last_undegraded":"2024-03-21T13:07:43.884282+0000","last_fullsized":"2024-03-21T13:07:43.884282+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:21.117370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.117370+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":230,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883231+0000","last_change":"2024-03-21T13:06:29.499863+0000","last_active":"2024-03-21T13:07:44.883231+0000","last_peered":"2024-03-21T13:07:44.883231+0000","last_clean":"2024-03-21T13:07:44.883231+0000","last_became_active":"2024-03-21T13:06:29.499497+0000","last_became_peered":"2024-03-21T13:06:29.499497+0000","last_unstale":"2024-03-21T13:07:44.883231+0000","last_undegraded":"2024-03-21T13:07:44.883231+0000","last_fullsized":"2024-03-21T13:07:44.883231+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:26.498073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:06:26.498073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116555+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":244,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.884239+0000","last_change":"2024-03-21T13:06:29.501937+0000","last_active":"2024-03-21T13:07:43.884239+0000","last_peered":"2024-03-21T13:07:43.884239+0000","last_clean":"2024-03-21T13:07:43.884239+0000","last_became_active":"2024-03-21T13:06:29.501232+0000","last_became_peered":"2024-03-21T13:06:29.501232+0000","last_unstale":"2024-03-21T13:07:43.884239+0000","last_undegraded":"2024-03-21T13:07:43.884239+0000","last_fullsized":"2024-03-21T13:07:43.884239+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.088812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.088812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":353,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.880582+0000","last_change":"2024-03-21T13:06:29.501866+0000","last_active":"2024-03-21T13:07:44.880582+0000","last_peered":"2024-03-21T13:07:44.880582+0000","last_clean":"2024-03-21T13:07:44.880582+0000","last_became_active":"2024-03-21T13:06:29.501471+0000","last_became_peered":"2024-03-21T13:06:29.501471+0000","last_unstale":"2024-03-21T13:07:44.880582+0000","last_undegraded":"2024-03-21T13:07:44.880582+0000","last_fullsized":"2024-03-21T13:07:44.880582+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:09.348646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:09.348646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'11","reported_seq":399,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882795+0000","last_change":"2024-03-21T13:05:49.982963+0000","last_active":"2024-03-21T13:07:43.882795+0000","last_peered":"2024-03-21T13:07:43.882795+0000","last_clean":"2024-03-21T13:07:43.882795+0000","last_became_active":"2024-03-21T13:02:26.254436+0000","last_became_peered":"2024-03-21T13:02:26.254436+0000","last_unstale":"2024-03-21T13:07:43.882795+0000","last_undegraded":"2024-03-21T13:07:43.882795+0000","last_fullsized":"2024-03-21T13:07:43.882795+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:49.982812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.982812+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269859+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":339,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883484+0000","last_change":"2024-03-21T13:06:29.980827+0000","last_active":"2024-03-21T13:07:43.883484+0000","last_peered":"2024-03-21T13:07:43.883484+0000","last_clean":"2024-03-21T13:07:43.883484+0000","last_became_active":"2024-03-21T13:06:29.503679+0000","last_became_peered":"2024-03-21T13:06:29.503679+0000","last_unstale":"2024-03-21T13:07:43.883484+0000","last_undegraded":"2024-03-21T13:07:43.883484+0000","last_fullsized":"2024-03-21T13:07:43.883484+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:29.980750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:06:29.980750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:44.698296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":339,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.885240+0000","last_change":"2024-03-21T13:05:42.990138+0000","last_active":"2024-03-21T13:07:43.885240+0000","last_peered":"2024-03-21T13:07:43.885240+0000","last_clean":"2024-03-21T13:07:43.885240+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:07:43.885240+0000","last_undegraded":"2024-03-21T13:07:43.885240+0000","last_fullsized":"2024-03-21T13:07:43.885240+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:42.990022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.990022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269417+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":216,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884298+0000","last_change":"2024-03-21T13:05:28.499322+0000","last_active":"2024-03-21T13:07:44.884298+0000","last_peered":"2024-03-21T13:07:44.884298+0000","last_clean":"2024-03-21T13:07:44.884298+0000","last_became_active":"2024-03-21T13:02:26.242973+0000","last_became_peered":"2024-03-21T13:02:26.242973+0000","last_unstale":"2024-03-21T13:07:44.884298+0000","last_undegraded":"2024-03-21T13:07:44.884298+0000","last_fullsized":"2024-03-21T13:07:44.884298+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.499185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.499185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":347,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.881154+0000","last_change":"2024-03-21T13:02:14.369668+0000","last_active":"2024-03-21T13:07:44.881154+0000","last_peered":"2024-03-21T13:07:44.881154+0000","last_clean":"2024-03-21T13:07:44.881154+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:07:44.881154+0000","last_undegraded":"2024-03-21T13:07:44.881154+0000","last_fullsized":"2024-03-21T13:07:44.881154+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-21T13:01:30.807979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:30.807979+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":329,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882166+0000","last_change":"2024-03-21T13:05:19.431337+0000","last_active":"2024-03-21T13:07:44.882166+0000","last_peered":"2024-03-21T13:07:44.882166+0000","last_clean":"2024-03-21T13:07:44.882166+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:07:44.882166+0000","last_undegraded":"2024-03-21T13:07:44.882166+0000","last_fullsized":"2024-03-21T13:07:44.882166+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:19.430947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:19.430947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":329,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882130+0000","last_change":"2024-03-21T13:03:43.905627+0000","last_active":"2024-03-21T13:07:43.882130+0000","last_peered":"2024-03-21T13:07:43.882130+0000","last_clean":"2024-03-21T13:07:43.882130+0000","last_became_active":"2024-03-21T12:58:41.164784+0000","last_became_peered":"2024-03-21T12:58:41.164784+0000","last_unstale":"2024-03-21T13:07:43.882130+0000","last_undegraded":"2024-03-21T13:07:43.882130+0000","last_fullsized":"2024-03-21T13:07:43.882130+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:43.905509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:03:43.905509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":334,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.884142+0000","last_change":"2024-03-21T13:06:29.501880+0000","last_active":"2024-03-21T13:07:43.884142+0000","last_peered":"2024-03-21T13:07:43.884142+0000","last_clean":"2024-03-21T13:07:43.884142+0000","last_became_active":"2024-03-21T13:06:29.501190+0000","last_became_peered":"2024-03-21T13:06:29.501190+0000","last_unstale":"2024-03-21T13:07:43.884142+0000","last_undegraded":"2024-03-21T13:07:43.884142+0000","last_fullsized":"2024-03-21T13:07:43.884142+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:02:21.087308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:02:21.087308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"71'11","reported_seq":71,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.881143+0000","last_change":"2024-03-21T13:06:29.532046+0000","last_active":"2024-03-21T13:07:44.881143+0000","last_peered":"2024-03-21T13:07:44.881143+0000","last_clean":"2024-03-21T13:07:44.881143+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:07:44.881143+0000","last_undegraded":"2024-03-21T13:07:44.881143+0000","last_fullsized":"2024-03-21T13:07:44.881143+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:40.262277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:40.262277+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":235,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882044+0000","last_change":"2024-03-21T13:05:16.850442+0000","last_active":"2024-03-21T13:07:43.882044+0000","last_peered":"2024-03-21T13:07:43.882044+0000","last_clean":"2024-03-21T13:07:43.882044+0000","last_became_active":"2024-03-21T13:02:26.252488+0000","last_became_peered":"2024-03-21T13:02:26.252488+0000","last_unstale":"2024-03-21T13:07:43.882044+0000","last_undegraded":"2024-03-21T13:07:43.882044+0000","last_fullsized":"2024-03-21T13:07:43.882044+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:16.850307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:16.850307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269223+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":345,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.881128+0000","last_change":"2024-03-21T13:05:31.496412+0000","last_active":"2024-03-21T13:07:44.881128+0000","last_peered":"2024-03-21T13:07:44.881128+0000","last_clean":"2024-03-21T13:07:44.881128+0000","last_became_active":"2024-03-21T13:02:26.232682+0000","last_became_peered":"2024-03-21T13:02:26.232682+0000","last_unstale":"2024-03-21T13:07:44.881128+0000","last_undegraded":"2024-03-21T13:07:44.881128+0000","last_fullsized":"2024-03-21T13:07:44.881128+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.495639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.495639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":337,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883224+0000","last_change":"2024-03-21T13:01:43.982673+0000","last_active":"2024-03-21T13:07:43.883224+0000","last_peered":"2024-03-21T13:07:43.883224+0000","last_clean":"2024-03-21T13:07:43.883224+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:07:43.883224+0000","last_undegraded":"2024-03-21T13:07:43.883224+0000","last_fullsized":"2024-03-21T13:07:43.883224+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:43.982553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:43.982553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"86'13","reported_seq":319,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.884566+0000","last_change":"2024-03-21T13:06:29.518281+0000","last_active":"2024-03-21T13:07:43.884566+0000","last_peered":"2024-03-21T13:07:43.884566+0000","last_clean":"2024-03-21T13:07:43.884566+0000","last_became_active":"2024-03-21T13:06:29.496686+0000","last_became_peered":"2024-03-21T13:06:29.496686+0000","last_unstale":"2024-03-21T13:07:43.884566+0000","last_undegraded":"2024-03-21T13:07:43.884566+0000","last_fullsized":"2024-03-21T13:07:43.884566+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:05:34.036430+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.036430+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269507+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":172,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882277+0000","last_change":"2024-03-21T13:05:29.031235+0000","last_active":"2024-03-21T13:07:43.882277+0000","last_peered":"2024-03-21T13:07:43.882277+0000","last_clean":"2024-03-21T13:07:43.882277+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:07:43.882277+0000","last_undegraded":"2024-03-21T13:07:43.882277+0000","last_fullsized":"2024-03-21T13:07:43.882277+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.031062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.031062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":208,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882646+0000","last_change":"2024-03-21T13:02:26.242867+0000","last_active":"2024-03-21T13:07:44.882646+0000","last_peered":"2024-03-21T13:07:44.882646+0000","last_clean":"2024-03-21T13:07:44.882646+0000","last_became_active":"2024-03-21T13:02:26.242491+0000","last_became_peered":"2024-03-21T13:02:26.242491+0000","last_unstale":"2024-03-21T13:07:44.882646+0000","last_undegraded":"2024-03-21T13:07:44.882646+0000","last_fullsized":"2024-03-21T13:07:44.882646+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:20.029629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:20.029629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":50,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882492+0000","last_change":"2024-03-21T13:06:45.406205+0000","last_active":"2024-03-21T13:07:43.882492+0000","last_peered":"2024-03-21T13:07:43.882492+0000","last_clean":"2024-03-21T13:07:43.882492+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:07:43.882492+0000","last_undegraded":"2024-03-21T13:07:43.882492+0000","last_fullsized":"2024-03-21T13:07:43.882492+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:32.151217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:04:32.151217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":418,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884028+0000","last_change":"2024-03-21T13:06:29.506628+0000","last_active":"2024-03-21T13:07:44.884028+0000","last_peered":"2024-03-21T13:07:44.884028+0000","last_clean":"2024-03-21T13:07:44.884028+0000","last_became_active":"2024-03-21T13:06:29.487725+0000","last_became_peered":"2024-03-21T13:06:29.487725+0000","last_unstale":"2024-03-21T13:07:44.884028+0000","last_undegraded":"2024-03-21T13:07:44.884028+0000","last_fullsized":"2024-03-21T13:07:44.884028+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:28.260333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.260333+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116365+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":238,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.884051+0000","last_change":"2024-03-21T13:04:31.805808+0000","last_active":"2024-03-21T13:07:43.884051+0000","last_peered":"2024-03-21T13:07:43.884051+0000","last_clean":"2024-03-21T13:07:43.884051+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:07:43.884051+0000","last_undegraded":"2024-03-21T13:07:43.884051+0000","last_fullsized":"2024-03-21T13:07:43.884051+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:31.805690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:31.805690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"125'10","reported_seq":453,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882215+0000","last_change":"2024-03-21T13:02:26.247826+0000","last_active":"2024-03-21T13:07:44.882215+0000","last_peered":"2024-03-21T13:07:44.882215+0000","last_clean":"2024-03-21T13:07:44.882215+0000","last_became_active":"2024-03-21T13:02:26.234494+0000","last_became_peered":"2024-03-21T13:02:26.234494+0000","last_unstale":"2024-03-21T13:07:44.882215+0000","last_undegraded":"2024-03-21T13:07:44.882215+0000","last_fullsized":"2024-03-21T13:07:44.882215+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:23.867564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:23.867564+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":351,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883112+0000","last_change":"2024-03-21T13:06:29.489915+0000","last_active":"2024-03-21T13:07:43.883112+0000","last_peered":"2024-03-21T13:07:43.883112+0000","last_clean":"2024-03-21T13:07:43.883112+0000","last_became_active":"2024-03-21T13:06:29.489392+0000","last_became_peered":"2024-03-21T13:06:29.489392+0000","last_unstale":"2024-03-21T13:07:43.883112+0000","last_undegraded":"2024-03-21T13:07:43.883112+0000","last_fullsized":"2024-03-21T13:07:43.883112+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.042372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.042372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"71'10","reported_seq":388,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883072+0000","last_change":"2024-03-21T13:06:29.530165+0000","last_active":"2024-03-21T13:07:44.883072+0000","last_peered":"2024-03-21T13:07:44.883072+0000","last_clean":"2024-03-21T13:07:44.883072+0000","last_became_active":"2024-03-21T13:06:29.506163+0000","last_became_peered":"2024-03-21T13:06:29.506163+0000","last_unstale":"2024-03-21T13:07:44.883072+0000","last_undegraded":"2024-03-21T13:07:44.883072+0000","last_fullsized":"2024-03-21T13:07:44.883072+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:23.379004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.379004+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":342,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883094+0000","last_change":"2024-03-21T13:06:45.402756+0000","last_active":"2024-03-21T13:07:43.883094+0000","last_peered":"2024-03-21T13:07:43.883094+0000","last_clean":"2024-03-21T13:07:43.883094+0000","last_became_active":"2024-03-21T13:06:45.402482+0000","last_became_peered":"2024-03-21T13:06:45.402482+0000","last_unstale":"2024-03-21T13:07:43.883094+0000","last_undegraded":"2024-03-21T13:07:43.883094+0000","last_fullsized":"2024-03-21T13:07:43.883094+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:21.157295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.157295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":454,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883173+0000","last_change":"2024-03-21T13:06:29.533651+0000","last_active":"2024-03-21T13:07:44.883173+0000","last_peered":"2024-03-21T13:07:44.883173+0000","last_clean":"2024-03-21T13:07:44.883173+0000","last_became_active":"2024-03-21T13:06:29.507429+0000","last_became_peered":"2024-03-21T13:06:29.507429+0000","last_unstale":"2024-03-21T13:07:44.883173+0000","last_undegraded":"2024-03-21T13:07:44.883173+0000","last_fullsized":"2024-03-21T13:07:44.883173+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:18.526428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:18.526428+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":238,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883225+0000","last_change":"2024-03-21T13:04:12.819697+0000","last_active":"2024-03-21T13:07:43.883225+0000","last_peered":"2024-03-21T13:07:43.883225+0000","last_clean":"2024-03-21T13:07:43.883225+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:07:43.883225+0000","last_undegraded":"2024-03-21T13:07:43.883225+0000","last_fullsized":"2024-03-21T13:07:43.883225+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:12.819556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:04:12.819556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"71'9","reported_seq":257,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883649+0000","last_change":"2024-03-21T13:04:13.827838+0000","last_active":"2024-03-21T13:07:43.883649+0000","last_peered":"2024-03-21T13:07:43.883649+0000","last_clean":"2024-03-21T13:07:43.883649+0000","last_became_active":"2024-03-21T13:02:26.239161+0000","last_became_peered":"2024-03-21T13:02:26.239161+0000","last_unstale":"2024-03-21T13:07:43.883649+0000","last_undegraded":"2024-03-21T13:07:43.883649+0000","last_fullsized":"2024-03-21T13:07:43.883649+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:04:13.827615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:13.827615+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":60,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883863+0000","last_change":"2024-03-21T13:06:29.489166+0000","last_active":"2024-03-21T13:07:44.883863+0000","last_peered":"2024-03-21T13:07:44.883863+0000","last_clean":"2024-03-21T13:07:44.883863+0000","last_became_active":"2024-03-21T13:06:29.488783+0000","last_became_peered":"2024-03-21T13:06:29.488783+0000","last_unstale":"2024-03-21T13:07:44.883863+0000","last_undegraded":"2024-03-21T13:07:44.883863+0000","last_fullsized":"2024-03-21T13:07:44.883863+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:42.187461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.187461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":339,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884540+0000","last_change":"2024-03-21T13:05:17.347764+0000","last_active":"2024-03-21T13:07:44.884540+0000","last_peered":"2024-03-21T13:07:44.884540+0000","last_clean":"2024-03-21T13:07:44.884540+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:07:44.884540+0000","last_undegraded":"2024-03-21T13:07:44.884540+0000","last_fullsized":"2024-03-21T13:07:44.884540+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:17.347604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:17.347604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":351,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882634+0000","last_change":"2024-03-21T13:05:23.049951+0000","last_active":"2024-03-21T13:07:43.882634+0000","last_peered":"2024-03-21T13:07:43.882634+0000","last_clean":"2024-03-21T13:07:43.882634+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:07:43.882634+0000","last_undegraded":"2024-03-21T13:07:43.882634+0000","last_fullsized":"2024-03-21T13:07:43.882634+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:05:23.049811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.049811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'9","reported_seq":267,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883739+0000","last_change":"2024-03-21T13:01:48.210983+0000","last_active":"2024-03-21T13:07:44.883739+0000","last_peered":"2024-03-21T13:07:44.883739+0000","last_clean":"2024-03-21T13:07:44.883739+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:07:44.883739+0000","last_undegraded":"2024-03-21T13:07:44.883739+0000","last_fullsized":"2024-03-21T13:07:44.883739+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:01:40.964615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.964615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":325,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883131+0000","last_change":"2024-03-21T13:03:41.952266+0000","last_active":"2024-03-21T13:07:43.883131+0000","last_peered":"2024-03-21T13:07:43.883131+0000","last_clean":"2024-03-21T13:07:43.883131+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:07:43.883131+0000","last_undegraded":"2024-03-21T13:07:43.883131+0000","last_fullsized":"2024-03-21T13:07:43.883131+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:03:41.952189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:03:41.952189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":330,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883898+0000","last_change":"2024-03-21T13:05:25.466812+0000","last_active":"2024-03-21T13:07:44.883898+0000","last_peered":"2024-03-21T13:07:44.883898+0000","last_clean":"2024-03-21T13:07:44.883898+0000","last_became_active":"2024-03-21T13:02:26.242502+0000","last_became_peered":"2024-03-21T13:02:26.242502+0000","last_unstale":"2024-03-21T13:07:44.883898+0000","last_undegraded":"2024-03-21T13:07:44.883898+0000","last_fullsized":"2024-03-21T13:07:44.883898+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:25.466695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:25.466695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":353,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882193+0000","last_change":"2024-03-21T13:05:41.004652+0000","last_active":"2024-03-21T13:07:43.882193+0000","last_peered":"2024-03-21T13:07:43.882193+0000","last_clean":"2024-03-21T13:07:43.882193+0000","last_became_active":"2024-03-21T13:02:26.252406+0000","last_became_peered":"2024-03-21T13:02:26.252406+0000","last_unstale":"2024-03-21T13:07:43.882193+0000","last_undegraded":"2024-03-21T13:07:43.882193+0000","last_fullsized":"2024-03-21T13:07:43.882193+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.004412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.004412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269693+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'6","reported_seq":374,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884444+0000","last_change":"2024-03-21T13:06:29.546967+0000","last_active":"2024-03-21T13:07:44.884444+0000","last_peered":"2024-03-21T13:07:44.884444+0000","last_clean":"2024-03-21T13:07:44.884444+0000","last_became_active":"2024-03-21T13:06:29.513331+0000","last_became_peered":"2024-03-21T13:06:29.513331+0000","last_unstale":"2024-03-21T13:07:44.884444+0000","last_undegraded":"2024-03-21T13:07:44.884444+0000","last_fullsized":"2024-03-21T13:07:44.884444+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T13:04:22.563861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:22.563861+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"219'14149","reported_seq":26981,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882997+0000","last_change":"2024-03-21T13:07:23.446606+0000","last_active":"2024-03-21T13:07:44.882997+0000","last_peered":"2024-03-21T13:07:44.882997+0000","last_clean":"2024-03-21T13:07:44.882997+0000","last_became_active":"2024-03-21T13:07:23.424956+0000","last_became_peered":"2024-03-21T13:07:23.424956+0000","last_unstale":"2024-03-21T13:07:44.882997+0000","last_undegraded":"2024-03-21T13:07:44.882997+0000","last_fullsized":"2024-03-21T13:07:44.882997+0000","mapping_epoch":209,"log_start":"202'10848","ondisk_log_start":"202'10848","created":168,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3301,"log_dups_size":0,"ondisk_log_size":3301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188940288,"num_objects":2883,"num_object_clones":0,"num_object_copies":8649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14149,"num_write_kb":545024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6351,"num_bytes_recovered":416219136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":218,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882055+0000","last_change":"2024-03-21T13:06:29.509970+0000","last_active":"2024-03-21T13:07:43.882055+0000","last_peered":"2024-03-21T13:07:43.882055+0000","last_clean":"2024-03-21T13:07:43.882055+0000","last_became_active":"2024-03-21T13:06:29.509616+0000","last_became_peered":"2024-03-21T13:06:29.509616+0000","last_unstale":"2024-03-21T13:07:43.882055+0000","last_undegraded":"2024-03-21T13:07:43.882055+0000","last_fullsized":"2024-03-21T13:07:43.882055+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.973796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.973796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269897+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":382,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883458+0000","last_change":"2024-03-21T13:06:29.498660+0000","last_active":"2024-03-21T13:07:43.883458+0000","last_peered":"2024-03-21T13:07:43.883458+0000","last_clean":"2024-03-21T13:07:43.883458+0000","last_became_active":"2024-03-21T13:06:29.487439+0000","last_became_peered":"2024-03-21T13:06:29.487439+0000","last_unstale":"2024-03-21T13:07:43.883458+0000","last_undegraded":"2024-03-21T13:07:43.883458+0000","last_fullsized":"2024-03-21T13:07:43.883458+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:05:44.816353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:44.816353+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'15","reported_seq":272,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.881965+0000","last_change":"2024-03-21T13:04:25.787896+0000","last_active":"2024-03-21T13:07:43.881965+0000","last_peered":"2024-03-21T13:07:43.881965+0000","last_clean":"2024-03-21T13:07:43.881965+0000","last_became_active":"2024-03-21T13:02:26.237800+0000","last_became_peered":"2024-03-21T13:02:26.237800+0000","last_unstale":"2024-03-21T13:07:43.881965+0000","last_undegraded":"2024-03-21T13:07:43.881965+0000","last_fullsized":"2024-03-21T13:07:43.881965+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:04:25.787703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:25.787703+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":449,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882800+0000","last_change":"2024-03-21T13:04:16.810359+0000","last_active":"2024-03-21T13:07:43.882800+0000","last_peered":"2024-03-21T13:07:43.882800+0000","last_clean":"2024-03-21T13:07:43.882800+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:07:43.882800+0000","last_undegraded":"2024-03-21T13:07:43.882800+0000","last_fullsized":"2024-03-21T13:07:43.882800+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:04:16.810269+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:04:16.810269+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"219'13914","reported_seq":14027,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.944044+0000","last_change":"2024-03-21T13:04:46.259252+0000","last_active":"2024-03-21T13:07:43.944044+0000","last_peered":"2024-03-21T13:07:43.944044+0000","last_clean":"2024-03-21T13:07:43.944044+0000","last_became_active":"2024-03-21T13:04:46.258930+0000","last_became_peered":"2024-03-21T13:04:46.258930+0000","last_unstale":"2024-03-21T13:07:43.944044+0000","last_undegraded":"2024-03-21T13:07:43.944044+0000","last_fullsized":"2024-03-21T13:07:43.944044+0000","mapping_epoch":168,"log_start":"202'10343","ondisk_log_start":"202'10343","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3571,"log_dups_size":0,"ondisk_log_size":3571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.267400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189530112,"num_objects":2892,"num_object_clones":0,"num_object_copies":8676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13914,"num_write_kb":537792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":336,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.881962+0000","last_change":"2024-03-21T13:05:43.596914+0000","last_active":"2024-03-21T13:07:44.881962+0000","last_peered":"2024-03-21T13:07:44.881962+0000","last_clean":"2024-03-21T13:07:44.881962+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:07:44.881962+0000","last_undegraded":"2024-03-21T13:07:44.881962+0000","last_fullsized":"2024-03-21T13:07:44.881962+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.596691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":352,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.884175+0000","last_change":"2024-03-21T13:05:09.138059+0000","last_active":"2024-03-21T13:07:43.884175+0000","last_peered":"2024-03-21T13:07:43.884175+0000","last_clean":"2024-03-21T13:07:43.884175+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:07:43.884175+0000","last_undegraded":"2024-03-21T13:07:43.884175+0000","last_fullsized":"2024-03-21T13:07:43.884175+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:09.137868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:09.137868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"71'12","reported_seq":272,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.881036+0000","last_change":"2024-03-21T13:02:14.383361+0000","last_active":"2024-03-21T13:07:44.881036+0000","last_peered":"2024-03-21T13:07:44.881036+0000","last_clean":"2024-03-21T13:07:44.881036+0000","last_became_active":"2024-03-21T13:02:14.347102+0000","last_became_peered":"2024-03-21T13:02:14.347102+0000","last_unstale":"2024-03-21T13:07:44.881036+0000","last_undegraded":"2024-03-21T13:07:44.881036+0000","last_fullsized":"2024-03-21T13:07:44.881036+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:00:32.864608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:00:32.864608+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":203,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882967+0000","last_change":"2024-03-21T13:06:29.487148+0000","last_active":"2024-03-21T13:07:44.882967+0000","last_peered":"2024-03-21T13:07:44.882967+0000","last_clean":"2024-03-21T13:07:44.882967+0000","last_became_active":"2024-03-21T13:06:29.486743+0000","last_became_peered":"2024-03-21T13:06:29.486743+0000","last_unstale":"2024-03-21T13:07:44.882967+0000","last_undegraded":"2024-03-21T13:07:44.882967+0000","last_fullsized":"2024-03-21T13:07:44.882967+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:21.170999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.170999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116149+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"219'14136","reported_seq":14249,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.973643+0000","last_change":"2024-03-21T13:04:46.257959+0000","last_active":"2024-03-21T13:07:43.973643+0000","last_peered":"2024-03-21T13:07:43.973643+0000","last_clean":"2024-03-21T13:07:43.973643+0000","last_became_active":"2024-03-21T13:04:46.257488+0000","last_became_peered":"2024-03-21T13:04:46.257488+0000","last_unstale":"2024-03-21T13:07:43.973643+0000","last_undegraded":"2024-03-21T13:07:43.973643+0000","last_fullsized":"2024-03-21T13:07:43.973643+0000","mapping_epoch":168,"log_start":"202'10597","ondisk_log_start":"202'10597","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3539,"log_dups_size":0,"ondisk_log_size":3539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.267361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192806912,"num_objects":2942,"num_object_clones":0,"num_object_copies":8826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14136,"num_write_kb":546496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":340,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.881547+0000","last_change":"2024-03-21T13:06:29.504532+0000","last_active":"2024-03-21T13:07:43.881547+0000","last_peered":"2024-03-21T13:07:43.881547+0000","last_clean":"2024-03-21T13:07:43.881547+0000","last_became_active":"2024-03-21T13:06:29.503980+0000","last_became_peered":"2024-03-21T13:06:29.503980+0000","last_unstale":"2024-03-21T13:07:43.881547+0000","last_undegraded":"2024-03-21T13:07:43.881547+0000","last_fullsized":"2024-03-21T13:07:43.881547+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:34.848616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:34.848616+0000","last_clean_scrub_stamp":"2024-03-21T13:04:34.848616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":195,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882956+0000","last_change":"2024-03-21T13:06:29.503568+0000","last_active":"2024-03-21T13:07:43.882956+0000","last_peered":"2024-03-21T13:07:43.882956+0000","last_clean":"2024-03-21T13:07:43.882956+0000","last_became_active":"2024-03-21T13:06:29.503252+0000","last_became_peered":"2024-03-21T13:06:29.503252+0000","last_unstale":"2024-03-21T13:07:43.882956+0000","last_undegraded":"2024-03-21T13:07:43.882956+0000","last_fullsized":"2024-03-21T13:07:43.882956+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.216547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.216547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":352,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883606+0000","last_change":"2024-03-21T13:05:22.078652+0000","last_active":"2024-03-21T13:07:43.883606+0000","last_peered":"2024-03-21T13:07:43.883606+0000","last_clean":"2024-03-21T13:07:43.883606+0000","last_became_active":"2024-03-21T13:02:26.236780+0000","last_became_peered":"2024-03-21T13:02:26.236780+0000","last_unstale":"2024-03-21T13:07:43.883606+0000","last_undegraded":"2024-03-21T13:07:43.883606+0000","last_fullsized":"2024-03-21T13:07:43.883606+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:22.078546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.078546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"71'19","reported_seq":427,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882267+0000","last_change":"2024-03-21T13:05:29.992376+0000","last_active":"2024-03-21T13:07:43.882267+0000","last_peered":"2024-03-21T13:07:43.882267+0000","last_clean":"2024-03-21T13:07:43.882267+0000","last_became_active":"2024-03-21T13:02:26.240350+0000","last_became_peered":"2024-03-21T13:02:26.240350+0000","last_unstale":"2024-03-21T13:07:43.882267+0000","last_undegraded":"2024-03-21T13:07:43.882267+0000","last_fullsized":"2024-03-21T13:07:43.882267+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:05:29.992207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.992207+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269569+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"220'13906","reported_seq":25899,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:45.718155+0000","last_change":"2024-03-21T13:07:26.326903+0000","last_active":"2024-03-21T13:07:45.718155+0000","last_peered":"2024-03-21T13:07:45.718155+0000","last_clean":"2024-03-21T13:07:45.718155+0000","last_became_active":"2024-03-21T13:07:26.308387+0000","last_became_peered":"2024-03-21T13:07:26.308387+0000","last_unstale":"2024-03-21T13:07:45.718155+0000","last_undegraded":"2024-03-21T13:07:45.718155+0000","last_fullsized":"2024-03-21T13:07:45.718155+0000","mapping_epoch":212,"log_start":"202'10779","ondisk_log_start":"202'10779","created":168,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185597952,"num_objects":2832,"num_object_clones":0,"num_object_copies":8496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13906,"num_write_kb":535616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5932,"num_bytes_recovered":388759552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":244,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882441+0000","last_change":"2024-03-21T13:06:29.501377+0000","last_active":"2024-03-21T13:07:43.882441+0000","last_peered":"2024-03-21T13:07:43.882441+0000","last_clean":"2024-03-21T13:07:43.882441+0000","last_became_active":"2024-03-21T13:06:29.500961+0000","last_became_peered":"2024-03-21T13:06:29.500961+0000","last_unstale":"2024-03-21T13:07:43.882441+0000","last_undegraded":"2024-03-21T13:07:43.882441+0000","last_fullsized":"2024-03-21T13:07:43.882441+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.926828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.926828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":60,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882991+0000","last_change":"2024-03-21T13:06:29.503278+0000","last_active":"2024-03-21T13:07:44.882991+0000","last_peered":"2024-03-21T13:07:44.882991+0000","last_clean":"2024-03-21T13:07:44.882991+0000","last_became_active":"2024-03-21T13:06:29.502745+0000","last_became_peered":"2024-03-21T13:06:29.502745+0000","last_unstale":"2024-03-21T13:07:44.882991+0000","last_undegraded":"2024-03-21T13:07:44.882991+0000","last_fullsized":"2024-03-21T13:07:44.882991+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:37.225558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:37.225558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":206,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882857+0000","last_change":"2024-03-21T13:02:26.244355+0000","last_active":"2024-03-21T13:07:44.882857+0000","last_peered":"2024-03-21T13:07:44.882857+0000","last_clean":"2024-03-21T13:07:44.882857+0000","last_became_active":"2024-03-21T13:02:26.243974+0000","last_became_peered":"2024-03-21T13:02:26.243974+0000","last_unstale":"2024-03-21T13:07:44.882857+0000","last_undegraded":"2024-03-21T13:07:44.882857+0000","last_fullsized":"2024-03-21T13:07:44.882857+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:33.998382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:33.998382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"86'13","reported_seq":304,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883578+0000","last_change":"2024-03-21T13:05:08.081782+0000","last_active":"2024-03-21T13:07:44.883578+0000","last_peered":"2024-03-21T13:07:44.883578+0000","last_clean":"2024-03-21T13:07:44.883578+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:07:44.883578+0000","last_undegraded":"2024-03-21T13:07:44.883578+0000","last_fullsized":"2024-03-21T13:07:44.883578+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:05:08.081600+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:05:08.081600+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116120+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.8","version":"220'14215","reported_seq":14422,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:45.706068+0000","last_change":"2024-03-21T13:06:29.535965+0000","last_active":"2024-03-21T13:07:45.706068+0000","last_peered":"2024-03-21T13:07:45.706068+0000","last_clean":"2024-03-21T13:07:45.706068+0000","last_became_active":"2024-03-21T13:04:46.257160+0000","last_became_peered":"2024-03-21T13:04:46.257160+0000","last_unstale":"2024-03-21T13:07:45.706068+0000","last_undegraded":"2024-03-21T13:07:45.706068+0000","last_fullsized":"2024-03-21T13:07:45.706068+0000","mapping_epoch":168,"log_start":"202'11123","ondisk_log_start":"202'11123","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'8553","last_scrub_stamp":"2024-03-21T13:06:29.535762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:06:29.535762+0000","objects_scrubbed":8459,"log_size":3092,"log_dups_size":0,"ondisk_log_size":3092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183304192,"num_objects":2797,"num_object_clones":0,"num_object_copies":8391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14215,"num_write_kb":544384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":322,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883985+0000","last_change":"2024-03-21T13:01:46.738822+0000","last_active":"2024-03-21T13:07:44.883985+0000","last_peered":"2024-03-21T13:07:44.883985+0000","last_clean":"2024-03-21T13:07:44.883985+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:07:44.883985+0000","last_undegraded":"2024-03-21T13:07:44.883985+0000","last_fullsized":"2024-03-21T13:07:44.883985+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.738659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":243,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882166+0000","last_change":"2024-03-21T13:05:43.961181+0000","last_active":"2024-03-21T13:07:43.882166+0000","last_peered":"2024-03-21T13:07:43.882166+0000","last_clean":"2024-03-21T13:07:43.882166+0000","last_became_active":"2024-03-21T13:02:26.243692+0000","last_became_peered":"2024-03-21T13:02:26.243692+0000","last_unstale":"2024-03-21T13:07:43.882166+0000","last_undegraded":"2024-03-21T13:07:43.882166+0000","last_fullsized":"2024-03-21T13:07:43.882166+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.961057+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269729+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":50,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.884659+0000","last_change":"2024-03-21T13:06:45.410372+0000","last_active":"2024-03-21T13:07:43.884659+0000","last_peered":"2024-03-21T13:07:43.884659+0000","last_clean":"2024-03-21T13:07:43.884659+0000","last_became_active":"2024-03-21T13:06:45.410017+0000","last_became_peered":"2024-03-21T13:06:45.410017+0000","last_unstale":"2024-03-21T13:07:43.884659+0000","last_undegraded":"2024-03-21T13:07:43.884659+0000","last_fullsized":"2024-03-21T13:07:43.884659+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:48.265982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:48.265982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"196'30","reported_seq":167,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883277+0000","last_change":"2024-03-21T13:06:29.547664+0000","last_active":"2024-03-21T13:07:44.883277+0000","last_peered":"2024-03-21T13:07:44.883277+0000","last_clean":"2024-03-21T13:07:44.883277+0000","last_became_active":"2024-03-21T13:06:29.508620+0000","last_became_peered":"2024-03-21T13:06:29.508620+0000","last_unstale":"2024-03-21T13:07:44.883277+0000","last_undegraded":"2024-03-21T13:07:44.883277+0000","last_fullsized":"2024-03-21T13:07:44.883277+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'29","last_scrub_stamp":"2024-03-21T13:05:22.201879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.201879+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":54,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.885460+0000","last_change":"2024-03-21T13:06:45.407111+0000","last_active":"2024-03-21T13:07:43.885460+0000","last_peered":"2024-03-21T13:07:43.885460+0000","last_clean":"2024-03-21T13:07:43.885460+0000","last_became_active":"2024-03-21T13:06:45.406734+0000","last_became_peered":"2024-03-21T13:06:45.406734+0000","last_unstale":"2024-03-21T13:07:43.885460+0000","last_undegraded":"2024-03-21T13:07:43.885460+0000","last_fullsized":"2024-03-21T13:07:43.885460+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.312583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.312583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.b","version":"220'14112","reported_seq":27327,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:45.712079+0000","last_change":"2024-03-21T13:07:19.414805+0000","last_active":"2024-03-21T13:07:45.712079+0000","last_peered":"2024-03-21T13:07:45.712079+0000","last_clean":"2024-03-21T13:07:45.712079+0000","last_became_active":"2024-03-21T13:07:19.393088+0000","last_became_peered":"2024-03-21T13:07:19.393088+0000","last_unstale":"2024-03-21T13:07:45.712079+0000","last_undegraded":"2024-03-21T13:07:45.712079+0000","last_fullsized":"2024-03-21T13:07:45.712079+0000","mapping_epoch":207,"log_start":"202'11060","ondisk_log_start":"202'11060","created":168,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3052,"log_dups_size":0,"ondisk_log_size":3052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183500800,"num_objects":2800,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14112,"num_write_kb":541184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6543,"num_bytes_recovered":428802048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":334,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883712+0000","last_change":"2024-03-21T13:05:08.172825+0000","last_active":"2024-03-21T13:07:43.883712+0000","last_peered":"2024-03-21T13:07:43.883712+0000","last_clean":"2024-03-21T13:07:43.883712+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:07:43.883712+0000","last_undegraded":"2024-03-21T13:07:43.883712+0000","last_fullsized":"2024-03-21T13:07:43.883712+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:08.172705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:08.172705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":60,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882556+0000","last_change":"2024-03-21T13:06:29.510026+0000","last_active":"2024-03-21T13:07:43.882556+0000","last_peered":"2024-03-21T13:07:43.882556+0000","last_clean":"2024-03-21T13:07:43.882556+0000","last_became_active":"2024-03-21T13:06:29.509674+0000","last_became_peered":"2024-03-21T13:06:29.509674+0000","last_unstale":"2024-03-21T13:07:43.882556+0000","last_undegraded":"2024-03-21T13:07:43.882556+0000","last_fullsized":"2024-03-21T13:07:43.882556+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.247107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.247107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.880593+0000","last_change":"2024-03-21T13:06:29.510171+0000","last_active":"2024-03-21T13:07:44.880593+0000","last_peered":"2024-03-21T13:07:44.880593+0000","last_clean":"2024-03-21T13:07:44.880593+0000","last_became_active":"2024-03-21T13:06:29.509315+0000","last_became_peered":"2024-03-21T13:06:29.509315+0000","last_unstale":"2024-03-21T13:07:44.880593+0000","last_undegraded":"2024-03-21T13:07:44.880593+0000","last_fullsized":"2024-03-21T13:07:44.880593+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:46.263644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:05:46.263644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"71'13","reported_seq":272,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883260+0000","last_change":"2024-03-21T13:05:15.119776+0000","last_active":"2024-03-21T13:07:43.883260+0000","last_peered":"2024-03-21T13:07:43.883260+0000","last_clean":"2024-03-21T13:07:43.883260+0000","last_became_active":"2024-03-21T13:02:26.234230+0000","last_became_peered":"2024-03-21T13:02:26.234230+0000","last_unstale":"2024-03-21T13:07:43.883260+0000","last_undegraded":"2024-03-21T13:07:43.883260+0000","last_fullsized":"2024-03-21T13:07:43.883260+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'13","last_scrub_stamp":"2024-03-21T13:05:15.119643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:15.119643+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":195,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.884329+0000","last_change":"2024-03-21T13:06:29.502194+0000","last_active":"2024-03-21T13:07:43.884329+0000","last_peered":"2024-03-21T13:07:43.884329+0000","last_clean":"2024-03-21T13:07:43.884329+0000","last_became_active":"2024-03-21T13:06:29.501625+0000","last_became_peered":"2024-03-21T13:06:29.501625+0000","last_unstale":"2024-03-21T13:07:43.884329+0000","last_undegraded":"2024-03-21T13:07:43.884329+0000","last_fullsized":"2024-03-21T13:07:43.884329+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:23.243468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.243468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.9","version":"219'14419","reported_seq":14532,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.988214+0000","last_change":"2024-03-21T13:04:46.260725+0000","last_active":"2024-03-21T13:07:43.988214+0000","last_peered":"2024-03-21T13:07:43.988214+0000","last_clean":"2024-03-21T13:07:43.988214+0000","last_became_active":"2024-03-21T13:04:46.260364+0000","last_became_peered":"2024-03-21T13:04:46.260364+0000","last_unstale":"2024-03-21T13:07:43.988214+0000","last_undegraded":"2024-03-21T13:07:43.988214+0000","last_fullsized":"2024-03-21T13:07:43.988214+0000","mapping_epoch":168,"log_start":"202'10902","ondisk_log_start":"202'10902","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189988864,"num_objects":2899,"num_object_clones":0,"num_object_copies":8697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14419,"num_write_kb":554176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":214,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884498+0000","last_change":"2024-03-21T13:04:27.549179+0000","last_active":"2024-03-21T13:07:44.884498+0000","last_peered":"2024-03-21T13:07:44.884498+0000","last_clean":"2024-03-21T13:07:44.884498+0000","last_became_active":"2024-03-21T13:02:26.235640+0000","last_became_peered":"2024-03-21T13:02:26.235640+0000","last_unstale":"2024-03-21T13:07:44.884498+0000","last_undegraded":"2024-03-21T13:07:44.884498+0000","last_fullsized":"2024-03-21T13:07:44.884498+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:27.549086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:27.549086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":595,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.881758+0000","last_change":"2024-03-21T13:04:11.818847+0000","last_active":"2024-03-21T13:07:43.881758+0000","last_peered":"2024-03-21T13:07:43.881758+0000","last_clean":"2024-03-21T13:07:43.881758+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:07:43.881758+0000","last_undegraded":"2024-03-21T13:07:43.881758+0000","last_fullsized":"2024-03-21T13:07:43.881758+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:04:11.818749+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:00:16.180126+0000","last_clean_scrub_stamp":"2024-03-21T13:04:11.818749+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":340,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.880776+0000","last_change":"2024-03-21T13:02:26.242969+0000","last_active":"2024-03-21T13:07:44.880776+0000","last_peered":"2024-03-21T13:07:44.880776+0000","last_clean":"2024-03-21T13:07:44.880776+0000","last_became_active":"2024-03-21T13:02:26.242456+0000","last_became_peered":"2024-03-21T13:02:26.242456+0000","last_unstale":"2024-03-21T13:07:44.880776+0000","last_undegraded":"2024-03-21T13:07:44.880776+0000","last_fullsized":"2024-03-21T13:07:44.880776+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:00:38.007486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:00:38.007486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"183'24","reported_seq":484,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.885829+0000","last_change":"2024-03-21T13:05:42.033620+0000","last_active":"2024-03-21T13:07:43.885829+0000","last_peered":"2024-03-21T13:07:43.885829+0000","last_clean":"2024-03-21T13:07:43.885829+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:07:43.885829+0000","last_undegraded":"2024-03-21T13:07:43.885829+0000","last_fullsized":"2024-03-21T13:07:43.885829+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'24","last_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.033266+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269795+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":350,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882677+0000","last_change":"2024-03-21T13:06:29.504771+0000","last_active":"2024-03-21T13:07:43.882677+0000","last_peered":"2024-03-21T13:07:43.882677+0000","last_clean":"2024-03-21T13:07:43.882677+0000","last_became_active":"2024-03-21T13:06:29.504250+0000","last_became_peered":"2024-03-21T13:06:29.504250+0000","last_unstale":"2024-03-21T13:07:43.882677+0000","last_undegraded":"2024-03-21T13:07:43.882677+0000","last_fullsized":"2024-03-21T13:07:43.882677+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:20.061709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.061709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"220'14187","reported_seq":14308,"reported_epoch":220,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-21T13:07:45.708069+0000","last_change":"2024-03-21T13:07:23.625692+0000","last_active":"2024-03-21T13:07:45.708069+0000","last_peered":"2024-03-21T13:07:45.708069+0000","last_clean":"2024-03-21T13:06:28.451225+0000","last_became_active":"2024-03-21T13:06:30.598008+0000","last_became_peered":"2024-03-21T13:06:30.598008+0000","last_unstale":"2024-03-21T13:07:45.708069+0000","last_undegraded":"2024-03-21T13:07:45.708069+0000","last_fullsized":"2024-03-21T13:07:45.708069+0000","mapping_epoch":196,"log_start":"202'11064","ondisk_log_start":"202'11064","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3123,"log_dups_size":0,"ondisk_log_size":3123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190775296,"num_objects":2911,"num_object_clones":0,"num_object_copies":8733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2911,"num_objects_unfound":0,"num_objects_dirty":2911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14187,"num_write_kb":547136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":2911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":216,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882329+0000","last_change":"2024-03-21T13:05:29.457708+0000","last_active":"2024-03-21T13:07:44.882329+0000","last_peered":"2024-03-21T13:07:44.882329+0000","last_clean":"2024-03-21T13:07:44.882329+0000","last_became_active":"2024-03-21T13:02:26.242708+0000","last_became_peered":"2024-03-21T13:02:26.242708+0000","last_unstale":"2024-03-21T13:07:44.882329+0000","last_undegraded":"2024-03-21T13:07:44.882329+0000","last_fullsized":"2024-03-21T13:07:44.882329+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.457597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.457597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":240,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882663+0000","last_change":"2024-03-21T13:05:11.035789+0000","last_active":"2024-03-21T13:07:44.882663+0000","last_peered":"2024-03-21T13:07:44.882663+0000","last_clean":"2024-03-21T13:07:44.882663+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:07:44.882663+0000","last_undegraded":"2024-03-21T13:07:44.882663+0000","last_fullsized":"2024-03-21T13:07:44.882663+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:11.035680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:05:11.035680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116303+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":897,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.881708+0000","last_change":"2024-03-21T13:03:45.889240+0000","last_active":"2024-03-21T13:07:43.881708+0000","last_peered":"2024-03-21T13:07:43.881708+0000","last_clean":"2024-03-21T13:07:43.881708+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:07:43.881708+0000","last_undegraded":"2024-03-21T13:07:43.881708+0000","last_fullsized":"2024-03-21T13:07:43.881708+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:03:45.889083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:03:45.889083+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":537,"num_read_kb":532,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":419,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884749+0000","last_change":"2024-03-21T13:06:33.328433+0000","last_active":"2024-03-21T13:07:44.884749+0000","last_peered":"2024-03-21T13:07:44.884749+0000","last_clean":"2024-03-21T13:07:44.884749+0000","last_became_active":"2024-03-21T13:06:29.526702+0000","last_became_peered":"2024-03-21T13:06:29.526702+0000","last_unstale":"2024-03-21T13:07:44.884749+0000","last_undegraded":"2024-03-21T13:07:44.884749+0000","last_fullsized":"2024-03-21T13:07:44.884749+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:06:33.328260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:06:33.328260+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":211,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883596+0000","last_change":"2024-03-21T13:05:13.085740+0000","last_active":"2024-03-21T13:07:43.883596+0000","last_peered":"2024-03-21T13:07:43.883596+0000","last_clean":"2024-03-21T13:07:43.883596+0000","last_became_active":"2024-03-21T13:02:26.232625+0000","last_became_peered":"2024-03-21T13:02:26.232625+0000","last_unstale":"2024-03-21T13:07:43.883596+0000","last_undegraded":"2024-03-21T13:07:43.883596+0000","last_fullsized":"2024-03-21T13:07:43.883596+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.085625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.085625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"220'13987","reported_seq":26061,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:45.706136+0000","last_change":"2024-03-21T13:07:25.483359+0000","last_active":"2024-03-21T13:07:45.706136+0000","last_peered":"2024-03-21T13:07:45.706136+0000","last_clean":"2024-03-21T13:07:45.706136+0000","last_became_active":"2024-03-21T13:07:25.458300+0000","last_became_peered":"2024-03-21T13:07:25.458300+0000","last_unstale":"2024-03-21T13:07:45.706136+0000","last_undegraded":"2024-03-21T13:07:45.706136+0000","last_fullsized":"2024-03-21T13:07:45.706136+0000","mapping_epoch":211,"log_start":"202'10881","ondisk_log_start":"202'10881","created":168,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3106,"log_dups_size":0,"ondisk_log_size":3106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181993472,"num_objects":2777,"num_object_clones":0,"num_object_copies":8331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13987,"num_write_kb":536448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5973,"num_bytes_recovered":391446528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":230,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.885143+0000","last_change":"2024-03-21T13:06:29.489827+0000","last_active":"2024-03-21T13:07:43.885143+0000","last_peered":"2024-03-21T13:07:43.885143+0000","last_clean":"2024-03-21T13:07:43.885143+0000","last_became_active":"2024-03-21T13:06:29.489464+0000","last_became_peered":"2024-03-21T13:06:29.489464+0000","last_unstale":"2024-03-21T13:07:43.885143+0000","last_undegraded":"2024-03-21T13:07:43.885143+0000","last_fullsized":"2024-03-21T13:07:43.885143+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:35.000684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:35.000684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269318+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":79,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882280+0000","last_change":"2024-03-21T13:06:29.484504+0000","last_active":"2024-03-21T13:07:43.882280+0000","last_peered":"2024-03-21T13:07:43.882280+0000","last_clean":"2024-03-21T13:07:43.882280+0000","last_became_active":"2024-03-21T13:06:29.484174+0000","last_became_peered":"2024-03-21T13:06:29.484174+0000","last_unstale":"2024-03-21T13:07:43.882280+0000","last_undegraded":"2024-03-21T13:07:43.882280+0000","last_fullsized":"2024-03-21T13:07:43.882280+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.282145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.282145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"71'19","reported_seq":309,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883382+0000","last_change":"2024-03-21T13:06:29.508182+0000","last_active":"2024-03-21T13:07:43.883382+0000","last_peered":"2024-03-21T13:07:43.883382+0000","last_clean":"2024-03-21T13:07:43.883382+0000","last_became_active":"2024-03-21T13:06:29.492612+0000","last_became_peered":"2024-03-21T13:06:29.492612+0000","last_unstale":"2024-03-21T13:07:43.883382+0000","last_undegraded":"2024-03-21T13:07:43.883382+0000","last_fullsized":"2024-03-21T13:07:43.883382+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:05:28.062237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.062237+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":352,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882608+0000","last_change":"2024-03-21T13:06:29.497191+0000","last_active":"2024-03-21T13:07:43.882608+0000","last_peered":"2024-03-21T13:07:43.882608+0000","last_clean":"2024-03-21T13:07:43.882608+0000","last_became_active":"2024-03-21T13:06:29.496679+0000","last_became_peered":"2024-03-21T13:06:29.496679+0000","last_unstale":"2024-03-21T13:07:43.882608+0000","last_undegraded":"2024-03-21T13:07:43.882608+0000","last_fullsized":"2024-03-21T13:07:43.882608+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:26.932188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.932188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269475+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"220'14173","reported_seq":24663,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:45.027181+0000","last_change":"2024-03-21T13:07:27.327646+0000","last_active":"2024-03-21T13:07:45.027181+0000","last_peered":"2024-03-21T13:07:45.027181+0000","last_clean":"2024-03-21T13:07:45.027181+0000","last_became_active":"2024-03-21T13:07:27.305219+0000","last_became_peered":"2024-03-21T13:07:27.305219+0000","last_unstale":"2024-03-21T13:07:45.027181+0000","last_undegraded":"2024-03-21T13:07:45.027181+0000","last_fullsized":"2024-03-21T13:07:45.027181+0000","mapping_epoch":213,"log_start":"201'10283","ondisk_log_start":"201'10283","created":168,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":299,"log_size":3890,"log_dups_size":0,"ondisk_log_size":3890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190513152,"num_objects":2907,"num_object_clones":0,"num_object_copies":8721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14173,"num_write_kb":546560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5174,"num_bytes_recovered":339017728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":238,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.880452+0000","last_change":"2024-03-21T13:06:29.502310+0000","last_active":"2024-03-21T13:07:44.880452+0000","last_peered":"2024-03-21T13:07:44.880452+0000","last_clean":"2024-03-21T13:07:44.880452+0000","last_became_active":"2024-03-21T13:06:29.501983+0000","last_became_peered":"2024-03-21T13:06:29.501983+0000","last_unstale":"2024-03-21T13:07:44.880452+0000","last_undegraded":"2024-03-21T13:07:44.880452+0000","last_fullsized":"2024-03-21T13:07:44.880452+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:31.121913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.121913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":220,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882932+0000","last_change":"2024-03-21T13:06:29.511420+0000","last_active":"2024-03-21T13:07:44.882932+0000","last_peered":"2024-03-21T13:07:44.882932+0000","last_clean":"2024-03-21T13:07:44.882932+0000","last_became_active":"2024-03-21T13:06:29.510833+0000","last_became_peered":"2024-03-21T13:06:29.510833+0000","last_unstale":"2024-03-21T13:07:44.882932+0000","last_undegraded":"2024-03-21T13:07:44.882932+0000","last_fullsized":"2024-03-21T13:07:44.882932+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.324701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.324701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"86'15","reported_seq":298,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882876+0000","last_change":"2024-03-21T13:06:29.513960+0000","last_active":"2024-03-21T13:07:44.882876+0000","last_peered":"2024-03-21T13:07:44.882876+0000","last_clean":"2024-03-21T13:07:44.882876+0000","last_became_active":"2024-03-21T13:06:29.492863+0000","last_became_peered":"2024-03-21T13:06:29.492863+0000","last_unstale":"2024-03-21T13:07:44.882876+0000","last_undegraded":"2024-03-21T13:07:44.882876+0000","last_fullsized":"2024-03-21T13:07:44.882876+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-03-21T13:05:42.308556+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.308556+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116496+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"125'12","reported_seq":347,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.884852+0000","last_change":"2024-03-21T13:05:13.903736+0000","last_active":"2024-03-21T13:07:43.884852+0000","last_peered":"2024-03-21T13:07:43.884852+0000","last_clean":"2024-03-21T13:07:43.884852+0000","last_became_active":"2024-03-21T13:02:26.253720+0000","last_became_peered":"2024-03-21T13:02:26.253720+0000","last_unstale":"2024-03-21T13:07:43.884852+0000","last_undegraded":"2024-03-21T13:07:43.884852+0000","last_fullsized":"2024-03-21T13:07:43.884852+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'12","last_scrub_stamp":"2024-03-21T13:05:13.903589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.903589+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269194+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"219'14292","reported_seq":28349,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.946472+0000","last_change":"2024-03-21T13:07:08.359129+0000","last_active":"2024-03-21T13:07:43.946472+0000","last_peered":"2024-03-21T13:07:43.946472+0000","last_clean":"2024-03-21T13:07:43.946472+0000","last_became_active":"2024-03-21T13:07:08.338910+0000","last_became_peered":"2024-03-21T13:07:08.338910+0000","last_unstale":"2024-03-21T13:07:43.946472+0000","last_undegraded":"2024-03-21T13:07:43.946472+0000","last_fullsized":"2024-03-21T13:07:43.946472+0000","mapping_epoch":203,"log_start":"202'10869","ondisk_log_start":"202'10869","created":168,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3423,"log_dups_size":0,"ondisk_log_size":3423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190119960,"num_objects":2902,"num_object_clones":0,"num_object_copies":8706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14292,"num_write_kb":550145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6964,"num_bytes_recovered":456327192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":340,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883473+0000","last_change":"2024-03-21T13:06:29.489617+0000","last_active":"2024-03-21T13:07:43.883473+0000","last_peered":"2024-03-21T13:07:43.883473+0000","last_clean":"2024-03-21T13:07:43.883473+0000","last_became_active":"2024-03-21T13:06:29.489289+0000","last_became_peered":"2024-03-21T13:06:29.489289+0000","last_unstale":"2024-03-21T13:07:43.883473+0000","last_undegraded":"2024-03-21T13:07:43.883473+0000","last_fullsized":"2024-03-21T13:07:43.883473+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.951655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.951655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":376,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884804+0000","last_change":"2024-03-21T13:06:41.354102+0000","last_active":"2024-03-21T13:07:44.884804+0000","last_peered":"2024-03-21T13:07:44.884804+0000","last_clean":"2024-03-21T13:07:44.884804+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:07:44.884804+0000","last_undegraded":"2024-03-21T13:07:44.884804+0000","last_fullsized":"2024-03-21T13:07:44.884804+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:06:41.353996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:41.353996+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"87'21","reported_seq":304,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.884796+0000","last_change":"2024-03-21T13:05:23.917855+0000","last_active":"2024-03-21T13:07:43.884796+0000","last_peered":"2024-03-21T13:07:43.884796+0000","last_clean":"2024-03-21T13:07:43.884796+0000","last_became_active":"2024-03-21T13:02:26.227165+0000","last_became_peered":"2024-03-21T13:02:26.227165+0000","last_unstale":"2024-03-21T13:07:43.884796+0000","last_undegraded":"2024-03-21T13:07:43.884796+0000","last_fullsized":"2024-03-21T13:07:43.884796+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'21","last_scrub_stamp":"2024-03-21T13:05:23.917691+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.917691+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269287+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"119'9","reported_seq":455,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884623+0000","last_change":"2024-03-21T13:02:14.375956+0000","last_active":"2024-03-21T13:07:44.884623+0000","last_peered":"2024-03-21T13:07:44.884623+0000","last_clean":"2024-03-21T13:07:44.884623+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:07:44.884623+0000","last_undegraded":"2024-03-21T13:07:44.884623+0000","last_fullsized":"2024-03-21T13:07:44.884623+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:01:25.700549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:01:25.700549+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"220'13994","reported_seq":14200,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:45.008403+0000","last_change":"2024-03-21T13:06:30.386663+0000","last_active":"2024-03-21T13:07:45.008403+0000","last_peered":"2024-03-21T13:07:45.008403+0000","last_clean":"2024-03-21T13:07:45.008403+0000","last_became_active":"2024-03-21T13:04:46.260642+0000","last_became_peered":"2024-03-21T13:04:46.260642+0000","last_unstale":"2024-03-21T13:07:45.008403+0000","last_undegraded":"2024-03-21T13:07:45.008403+0000","last_fullsized":"2024-03-21T13:07:45.008403+0000","mapping_epoch":168,"log_start":"201'10159","ondisk_log_start":"201'10159","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'8456","last_scrub_stamp":"2024-03-21T13:06:30.386488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:06:30.386488+0000","objects_scrubbed":8380,"log_size":3835,"log_dups_size":0,"ondisk_log_size":3835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:54.533306+0000","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186253312,"num_objects":2842,"num_object_clones":0,"num_object_copies":8526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13994,"num_write_kb":538752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":261,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882221+0000","last_change":"2024-03-21T13:06:29.491923+0000","last_active":"2024-03-21T13:07:44.882221+0000","last_peered":"2024-03-21T13:07:44.882221+0000","last_clean":"2024-03-21T13:07:44.882221+0000","last_became_active":"2024-03-21T13:06:29.491598+0000","last_became_peered":"2024-03-21T13:06:29.491598+0000","last_unstale":"2024-03-21T13:07:44.882221+0000","last_undegraded":"2024-03-21T13:07:44.882221+0000","last_fullsized":"2024-03-21T13:07:44.882221+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.416262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.416262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":342,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884355+0000","last_change":"2024-03-21T13:04:20.538204+0000","last_active":"2024-03-21T13:07:44.884355+0000","last_peered":"2024-03-21T13:07:44.884355+0000","last_clean":"2024-03-21T13:07:44.884355+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:07:44.884355+0000","last_undegraded":"2024-03-21T13:07:44.884355+0000","last_fullsized":"2024-03-21T13:07:44.884355+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:20.538115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:04:20.538115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"71'18","reported_seq":370,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884152+0000","last_change":"2024-03-21T13:05:12.322522+0000","last_active":"2024-03-21T13:07:44.884152+0000","last_peered":"2024-03-21T13:07:44.884152+0000","last_clean":"2024-03-21T13:07:44.884152+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:07:44.884152+0000","last_undegraded":"2024-03-21T13:07:44.884152+0000","last_fullsized":"2024-03-21T13:07:44.884152+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T13:05:12.322284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:12.322284+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":334,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882357+0000","last_change":"2024-03-21T13:05:07.086071+0000","last_active":"2024-03-21T13:07:44.882357+0000","last_peered":"2024-03-21T13:07:44.882357+0000","last_clean":"2024-03-21T13:07:44.882357+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:07:44.882357+0000","last_undegraded":"2024-03-21T13:07:44.882357+0000","last_fullsized":"2024-03-21T13:07:44.882357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.085992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.085992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116209+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"219'14074","reported_seq":28013,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.967460+0000","last_change":"2024-03-21T13:07:09.355754+0000","last_active":"2024-03-21T13:07:43.967460+0000","last_peered":"2024-03-21T13:07:43.967460+0000","last_clean":"2024-03-21T13:07:43.967460+0000","last_became_active":"2024-03-21T13:07:09.337338+0000","last_became_peered":"2024-03-21T13:07:09.337338+0000","last_unstale":"2024-03-21T13:07:43.967460+0000","last_undegraded":"2024-03-21T13:07:43.967460+0000","last_fullsized":"2024-03-21T13:07:43.967460+0000","mapping_epoch":204,"log_start":"202'10525","ondisk_log_start":"202'10525","created":168,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3549,"log_dups_size":0,"ondisk_log_size":3549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.267432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189399040,"num_objects":2890,"num_object_clones":0,"num_object_copies":8670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14074,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6904,"num_bytes_recovered":452395008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":180,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.885060+0000","last_change":"2024-03-21T13:06:29.489514+0000","last_active":"2024-03-21T13:07:43.885060+0000","last_peered":"2024-03-21T13:07:43.885060+0000","last_clean":"2024-03-21T13:07:43.885060+0000","last_became_active":"2024-03-21T13:06:29.488826+0000","last_became_peered":"2024-03-21T13:06:29.488826+0000","last_unstale":"2024-03-21T13:07:43.885060+0000","last_undegraded":"2024-03-21T13:07:43.885060+0000","last_fullsized":"2024-03-21T13:07:43.885060+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.328925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.328925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269825+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.e","version":"196'11","reported_seq":334,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884920+0000","last_change":"2024-03-21T13:06:34.322811+0000","last_active":"2024-03-21T13:07:44.884920+0000","last_peered":"2024-03-21T13:07:44.884920+0000","last_clean":"2024-03-21T13:07:44.884920+0000","last_became_active":"2024-03-21T13:06:29.505444+0000","last_became_peered":"2024-03-21T13:06:29.505444+0000","last_unstale":"2024-03-21T13:07:44.884920+0000","last_undegraded":"2024-03-21T13:07:44.884920+0000","last_fullsized":"2024-03-21T13:07:44.884920+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'11","last_scrub_stamp":"2024-03-21T13:06:34.322163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:06:34.322163+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:08:02.271008+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":348,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884060+0000","last_change":"2024-03-21T13:04:19.520964+0000","last_active":"2024-03-21T13:07:44.884060+0000","last_peered":"2024-03-21T13:07:44.884060+0000","last_clean":"2024-03-21T13:07:44.884060+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:07:44.884060+0000","last_undegraded":"2024-03-21T13:07:44.884060+0000","last_fullsized":"2024-03-21T13:07:44.884060+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:19.520824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:04:19.520824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":278,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884400+0000","last_change":"2024-03-21T13:02:14.377274+0000","last_active":"2024-03-21T13:07:44.884400+0000","last_peered":"2024-03-21T13:07:44.884400+0000","last_clean":"2024-03-21T13:07:44.884400+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:07:44.884400+0000","last_undegraded":"2024-03-21T13:07:44.884400+0000","last_fullsized":"2024-03-21T13:07:44.884400+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:01:46.167736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.167736+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"219'14122","reported_seq":14237,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882383+0000","last_change":"2024-03-21T13:04:46.261080+0000","last_active":"2024-03-21T13:07:44.882383+0000","last_peered":"2024-03-21T13:07:44.882383+0000","last_clean":"2024-03-21T13:07:44.882383+0000","last_became_active":"2024-03-21T13:04:46.260735+0000","last_became_peered":"2024-03-21T13:04:46.260735+0000","last_unstale":"2024-03-21T13:07:44.882383+0000","last_undegraded":"2024-03-21T13:07:44.882383+0000","last_fullsized":"2024-03-21T13:07:44.882383+0000","mapping_epoch":168,"log_start":"202'10814","ondisk_log_start":"202'10814","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3308,"log_dups_size":0,"ondisk_log_size":3308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186122240,"num_objects":2840,"num_object_clones":0,"num_object_copies":8520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14122,"num_write_kb":542784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":341,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882471+0000","last_change":"2024-03-21T13:05:39.518051+0000","last_active":"2024-03-21T13:07:44.882471+0000","last_peered":"2024-03-21T13:07:44.882471+0000","last_clean":"2024-03-21T13:07:44.882471+0000","last_became_active":"2024-03-21T13:02:26.243985+0000","last_became_peered":"2024-03-21T13:02:26.243985+0000","last_unstale":"2024-03-21T13:07:44.882471+0000","last_undegraded":"2024-03-21T13:07:44.882471+0000","last_fullsized":"2024-03-21T13:07:44.882471+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:39.517591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.517591+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":228,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.884192+0000","last_change":"2024-03-21T13:06:29.501672+0000","last_active":"2024-03-21T13:07:44.884192+0000","last_peered":"2024-03-21T13:07:44.884192+0000","last_clean":"2024-03-21T13:07:44.884192+0000","last_became_active":"2024-03-21T13:06:29.501230+0000","last_became_peered":"2024-03-21T13:06:29.501230+0000","last_unstale":"2024-03-21T13:07:44.884192+0000","last_undegraded":"2024-03-21T13:07:44.884192+0000","last_fullsized":"2024-03-21T13:07:44.884192+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:38.439435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.439435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116268+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":356,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882460+0000","last_change":"2024-03-21T13:06:29.499616+0000","last_active":"2024-03-21T13:07:44.882460+0000","last_peered":"2024-03-21T13:07:44.882460+0000","last_clean":"2024-03-21T13:07:44.882460+0000","last_became_active":"2024-03-21T13:06:29.499291+0000","last_became_peered":"2024-03-21T13:06:29.499291+0000","last_unstale":"2024-03-21T13:07:44.882460+0000","last_undegraded":"2024-03-21T13:07:44.882460+0000","last_fullsized":"2024-03-21T13:07:44.882460+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:14.020911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:14.020911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'11","reported_seq":88,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.882431+0000","last_change":"2024-03-21T13:06:45.412306+0000","last_active":"2024-03-21T13:07:43.882431+0000","last_peered":"2024-03-21T13:07:43.882431+0000","last_clean":"2024-03-21T13:07:43.882431+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:07:43.882431+0000","last_undegraded":"2024-03-21T13:07:43.882431+0000","last_fullsized":"2024-03-21T13:07:43.882431+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:26.195507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.195507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"220'14314","reported_seq":23108,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:45.688622+0000","last_change":"2024-03-21T13:07:44.899912+0000","last_active":"2024-03-21T13:07:45.688622+0000","last_peered":"2024-03-21T13:07:45.688622+0000","last_clean":"2024-03-21T13:07:45.688622+0000","last_became_active":"2024-03-21T13:07:44.885457+0000","last_became_peered":"2024-03-21T13:07:44.885457+0000","last_unstale":"2024-03-21T13:07:45.688622+0000","last_undegraded":"2024-03-21T13:07:45.688622+0000","last_fullsized":"2024-03-21T13:07:45.688622+0000","mapping_epoch":219,"log_start":"202'11265","ondisk_log_start":"202'11265","created":168,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3049,"log_dups_size":0,"ondisk_log_size":3049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183762944,"num_objects":2804,"num_object_clones":0,"num_object_copies":8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14314,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4334,"num_bytes_recovered":284033024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":60,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883657+0000","last_change":"2024-03-21T13:06:29.508869+0000","last_active":"2024-03-21T13:07:44.883657+0000","last_peered":"2024-03-21T13:07:44.883657+0000","last_clean":"2024-03-21T13:07:44.883657+0000","last_became_active":"2024-03-21T13:06:29.508508+0000","last_became_peered":"2024-03-21T13:06:29.508508+0000","last_unstale":"2024-03-21T13:07:44.883657+0000","last_undegraded":"2024-03-21T13:07:44.883657+0000","last_fullsized":"2024-03-21T13:07:44.883657+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:36.362197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:05:36.362197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":234,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.883436+0000","last_change":"2024-03-21T13:05:33.327069+0000","last_active":"2024-03-21T13:07:44.883436+0000","last_peered":"2024-03-21T13:07:44.883436+0000","last_clean":"2024-03-21T13:07:44.883436+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:07:44.883436+0000","last_undegraded":"2024-03-21T13:07:44.883436+0000","last_fullsized":"2024-03-21T13:07:44.883436+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:33.326990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:33.326990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.116240+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":362,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.884390+0000","last_change":"2024-03-21T13:06:30.274121+0000","last_active":"2024-03-21T13:07:43.884390+0000","last_peered":"2024-03-21T13:07:43.884390+0000","last_clean":"2024-03-21T13:07:43.884390+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:07:43.884390+0000","last_undegraded":"2024-03-21T13:07:43.884390+0000","last_fullsized":"2024-03-21T13:07:43.884390+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:30.274033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:30.274033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":160,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.885316+0000","last_change":"2024-03-21T13:06:29.531422+0000","last_active":"2024-03-21T13:07:43.885316+0000","last_peered":"2024-03-21T13:07:43.885316+0000","last_clean":"2024-03-21T13:07:43.885316+0000","last_became_active":"2024-03-21T13:06:29.489192+0000","last_became_peered":"2024-03-21T13:06:29.489192+0000","last_unstale":"2024-03-21T13:07:43.885316+0000","last_undegraded":"2024-03-21T13:07:43.885316+0000","last_fullsized":"2024-03-21T13:07:43.885316+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-21T13:05:38.215693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.215693+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:07:48.269350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.0","version":"220'14027","reported_seq":14142,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:45.704281+0000","last_change":"2024-03-21T13:04:46.258753+0000","last_active":"2024-03-21T13:07:45.704281+0000","last_peered":"2024-03-21T13:07:45.704281+0000","last_clean":"2024-03-21T13:07:45.704281+0000","last_became_active":"2024-03-21T13:04:46.258434+0000","last_became_peered":"2024-03-21T13:04:46.258434+0000","last_unstale":"2024-03-21T13:07:45.704281+0000","last_undegraded":"2024-03-21T13:07:45.704281+0000","last_fullsized":"2024-03-21T13:07:45.704281+0000","mapping_epoch":168,"log_start":"202'10921","ondisk_log_start":"202'10921","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3106,"log_dups_size":0,"ondisk_log_size":3106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183697408,"num_objects":2803,"num_object_clones":0,"num_object_copies":8409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14027,"num_write_kb":538560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":81,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883586+0000","last_change":"2024-03-21T13:06:29.497122+0000","last_active":"2024-03-21T13:07:43.883586+0000","last_peered":"2024-03-21T13:07:43.883586+0000","last_clean":"2024-03-21T13:07:43.883586+0000","last_became_active":"2024-03-21T13:06:29.496779+0000","last_became_peered":"2024-03-21T13:06:29.496779+0000","last_unstale":"2024-03-21T13:07:43.883586+0000","last_undegraded":"2024-03-21T13:07:43.883586+0000","last_fullsized":"2024-03-21T13:07:43.883586+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.291369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.291369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"196'11","reported_seq":284,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-21T13:07:43.883430+0000","last_change":"2024-03-21T13:06:29.522757+0000","last_active":"2024-03-21T13:07:43.883430+0000","last_peered":"2024-03-21T13:07:43.883430+0000","last_clean":"2024-03-21T13:07:43.883430+0000","last_became_active":"2024-03-21T13:06:29.501395+0000","last_became_peered":"2024-03-21T13:06:29.501395+0000","last_unstale":"2024-03-21T13:07:43.883430+0000","last_undegraded":"2024-03-21T13:07:43.883430+0000","last_fullsized":"2024-03-21T13:07:43.883430+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:05:30.300605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:30.300605+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":342,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.881041+0000","last_change":"2024-03-21T13:05:38.492137+0000","last_active":"2024-03-21T13:07:44.881041+0000","last_peered":"2024-03-21T13:07:44.881041+0000","last_clean":"2024-03-21T13:07:44.881041+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:07:44.881041+0000","last_undegraded":"2024-03-21T13:07:44.881041+0000","last_fullsized":"2024-03-21T13:07:44.881041+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:38.492015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.492015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":288,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-03-21T13:07:44.882720+0000","last_change":"2024-03-21T13:05:22.406498+0000","last_active":"2024-03-21T13:07:44.882720+0000","last_peered":"2024-03-21T13:07:44.882720+0000","last_clean":"2024-03-21T13:07:44.882720+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:07:44.882720+0000","last_undegraded":"2024-03-21T13:07:44.882720+0000","last_fullsized":"2024-03-21T13:07:44.882720+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:22.406367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.406367+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2996305944,"num_objects":45721,"num_object_clones":0,"num_object_copies":137163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2911,"num_objects_unfound":0,"num_objects_dirty":45721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226021,"num_write_kb":8695681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48175,"num_bytes_recovered":3157000216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9092804608,"data_stored":9092792392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53588,"ondisk_log_size":53588,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":537,"num_read_kb":532,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":5213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":682,"num_read_kb":439,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4428,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":15782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":433,"ondisk_log_size":433,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168690,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1674476,"kb_used_data":1300384,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92070676,"statfs":{"total":95995035648,"available":94280372224,"internally_reserved":0,"allocated":1331593216,"data_stored":1331166741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":383056330},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463032,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2273624,"kb_used_data":1855116,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":91471528,"statfs":{"total":95995035648,"available":93666844672,"internally_reserved":0,"allocated":1899638784,"data_stored":1899232061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":428538310},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574992,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738518,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1603916,"kb_used_data":1297856,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92141236,"statfs":{"total":95995035648,"available":94352625664,"internally_reserved":0,"allocated":1329004544,"data_stored":1328609284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411468,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182535,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2754600,"kb_used_data":2222236,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":90990552,"statfs":{"total":95995035648,"available":93174325248,"internally_reserved":0,"allocated":2275569664,"data_stored":2275146343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":545126855},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019009,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2526924,"kb_used_data":2040384,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91218228,"statfs":{"total":95995035648,"available":93407465472,"internally_reserved":0,"allocated":2089353216,"data_stored":2088958093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724668,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1321009152,"data_stored":1321009152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":248578048,"data_stored":248578048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2206203904,"data_stored":2206203904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2088570880,"data_stored":2088566808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1898254336,"data_stored":1898250264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1330188288,"data_stored":1330184216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:07:59.598 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-21T13:07:59.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:59.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:59.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:07:59.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:07:59.741 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:07:59.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:59.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:07:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:59 smithi204 ceph-mon[33153]: 4.b scrub starts 2024-03-21T13:07:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:07:59 smithi204 ceph-mon[33153]: 4.b scrub ok 2024-03-21T13:07:59.843 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:07:59.874 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:07:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:07:59.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:07:59.976 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:00.011 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:00.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:00.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:00.148 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:00.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:00.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:00.250 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:00.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:00.296 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:00.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:00.299 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:08:00.299 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:08:00.299 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-03-21T13:08:00.299 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-03-21T13:08:00.299 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:08:00.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:00.433 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:00.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:00.535 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:00.570 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:00.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:00.642 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:00 smithi163 ceph-mon[29621]: pgmap v405: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.9 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.44k op/s; 1755/94545 objects misplaced (1.856%); 1.5 MiB/s, 24 objects/s recovering 2024-03-21T13:08:00.643 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:00 smithi163 ceph-mon[29621]: 5.8 scrub starts 2024-03-21T13:08:00.643 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:00 smithi163 ceph-mon[29621]: 5.8 scrub ok 2024-03-21T13:08:00.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:00 smithi163 ceph-mon[40675]: pgmap v405: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.9 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.44k op/s; 1755/94545 objects misplaced (1.856%); 1.5 MiB/s, 24 objects/s recovering 2024-03-21T13:08:00.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:00 smithi163 ceph-mon[40675]: 5.8 scrub starts 2024-03-21T13:08:00.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:00 smithi163 ceph-mon[40675]: 5.8 scrub ok 2024-03-21T13:08:00.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:00.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:00.708 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:00.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:00.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:00 smithi204 ceph-mon[33153]: pgmap v405: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.9 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.44k op/s; 1755/94545 objects misplaced (1.856%); 1.5 MiB/s, 24 objects/s recovering 2024-03-21T13:08:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:00 smithi204 ceph-mon[33153]: 5.8 scrub starts 2024-03-21T13:08:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:00 smithi204 ceph-mon[33153]: 5.8 scrub ok 2024-03-21T13:08:00.809 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:00.850 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:00.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:00.951 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:00.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:00.991 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:00.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:01.093 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:01.133 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:01.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:01.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:01.284 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:01.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:01.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:01.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:01.413 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-21T13:08:01.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:01.433 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:01.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:01.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:01.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:01.568 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:01.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:01.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:01.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:01 smithi163 ceph-mon[29621]: 4.18 scrub starts 2024-03-21T13:08:01.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:01 smithi163 ceph-mon[29621]: 4.18 scrub ok 2024-03-21T13:08:01.690 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:01 smithi163 ceph-mon[40675]: 4.18 scrub starts 2024-03-21T13:08:01.690 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:01 smithi163 ceph-mon[40675]: 4.18 scrub ok 2024-03-21T13:08:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:01.701 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:01.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:01.725 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:01 smithi204 ceph-mon[33153]: 4.18 scrub starts 2024-03-21T13:08:01.725 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:01 smithi204 ceph-mon[33153]: 4.18 scrub ok 2024-03-21T13:08:01.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:01.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:01.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:01.939 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:01.980 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:01.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:02.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:08:01 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:08:02.082 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:02.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:02.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:02.221 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:02.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:02.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:02.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:02.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:02.414 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:02.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:02.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:02.516 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:02.540 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:02.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:02.542 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:02 smithi163 ceph-mon[29621]: pgmap v406: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.4 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 976 B/s rd, 0 B/s wr, 1.74k op/s; 713/70161 objects misplaced (1.016%); 5.4 MiB/s, 86 objects/s recovering 2024-03-21T13:08:02.542 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:02 smithi163 ceph-mon[40675]: pgmap v406: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.4 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 976 B/s rd, 0 B/s wr, 1.74k op/s; 713/70161 objects misplaced (1.016%); 5.4 MiB/s, 86 objects/s recovering 2024-03-21T13:08:02.641 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:02.678 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:02.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:02 smithi204 ceph-mon[33153]: pgmap v406: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.4 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 976 B/s rd, 0 B/s wr, 1.74k op/s; 713/70161 objects misplaced (1.016%); 5.4 MiB/s, 86 objects/s recovering 2024-03-21T13:08:02.779 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:02.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:02.918 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:02.952 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:02.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:03.053 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:03.090 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:03.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:03.192 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:03.223 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:03.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:03.257 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:08:03.257 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:08:03.257 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:08:03.257 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:08:03.257 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:08:03.257 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:08:03.257 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:08:03.257 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:08:03.258 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:08:03.258 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:08:03.258 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:08:03.258 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:08:03.258 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:08:03.258 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:08:03.258 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:08:03.258 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:08:03.325 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:03.363 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:03.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:03.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:03.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:03.505 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:03.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:03.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:03.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:03 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:08:03.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:03 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:08:03.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:03 smithi163 ceph-mon[29621]: osdmap e222: 8 total, 8 up, 5 in 2024-03-21T13:08:03.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:03 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2374541195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:03.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:03.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:03 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:08:03.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:03 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:08:03.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:03 smithi163 ceph-mon[40675]: osdmap e222: 8 total, 8 up, 5 in 2024-03-21T13:08:03.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:03 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2374541195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:03.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:03.641 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:03.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:03.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:03.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:03 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:08:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:03 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:08:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:03 smithi204 ceph-mon[33153]: osdmap e222: 8 total, 8 up, 5 in 2024-03-21T13:08:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:03 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2374541195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:03.779 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:03.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:03.880 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:03.916 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:03.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:04.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:04.053 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:04.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:04.136 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:04.154 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:04.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:04.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:04.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:04.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:04.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:04.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:04.429 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:04.468 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:04.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:04.570 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:04.599 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:08:04.599 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:08:04.599 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-21T13:08:04.600 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-21T13:08:04.604 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:04.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:04.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:04.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:04 smithi163 ceph-mon[40675]: pgmap v408: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.4 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s; 713/70161 objects misplaced (1.016%); 5.7 MiB/s, 90 objects/s recovering 2024-03-21T13:08:04.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:04 smithi163 ceph-mon[40675]: 5.10 scrub starts 2024-03-21T13:08:04.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:04 smithi163 ceph-mon[40675]: 5.10 scrub ok 2024-03-21T13:08:04.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:04 smithi163 ceph-mon[29621]: pgmap v408: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.4 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s; 713/70161 objects misplaced (1.016%); 5.7 MiB/s, 90 objects/s recovering 2024-03-21T13:08:04.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:04 smithi163 ceph-mon[29621]: 5.10 scrub starts 2024-03-21T13:08:04.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:04 smithi163 ceph-mon[29621]: 5.10 scrub ok 2024-03-21T13:08:04.746 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:04.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:04.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:04.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:04 smithi204 ceph-mon[33153]: pgmap v408: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.4 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s; 713/70161 objects misplaced (1.016%); 5.7 MiB/s, 90 objects/s recovering 2024-03-21T13:08:04.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:04 smithi204 ceph-mon[33153]: 5.10 scrub starts 2024-03-21T13:08:04.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:04 smithi204 ceph-mon[33153]: 5.10 scrub ok 2024-03-21T13:08:04.847 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:04.884 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:04.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:04.986 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:05.021 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:05.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:05.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:05.122 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:05.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:05.259 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:05.298 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:05.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:05.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:05.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:05.434 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:05.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:05.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:05.464 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:05 smithi163 ceph-mon[29621]: 5.1 scrub starts 2024-03-21T13:08:05.476 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:08:05.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:05.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:05.569 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:05.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:05.669 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:05.694 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:05.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:05 smithi204 ceph-mon[33153]: 5.1 scrub starts 2024-03-21T13:08:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:05 smithi204 ceph-mon[33153]: 5.1 scrub ok 2024-03-21T13:08:05.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:05 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2111614650' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:08:05.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:05 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:08:05.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:05.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:08:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:05 smithi163 ceph-mon[29621]: 5.1 scrub ok 2024-03-21T13:08:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:05 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2111614650' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:08:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:05 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:08:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:08:05.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:08:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:08:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:08:05.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:05 smithi163 ceph-mon[40675]: 5.1 scrub starts 2024-03-21T13:08:05.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:05 smithi163 ceph-mon[40675]: 5.1 scrub ok 2024-03-21T13:08:05.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:05 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2111614650' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:08:05.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:05 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:08:05.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:05.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:08:05.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:05.820 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:05.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:05.921 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:05.946 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:05.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:06.047 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:06.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:06.173 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:06.197 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:06.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:06.256 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:06.298 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:06.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:06.325 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:06.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:06.426 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:06.451 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:06.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:06.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:06.586 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:06.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:06.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:06.653 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:08:06.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:06.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[29621]: pgmap v409: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.4 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s; 713/70161 objects misplaced (1.016%); 5.7 MiB/s, 90 objects/s recovering 2024-03-21T13:08:06.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[29621]: 5.e scrub starts 2024-03-21T13:08:06.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[29621]: 5.e scrub ok 2024-03-21T13:08:06.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[29621]: 5.c scrub starts 2024-03-21T13:08:06.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[29621]: 5.c scrub ok 2024-03-21T13:08:06.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[29621]: 2.0 scrub starts 2024-03-21T13:08:06.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[29621]: 2.0 scrub ok 2024-03-21T13:08:06.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:08:06.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:08:06.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:08:06.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[29621]: osdmap e223: 8 total, 8 up, 5 in 2024-03-21T13:08:06.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[29621]: 5.12 scrub starts 2024-03-21T13:08:06.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[29621]: 5.12 scrub ok 2024-03-21T13:08:06.690 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[40675]: pgmap v409: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.4 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s; 713/70161 objects misplaced (1.016%); 5.7 MiB/s, 90 objects/s recovering 2024-03-21T13:08:06.690 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[40675]: 5.e scrub starts 2024-03-21T13:08:06.690 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[40675]: 5.e scrub ok 2024-03-21T13:08:06.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[40675]: 5.c scrub starts 2024-03-21T13:08:06.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[40675]: 5.c scrub ok 2024-03-21T13:08:06.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[40675]: 2.0 scrub starts 2024-03-21T13:08:06.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[40675]: 2.0 scrub ok 2024-03-21T13:08:06.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:08:06.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:08:06.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:08:06.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[40675]: osdmap e223: 8 total, 8 up, 5 in 2024-03-21T13:08:06.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[40675]: 5.12 scrub starts 2024-03-21T13:08:06.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:06 smithi163 ceph-mon[40675]: 5.12 scrub ok 2024-03-21T13:08:06.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:06.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:06.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:06 smithi204 ceph-mon[33153]: pgmap v409: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.4 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s; 713/70161 objects misplaced (1.016%); 5.7 MiB/s, 90 objects/s recovering 2024-03-21T13:08:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:06 smithi204 ceph-mon[33153]: 5.e scrub starts 2024-03-21T13:08:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:06 smithi204 ceph-mon[33153]: 5.e scrub ok 2024-03-21T13:08:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:06 smithi204 ceph-mon[33153]: 5.c scrub starts 2024-03-21T13:08:06.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:06 smithi204 ceph-mon[33153]: 5.c scrub ok 2024-03-21T13:08:06.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:06 smithi204 ceph-mon[33153]: 2.0 scrub starts 2024-03-21T13:08:06.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:06 smithi204 ceph-mon[33153]: 2.0 scrub ok 2024-03-21T13:08:06.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:06 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:08:06.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:06 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:08:06.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:06 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:08:06.755 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:06 smithi204 ceph-mon[33153]: osdmap e223: 8 total, 8 up, 5 in 2024-03-21T13:08:06.755 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:06 smithi204 ceph-mon[33153]: 5.12 scrub starts 2024-03-21T13:08:06.755 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:06 smithi204 ceph-mon[33153]: 5.12 scrub ok 2024-03-21T13:08:06.825 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:06.878 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:06.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:06.979 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:07.014 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:07.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:07.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:07.116 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:07.153 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:07.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:07.255 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:07.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:07.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:07.392 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:07.427 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:07.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:07.528 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:07.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:07.563 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:07.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:07.664 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:07.705 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:07.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:07.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:07 smithi204 ceph-mon[33153]: osdmap e224: 8 total, 8 up, 5 in 2024-03-21T13:08:07.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:07 smithi204 ceph-mon[33153]: pgmap v412: 148 pgs: 1 peering, 147 active+clean; 953 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s; 6.2 MiB/s, 98 objects/s recovering 2024-03-21T13:08:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:07 smithi163 ceph-mon[29621]: osdmap e224: 8 total, 8 up, 5 in 2024-03-21T13:08:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:07 smithi163 ceph-mon[29621]: pgmap v412: 148 pgs: 1 peering, 147 active+clean; 953 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s; 6.2 MiB/s, 98 objects/s recovering 2024-03-21T13:08:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:07 smithi163 ceph-mon[40675]: osdmap e224: 8 total, 8 up, 5 in 2024-03-21T13:08:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:07 smithi163 ceph-mon[40675]: pgmap v412: 148 pgs: 1 peering, 147 active+clean; 953 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s; 6.2 MiB/s, 98 objects/s recovering 2024-03-21T13:08:07.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:07.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:07.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:07.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:07.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:07.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:08.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:08.123 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:08.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:08.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:08.225 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:08.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:08.262 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:08.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:08.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:08.399 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:08.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:08.500 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:08.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:08.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:08.638 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:08.671 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:08.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:08 smithi204 ceph-mon[33153]: 1.0 deep-scrub starts 2024-03-21T13:08:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:08 smithi204 ceph-mon[33153]: 1.0 deep-scrub ok 2024-03-21T13:08:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:08 smithi163 ceph-mon[40675]: 1.0 deep-scrub starts 2024-03-21T13:08:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:08 smithi163 ceph-mon[40675]: 1.0 deep-scrub ok 2024-03-21T13:08:08.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:08 smithi163 ceph-mon[29621]: 1.0 deep-scrub starts 2024-03-21T13:08:08.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:08 smithi163 ceph-mon[29621]: 1.0 deep-scrub ok 2024-03-21T13:08:08.772 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:08.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:08.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:08.857 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:08:08.911 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:08.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:08.981 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:08:08.981 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:08:09.052 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:09.088 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:09.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:09.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:09.190 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:09.226 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:09.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:09.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:09.368 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:09.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:09.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:09.470 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:09.504 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:09.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:09.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:09.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:09.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:09 smithi163 ceph-mon[29621]: 5.f scrub starts 2024-03-21T13:08:09.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:09 smithi163 ceph-mon[29621]: 5.f scrub ok 2024-03-21T13:08:09.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:09 smithi163 ceph-mon[29621]: 4.15 scrub starts 2024-03-21T13:08:09.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:09 smithi163 ceph-mon[29621]: 4.15 scrub ok 2024-03-21T13:08:09.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:09 smithi163 ceph-mon[29621]: pgmap v413: 148 pgs: 1 peering, 147 active+clean; 953 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 805 B/s rd, 0 B/s wr, 1.25k op/s; 5.8 MiB/s, 93 objects/s recovering 2024-03-21T13:08:09.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:09 smithi163 ceph-mon[40675]: 5.f scrub starts 2024-03-21T13:08:09.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:09 smithi163 ceph-mon[40675]: 5.f scrub ok 2024-03-21T13:08:09.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:09 smithi163 ceph-mon[40675]: 4.15 scrub starts 2024-03-21T13:08:09.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:09 smithi163 ceph-mon[40675]: 4.15 scrub ok 2024-03-21T13:08:09.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:09 smithi163 ceph-mon[40675]: pgmap v413: 148 pgs: 1 peering, 147 active+clean; 953 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 805 B/s rd, 0 B/s wr, 1.25k op/s; 5.8 MiB/s, 93 objects/s recovering 2024-03-21T13:08:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:09.643 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:09.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:09.668 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-21T13:08:09.744 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:09.781 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:09.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:09.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:09.882 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:09.921 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:09.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:10.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:09 smithi204 ceph-mon[33153]: 5.f scrub starts 2024-03-21T13:08:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:09 smithi204 ceph-mon[33153]: 5.f scrub ok 2024-03-21T13:08:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:09 smithi204 ceph-mon[33153]: 4.15 scrub starts 2024-03-21T13:08:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:09 smithi204 ceph-mon[33153]: 4.15 scrub ok 2024-03-21T13:08:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:09 smithi204 ceph-mon[33153]: pgmap v413: 148 pgs: 1 peering, 147 active+clean; 953 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 805 B/s rd, 0 B/s wr, 1.25k op/s; 5.8 MiB/s, 93 objects/s recovering 2024-03-21T13:08:10.022 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:10.057 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:10.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:10.159 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:10.195 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:10.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:10.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:10.224 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:10.288 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:10.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:10.304 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:10.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:10.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:10.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:10.430 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:10.465 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:10.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:10.567 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:10.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:10.611 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:10.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:08:10.660 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:08:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 224, 2024-03-21T13:08:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "d4bf79c0-e781-11ee-95cd-87774f69a715", 2024-03-21T13:08:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "created": "2024-03-21T12:52:56.031419+0000", 2024-03-21T13:08:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "modified": "2024-03-21T13:08:06.463752+0000", 2024-03-21T13:08:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "last_up_change": "2024-03-21T12:58:23.775302+0000", 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "last_in_change": "2024-03-21T13:06:28.461702+0000", 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_num": 5799936, 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_set": [ 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "pglog_hardlimit", 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "purged_snapdirs", 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_deletes", 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "sortbitwise" 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_version": 18, 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "full_ratio": 0.94999998807907104, 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-21T13:08:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_snapshot": "", 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_max": 8, 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd": 8, 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "require_min_compat_client": "luminous", 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "min_compat_client": "luminous", 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "require_osd_release": "squid", 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "allow_crimson": false, 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "pools": [ 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 1, 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".mgr", 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:56:20.557490+0000", 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:08:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 1, 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 1, 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 1, 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 1, 2024-03-21T13:08:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 1, 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "21", 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:08:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:08:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:08:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:08:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:08:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:08:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:08:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:08:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:08:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-03-21T13:08:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_max": 32, 2024-03-21T13:08:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_min": 1 2024-03-21T13:08:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:08:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": {} 2024-03-21T13:08:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 7.8899998664855957, 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 7.8899998664855957, 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.37999999523162842, 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 3, 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 3, 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 2, 2024-03-21T13:08:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "datapool", 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:35.713383+0000", 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:08:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 3, 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 3, 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 3, 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 3, 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 3, 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "67", 2024-03-21T13:08:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:08:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:08:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:08:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:08:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:08:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 2, 2024-03-21T13:08:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 67, 2024-03-21T13:08:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:08:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:08:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:08:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:08:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:08:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:08:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:08:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:08:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:08:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:08:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:08:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:08:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:08:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:08:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:08:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:08:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:08:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:08:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd": {} 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.6600000858306885, 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.6600000858306885, 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.5, 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.3300000429153442, 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.3300000429153442, 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 3, 2024-03-21T13:08:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".rgw.root", 2024-03-21T13:08:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:36.436384+0000", 2024-03-21T13:08:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:08:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:08:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:08:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:08:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:08:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:08:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:08:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:08:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:08:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:08:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:08:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:08:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:08:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:08:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:08:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:08:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:08:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:08:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:08:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:08:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:08:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:08:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:08:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:08:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "65", 2024-03-21T13:08:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:08:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:08:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:08:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:08:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:08:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:08:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:08:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:08:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:08:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:08:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:08:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:08:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:08:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:08:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.4800000190734863, 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.4800000190734863, 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.62999999523162842, 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.559999942779541, 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.559999942779541, 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 4, 2024-03-21T13:08:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.log", 2024-03-21T13:08:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:38.049304+0000", 2024-03-21T13:08:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:08:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:08:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:08:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:08:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:08:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:08:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:08:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:08:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:08:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:08:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:08:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "67", 2024-03-21T13:08:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:08:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:08:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:08:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:08:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:08:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:08:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:08:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:08:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:08:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:08:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:08:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:08:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:08:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:08:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:08:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:08:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:08:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 3.7100000381469727, 2024-03-21T13:08:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 3.7100000381469727, 2024-03-21T13:08:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.62999999523162842, 2024-03-21T13:08:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 2.3399999141693115, 2024-03-21T13:08:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 2.3399999141693115, 2024-03-21T13:08:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:08:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:08:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:08:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 5, 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.control", 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:39.990512+0000", 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:08:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:08:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:08:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "69", 2024-03-21T13:08:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:08:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:08:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:08:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:08:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:08:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:08:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:08:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:08:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:08:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:08:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:08:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:08:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:08:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:08:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:08:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:08:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.4800000190734863, 2024-03-21T13:08:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.4800000190734863, 2024-03-21T13:08:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.62999999523162842, 2024-03-21T13:08:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.559999942779541, 2024-03-21T13:08:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.559999942779541, 2024-03-21T13:08:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:08:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:08:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:08:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 6, 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.meta", 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:41.463686+0000", 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:08:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:08:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "71", 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:08:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:08:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:08:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_bias": 4 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 3.2200000286102295, 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 3.2200000286102295, 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.62999999523162842, 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 2.0299999713897705, 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 2.0299999713897705, 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:08:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 8, 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "unique_pool_1", 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T13:04:45.973511+0000", 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:08:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 16, 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 16, 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 16, 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 16, 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 16, 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:08:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "173", 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:08:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:08:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:08:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 3.4800000190734863, 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 3.4800000190734863, 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.62999999523162842, 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 2.190000057220459, 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 2.190000057220459, 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:08:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f", 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0.0852813720703125, 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 8, 2024-03-21T13:08:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 213, 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6801", 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6802", 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6804", 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:08:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6803", 2024-03-21T13:08:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:08:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6801/3086410981", 2024-03-21T13:08:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6802/3086410981", 2024-03-21T13:08:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6804/3086410981", 2024-03-21T13:08:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6803/3086410981", 2024-03-21T13:08:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:08:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:08:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:08:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "36e05f89-c238-41f7-be65-1553b529e8ce", 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 0, 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 0, 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 13, 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 110, 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:08:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6805", 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6806", 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6808", 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6807", 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6805/3666062784", 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6806/3666062784", 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6808/3666062784", 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6807/3666062784", 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:08:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f", 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 0, 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 0, 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 18, 2024-03-21T13:08:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 168, 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6809", 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6810", 2024-03-21T13:08:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:08:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:08:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6812", 2024-03-21T13:08:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:08:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:08:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6811", 2024-03-21T13:08:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:08:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6809/4009213507", 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6810/4009213507", 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6812/4009213507", 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6811/4009213507", 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d", 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:08:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 25, 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 223, 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6813", 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6814", 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6816", 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6815", 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6813/3981878189", 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6814/3981878189", 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6816/3981878189", 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6815/3981878189", 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471", 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:08:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0.578704833984375, 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 30, 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 209, 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6800", 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:08:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6801", 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6803", 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6802", 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6800/719276054", 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6801/719276054", 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6803/719276054", 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6802/719276054", 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1", 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:08:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 0, 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 0, 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 39, 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 123, 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6804", 2024-03-21T13:08:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:08:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:08:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6805", 2024-03-21T13:08:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:08:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6807", 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6806", 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:08:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6804/1346947668", 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6805/1346947668", 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6807/1346947668", 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6806/1346947668", 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-03-21T13:08:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b", 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 47, 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 204, 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6808", 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6809", 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:08:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6811", 2024-03-21T13:08:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:08:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:08:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6810", 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6808/200077988", 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6809/200077988", 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6811/200077988", 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6810/200077988", 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-03-21T13:08:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49", 2024-03-21T13:08:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:08:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:08:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:08:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:08:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:08:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:08:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 56, 2024-03-21T13:08:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 203, 2024-03-21T13:08:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:08:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:08:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:08:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6812", 2024-03-21T13:08:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:08:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:08:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6813", 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6815", 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6814", 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6812/315664888", 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6813/315664888", 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6815/315664888", 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6814/315664888", 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:08:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_xinfo": [ 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:07:33.010762+0000", 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:08:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 65536, 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:07:30.860144+0000", 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:08:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 65536, 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:07:49.384732+0000", 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:08:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:07:16.773470+0000", 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:07:29.252857+0000", 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 65536, 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:08:01.728919+0000", 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:08:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:06:51.232427+0000", 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:06:56.228460+0000", 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:08:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap": [ 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.13", 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: 0, 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: 5, 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: 6 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "6.11", 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: 7, 2024-03-21T13:08:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: 4 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_items": [ 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "3.19", 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 4, 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 7 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "6.1c", 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 7, 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 0 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_primaries": [], 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_temp": [], 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_temp": [], 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: "blocklist": { 2024-03-21T13:08:10.749 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3453855427": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/2571244897": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/3073378854": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1334591766": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/116934309": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3890775846": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1783921524": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2553440211": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1812694077": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1070540283": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/2354157786": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1591817392": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/4152198951": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/1798115633": "2024-03-22T12:54:00.224135+0000" 2024-03-21T13:08:10.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: "range_blocklist": {}, 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profiles": { 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: "default": { 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van" 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps_queue": [], 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: "new_removed_snaps": [], 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: "new_purged_snaps": [], 2024-03-21T13:08:10.751 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_node_flags": {}, 2024-03-21T13:08:10.752 INFO:teuthology.orchestra.run.smithi163.stdout: "device_class_flags": {}, 2024-03-21T13:08:10.752 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode": { 2024-03-21T13:08:10.752 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_enabled": false, 2024-03-21T13:08:10.752 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_bucket_count": 0, 2024-03-21T13:08:10.752 INFO:teuthology.orchestra.run.smithi163.stdout: "degraded_stretch_mode": 0, 2024-03-21T13:08:10.752 INFO:teuthology.orchestra.run.smithi163.stdout: "recovering_stretch_mode": 0, 2024-03-21T13:08:10.752 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_bucket": 0 2024-03-21T13:08:10.752 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:08:10.752 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:08:10.752 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:10 smithi163 ceph-mon[29621]: 5.1b scrub starts 2024-03-21T13:08:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:10 smithi163 ceph-mon[29621]: 5.1b scrub ok 2024-03-21T13:08:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:10 smithi163 ceph-mon[29621]: 8.6 scrub starts 2024-03-21T13:08:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:10 smithi163 ceph-mon[29621]: 5.1a scrub starts 2024-03-21T13:08:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:10 smithi163 ceph-mon[29621]: 5.1a scrub ok 2024-03-21T13:08:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:10 smithi163 ceph-mon[29621]: 8.6 scrub ok 2024-03-21T13:08:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:10 smithi163 ceph-mon[40675]: 5.1b scrub starts 2024-03-21T13:08:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:10 smithi163 ceph-mon[40675]: 5.1b scrub ok 2024-03-21T13:08:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:10 smithi163 ceph-mon[40675]: 8.6 scrub starts 2024-03-21T13:08:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:10 smithi163 ceph-mon[40675]: 5.1a scrub starts 2024-03-21T13:08:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:10 smithi163 ceph-mon[40675]: 5.1a scrub ok 2024-03-21T13:08:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:10 smithi163 ceph-mon[40675]: 8.6 scrub ok 2024-03-21T13:08:10.788 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:10.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:10.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:10.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:10.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:10.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:10 smithi204 ceph-mon[33153]: 5.1b scrub starts 2024-03-21T13:08:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:10 smithi204 ceph-mon[33153]: 5.1b scrub ok 2024-03-21T13:08:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:10 smithi204 ceph-mon[33153]: 8.6 scrub starts 2024-03-21T13:08:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:10 smithi204 ceph-mon[33153]: 5.1a scrub starts 2024-03-21T13:08:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:10 smithi204 ceph-mon[33153]: 5.1a scrub ok 2024-03-21T13:08:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:10 smithi204 ceph-mon[33153]: 8.6 scrub ok 2024-03-21T13:08:11.026 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:11.059 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:11.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:11.161 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:11.196 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:11.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:11.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:11.339 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:11.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:11.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:11.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:11.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:11.577 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:11.608 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:11.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:11.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:11.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:11.750 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:11.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:11 smithi163 ceph-mon[29621]: 4.8 scrub starts 2024-03-21T13:08:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:11 smithi163 ceph-mon[29621]: 4.8 scrub ok 2024-03-21T13:08:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:11 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1951250388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:08:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:11 smithi163 ceph-mon[29621]: pgmap v414: 148 pgs: 148 active+clean; 501 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.90k op/s; 4.6 MiB/s, 74 objects/s recovering 2024-03-21T13:08:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:11 smithi163 ceph-mon[40675]: 4.8 scrub starts 2024-03-21T13:08:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:11 smithi163 ceph-mon[40675]: 4.8 scrub ok 2024-03-21T13:08:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:11 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1951250388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:08:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:11 smithi163 ceph-mon[40675]: pgmap v414: 148 pgs: 148 active+clean; 501 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.90k op/s; 4.6 MiB/s, 74 objects/s recovering 2024-03-21T13:08:11.851 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:11.892 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:11.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:11.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:11 smithi204 ceph-mon[33153]: 4.8 scrub starts 2024-03-21T13:08:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:11 smithi204 ceph-mon[33153]: 4.8 scrub ok 2024-03-21T13:08:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:11 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1951250388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:08:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:11 smithi204 ceph-mon[33153]: pgmap v414: 148 pgs: 148 active+clean; 501 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.90k op/s; 4.6 MiB/s, 74 objects/s recovering 2024-03-21T13:08:12.003 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:08:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:08:12.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:12.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:12.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:12.092 INFO:tasks.thrashosds.thrasher:j is {'epoch': 224, 'fsid': 'd4bf79c0-e781-11ee-95cd-87774f69a715', 'created': '2024-03-21T12:52:56.031419+0000', 'modified': '2024-03-21T13:08:06.463752+0000', 'last_up_change': '2024-03-21T12:58:23.775302+0000', 'last_in_change': '2024-03-21T13:06:28.461702+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T12:56:20.557490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-21T12:58:35.713383+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6600000858306885, 'score_stable': 2.6600000858306885, 'optimal_score': 0.5, 'raw_score_acting': 1.3300000429153442, 'raw_score_stable': 1.3300000429153442, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-21T12:58:36.436384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4800000190734863, 'score_stable': 2.4800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-21T12:58:38.049304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.7100000381469727, 'score_stable': 3.7100000381469727, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.3399999141693115, 'raw_score_stable': 2.3399999141693115, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-21T12:58:39.990512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4800000190734863, 'score_stable': 2.4800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-21T12:58:41.463686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2200000286102295, 'score_stable': 3.2200000286102295, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.0299999713897705, 'raw_score_stable': 2.0299999713897705, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-21T13:04:45.973511+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '173', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.4800000190734863, 'score_stable': 3.4800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'c34a3b25-9e02-46e6-8f9a-d0af2615258f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.0852813720703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 213, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6801', 'nonce': 3086410981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6802', 'nonce': 3086410981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6804', 'nonce': 3086410981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6803', 'nonce': 3086410981}]}, 'public_addr': '172.21.15.163:6801/3086410981', 'cluster_addr': '172.21.15.163:6802/3086410981', 'heartbeat_back_addr': '172.21.15.163:6804/3086410981', 'heartbeat_front_addr': '172.21.15.163:6803/3086410981', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '36e05f89-c238-41f7-be65-1553b529e8ce', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6805', 'nonce': 3666062784}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6806', 'nonce': 3666062784}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6808', 'nonce': 3666062784}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6807', 'nonce': 3666062784}]}, 'public_addr': '172.21.15.163:6805/3666062784', 'cluster_addr': '172.21.15.163:6806/3666062784', 'heartbeat_back_addr': '172.21.15.163:6808/3666062784', 'heartbeat_front_addr': '172.21.15.163:6807/3666062784', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '968ec78d-25b8-4905-b328-d490c38ca85f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 168, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6809', 'nonce': 4009213507}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6810', 'nonce': 4009213507}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6812', 'nonce': 4009213507}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6811', 'nonce': 4009213507}]}, 'public_addr': '172.21.15.163:6809/4009213507', 'cluster_addr': '172.21.15.163:6810/4009213507', 'heartbeat_back_addr': '172.21.15.163:6812/4009213507', 'heartbeat_front_addr': '172.21.15.163:6811/4009213507', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '67dfa4f6-1709-48f6-971d-c6287233492d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 223, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6813', 'nonce': 3981878189}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6814', 'nonce': 3981878189}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6816', 'nonce': 3981878189}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6815', 'nonce': 3981878189}]}, 'public_addr': '172.21.15.163:6813/3981878189', 'cluster_addr': '172.21.15.163:6814/3981878189', 'heartbeat_back_addr': '172.21.15.163:6816/3981878189', 'heartbeat_front_addr': '172.21.15.163:6815/3981878189', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7fd311a9-f3c2-456d-8ab1-341cf5b33471', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.578704833984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 719276054}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6801', 'nonce': 719276054}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6803', 'nonce': 719276054}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 719276054}]}, 'public_addr': '172.21.15.204:6800/719276054', 'cluster_addr': '172.21.15.204:6801/719276054', 'heartbeat_back_addr': '172.21.15.204:6803/719276054', 'heartbeat_front_addr': '172.21.15.204:6802/719276054', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a87759c8-251c-415d-9387-a49f7d2f36d1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 123, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 1346947668}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6805', 'nonce': 1346947668}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6807', 'nonce': 1346947668}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 1346947668}]}, 'public_addr': '172.21.15.204:6804/1346947668', 'cluster_addr': '172.21.15.204:6805/1346947668', 'heartbeat_back_addr': '172.21.15.204:6807/1346947668', 'heartbeat_front_addr': '172.21.15.204:6806/1346947668', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5eb135cd-f677-4d0c-b23a-738f31615b6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 204, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 200077988}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6809', 'nonce': 200077988}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6811', 'nonce': 200077988}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 200077988}]}, 'public_addr': '172.21.15.204:6808/200077988', 'cluster_addr': '172.21.15.204:6809/200077988', 'heartbeat_back_addr': '172.21.15.204:6811/200077988', 'heartbeat_front_addr': '172.21.15.204:6810/200077988', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4f528e10-2c93-4bc2-a54c-06ca4132ed49', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 203, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 315664888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6813', 'nonce': 315664888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6815', 'nonce': 315664888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 315664888}]}, 'public_addr': '172.21.15.204:6812/315664888', 'cluster_addr': '172.21.15.204:6813/315664888', 'heartbeat_back_addr': '172.21.15.204:6815/315664888', 'heartbeat_front_addr': '172.21.15.204:6814/315664888', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:07:33.010762+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-21T13:07:30.860144+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-21T13:07:49.384732+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:07:16.773470+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:07:29.252857+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-21T13:08:01.728919+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:06:51.232427+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:06:56.228460+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.13', 'osds': [0, 5, 6]}, {'pgid': '6.11', 'osds': [7, 3, 4]}], 'pg_upmap_items': [{'pgid': '3.19', 'mappings': [{'from': 4, 'to': 7}]}, {'pgid': '6.1c', 'mappings': [{'from': 7, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.163:0/3453855427': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/2571244897': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:6800/3073378854': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/1334591766': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/116934309': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:0/3890775846': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/1783921524': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:6800/2553440211': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1812694077': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1070540283': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/2354157786': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1591817392': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/4152198951': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:6800/1798115633': '2024-03-22T12:54:00.224135+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-21T13:08:12.092 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:08:12.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:12.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:12.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:12.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:12.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:12.308 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:12.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:12.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:12.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:12.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:12.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:12.550 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:12.590 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:12.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:12.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:12.729 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:12.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:12 smithi163 ceph-mon[29621]: 3.1 scrub starts 2024-03-21T13:08:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:12 smithi163 ceph-mon[29621]: 3.1 scrub ok 2024-03-21T13:08:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:12 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:12 smithi163 ceph-mon[40675]: 3.1 scrub starts 2024-03-21T13:08:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:12 smithi163 ceph-mon[40675]: 3.1 scrub ok 2024-03-21T13:08:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:12 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:12.832 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:12.867 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:12.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:12.903 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:12 smithi204 ceph-mon[33153]: 3.1 scrub starts 2024-03-21T13:08:12.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:12 smithi204 ceph-mon[33153]: 3.1 scrub ok 2024-03-21T13:08:12.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:12 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:12.969 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:13.005 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:13.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:13.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:13.145 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:13.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:13.246 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:13.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:13.289 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:13.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:13.391 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:13.429 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:13.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:13.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:13.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:13.568 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:13.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:13.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:13.670 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:13.707 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:13.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:08:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:08:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[29621]: 5.a scrub starts 2024-03-21T13:08:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[29621]: 5.a scrub ok 2024-03-21T13:08:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[29621]: 8.5 scrub starts 2024-03-21T13:08:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[29621]: 8.5 scrub ok 2024-03-21T13:08:13.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[29621]: osdmap e225: 8 total, 8 up, 5 in 2024-03-21T13:08:13.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[29621]: pgmap v416: 148 pgs: 148 active+clean; 501 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 697 B/s rd, 0 B/s wr, 1.44k op/s; 5.0 MiB/s, 80 objects/s recovering 2024-03-21T13:08:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:08:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:08:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[40675]: 5.a scrub starts 2024-03-21T13:08:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[40675]: 5.a scrub ok 2024-03-21T13:08:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[40675]: 8.5 scrub starts 2024-03-21T13:08:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[40675]: 8.5 scrub ok 2024-03-21T13:08:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[40675]: osdmap e225: 8 total, 8 up, 5 in 2024-03-21T13:08:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:13 smithi163 ceph-mon[40675]: pgmap v416: 148 pgs: 148 active+clean; 501 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 697 B/s rd, 0 B/s wr, 1.44k op/s; 5.0 MiB/s, 80 objects/s recovering 2024-03-21T13:08:13.808 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:13.843 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:13.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:13.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:13.981 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:13.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:14.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:13 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:08:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:13 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:08:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:13 smithi204 ceph-mon[33153]: 5.a scrub starts 2024-03-21T13:08:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:13 smithi204 ceph-mon[33153]: 5.a scrub ok 2024-03-21T13:08:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:13 smithi204 ceph-mon[33153]: 8.5 scrub starts 2024-03-21T13:08:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:13 smithi204 ceph-mon[33153]: 8.5 scrub ok 2024-03-21T13:08:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:13 smithi204 ceph-mon[33153]: osdmap e225: 8 total, 8 up, 5 in 2024-03-21T13:08:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:13 smithi204 ceph-mon[33153]: pgmap v416: 148 pgs: 148 active+clean; 501 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 697 B/s rd, 0 B/s wr, 1.44k op/s; 5.0 MiB/s, 80 objects/s recovering 2024-03-21T13:08:14.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:14.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:14.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:14.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:14.224 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:14.258 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:14.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:14.359 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:14.394 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:14.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:14.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:14.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:14.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:14.536 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:14.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:14.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:14.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:14.679 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:14.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:14.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:14.813 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:14.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:14.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:14.954 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:14.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:15.004 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Removed 135870 objects 2024-03-21T13:08:15.004 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Clean up completed and total clean up time :106.911 2024-03-21T13:08:15.036 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:08:15.055 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:15.056 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:08:15.075 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:08:15.079 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:15.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:15.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:15.095 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:08:15.121 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:08:15.142 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:08:15.161 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:08:15.181 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:15.182 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:08:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:08:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:08:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:08:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:08:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:08:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:08:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:08:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:08:15.206 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:15.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:15.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:15.331 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:15.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:15.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:15.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:15.458 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:15.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:15.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:15.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:15.585 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:15.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:15.686 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:15.711 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:15.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:15.736 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:08:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:08:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:08:15.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:15.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:15.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:15.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:15.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:15.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:15.964 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:15.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:16.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:15 smithi163 ceph-mon[29621]: 8.3 scrub starts 2024-03-21T13:08:16.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:15 smithi163 ceph-mon[29621]: 8.3 scrub ok 2024-03-21T13:08:16.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:15 smithi163 ceph-mon[29621]: 6.8 deep-scrub starts 2024-03-21T13:08:16.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:15 smithi163 ceph-mon[29621]: 6.8 deep-scrub ok 2024-03-21T13:08:16.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:15 smithi163 ceph-mon[29621]: pgmap v417: 148 pgs: 148 active+clean; 501 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 614 B/s rd, 0 B/s wr, 867 op/s 2024-03-21T13:08:16.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:15 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4016046559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:16.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:15 smithi163 ceph-mon[40675]: 8.3 scrub starts 2024-03-21T13:08:16.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:15 smithi163 ceph-mon[40675]: 8.3 scrub ok 2024-03-21T13:08:16.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:15 smithi163 ceph-mon[40675]: 6.8 deep-scrub starts 2024-03-21T13:08:16.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:15 smithi163 ceph-mon[40675]: 6.8 deep-scrub ok 2024-03-21T13:08:16.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:15 smithi163 ceph-mon[40675]: pgmap v417: 148 pgs: 148 active+clean; 501 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 614 B/s rd, 0 B/s wr, 867 op/s 2024-03-21T13:08:16.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:15 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4016046559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:16.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:16.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:16.090 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:16.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:16.103 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:16.191 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:16.216 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:16.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:16.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:16.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:15 smithi204 ceph-mon[33153]: 8.3 scrub starts 2024-03-21T13:08:16.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:15 smithi204 ceph-mon[33153]: 8.3 scrub ok 2024-03-21T13:08:16.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:15 smithi204 ceph-mon[33153]: 6.8 deep-scrub starts 2024-03-21T13:08:16.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:15 smithi204 ceph-mon[33153]: 6.8 deep-scrub ok 2024-03-21T13:08:16.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:15 smithi204 ceph-mon[33153]: pgmap v417: 148 pgs: 148 active+clean; 501 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 614 B/s rd, 0 B/s wr, 867 op/s 2024-03-21T13:08:16.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:15 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4016046559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:16.317 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:16.343 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:16.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:16.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:16.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:16.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:16.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:16.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:16.597 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:16.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:16.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:16.723 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:16.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:16.824 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:16.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:16.848 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:16.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:16.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[29621]: 8.9 scrub starts 2024-03-21T13:08:16.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[29621]: 8.9 scrub ok 2024-03-21T13:08:16.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:08:16.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:08:16.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[29621]: 5.15 scrub starts 2024-03-21T13:08:16.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[29621]: 5.15 scrub ok 2024-03-21T13:08:16.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[29621]: osdmap e226: 8 total, 8 up, 5 in 2024-03-21T13:08:16.909 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[40675]: 8.9 scrub starts 2024-03-21T13:08:16.909 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[40675]: 8.9 scrub ok 2024-03-21T13:08:16.909 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:08:16.909 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:08:16.909 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[40675]: 5.15 scrub starts 2024-03-21T13:08:16.909 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[40675]: 5.15 scrub ok 2024-03-21T13:08:16.909 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:16 smithi163 ceph-mon[40675]: osdmap e226: 8 total, 8 up, 5 in 2024-03-21T13:08:16.950 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:16.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:16.974 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:16.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:17.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:17.103 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:17.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:17.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:17.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:17.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:17.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:16 smithi204 ceph-mon[33153]: 8.9 scrub starts 2024-03-21T13:08:17.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:16 smithi204 ceph-mon[33153]: 8.9 scrub ok 2024-03-21T13:08:17.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:16 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:08:17.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:16 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:08:17.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:16 smithi204 ceph-mon[33153]: 5.15 scrub starts 2024-03-21T13:08:17.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:16 smithi204 ceph-mon[33153]: 5.15 scrub ok 2024-03-21T13:08:17.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:16 smithi204 ceph-mon[33153]: osdmap e226: 8 total, 8 up, 5 in 2024-03-21T13:08:17.335 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:17.361 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:17.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:17.462 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:17.491 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:17.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:17.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:17.581 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:08:17.581 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:08:17.581 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-03-21T13:08:17.582 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:08:17.593 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:17.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:17.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:17.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:17.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:17.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:17.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:17.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:17.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:17.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:17.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:17.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:18.010 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:18.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:18.039 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:08:18.112 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:18.136 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:18.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:18.238 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:18.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:17 smithi204 ceph-mon[33153]: 4.13 scrub starts 2024-03-21T13:08:18.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:17 smithi204 ceph-mon[33153]: 4.13 scrub ok 2024-03-21T13:08:18.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:17 smithi204 ceph-mon[33153]: 5.0 deep-scrub starts 2024-03-21T13:08:18.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:17 smithi204 ceph-mon[33153]: 5.0 deep-scrub ok 2024-03-21T13:08:18.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:17 smithi204 ceph-mon[33153]: pgmap v419: 148 pgs: 148 active+clean; 6.2 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.89k op/s 2024-03-21T13:08:18.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:17 smithi163 ceph-mon[29621]: 4.13 scrub starts 2024-03-21T13:08:18.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:17 smithi163 ceph-mon[29621]: 4.13 scrub ok 2024-03-21T13:08:18.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:17 smithi163 ceph-mon[29621]: 5.0 deep-scrub starts 2024-03-21T13:08:18.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:17 smithi163 ceph-mon[29621]: 5.0 deep-scrub ok 2024-03-21T13:08:18.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:17 smithi163 ceph-mon[29621]: pgmap v419: 148 pgs: 148 active+clean; 6.2 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.89k op/s 2024-03-21T13:08:18.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:17 smithi163 ceph-mon[40675]: 4.13 scrub starts 2024-03-21T13:08:18.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:17 smithi163 ceph-mon[40675]: 4.13 scrub ok 2024-03-21T13:08:18.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:17 smithi163 ceph-mon[40675]: 5.0 deep-scrub starts 2024-03-21T13:08:18.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:17 smithi163 ceph-mon[40675]: 5.0 deep-scrub ok 2024-03-21T13:08:18.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:17 smithi163 ceph-mon[40675]: pgmap v419: 148 pgs: 148 active+clean; 6.2 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.89k op/s 2024-03-21T13:08:18.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:18.263 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:18.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:18.364 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:18.394 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:18.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:18.495 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:18.524 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:18.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:18.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:18.655 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:18.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:18.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:18.783 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:18.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:18.884 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:18.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:18.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:19.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:19.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:19.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:19.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:19.160 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:19.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:19.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:19.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:18 smithi204 ceph-mon[33153]: 6.19 scrub starts 2024-03-21T13:08:19.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:18 smithi204 ceph-mon[33153]: 6.19 scrub ok 2024-03-21T13:08:19.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:18 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:19.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:18 smithi163 ceph-mon[29621]: 6.19 scrub starts 2024-03-21T13:08:19.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:18 smithi163 ceph-mon[29621]: 6.19 scrub ok 2024-03-21T13:08:19.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:18 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:19.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:18 smithi163 ceph-mon[40675]: 6.19 scrub starts 2024-03-21T13:08:19.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:18 smithi163 ceph-mon[40675]: 6.19 scrub ok 2024-03-21T13:08:19.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:18 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:19.261 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:19.285 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:19.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:19.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:19.362 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-21T13:08:19.363 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-21T13:08:19.387 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:19.412 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:19.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:19.513 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:19.539 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:19.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:19.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:19.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:19.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:19.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:19.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:19.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:19.791 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:19.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:19.892 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:19.918 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:19.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:20.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:20.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:20.044 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:20.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:20.145 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:20.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: 3.1c scrub starts 2024-03-21T13:08:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: 3.1c scrub ok 2024-03-21T13:08:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: 5.3 scrub starts 2024-03-21T13:08:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: 5.3 scrub ok 2024-03-21T13:08:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: 3.14 scrub starts 2024-03-21T13:08:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: 3.14 scrub ok 2024-03-21T13:08:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: 4.10 scrub starts 2024-03-21T13:08:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: 4.10 scrub ok 2024-03-21T13:08:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: pgmap v420: 148 pgs: 148 active+clean; 6.2 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 990 op/s 2024-03-21T13:08:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.7", "id": [0, 6]}]: dispatch 2024-03-21T13:08:20.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 0]}]: dispatch 2024-03-21T13:08:20.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [4, 3]}]: dispatch 2024-03-21T13:08:20.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:20.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:08:20.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: 3.1c scrub starts 2024-03-21T13:08:20.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: 3.1c scrub ok 2024-03-21T13:08:20.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: 5.3 scrub starts 2024-03-21T13:08:20.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: 5.3 scrub ok 2024-03-21T13:08:20.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: 3.14 scrub starts 2024-03-21T13:08:20.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: 3.14 scrub ok 2024-03-21T13:08:20.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: 4.10 scrub starts 2024-03-21T13:08:20.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: 4.10 scrub ok 2024-03-21T13:08:20.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: pgmap v420: 148 pgs: 148 active+clean; 6.2 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 990 op/s 2024-03-21T13:08:20.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.7", "id": [0, 6]}]: dispatch 2024-03-21T13:08:20.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 0]}]: dispatch 2024-03-21T13:08:20.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [4, 3]}]: dispatch 2024-03-21T13:08:20.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:20.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: 3.1c scrub starts 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: 3.1c scrub ok 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: 5.3 scrub starts 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: 5.3 scrub ok 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: 3.14 scrub starts 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: 3.14 scrub ok 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: 4.10 scrub starts 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: 4.10 scrub ok 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: pgmap v420: 148 pgs: 148 active+clean; 6.2 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 990 op/s 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.7", "id": [0, 6]}]: dispatch 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 0]}]: dispatch 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [4, 3]}]: dispatch 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:20.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:08:20.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:20.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:20.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:20.402 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:20.429 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:20.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:20.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:20.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:20.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:20.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:20.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:20.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:20.679 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:20.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:20.780 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:20.809 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:20.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:20.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:20.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:20.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:20.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:20.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:21.038 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:21.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:21.067 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:21.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:21.168 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:21.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:21.196 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:21.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:21.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:20 smithi204 ceph-mon[33153]: 5.1d scrub starts 2024-03-21T13:08:21.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:20 smithi204 ceph-mon[33153]: 5.1d scrub ok 2024-03-21T13:08:21.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.7", "id": [0, 6]}]': finished 2024-03-21T13:08:21.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 0]}]': finished 2024-03-21T13:08:21.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [4, 3]}]': finished 2024-03-21T13:08:21.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:20 smithi204 ceph-mon[33153]: osdmap e227: 8 total, 8 up, 5 in 2024-03-21T13:08:21.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:20 smithi163 ceph-mon[29621]: 5.1d scrub starts 2024-03-21T13:08:21.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:20 smithi163 ceph-mon[29621]: 5.1d scrub ok 2024-03-21T13:08:21.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.7", "id": [0, 6]}]': finished 2024-03-21T13:08:21.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 0]}]': finished 2024-03-21T13:08:21.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [4, 3]}]': finished 2024-03-21T13:08:21.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:20 smithi163 ceph-mon[29621]: osdmap e227: 8 total, 8 up, 5 in 2024-03-21T13:08:21.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:20 smithi163 ceph-mon[40675]: 5.1d scrub starts 2024-03-21T13:08:21.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:20 smithi163 ceph-mon[40675]: 5.1d scrub ok 2024-03-21T13:08:21.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.7", "id": [0, 6]}]': finished 2024-03-21T13:08:21.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 0]}]': finished 2024-03-21T13:08:21.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [4, 3]}]': finished 2024-03-21T13:08:21.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:20 smithi163 ceph-mon[40675]: osdmap e227: 8 total, 8 up, 5 in 2024-03-21T13:08:21.298 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:21.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:21.326 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:21.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:21.427 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:21.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:21.453 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:21.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:21.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:21.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:21.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:21.680 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:21.709 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:21.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:21.810 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:21.836 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:21.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:21.937 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:21.965 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:21.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:21.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:22.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:08:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:08:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:21 smithi204 ceph-mon[33153]: 6.10 scrub starts 2024-03-21T13:08:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:21 smithi204 ceph-mon[33153]: 6.10 scrub ok 2024-03-21T13:08:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:21 smithi204 ceph-mon[33153]: pgmap v422: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-21T13:08:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:21 smithi204 ceph-mon[33153]: osdmap e228: 8 total, 8 up, 5 in 2024-03-21T13:08:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:21 smithi204 ceph-mon[33153]: 3.1a deep-scrub starts 2024-03-21T13:08:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:21 smithi204 ceph-mon[33153]: 3.1a deep-scrub ok 2024-03-21T13:08:22.067 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:22.091 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:22.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:22.178 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:21 smithi163 ceph-mon[40675]: 6.10 scrub starts 2024-03-21T13:08:22.178 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:21 smithi163 ceph-mon[40675]: 6.10 scrub ok 2024-03-21T13:08:22.178 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:21 smithi163 ceph-mon[40675]: pgmap v422: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-21T13:08:22.179 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:21 smithi163 ceph-mon[40675]: osdmap e228: 8 total, 8 up, 5 in 2024-03-21T13:08:22.179 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:21 smithi163 ceph-mon[40675]: 3.1a deep-scrub starts 2024-03-21T13:08:22.179 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:21 smithi163 ceph-mon[40675]: 3.1a deep-scrub ok 2024-03-21T13:08:22.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:21 smithi163 ceph-mon[29621]: 6.10 scrub starts 2024-03-21T13:08:22.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:21 smithi163 ceph-mon[29621]: 6.10 scrub ok 2024-03-21T13:08:22.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:21 smithi163 ceph-mon[29621]: pgmap v422: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-21T13:08:22.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:21 smithi163 ceph-mon[29621]: osdmap e228: 8 total, 8 up, 5 in 2024-03-21T13:08:22.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:21 smithi163 ceph-mon[29621]: 3.1a deep-scrub starts 2024-03-21T13:08:22.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:21 smithi163 ceph-mon[29621]: 3.1a deep-scrub ok 2024-03-21T13:08:22.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:22.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:22.217 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:22.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:22.319 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:22.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:22.343 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:22.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:22.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:22.469 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:22.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:22.570 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:22.578 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-21T13:08:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:22.596 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:22.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:22.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:22.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:22.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:22.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:22.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:22.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:22.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:22.896 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:08:22.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:22.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:22.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:22.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:23.076 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:23.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:23.101 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:23.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:23.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[29621]: 3.12 deep-scrub starts 2024-03-21T13:08:23.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[29621]: 3.12 deep-scrub ok 2024-03-21T13:08:23.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:23.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4134318845' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:08:23.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:08:23.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:23.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:23.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:23.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[40675]: 3.12 deep-scrub starts 2024-03-21T13:08:23.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[40675]: 3.12 deep-scrub ok 2024-03-21T13:08:23.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:23.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4134318845' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:08:23.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:08:23.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:23.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:23.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:22 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:23.202 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:23.225 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:23.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:23.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:22 smithi204 ceph-mon[33153]: 3.12 deep-scrub starts 2024-03-21T13:08:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:22 smithi204 ceph-mon[33153]: 3.12 deep-scrub ok 2024-03-21T13:08:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:22 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:22 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4134318845' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:08:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:22 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:08:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:22 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:22 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:22 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:23.328 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:23.354 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:23.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:23.455 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:23.479 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:23.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:23.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:23.581 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:23.608 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:23.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:23.714 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:23.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:23.744 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:23.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:23.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:23.870 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:23.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:23.972 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:23.973 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:23.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:23.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:24.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:24.126 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:24.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:24.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[29621]: 3.4 scrub starts 2024-03-21T13:08:24.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[29621]: 3.4 scrub ok 2024-03-21T13:08:24.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[29621]: 4.f deep-scrub starts 2024-03-21T13:08:24.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[29621]: 4.f deep-scrub ok 2024-03-21T13:08:24.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[29621]: pgmap v424: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 737 B/s rd, 0 B/s wr, 13 op/s 2024-03-21T13:08:24.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:08:24.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:08:24.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[29621]: osdmap e229: 8 total, 8 up, 5 in 2024-03-21T13:08:24.150 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[40675]: 3.4 scrub starts 2024-03-21T13:08:24.150 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[40675]: 3.4 scrub ok 2024-03-21T13:08:24.150 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[40675]: 4.f deep-scrub starts 2024-03-21T13:08:24.151 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[40675]: 4.f deep-scrub ok 2024-03-21T13:08:24.151 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[40675]: pgmap v424: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 737 B/s rd, 0 B/s wr, 13 op/s 2024-03-21T13:08:24.151 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:08:24.151 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:08:24.151 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:23 smithi163 ceph-mon[40675]: osdmap e229: 8 total, 8 up, 5 in 2024-03-21T13:08:24.158 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:08:24.162 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:08:24.227 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:24.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:23 smithi204 ceph-mon[33153]: 3.4 scrub starts 2024-03-21T13:08:24.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:23 smithi204 ceph-mon[33153]: 3.4 scrub ok 2024-03-21T13:08:24.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:23 smithi204 ceph-mon[33153]: 4.f deep-scrub starts 2024-03-21T13:08:24.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:23 smithi204 ceph-mon[33153]: 4.f deep-scrub ok 2024-03-21T13:08:24.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:23 smithi204 ceph-mon[33153]: pgmap v424: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 737 B/s rd, 0 B/s wr, 13 op/s 2024-03-21T13:08:24.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:23 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:08:24.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:23 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:08:24.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:23 smithi204 ceph-mon[33153]: osdmap e229: 8 total, 8 up, 5 in 2024-03-21T13:08:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:24.256 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:24.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:24.357 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:24.386 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:24.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:24.487 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:24.511 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:24.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:24.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:24.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:24.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:24.638 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:24.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:24.741 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:24.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:24.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:24.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:24.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:24.893 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:24.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:24.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:24.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:24 smithi163 ceph-mon[29621]: 5.9 scrub starts 2024-03-21T13:08:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:24 smithi163 ceph-mon[29621]: from='client.25169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:08:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:24 smithi163 ceph-mon[40675]: 5.9 scrub starts 2024-03-21T13:08:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:24 smithi163 ceph-mon[40675]: from='client.25169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:08:25.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:25.021 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:25.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:25.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:25.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:25.148 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:25.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:25.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:25.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:24 smithi204 ceph-mon[33153]: 5.9 scrub starts 2024-03-21T13:08:25.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:24 smithi204 ceph-mon[33153]: from='client.25169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:08:25.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:25.275 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:25.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:25.376 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:25.401 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:25.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:25.502 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:25.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:25.528 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:25.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:25.629 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:25.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:25.655 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:25.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:25.756 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:25.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:25.781 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:25.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:25.882 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:25.892 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:08:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:08:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:08:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:25.907 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:25.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:26.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:26.037 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:26.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:26.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:26.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:26.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:25 smithi204 ceph-mon[33153]: pgmap v426: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 15 op/s 2024-03-21T13:08:26.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:25 smithi163 ceph-mon[29621]: pgmap v426: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 15 op/s 2024-03-21T13:08:26.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:25 smithi163 ceph-mon[40675]: pgmap v426: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 15 op/s 2024-03-21T13:08:26.264 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:26.266 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:26.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:26.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:26.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:26.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:26.420 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:26.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:26.521 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:26.546 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:26.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:26.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:26.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:26.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:26.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:26.789 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:08:26.799 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-03-21T13:08:22.890086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1491,"num_read_kb":1337,"num_write":272357,"num_write_kb":8698142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49917,"num_bytes_recovered":3257146833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54269,"ondisk_log_size":54269,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":2213848,"kb_used_data":5940,"kb_used_omap":7,"kb_used_meta":2207736,"kb_avail":466511912,"statfs":{"total":479975178240,"available":477708197888,"internally_reserved":0,"allocated":6082560,"data_stored":3909075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7967,"internal_metadata":2260721889},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5898264,"num_objects":-91,"num_object_clones":0,"num_object_copies":-273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-91,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":91,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.093250"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":202,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004854+0000","last_change":"2024-03-21T13:06:45.404501+0000","last_active":"2024-03-21T13:08:16.004854+0000","last_peered":"2024-03-21T13:08:16.004854+0000","last_clean":"2024-03-21T13:08:16.004854+0000","last_became_active":"2024-03-21T13:06:45.404217+0000","last_became_peered":"2024-03-21T13:06:45.404217+0000","last_unstale":"2024-03-21T13:08:16.004854+0000","last_undegraded":"2024-03-21T13:08:16.004854+0000","last_fullsized":"2024-03-21T13:08:16.004854+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:20.083111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.083111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":242,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004660+0000","last_change":"2024-03-21T13:07:54.361035+0000","last_active":"2024-03-21T13:08:16.004660+0000","last_peered":"2024-03-21T13:08:16.004660+0000","last_clean":"2024-03-21T13:08:16.004660+0000","last_became_active":"2024-03-21T13:06:29.490568+0000","last_became_peered":"2024-03-21T13:06:29.490568+0000","last_unstale":"2024-03-21T13:08:16.004660+0000","last_undegraded":"2024-03-21T13:08:16.004660+0000","last_fullsized":"2024-03-21T13:08:16.004660+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:54.360939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:07:54.360939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:12.341943+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":76,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002947+0000","last_change":"2024-03-21T13:06:29.484964+0000","last_active":"2024-03-21T13:08:16.002947+0000","last_peered":"2024-03-21T13:08:16.002947+0000","last_clean":"2024-03-21T13:08:16.002947+0000","last_became_active":"2024-03-21T13:06:29.484528+0000","last_became_peered":"2024-03-21T13:06:29.484528+0000","last_unstale":"2024-03-21T13:08:16.002947+0000","last_undegraded":"2024-03-21T13:08:16.002947+0000","last_fullsized":"2024-03-21T13:08:16.002947+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:49.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.260693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"89'13","reported_seq":419,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.875289+0000","last_change":"2024-03-21T13:08:00.054633+0000","last_active":"2024-03-21T13:08:19.875289+0000","last_peered":"2024-03-21T13:08:19.875289+0000","last_clean":"2024-03-21T13:08:19.875289+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:08:19.875289+0000","last_undegraded":"2024-03-21T13:08:19.875289+0000","last_fullsized":"2024-03-21T13:08:19.875289+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-03-21T13:08:00.054482+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:08:00.054482+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:21.647462+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":343,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873171+0000","last_change":"2024-03-21T13:05:07.334438+0000","last_active":"2024-03-21T13:08:19.873171+0000","last_peered":"2024-03-21T13:08:19.873171+0000","last_clean":"2024-03-21T13:08:19.873171+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:08:19.873171+0000","last_undegraded":"2024-03-21T13:08:19.873171+0000","last_fullsized":"2024-03-21T13:08:19.873171+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.330296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.330296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":286,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.001820+0000","last_change":"2024-03-21T13:06:29.490705+0000","last_active":"2024-03-21T13:08:16.001820+0000","last_peered":"2024-03-21T13:08:16.001820+0000","last_clean":"2024-03-21T13:08:16.001820+0000","last_became_active":"2024-03-21T13:06:29.490004+0000","last_became_peered":"2024-03-21T13:06:29.490004+0000","last_unstale":"2024-03-21T13:08:16.001820+0000","last_undegraded":"2024-03-21T13:08:16.001820+0000","last_fullsized":"2024-03-21T13:08:16.001820+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:50.991735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.991735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":365,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873470+0000","last_change":"2024-03-21T13:06:29.542506+0000","last_active":"2024-03-21T13:08:19.873470+0000","last_peered":"2024-03-21T13:08:19.873470+0000","last_clean":"2024-03-21T13:08:19.873470+0000","last_became_active":"2024-03-21T13:06:29.511102+0000","last_became_peered":"2024-03-21T13:06:29.511102+0000","last_unstale":"2024-03-21T13:08:19.873470+0000","last_undegraded":"2024-03-21T13:08:19.873470+0000","last_fullsized":"2024-03-21T13:08:19.873470+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_clean_scrub_stamp":"2024-03-21T13:01:38.765896+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"71'15","reported_seq":398,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872169+0000","last_change":"2024-03-21T13:06:29.546224+0000","last_active":"2024-03-21T13:08:19.872169+0000","last_peered":"2024-03-21T13:08:19.872169+0000","last_clean":"2024-03-21T13:08:19.872169+0000","last_became_active":"2024-03-21T13:06:29.509791+0000","last_became_peered":"2024-03-21T13:06:29.509791+0000","last_unstale":"2024-03-21T13:08:19.872169+0000","last_undegraded":"2024-03-21T13:08:19.872169+0000","last_fullsized":"2024-03-21T13:08:19.872169+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:26.476043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.476043+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":246,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004213+0000","last_change":"2024-03-21T13:06:29.489731+0000","last_active":"2024-03-21T13:08:16.004213+0000","last_peered":"2024-03-21T13:08:16.004213+0000","last_clean":"2024-03-21T13:08:16.004213+0000","last_became_active":"2024-03-21T13:06:29.487372+0000","last_became_peered":"2024-03-21T13:06:29.487372+0000","last_unstale":"2024-03-21T13:08:16.004213+0000","last_undegraded":"2024-03-21T13:08:16.004213+0000","last_fullsized":"2024-03-21T13:08:16.004213+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:32.834646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:32.834646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":252,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.879977+0000","last_change":"2024-03-21T13:08:08.963344+0000","last_active":"2024-03-21T13:08:19.879977+0000","last_peered":"2024-03-21T13:08:19.879977+0000","last_clean":"2024-03-21T13:08:19.879977+0000","last_became_active":"2024-03-21T13:02:26.236707+0000","last_became_peered":"2024-03-21T13:02:26.236707+0000","last_unstale":"2024-03-21T13:08:19.879977+0000","last_undegraded":"2024-03-21T13:08:19.879977+0000","last_fullsized":"2024-03-21T13:08:19.879977+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:08.963164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:08:08.963164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:30.848955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":226,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002904+0000","last_change":"2024-03-21T13:05:45.989199+0000","last_active":"2024-03-21T13:08:16.002904+0000","last_peered":"2024-03-21T13:08:16.002904+0000","last_clean":"2024-03-21T13:08:16.002904+0000","last_became_active":"2024-03-21T13:02:26.245402+0000","last_became_peered":"2024-03-21T13:02:26.245402+0000","last_unstale":"2024-03-21T13:08:16.002904+0000","last_undegraded":"2024-03-21T13:08:16.002904+0000","last_fullsized":"2024-03-21T13:08:16.002904+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:45.989114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:45.989114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":345,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872713+0000","last_change":"2024-03-21T13:05:51.580203+0000","last_active":"2024-03-21T13:08:19.872713+0000","last_peered":"2024-03-21T13:08:19.872713+0000","last_clean":"2024-03-21T13:08:19.872713+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:08:19.872713+0000","last_undegraded":"2024-03-21T13:08:19.872713+0000","last_fullsized":"2024-03-21T13:08:19.872713+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-21T13:05:51.580074+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:05:51.580074+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":354,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.871552+0000","last_change":"2024-03-21T13:08:17.747461+0000","last_active":"2024-03-21T13:08:19.871552+0000","last_peered":"2024-03-21T13:08:19.871552+0000","last_clean":"2024-03-21T13:08:19.871552+0000","last_became_active":"2024-03-21T13:06:29.505469+0000","last_became_peered":"2024-03-21T13:06:29.505469+0000","last_unstale":"2024-03-21T13:08:19.871552+0000","last_undegraded":"2024-03-21T13:08:19.871552+0000","last_fullsized":"2024-03-21T13:08:19.871552+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:17.747360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:17.747360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:40.450392+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":365,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004982+0000","last_change":"2024-03-21T13:08:09.443525+0000","last_active":"2024-03-21T13:08:16.004982+0000","last_peered":"2024-03-21T13:08:16.004982+0000","last_clean":"2024-03-21T13:08:16.004982+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:08:16.004982+0000","last_undegraded":"2024-03-21T13:08:16.004982+0000","last_fullsized":"2024-03-21T13:08:16.004982+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:09.443411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:09.443411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:10.311895+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":376,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.875910+0000","last_change":"2024-03-21T13:08:17.962862+0000","last_active":"2024-03-21T13:08:19.875910+0000","last_peered":"2024-03-21T13:08:19.875910+0000","last_clean":"2024-03-21T13:08:19.875910+0000","last_became_active":"2024-03-21T13:02:26.230057+0000","last_became_peered":"2024-03-21T13:02:26.230057+0000","last_unstale":"2024-03-21T13:08:19.875910+0000","last_undegraded":"2024-03-21T13:08:19.875910+0000","last_fullsized":"2024-03-21T13:08:19.875910+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:17.962759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:17.962759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:26.948935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872440+0000","last_change":"2024-03-21T13:06:29.517935+0000","last_active":"2024-03-21T13:08:19.872440+0000","last_peered":"2024-03-21T13:08:19.872440+0000","last_clean":"2024-03-21T13:08:19.872440+0000","last_became_active":"2024-03-21T13:06:29.506772+0000","last_became_peered":"2024-03-21T13:06:29.506772+0000","last_unstale":"2024-03-21T13:08:19.872440+0000","last_undegraded":"2024-03-21T13:08:19.872440+0000","last_fullsized":"2024-03-21T13:08:19.872440+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T13:05:16.384676+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:05:16.384676+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":374,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.876947+0000","last_change":"2024-03-21T13:07:47.084827+0000","last_active":"2024-03-21T13:08:19.876947+0000","last_peered":"2024-03-21T13:08:19.876947+0000","last_clean":"2024-03-21T13:08:19.876947+0000","last_became_active":"2024-03-21T13:02:26.226675+0000","last_became_peered":"2024-03-21T13:02:26.226675+0000","last_unstale":"2024-03-21T13:08:19.876947+0000","last_undegraded":"2024-03-21T13:08:19.876947+0000","last_fullsized":"2024-03-21T13:08:19.876947+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:47.084713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:47.084713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:14.317582+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":262,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.877088+0000","last_change":"2024-03-21T13:08:18.994766+0000","last_active":"2024-03-21T13:08:19.877088+0000","last_peered":"2024-03-21T13:08:19.877088+0000","last_clean":"2024-03-21T13:08:19.877088+0000","last_became_active":"2024-03-21T13:02:26.231879+0000","last_became_peered":"2024-03-21T13:02:26.231879+0000","last_unstale":"2024-03-21T13:08:19.877088+0000","last_undegraded":"2024-03-21T13:08:19.877088+0000","last_fullsized":"2024-03-21T13:08:19.877088+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:18.994687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.994687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:31.114182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":252,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005164+0000","last_change":"2024-03-21T13:06:28.985227+0000","last_active":"2024-03-21T13:08:16.005164+0000","last_peered":"2024-03-21T13:08:16.005164+0000","last_clean":"2024-03-21T13:08:16.005164+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:08:16.005164+0000","last_undegraded":"2024-03-21T13:08:16.005164+0000","last_fullsized":"2024-03-21T13:08:16.005164+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:28.985145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:28.985145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":134,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873433+0000","last_change":"2024-03-21T13:06:29.540373+0000","last_active":"2024-03-21T13:08:19.873433+0000","last_peered":"2024-03-21T13:08:19.873433+0000","last_clean":"2024-03-21T13:08:19.873433+0000","last_became_active":"2024-03-21T13:06:29.504235+0000","last_became_peered":"2024-03-21T13:06:29.504235+0000","last_unstale":"2024-03-21T13:08:19.873433+0000","last_undegraded":"2024-03-21T13:08:19.873433+0000","last_fullsized":"2024-03-21T13:08:19.873433+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:20.158184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.158184+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":356,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873791+0000","last_change":"2024-03-21T13:07:44.816849+0000","last_active":"2024-03-21T13:08:19.873791+0000","last_peered":"2024-03-21T13:08:19.873791+0000","last_clean":"2024-03-21T13:08:19.873791+0000","last_became_active":"2024-03-21T13:06:29.512081+0000","last_became_peered":"2024-03-21T13:06:29.512081+0000","last_unstale":"2024-03-21T13:08:19.873791+0000","last_undegraded":"2024-03-21T13:08:19.873791+0000","last_fullsized":"2024-03-21T13:08:19.873791+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:44.816722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:44.816722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:06.483247+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":274,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872984+0000","last_change":"2024-03-21T13:06:29.500559+0000","last_active":"2024-03-21T13:08:19.872984+0000","last_peered":"2024-03-21T13:08:19.872984+0000","last_clean":"2024-03-21T13:08:19.872984+0000","last_became_active":"2024-03-21T13:06:29.498872+0000","last_became_peered":"2024-03-21T13:06:29.498872+0000","last_unstale":"2024-03-21T13:08:19.872984+0000","last_undegraded":"2024-03-21T13:08:19.872984+0000","last_fullsized":"2024-03-21T13:08:19.872984+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:29.464856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:04:29.464856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":383,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.875752+0000","last_change":"2024-03-21T13:06:29.488224+0000","last_active":"2024-03-21T13:08:19.875752+0000","last_peered":"2024-03-21T13:08:19.875752+0000","last_clean":"2024-03-21T13:08:19.875752+0000","last_became_active":"2024-03-21T13:06:29.487644+0000","last_became_peered":"2024-03-21T13:06:29.487644+0000","last_unstale":"2024-03-21T13:08:19.875752+0000","last_undegraded":"2024-03-21T13:08:19.875752+0000","last_fullsized":"2024-03-21T13:08:19.875752+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.354575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:54.948081+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.354575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'12","reported_seq":402,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872945+0000","last_change":"2024-03-21T13:02:14.362313+0000","last_active":"2024-03-21T13:08:19.872945+0000","last_peered":"2024-03-21T13:08:19.872945+0000","last_clean":"2024-03-21T13:08:19.872945+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:08:19.872945+0000","last_undegraded":"2024-03-21T13:08:19.872945+0000","last_fullsized":"2024-03-21T13:08:19.872945+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:01:39.755345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.755345+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":209,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872407+0000","last_change":"2024-03-21T13:06:29.537271+0000","last_active":"2024-03-21T13:08:19.872407+0000","last_peered":"2024-03-21T13:08:19.872407+0000","last_clean":"2024-03-21T13:08:19.872407+0000","last_became_active":"2024-03-21T13:06:29.504570+0000","last_became_peered":"2024-03-21T13:06:29.504570+0000","last_unstale":"2024-03-21T13:08:19.872407+0000","last_undegraded":"2024-03-21T13:08:19.872407+0000","last_fullsized":"2024-03-21T13:08:19.872407+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:50.295057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.295057+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":482,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.003109+0000","last_change":"2024-03-21T13:05:38.022572+0000","last_active":"2024-03-21T13:08:16.003109+0000","last_peered":"2024-03-21T13:08:16.003109+0000","last_clean":"2024-03-21T13:08:16.003109+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:08:16.003109+0000","last_undegraded":"2024-03-21T13:08:16.003109+0000","last_fullsized":"2024-03-21T13:08:16.003109+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:05:38.022439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.022439+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":254,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.871726+0000","last_change":"2024-03-21T13:04:20.361810+0000","last_active":"2024-03-21T13:08:19.871726+0000","last_peered":"2024-03-21T13:08:19.871726+0000","last_clean":"2024-03-21T13:08:19.871726+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:08:19.871726+0000","last_undegraded":"2024-03-21T13:08:19.871726+0000","last_fullsized":"2024-03-21T13:08:19.871726+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:01:37.244557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:37.244557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"71'10","reported_seq":287,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.006073+0000","last_change":"2024-03-21T13:05:21.117502+0000","last_active":"2024-03-21T13:08:16.006073+0000","last_peered":"2024-03-21T13:08:16.006073+0000","last_clean":"2024-03-21T13:08:16.006073+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:08:16.006073+0000","last_undegraded":"2024-03-21T13:08:16.006073+0000","last_fullsized":"2024-03-21T13:08:16.006073+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:21.117370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.117370+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":244,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.876424+0000","last_change":"2024-03-21T13:06:29.499863+0000","last_active":"2024-03-21T13:08:19.876424+0000","last_peered":"2024-03-21T13:08:19.876424+0000","last_clean":"2024-03-21T13:08:19.876424+0000","last_became_active":"2024-03-21T13:06:29.499497+0000","last_became_peered":"2024-03-21T13:06:29.499497+0000","last_unstale":"2024-03-21T13:08:19.876424+0000","last_undegraded":"2024-03-21T13:08:19.876424+0000","last_fullsized":"2024-03-21T13:08:19.876424+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:26.498073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:06:26.498073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":259,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005784+0000","last_change":"2024-03-21T13:06:29.501937+0000","last_active":"2024-03-21T13:08:16.005784+0000","last_peered":"2024-03-21T13:08:16.005784+0000","last_clean":"2024-03-21T13:08:16.005784+0000","last_became_active":"2024-03-21T13:06:29.501232+0000","last_became_peered":"2024-03-21T13:06:29.501232+0000","last_unstale":"2024-03-21T13:08:16.005784+0000","last_undegraded":"2024-03-21T13:08:16.005784+0000","last_fullsized":"2024-03-21T13:08:16.005784+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.088812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.088812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":367,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.871670+0000","last_change":"2024-03-21T13:06:29.501866+0000","last_active":"2024-03-21T13:08:19.871670+0000","last_peered":"2024-03-21T13:08:19.871670+0000","last_clean":"2024-03-21T13:08:19.871670+0000","last_became_active":"2024-03-21T13:06:29.501471+0000","last_became_peered":"2024-03-21T13:06:29.501471+0000","last_unstale":"2024-03-21T13:08:19.871670+0000","last_undegraded":"2024-03-21T13:08:19.871670+0000","last_fullsized":"2024-03-21T13:08:19.871670+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:09.348646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:09.348646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'11","reported_seq":413,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.003129+0000","last_change":"2024-03-21T13:05:49.982963+0000","last_active":"2024-03-21T13:08:16.003129+0000","last_peered":"2024-03-21T13:08:16.003129+0000","last_clean":"2024-03-21T13:08:16.003129+0000","last_became_active":"2024-03-21T13:02:26.254436+0000","last_became_peered":"2024-03-21T13:02:26.254436+0000","last_unstale":"2024-03-21T13:08:16.003129+0000","last_undegraded":"2024-03-21T13:08:16.003129+0000","last_fullsized":"2024-03-21T13:08:16.003129+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:49.982812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.982812+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":354,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005025+0000","last_change":"2024-03-21T13:06:29.980827+0000","last_active":"2024-03-21T13:08:16.005025+0000","last_peered":"2024-03-21T13:08:16.005025+0000","last_clean":"2024-03-21T13:08:16.005025+0000","last_became_active":"2024-03-21T13:06:29.503679+0000","last_became_peered":"2024-03-21T13:06:29.503679+0000","last_unstale":"2024-03-21T13:08:16.005025+0000","last_undegraded":"2024-03-21T13:08:16.005025+0000","last_fullsized":"2024-03-21T13:08:16.005025+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:29.980750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:06:29.980750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":354,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002641+0000","last_change":"2024-03-21T13:05:42.990138+0000","last_active":"2024-03-21T13:08:16.002641+0000","last_peered":"2024-03-21T13:08:16.002641+0000","last_clean":"2024-03-21T13:08:16.002641+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:08:16.002641+0000","last_undegraded":"2024-03-21T13:08:16.002641+0000","last_fullsized":"2024-03-21T13:08:16.002641+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:42.990022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.990022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":230,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873944+0000","last_change":"2024-03-21T13:05:28.499322+0000","last_active":"2024-03-21T13:08:19.873944+0000","last_peered":"2024-03-21T13:08:19.873944+0000","last_clean":"2024-03-21T13:08:19.873944+0000","last_became_active":"2024-03-21T13:02:26.242973+0000","last_became_peered":"2024-03-21T13:02:26.242973+0000","last_unstale":"2024-03-21T13:08:19.873944+0000","last_undegraded":"2024-03-21T13:08:19.873944+0000","last_fullsized":"2024-03-21T13:08:19.873944+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.499185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.499185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":370,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.871799+0000","last_change":"2024-03-21T13:08:18.740907+0000","last_active":"2024-03-21T13:08:19.871799+0000","last_peered":"2024-03-21T13:08:19.871799+0000","last_clean":"2024-03-21T13:08:19.871799+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:08:19.871799+0000","last_undegraded":"2024-03-21T13:08:19.871799+0000","last_fullsized":"2024-03-21T13:08:19.871799+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-21T13:08:18.740815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.740815+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:26.851401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":344,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873357+0000","last_change":"2024-03-21T13:05:19.431337+0000","last_active":"2024-03-21T13:08:19.873357+0000","last_peered":"2024-03-21T13:08:19.873357+0000","last_clean":"2024-03-21T13:08:19.873357+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:08:19.873357+0000","last_undegraded":"2024-03-21T13:08:19.873357+0000","last_fullsized":"2024-03-21T13:08:19.873357+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:19.430947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:19.430947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":352,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004520+0000","last_change":"2024-03-21T13:08:03.435009+0000","last_active":"2024-03-21T13:08:16.004520+0000","last_peered":"2024-03-21T13:08:16.004520+0000","last_clean":"2024-03-21T13:08:16.004520+0000","last_became_active":"2024-03-21T12:58:41.164784+0000","last_became_peered":"2024-03-21T12:58:41.164784+0000","last_unstale":"2024-03-21T13:08:16.004520+0000","last_undegraded":"2024-03-21T13:08:16.004520+0000","last_fullsized":"2024-03-21T13:08:16.004520+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:03.434682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:03.434682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:31.140816+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":356,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005668+0000","last_change":"2024-03-21T13:07:50.428363+0000","last_active":"2024-03-21T13:08:16.005668+0000","last_peered":"2024-03-21T13:08:16.005668+0000","last_clean":"2024-03-21T13:08:16.005668+0000","last_became_active":"2024-03-21T13:06:29.501190+0000","last_became_peered":"2024-03-21T13:06:29.501190+0000","last_unstale":"2024-03-21T13:08:16.005668+0000","last_undegraded":"2024-03-21T13:08:16.005668+0000","last_fullsized":"2024-03-21T13:08:16.005668+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:50.428251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:50.428251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:18.954380+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"71'11","reported_seq":85,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.871980+0000","last_change":"2024-03-21T13:06:29.532046+0000","last_active":"2024-03-21T13:08:19.871980+0000","last_peered":"2024-03-21T13:08:19.871980+0000","last_clean":"2024-03-21T13:08:19.871980+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:08:19.871980+0000","last_undegraded":"2024-03-21T13:08:19.871980+0000","last_fullsized":"2024-03-21T13:08:19.871980+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:40.262277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:40.262277+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":249,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.001698+0000","last_change":"2024-03-21T13:05:16.850442+0000","last_active":"2024-03-21T13:08:16.001698+0000","last_peered":"2024-03-21T13:08:16.001698+0000","last_clean":"2024-03-21T13:08:16.001698+0000","last_became_active":"2024-03-21T13:02:26.252488+0000","last_became_peered":"2024-03-21T13:02:26.252488+0000","last_unstale":"2024-03-21T13:08:16.001698+0000","last_undegraded":"2024-03-21T13:08:16.001698+0000","last_fullsized":"2024-03-21T13:08:16.001698+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:16.850307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:16.850307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":360,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873461+0000","last_change":"2024-03-21T13:05:31.496412+0000","last_active":"2024-03-21T13:08:19.873461+0000","last_peered":"2024-03-21T13:08:19.873461+0000","last_clean":"2024-03-21T13:08:19.873461+0000","last_became_active":"2024-03-21T13:02:26.232682+0000","last_became_peered":"2024-03-21T13:02:26.232682+0000","last_unstale":"2024-03-21T13:08:19.873461+0000","last_undegraded":"2024-03-21T13:08:19.873461+0000","last_fullsized":"2024-03-21T13:08:19.873461+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.495639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.495639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":360,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005346+0000","last_change":"2024-03-21T13:07:48.347291+0000","last_active":"2024-03-21T13:08:16.005346+0000","last_peered":"2024-03-21T13:08:16.005346+0000","last_clean":"2024-03-21T13:08:16.005346+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:08:16.005346+0000","last_undegraded":"2024-03-21T13:08:16.005346+0000","last_fullsized":"2024-03-21T13:08:16.005346+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:48.347175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:48.347175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:16.886257+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"86'13","reported_seq":333,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002694+0000","last_change":"2024-03-21T13:06:29.518281+0000","last_active":"2024-03-21T13:08:16.002694+0000","last_peered":"2024-03-21T13:08:16.002694+0000","last_clean":"2024-03-21T13:08:16.002694+0000","last_became_active":"2024-03-21T13:06:29.496686+0000","last_became_peered":"2024-03-21T13:06:29.496686+0000","last_unstale":"2024-03-21T13:08:16.002694+0000","last_undegraded":"2024-03-21T13:08:16.002694+0000","last_fullsized":"2024-03-21T13:08:16.002694+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:05:34.036430+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.036430+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":187,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004426+0000","last_change":"2024-03-21T13:05:29.031235+0000","last_active":"2024-03-21T13:08:16.004426+0000","last_peered":"2024-03-21T13:08:16.004426+0000","last_clean":"2024-03-21T13:08:16.004426+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:08:16.004426+0000","last_undegraded":"2024-03-21T13:08:16.004426+0000","last_fullsized":"2024-03-21T13:08:16.004426+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.031062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.031062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":230,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873108+0000","last_change":"2024-03-21T13:08:05.697231+0000","last_active":"2024-03-21T13:08:19.873108+0000","last_peered":"2024-03-21T13:08:19.873108+0000","last_clean":"2024-03-21T13:08:19.873108+0000","last_became_active":"2024-03-21T13:02:26.242491+0000","last_became_peered":"2024-03-21T13:02:26.242491+0000","last_unstale":"2024-03-21T13:08:19.873108+0000","last_undegraded":"2024-03-21T13:08:19.873108+0000","last_fullsized":"2024-03-21T13:08:19.873108+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:05.697141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:05.697141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:28.363142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":72,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:18.389506+0000","last_change":"2024-03-21T13:08:18.389543+0000","last_active":"2024-03-21T13:08:18.389506+0000","last_peered":"2024-03-21T13:08:18.389506+0000","last_clean":"2024-03-21T13:08:18.389506+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:08:18.389506+0000","last_undegraded":"2024-03-21T13:08:18.389506+0000","last_fullsized":"2024-03-21T13:08:18.389506+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:18.389423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.389423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:18.843144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":441,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.876733+0000","last_change":"2024-03-21T13:08:16.007543+0000","last_active":"2024-03-21T13:08:19.876733+0000","last_peered":"2024-03-21T13:08:19.876733+0000","last_clean":"2024-03-21T13:08:19.876733+0000","last_became_active":"2024-03-21T13:06:29.487725+0000","last_became_peered":"2024-03-21T13:06:29.487725+0000","last_unstale":"2024-03-21T13:08:19.876733+0000","last_undegraded":"2024-03-21T13:08:19.876733+0000","last_fullsized":"2024-03-21T13:08:19.876733+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:08:16.007433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:16.007433+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:41.300649+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":252,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005669+0000","last_change":"2024-03-21T13:04:31.805808+0000","last_active":"2024-03-21T13:08:16.005669+0000","last_peered":"2024-03-21T13:08:16.005669+0000","last_clean":"2024-03-21T13:08:16.005669+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:08:16.005669+0000","last_undegraded":"2024-03-21T13:08:16.005669+0000","last_fullsized":"2024-03-21T13:08:16.005669+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:31.805690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:31.805690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"125'10","reported_seq":482,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872479+0000","last_change":"2024-03-21T13:08:15.777606+0000","last_active":"2024-03-21T13:08:19.872479+0000","last_peered":"2024-03-21T13:08:19.872479+0000","last_clean":"2024-03-21T13:08:19.872479+0000","last_became_active":"2024-03-21T13:02:26.234494+0000","last_became_peered":"2024-03-21T13:02:26.234494+0000","last_unstale":"2024-03-21T13:08:19.872479+0000","last_undegraded":"2024-03-21T13:08:19.872479+0000","last_fullsized":"2024-03-21T13:08:19.872479+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:08:15.777522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:15.777522+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:35.180586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":366,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005028+0000","last_change":"2024-03-21T13:06:29.489915+0000","last_active":"2024-03-21T13:08:16.005028+0000","last_peered":"2024-03-21T13:08:16.005028+0000","last_clean":"2024-03-21T13:08:16.005028+0000","last_became_active":"2024-03-21T13:06:29.489392+0000","last_became_peered":"2024-03-21T13:06:29.489392+0000","last_unstale":"2024-03-21T13:08:16.005028+0000","last_undegraded":"2024-03-21T13:08:16.005028+0000","last_fullsized":"2024-03-21T13:08:16.005028+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.042372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.042372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"71'10","reported_seq":403,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873554+0000","last_change":"2024-03-21T13:06:29.530165+0000","last_active":"2024-03-21T13:08:19.873554+0000","last_peered":"2024-03-21T13:08:19.873554+0000","last_clean":"2024-03-21T13:08:19.873554+0000","last_became_active":"2024-03-21T13:06:29.506163+0000","last_became_peered":"2024-03-21T13:06:29.506163+0000","last_unstale":"2024-03-21T13:08:19.873554+0000","last_undegraded":"2024-03-21T13:08:19.873554+0000","last_fullsized":"2024-03-21T13:08:19.873554+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:23.379004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.379004+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":356,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005481+0000","last_change":"2024-03-21T13:06:45.402756+0000","last_active":"2024-03-21T13:08:16.005481+0000","last_peered":"2024-03-21T13:08:16.005481+0000","last_clean":"2024-03-21T13:08:16.005481+0000","last_became_active":"2024-03-21T13:06:45.402482+0000","last_became_peered":"2024-03-21T13:06:45.402482+0000","last_unstale":"2024-03-21T13:08:16.005481+0000","last_undegraded":"2024-03-21T13:08:16.005481+0000","last_fullsized":"2024-03-21T13:08:16.005481+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:21.157295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.157295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":484,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873928+0000","last_change":"2024-03-21T13:07:52.673401+0000","last_active":"2024-03-21T13:08:19.873928+0000","last_peered":"2024-03-21T13:08:19.873928+0000","last_clean":"2024-03-21T13:08:19.873928+0000","last_became_active":"2024-03-21T13:06:29.507429+0000","last_became_peered":"2024-03-21T13:06:29.507429+0000","last_unstale":"2024-03-21T13:08:19.873928+0000","last_undegraded":"2024-03-21T13:08:19.873928+0000","last_fullsized":"2024-03-21T13:08:19.873928+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-03-21T13:07:52.673289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:07:52.673289+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:08:53.864669+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":252,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004885+0000","last_change":"2024-03-21T13:04:12.819697+0000","last_active":"2024-03-21T13:08:16.004885+0000","last_peered":"2024-03-21T13:08:16.004885+0000","last_clean":"2024-03-21T13:08:16.004885+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:08:16.004885+0000","last_undegraded":"2024-03-21T13:08:16.004885+0000","last_fullsized":"2024-03-21T13:08:16.004885+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:12.819556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:04:12.819556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"71'9","reported_seq":279,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005747+0000","last_change":"2024-03-21T13:08:08.413361+0000","last_active":"2024-03-21T13:08:16.005747+0000","last_peered":"2024-03-21T13:08:16.005747+0000","last_clean":"2024-03-21T13:08:16.005747+0000","last_became_active":"2024-03-21T13:02:26.239161+0000","last_became_peered":"2024-03-21T13:02:26.239161+0000","last_unstale":"2024-03-21T13:08:16.005747+0000","last_undegraded":"2024-03-21T13:08:16.005747+0000","last_fullsized":"2024-03-21T13:08:16.005747+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:08:08.413234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:08.413234+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:37.899511+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":74,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.876136+0000","last_change":"2024-03-21T13:06:29.489166+0000","last_active":"2024-03-21T13:08:19.876136+0000","last_peered":"2024-03-21T13:08:19.876136+0000","last_clean":"2024-03-21T13:08:19.876136+0000","last_became_active":"2024-03-21T13:06:29.488783+0000","last_became_peered":"2024-03-21T13:06:29.488783+0000","last_unstale":"2024-03-21T13:08:19.876136+0000","last_undegraded":"2024-03-21T13:08:19.876136+0000","last_fullsized":"2024-03-21T13:08:19.876136+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:42.187461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.187461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":354,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.874250+0000","last_change":"2024-03-21T13:05:17.347764+0000","last_active":"2024-03-21T13:08:19.874250+0000","last_peered":"2024-03-21T13:08:19.874250+0000","last_clean":"2024-03-21T13:08:19.874250+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:08:19.874250+0000","last_undegraded":"2024-03-21T13:08:19.874250+0000","last_fullsized":"2024-03-21T13:08:19.874250+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:17.347604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:17.347604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":366,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004553+0000","last_change":"2024-03-21T13:05:23.049951+0000","last_active":"2024-03-21T13:08:16.004553+0000","last_peered":"2024-03-21T13:08:16.004553+0000","last_clean":"2024-03-21T13:08:16.004553+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:08:16.004553+0000","last_undegraded":"2024-03-21T13:08:16.004553+0000","last_fullsized":"2024-03-21T13:08:16.004553+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:05:23.049811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.049811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'9","reported_seq":281,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873071+0000","last_change":"2024-03-21T13:01:48.210983+0000","last_active":"2024-03-21T13:08:19.873071+0000","last_peered":"2024-03-21T13:08:19.873071+0000","last_clean":"2024-03-21T13:08:19.873071+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:08:19.873071+0000","last_undegraded":"2024-03-21T13:08:19.873071+0000","last_fullsized":"2024-03-21T13:08:19.873071+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:01:40.964615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.964615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":347,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005183+0000","last_change":"2024-03-21T13:07:52.425295+0000","last_active":"2024-03-21T13:08:16.005183+0000","last_peered":"2024-03-21T13:08:16.005183+0000","last_clean":"2024-03-21T13:08:16.005183+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:08:16.005183+0000","last_undegraded":"2024-03-21T13:08:16.005183+0000","last_fullsized":"2024-03-21T13:08:16.005183+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:52.424928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:52.424928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:11.921996+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":344,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873667+0000","last_change":"2024-03-21T13:05:25.466812+0000","last_active":"2024-03-21T13:08:19.873667+0000","last_peered":"2024-03-21T13:08:19.873667+0000","last_clean":"2024-03-21T13:08:19.873667+0000","last_became_active":"2024-03-21T13:02:26.242502+0000","last_became_peered":"2024-03-21T13:02:26.242502+0000","last_unstale":"2024-03-21T13:08:19.873667+0000","last_undegraded":"2024-03-21T13:08:19.873667+0000","last_fullsized":"2024-03-21T13:08:19.873667+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:25.466695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:25.466695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":368,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.001814+0000","last_change":"2024-03-21T13:05:41.004652+0000","last_active":"2024-03-21T13:08:16.001814+0000","last_peered":"2024-03-21T13:08:16.001814+0000","last_clean":"2024-03-21T13:08:16.001814+0000","last_became_active":"2024-03-21T13:02:26.252406+0000","last_became_peered":"2024-03-21T13:02:26.252406+0000","last_unstale":"2024-03-21T13:08:16.001814+0000","last_undegraded":"2024-03-21T13:08:16.001814+0000","last_fullsized":"2024-03-21T13:08:16.001814+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.004412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.004412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'6","reported_seq":389,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.874152+0000","last_change":"2024-03-21T13:06:29.546967+0000","last_active":"2024-03-21T13:08:19.874152+0000","last_peered":"2024-03-21T13:08:19.874152+0000","last_clean":"2024-03-21T13:08:19.874152+0000","last_became_active":"2024-03-21T13:06:29.513331+0000","last_became_peered":"2024-03-21T13:06:29.513331+0000","last_unstale":"2024-03-21T13:08:19.874152+0000","last_undegraded":"2024-03-21T13:08:19.874152+0000","last_fullsized":"2024-03-21T13:08:19.874152+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T13:04:22.563861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:22.563861+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"225'17032","reported_seq":29908,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.875793+0000","last_change":"2024-03-21T13:07:52.899284+0000","last_active":"2024-03-21T13:08:19.875793+0000","last_peered":"2024-03-21T13:08:19.875793+0000","last_clean":"2024-03-21T13:08:19.875793+0000","last_became_active":"2024-03-21T13:07:23.424956+0000","last_became_peered":"2024-03-21T13:07:23.424956+0000","last_unstale":"2024-03-21T13:08:19.875793+0000","last_undegraded":"2024-03-21T13:08:19.875793+0000","last_fullsized":"2024-03-21T13:08:19.875793+0000","mapping_epoch":209,"log_start":"218'13648","ondisk_log_start":"218'13648","created":168,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'14875","last_scrub_stamp":"2024-03-21T13:07:52.899024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:07:52.899024+0000","objects_scrubbed":2203,"log_size":3384,"log_dups_size":0,"ondisk_log_size":3384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:08:55.572271+0000","scrub_duration":775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17032,"num_write_kb":545024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6351,"num_bytes_recovered":416219136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":232,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002043+0000","last_change":"2024-03-21T13:06:29.509970+0000","last_active":"2024-03-21T13:08:16.002043+0000","last_peered":"2024-03-21T13:08:16.002043+0000","last_clean":"2024-03-21T13:08:16.002043+0000","last_became_active":"2024-03-21T13:06:29.509616+0000","last_became_peered":"2024-03-21T13:06:29.509616+0000","last_unstale":"2024-03-21T13:08:16.002043+0000","last_undegraded":"2024-03-21T13:08:16.002043+0000","last_fullsized":"2024-03-21T13:08:16.002043+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.973796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.973796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":396,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005731+0000","last_change":"2024-03-21T13:06:29.498660+0000","last_active":"2024-03-21T13:08:16.005731+0000","last_peered":"2024-03-21T13:08:16.005731+0000","last_clean":"2024-03-21T13:08:16.005731+0000","last_became_active":"2024-03-21T13:06:29.487439+0000","last_became_peered":"2024-03-21T13:06:29.487439+0000","last_unstale":"2024-03-21T13:08:16.005731+0000","last_undegraded":"2024-03-21T13:08:16.005731+0000","last_fullsized":"2024-03-21T13:08:16.005731+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:05:44.816353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:44.816353+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'15","reported_seq":294,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004498+0000","last_change":"2024-03-21T13:08:10.436112+0000","last_active":"2024-03-21T13:08:16.004498+0000","last_peered":"2024-03-21T13:08:16.004498+0000","last_clean":"2024-03-21T13:08:16.004498+0000","last_became_active":"2024-03-21T13:02:26.237800+0000","last_became_peered":"2024-03-21T13:02:26.237800+0000","last_unstale":"2024-03-21T13:08:16.004498+0000","last_undegraded":"2024-03-21T13:08:16.004498+0000","last_fullsized":"2024-03-21T13:08:16.004498+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:08:10.435742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:10.435742+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:37.629108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":470,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004677+0000","last_change":"2024-03-21T13:04:16.810359+0000","last_active":"2024-03-21T13:08:16.004677+0000","last_peered":"2024-03-21T13:08:16.004677+0000","last_clean":"2024-03-21T13:08:16.004677+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:08:16.004677+0000","last_undegraded":"2024-03-21T13:08:16.004677+0000","last_fullsized":"2024-03-21T13:08:16.004677+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:04:16.810269+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:04:16.810269+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"225'16806","reported_seq":16944,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002852+0000","last_change":"2024-03-21T13:08:12.301921+0000","last_active":"2024-03-21T13:08:16.002852+0000","last_peered":"2024-03-21T13:08:16.002852+0000","last_clean":"2024-03-21T13:08:16.002852+0000","last_became_active":"2024-03-21T13:04:46.258930+0000","last_became_peered":"2024-03-21T13:04:46.258930+0000","last_unstale":"2024-03-21T13:08:16.002852+0000","last_undegraded":"2024-03-21T13:08:16.002852+0000","last_fullsized":"2024-03-21T13:08:16.002852+0000","mapping_epoch":168,"log_start":"218'13243","ondisk_log_start":"218'13243","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'16546","last_scrub_stamp":"2024-03-21T13:08:12.301751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:08:12.301751+0000","objects_scrubbed":268,"log_size":3563,"log_dups_size":0,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:21.598383+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16806,"num_write_kb":537792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":350,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873263+0000","last_change":"2024-03-21T13:05:43.596914+0000","last_active":"2024-03-21T13:08:19.873263+0000","last_peered":"2024-03-21T13:08:19.873263+0000","last_clean":"2024-03-21T13:08:19.873263+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:08:19.873263+0000","last_undegraded":"2024-03-21T13:08:19.873263+0000","last_fullsized":"2024-03-21T13:08:19.873263+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.596691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":366,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005750+0000","last_change":"2024-03-21T13:05:09.138059+0000","last_active":"2024-03-21T13:08:16.005750+0000","last_peered":"2024-03-21T13:08:16.005750+0000","last_clean":"2024-03-21T13:08:16.005750+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:08:16.005750+0000","last_undegraded":"2024-03-21T13:08:16.005750+0000","last_fullsized":"2024-03-21T13:08:16.005750+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:09.137868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:09.137868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"71'12","reported_seq":294,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.871865+0000","last_change":"2024-03-21T13:07:48.778302+0000","last_active":"2024-03-21T13:08:19.871865+0000","last_peered":"2024-03-21T13:08:19.871865+0000","last_clean":"2024-03-21T13:08:19.871865+0000","last_became_active":"2024-03-21T13:02:14.347102+0000","last_became_peered":"2024-03-21T13:02:14.347102+0000","last_unstale":"2024-03-21T13:08:19.871865+0000","last_undegraded":"2024-03-21T13:08:19.871865+0000","last_fullsized":"2024-03-21T13:08:19.871865+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:07:48.778145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:07:48.778145+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:08:54.935237+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":225,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.875125+0000","last_change":"2024-03-21T13:07:59.084839+0000","last_active":"2024-03-21T13:08:19.875125+0000","last_peered":"2024-03-21T13:08:19.875125+0000","last_clean":"2024-03-21T13:08:19.875125+0000","last_became_active":"2024-03-21T13:06:29.486743+0000","last_became_peered":"2024-03-21T13:06:29.486743+0000","last_unstale":"2024-03-21T13:08:19.875125+0000","last_undegraded":"2024-03-21T13:08:19.875125+0000","last_fullsized":"2024-03-21T13:08:19.875125+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:59.082703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:07:59.082703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:09.013939+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"225'17078","reported_seq":17220,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002324+0000","last_change":"2024-03-21T13:08:09.468194+0000","last_active":"2024-03-21T13:08:16.002324+0000","last_peered":"2024-03-21T13:08:16.002324+0000","last_clean":"2024-03-21T13:08:16.002324+0000","last_became_active":"2024-03-21T13:04:46.257488+0000","last_became_peered":"2024-03-21T13:04:46.257488+0000","last_unstale":"2024-03-21T13:08:16.002324+0000","last_undegraded":"2024-03-21T13:08:16.002324+0000","last_fullsized":"2024-03-21T13:08:16.002324+0000","mapping_epoch":168,"log_start":"218'13497","ondisk_log_start":"218'13497","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'16503","last_scrub_stamp":"2024-03-21T13:08:09.467928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:08:09.467928+0000","objects_scrubbed":584,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:14.486431+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17078,"num_write_kb":546496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":362,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004223+0000","last_change":"2024-03-21T13:08:14.371833+0000","last_active":"2024-03-21T13:08:16.004223+0000","last_peered":"2024-03-21T13:08:16.004223+0000","last_clean":"2024-03-21T13:08:16.004223+0000","last_became_active":"2024-03-21T13:06:29.503980+0000","last_became_peered":"2024-03-21T13:06:29.503980+0000","last_unstale":"2024-03-21T13:08:16.004223+0000","last_undegraded":"2024-03-21T13:08:16.004223+0000","last_fullsized":"2024-03-21T13:08:16.004223+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:08:14.371718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:15.534667+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":209,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004970+0000","last_change":"2024-03-21T13:06:29.503568+0000","last_active":"2024-03-21T13:08:16.004970+0000","last_peered":"2024-03-21T13:08:16.004970+0000","last_clean":"2024-03-21T13:08:16.004970+0000","last_became_active":"2024-03-21T13:06:29.503252+0000","last_became_peered":"2024-03-21T13:06:29.503252+0000","last_unstale":"2024-03-21T13:08:16.004970+0000","last_undegraded":"2024-03-21T13:08:16.004970+0000","last_fullsized":"2024-03-21T13:08:16.004970+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.216547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.216547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":366,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005659+0000","last_change":"2024-03-21T13:05:22.078652+0000","last_active":"2024-03-21T13:08:16.005659+0000","last_peered":"2024-03-21T13:08:16.005659+0000","last_clean":"2024-03-21T13:08:16.005659+0000","last_became_active":"2024-03-21T13:02:26.236780+0000","last_became_peered":"2024-03-21T13:02:26.236780+0000","last_unstale":"2024-03-21T13:08:16.005659+0000","last_undegraded":"2024-03-21T13:08:16.005659+0000","last_fullsized":"2024-03-21T13:08:16.005659+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:22.078546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.078546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"71'19","reported_seq":441,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002299+0000","last_change":"2024-03-21T13:05:29.992376+0000","last_active":"2024-03-21T13:08:16.002299+0000","last_peered":"2024-03-21T13:08:16.002299+0000","last_clean":"2024-03-21T13:08:16.002299+0000","last_became_active":"2024-03-21T13:02:26.240350+0000","last_became_peered":"2024-03-21T13:02:26.240350+0000","last_unstale":"2024-03-21T13:08:16.002299+0000","last_undegraded":"2024-03-21T13:08:16.002299+0000","last_fullsized":"2024-03-21T13:08:16.002299+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:05:29.992207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.992207+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"225'16738","reported_seq":28745,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873599+0000","last_change":"2024-03-21T13:07:26.326903+0000","last_active":"2024-03-21T13:08:19.873599+0000","last_peered":"2024-03-21T13:08:19.873599+0000","last_clean":"2024-03-21T13:08:19.873599+0000","last_became_active":"2024-03-21T13:07:26.308387+0000","last_became_peered":"2024-03-21T13:07:26.308387+0000","last_unstale":"2024-03-21T13:08:19.873599+0000","last_undegraded":"2024-03-21T13:08:19.873599+0000","last_fullsized":"2024-03-21T13:08:19.873599+0000","mapping_epoch":212,"log_start":"218'13679","ondisk_log_start":"218'13679","created":168,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3059,"log_dups_size":0,"ondisk_log_size":3059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16738,"num_write_kb":535616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5932,"num_bytes_recovered":388759552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":259,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004481+0000","last_change":"2024-03-21T13:06:29.501377+0000","last_active":"2024-03-21T13:08:16.004481+0000","last_peered":"2024-03-21T13:08:16.004481+0000","last_clean":"2024-03-21T13:08:16.004481+0000","last_became_active":"2024-03-21T13:06:29.500961+0000","last_became_peered":"2024-03-21T13:06:29.500961+0000","last_unstale":"2024-03-21T13:08:16.004481+0000","last_undegraded":"2024-03-21T13:08:16.004481+0000","last_fullsized":"2024-03-21T13:08:16.004481+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.926828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.926828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":82,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.876150+0000","last_change":"2024-03-21T13:08:12.031223+0000","last_active":"2024-03-21T13:08:19.876150+0000","last_peered":"2024-03-21T13:08:19.876150+0000","last_clean":"2024-03-21T13:08:19.876150+0000","last_became_active":"2024-03-21T13:06:29.502745+0000","last_became_peered":"2024-03-21T13:06:29.502745+0000","last_unstale":"2024-03-21T13:08:19.876150+0000","last_undegraded":"2024-03-21T13:08:19.876150+0000","last_fullsized":"2024-03-21T13:08:19.876150+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:12.031099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:12.031099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:13.044020+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":220,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873334+0000","last_change":"2024-03-21T13:02:26.244355+0000","last_active":"2024-03-21T13:08:19.873334+0000","last_peered":"2024-03-21T13:08:19.873334+0000","last_clean":"2024-03-21T13:08:19.873334+0000","last_became_active":"2024-03-21T13:02:26.243974+0000","last_became_peered":"2024-03-21T13:02:26.243974+0000","last_unstale":"2024-03-21T13:08:19.873334+0000","last_undegraded":"2024-03-21T13:08:19.873334+0000","last_fullsized":"2024-03-21T13:08:19.873334+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:33.998382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:01:33.998382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"86'13","reported_seq":326,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.876458+0000","last_change":"2024-03-21T13:07:58.117288+0000","last_active":"2024-03-21T13:08:19.876458+0000","last_peered":"2024-03-21T13:08:19.876458+0000","last_clean":"2024-03-21T13:08:19.876458+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:08:19.876458+0000","last_undegraded":"2024-03-21T13:08:19.876458+0000","last_fullsized":"2024-03-21T13:08:19.876458+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:07:58.117164+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:07:58.117164+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:18.674340+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.8","version":"225'17012","reported_seq":17233,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.871265+0000","last_change":"2024-03-21T13:06:29.535965+0000","last_active":"2024-03-21T13:08:19.871265+0000","last_peered":"2024-03-21T13:08:19.871265+0000","last_clean":"2024-03-21T13:08:19.871265+0000","last_became_active":"2024-03-21T13:04:46.257160+0000","last_became_peered":"2024-03-21T13:04:46.257160+0000","last_unstale":"2024-03-21T13:08:19.871265+0000","last_undegraded":"2024-03-21T13:08:19.871265+0000","last_fullsized":"2024-03-21T13:08:19.871265+0000","mapping_epoch":168,"log_start":"218'13923","ondisk_log_start":"218'13923","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'8553","last_scrub_stamp":"2024-03-21T13:06:29.535762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:06:29.535762+0000","objects_scrubbed":8459,"log_size":3089,"log_dups_size":0,"ondisk_log_size":3089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17012,"num_write_kb":544384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":336,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873680+0000","last_change":"2024-03-21T13:01:46.738822+0000","last_active":"2024-03-21T13:08:19.873680+0000","last_peered":"2024-03-21T13:08:19.873680+0000","last_clean":"2024-03-21T13:08:19.873680+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:08:19.873680+0000","last_undegraded":"2024-03-21T13:08:19.873680+0000","last_fullsized":"2024-03-21T13:08:19.873680+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.738659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":257,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002096+0000","last_change":"2024-03-21T13:05:43.961181+0000","last_active":"2024-03-21T13:08:16.002096+0000","last_peered":"2024-03-21T13:08:16.002096+0000","last_clean":"2024-03-21T13:08:16.002096+0000","last_became_active":"2024-03-21T13:02:26.243692+0000","last_became_peered":"2024-03-21T13:02:26.243692+0000","last_unstale":"2024-03-21T13:08:16.002096+0000","last_undegraded":"2024-03-21T13:08:16.002096+0000","last_fullsized":"2024-03-21T13:08:16.002096+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.961057+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":64,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002065+0000","last_change":"2024-03-21T13:06:45.410372+0000","last_active":"2024-03-21T13:08:16.002065+0000","last_peered":"2024-03-21T13:08:16.002065+0000","last_clean":"2024-03-21T13:08:16.002065+0000","last_became_active":"2024-03-21T13:06:45.410017+0000","last_became_peered":"2024-03-21T13:06:45.410017+0000","last_unstale":"2024-03-21T13:08:16.002065+0000","last_undegraded":"2024-03-21T13:08:16.002065+0000","last_fullsized":"2024-03-21T13:08:16.002065+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:48.265982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:48.265982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"196'30","reported_seq":188,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.874061+0000","last_change":"2024-03-21T13:06:29.547664+0000","last_active":"2024-03-21T13:08:19.874061+0000","last_peered":"2024-03-21T13:08:19.874061+0000","last_clean":"2024-03-21T13:08:19.874061+0000","last_became_active":"2024-03-21T13:06:29.508620+0000","last_became_peered":"2024-03-21T13:06:29.508620+0000","last_unstale":"2024-03-21T13:08:19.874061+0000","last_undegraded":"2024-03-21T13:08:19.874061+0000","last_fullsized":"2024-03-21T13:08:19.874061+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'29","last_scrub_stamp":"2024-03-21T13:05:22.201879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.201879+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":68,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002818+0000","last_change":"2024-03-21T13:06:45.407111+0000","last_active":"2024-03-21T13:08:16.002818+0000","last_peered":"2024-03-21T13:08:16.002818+0000","last_clean":"2024-03-21T13:08:16.002818+0000","last_became_active":"2024-03-21T13:06:45.406734+0000","last_became_peered":"2024-03-21T13:06:45.406734+0000","last_unstale":"2024-03-21T13:08:16.002818+0000","last_undegraded":"2024-03-21T13:08:16.002818+0000","last_fullsized":"2024-03-21T13:08:16.002818+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.312583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.312583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.b","version":"225'16912","reported_seq":30141,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872261+0000","last_change":"2024-03-21T13:07:19.414805+0000","last_active":"2024-03-21T13:08:19.872261+0000","last_peered":"2024-03-21T13:08:19.872261+0000","last_clean":"2024-03-21T13:08:19.872261+0000","last_became_active":"2024-03-21T13:07:19.393088+0000","last_became_peered":"2024-03-21T13:07:19.393088+0000","last_unstale":"2024-03-21T13:08:19.872261+0000","last_undegraded":"2024-03-21T13:08:19.872261+0000","last_fullsized":"2024-03-21T13:08:19.872261+0000","mapping_epoch":207,"log_start":"218'13860","ondisk_log_start":"218'13860","created":168,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3052,"log_dups_size":0,"ondisk_log_size":3052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16912,"num_write_kb":541184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6543,"num_bytes_recovered":428802048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":348,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005769+0000","last_change":"2024-03-21T13:05:08.172825+0000","last_active":"2024-03-21T13:08:16.005769+0000","last_peered":"2024-03-21T13:08:16.005769+0000","last_clean":"2024-03-21T13:08:16.005769+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:08:16.005769+0000","last_undegraded":"2024-03-21T13:08:16.005769+0000","last_fullsized":"2024-03-21T13:08:16.005769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:08.172705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:08.172705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":74,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002084+0000","last_change":"2024-03-21T13:06:29.510026+0000","last_active":"2024-03-21T13:08:16.002084+0000","last_peered":"2024-03-21T13:08:16.002084+0000","last_clean":"2024-03-21T13:08:16.002084+0000","last_became_active":"2024-03-21T13:06:29.509674+0000","last_became_peered":"2024-03-21T13:06:29.509674+0000","last_unstale":"2024-03-21T13:08:16.002084+0000","last_undegraded":"2024-03-21T13:08:16.002084+0000","last_fullsized":"2024-03-21T13:08:16.002084+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.247107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.247107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":74,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.871612+0000","last_change":"2024-03-21T13:06:29.510171+0000","last_active":"2024-03-21T13:08:19.871612+0000","last_peered":"2024-03-21T13:08:19.871612+0000","last_clean":"2024-03-21T13:08:19.871612+0000","last_became_active":"2024-03-21T13:06:29.509315+0000","last_became_peered":"2024-03-21T13:06:29.509315+0000","last_unstale":"2024-03-21T13:08:19.871612+0000","last_undegraded":"2024-03-21T13:08:19.871612+0000","last_fullsized":"2024-03-21T13:08:19.871612+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:46.263644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:05:46.263644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"71'13","reported_seq":286,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005541+0000","last_change":"2024-03-21T13:05:15.119776+0000","last_active":"2024-03-21T13:08:16.005541+0000","last_peered":"2024-03-21T13:08:16.005541+0000","last_clean":"2024-03-21T13:08:16.005541+0000","last_became_active":"2024-03-21T13:02:26.234230+0000","last_became_peered":"2024-03-21T13:02:26.234230+0000","last_unstale":"2024-03-21T13:08:16.005541+0000","last_undegraded":"2024-03-21T13:08:16.005541+0000","last_fullsized":"2024-03-21T13:08:16.005541+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'13","last_scrub_stamp":"2024-03-21T13:05:15.119643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:15.119643+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":209,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.006110+0000","last_change":"2024-03-21T13:06:29.502194+0000","last_active":"2024-03-21T13:08:16.006110+0000","last_peered":"2024-03-21T13:08:16.006110+0000","last_clean":"2024-03-21T13:08:16.006110+0000","last_became_active":"2024-03-21T13:06:29.501625+0000","last_became_peered":"2024-03-21T13:06:29.501625+0000","last_unstale":"2024-03-21T13:08:16.006110+0000","last_undegraded":"2024-03-21T13:08:16.006110+0000","last_fullsized":"2024-03-21T13:08:16.006110+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:23.243468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.243468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.9","version":"225'17318","reported_seq":17453,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002004+0000","last_change":"2024-03-21T13:08:15.135104+0000","last_active":"2024-03-21T13:08:16.002004+0000","last_peered":"2024-03-21T13:08:16.002004+0000","last_clean":"2024-03-21T13:08:16.002004+0000","last_became_active":"2024-03-21T13:04:46.260364+0000","last_became_peered":"2024-03-21T13:04:46.260364+0000","last_unstale":"2024-03-21T13:08:16.002004+0000","last_undegraded":"2024-03-21T13:08:16.002004+0000","last_fullsized":"2024-03-21T13:08:16.002004+0000","mapping_epoch":168,"log_start":"218'13802","ondisk_log_start":"218'13802","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'17318","last_scrub_stamp":"2024-03-21T13:08:15.134976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:08:15.134976+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:15.532400+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17318,"num_write_kb":554176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":228,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.874162+0000","last_change":"2024-03-21T13:04:27.549179+0000","last_active":"2024-03-21T13:08:19.874162+0000","last_peered":"2024-03-21T13:08:19.874162+0000","last_clean":"2024-03-21T13:08:19.874162+0000","last_became_active":"2024-03-21T13:02:26.235640+0000","last_became_peered":"2024-03-21T13:02:26.235640+0000","last_unstale":"2024-03-21T13:08:19.874162+0000","last_undegraded":"2024-03-21T13:08:19.874162+0000","last_fullsized":"2024-03-21T13:08:19.874162+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:27.549086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:27.549086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":617,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.004330+0000","last_change":"2024-03-21T13:08:07.454234+0000","last_active":"2024-03-21T13:08:16.004330+0000","last_peered":"2024-03-21T13:08:16.004330+0000","last_clean":"2024-03-21T13:08:16.004330+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:08:16.004330+0000","last_undegraded":"2024-03-21T13:08:16.004330+0000","last_fullsized":"2024-03-21T13:08:16.004330+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:08:07.454016+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:19.144146+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":362,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872604+0000","last_change":"2024-03-21T13:07:49.758585+0000","last_active":"2024-03-21T13:08:19.872604+0000","last_peered":"2024-03-21T13:08:19.872604+0000","last_clean":"2024-03-21T13:08:19.872604+0000","last_became_active":"2024-03-21T13:02:26.242456+0000","last_became_peered":"2024-03-21T13:02:26.242456+0000","last_unstale":"2024-03-21T13:08:19.872604+0000","last_undegraded":"2024-03-21T13:08:19.872604+0000","last_fullsized":"2024-03-21T13:08:19.872604+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:49.758485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:49.758485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:07.636831+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"183'24","reported_seq":505,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.003058+0000","last_change":"2024-03-21T13:05:42.033620+0000","last_active":"2024-03-21T13:08:16.003058+0000","last_peered":"2024-03-21T13:08:16.003058+0000","last_clean":"2024-03-21T13:08:16.003058+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:08:16.003058+0000","last_undegraded":"2024-03-21T13:08:16.003058+0000","last_fullsized":"2024-03-21T13:08:16.003058+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'24","last_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.033266+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":364,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005350+0000","last_change":"2024-03-21T13:06:29.504771+0000","last_active":"2024-03-21T13:08:16.005350+0000","last_peered":"2024-03-21T13:08:16.005350+0000","last_clean":"2024-03-21T13:08:16.005350+0000","last_became_active":"2024-03-21T13:06:29.504250+0000","last_became_peered":"2024-03-21T13:06:29.504250+0000","last_unstale":"2024-03-21T13:08:16.005350+0000","last_undegraded":"2024-03-21T13:08:16.005350+0000","last_fullsized":"2024-03-21T13:08:16.005350+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:20.061709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.061709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"225'17098","reported_seq":20246,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872529+0000","last_change":"2024-03-21T13:08:06.489155+0000","last_active":"2024-03-21T13:08:19.872529+0000","last_peered":"2024-03-21T13:08:19.872529+0000","last_clean":"2024-03-21T13:08:19.872529+0000","last_became_active":"2024-03-21T13:08:06.474319+0000","last_became_peered":"2024-03-21T13:08:06.474319+0000","last_unstale":"2024-03-21T13:08:19.872529+0000","last_undegraded":"2024-03-21T13:08:19.872529+0000","last_fullsized":"2024-03-21T13:08:19.872529+0000","mapping_epoch":223,"log_start":"218'14064","ondisk_log_start":"218'14064","created":168,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3034,"log_dups_size":0,"ondisk_log_size":3034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17098,"num_write_kb":547136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1501,"num_bytes_recovered":98369536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":230,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872534+0000","last_change":"2024-03-21T13:05:29.457708+0000","last_active":"2024-03-21T13:08:19.872534+0000","last_peered":"2024-03-21T13:08:19.872534+0000","last_clean":"2024-03-21T13:08:19.872534+0000","last_became_active":"2024-03-21T13:02:26.242708+0000","last_became_peered":"2024-03-21T13:02:26.242708+0000","last_unstale":"2024-03-21T13:08:19.872534+0000","last_undegraded":"2024-03-21T13:08:19.872534+0000","last_fullsized":"2024-03-21T13:08:19.872534+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.457597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.457597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":263,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.875626+0000","last_change":"2024-03-21T13:08:11.057903+0000","last_active":"2024-03-21T13:08:19.875626+0000","last_peered":"2024-03-21T13:08:19.875626+0000","last_clean":"2024-03-21T13:08:19.875626+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:08:19.875626+0000","last_undegraded":"2024-03-21T13:08:19.875626+0000","last_fullsized":"2024-03-21T13:08:19.875626+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:11.057788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:08:11.057788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:17.241879+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":954,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:17.938315+0000","last_change":"2024-03-21T13:08:05.448565+0000","last_active":"2024-03-21T13:08:17.938315+0000","last_peered":"2024-03-21T13:08:17.938315+0000","last_clean":"2024-03-21T13:08:17.938315+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:08:17.938315+0000","last_undegraded":"2024-03-21T13:08:17.938315+0000","last_fullsized":"2024-03-21T13:08:17.938315+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:08:05.448403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:08:05.448403+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:26.024762+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":572,"num_read_kb":567,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":441,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.880882+0000","last_change":"2024-03-21T13:07:50.821737+0000","last_active":"2024-03-21T13:08:19.880882+0000","last_peered":"2024-03-21T13:08:19.880882+0000","last_clean":"2024-03-21T13:08:19.880882+0000","last_became_active":"2024-03-21T13:06:29.526702+0000","last_became_peered":"2024-03-21T13:06:29.526702+0000","last_unstale":"2024-03-21T13:08:19.880882+0000","last_undegraded":"2024-03-21T13:08:19.880882+0000","last_fullsized":"2024-03-21T13:08:19.880882+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:07:50.821119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:07:50.821119+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:15.038128+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":225,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005732+0000","last_change":"2024-03-21T13:05:13.085740+0000","last_active":"2024-03-21T13:08:16.005732+0000","last_peered":"2024-03-21T13:08:16.005732+0000","last_clean":"2024-03-21T13:08:16.005732+0000","last_became_active":"2024-03-21T13:02:26.232625+0000","last_became_peered":"2024-03-21T13:02:26.232625+0000","last_unstale":"2024-03-21T13:08:16.005732+0000","last_undegraded":"2024-03-21T13:08:16.005732+0000","last_fullsized":"2024-03-21T13:08:16.005732+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.085625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.085625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"225'16764","reported_seq":28852,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873588+0000","last_change":"2024-03-21T13:07:25.483359+0000","last_active":"2024-03-21T13:08:19.873588+0000","last_peered":"2024-03-21T13:08:19.873588+0000","last_clean":"2024-03-21T13:08:19.873588+0000","last_became_active":"2024-03-21T13:07:25.458300+0000","last_became_peered":"2024-03-21T13:07:25.458300+0000","last_unstale":"2024-03-21T13:08:19.873588+0000","last_undegraded":"2024-03-21T13:08:19.873588+0000","last_fullsized":"2024-03-21T13:08:19.873588+0000","mapping_epoch":211,"log_start":"218'13681","ondisk_log_start":"218'13681","created":168,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16764,"num_write_kb":536448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5973,"num_bytes_recovered":391446528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":244,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002445+0000","last_change":"2024-03-21T13:06:29.489827+0000","last_active":"2024-03-21T13:08:16.002445+0000","last_peered":"2024-03-21T13:08:16.002445+0000","last_clean":"2024-03-21T13:08:16.002445+0000","last_became_active":"2024-03-21T13:06:29.489464+0000","last_became_peered":"2024-03-21T13:06:29.489464+0000","last_unstale":"2024-03-21T13:08:16.002445+0000","last_undegraded":"2024-03-21T13:08:16.002445+0000","last_fullsized":"2024-03-21T13:08:16.002445+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:35.000684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:35.000684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":93,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002251+0000","last_change":"2024-03-21T13:06:29.484504+0000","last_active":"2024-03-21T13:08:16.002251+0000","last_peered":"2024-03-21T13:08:16.002251+0000","last_clean":"2024-03-21T13:08:16.002251+0000","last_became_active":"2024-03-21T13:06:29.484174+0000","last_became_peered":"2024-03-21T13:06:29.484174+0000","last_unstale":"2024-03-21T13:08:16.002251+0000","last_undegraded":"2024-03-21T13:08:16.002251+0000","last_fullsized":"2024-03-21T13:08:16.002251+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.282145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.282145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"71'19","reported_seq":323,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005193+0000","last_change":"2024-03-21T13:06:29.508182+0000","last_active":"2024-03-21T13:08:16.005193+0000","last_peered":"2024-03-21T13:08:16.005193+0000","last_clean":"2024-03-21T13:08:16.005193+0000","last_became_active":"2024-03-21T13:06:29.492612+0000","last_became_peered":"2024-03-21T13:06:29.492612+0000","last_unstale":"2024-03-21T13:08:16.005193+0000","last_undegraded":"2024-03-21T13:08:16.005193+0000","last_fullsized":"2024-03-21T13:08:16.005193+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:05:28.062237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.062237+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":366,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002460+0000","last_change":"2024-03-21T13:06:29.497191+0000","last_active":"2024-03-21T13:08:16.002460+0000","last_peered":"2024-03-21T13:08:16.002460+0000","last_clean":"2024-03-21T13:08:16.002460+0000","last_became_active":"2024-03-21T13:06:29.496679+0000","last_became_peered":"2024-03-21T13:06:29.496679+0000","last_unstale":"2024-03-21T13:08:16.002460+0000","last_undegraded":"2024-03-21T13:08:16.002460+0000","last_fullsized":"2024-03-21T13:08:16.002460+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:26.932188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.932188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"225'17080","reported_seq":27618,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.880716+0000","last_change":"2024-03-21T13:07:50.725881+0000","last_active":"2024-03-21T13:08:19.880716+0000","last_peered":"2024-03-21T13:08:19.880716+0000","last_clean":"2024-03-21T13:08:19.880716+0000","last_became_active":"2024-03-21T13:07:27.305219+0000","last_became_peered":"2024-03-21T13:07:27.305219+0000","last_unstale":"2024-03-21T13:08:19.880716+0000","last_undegraded":"2024-03-21T13:08:19.880716+0000","last_fullsized":"2024-03-21T13:08:19.880716+0000","mapping_epoch":213,"log_start":"217'13183","ondisk_log_start":"217'13183","created":168,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'14693","last_scrub_stamp":"2024-03-21T13:07:50.725717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:07:50.725717+0000","objects_scrubbed":2422,"log_size":3897,"log_dups_size":0,"ondisk_log_size":3897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:00.774520+0000","scrub_duration":890,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17080,"num_write_kb":546560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5174,"num_bytes_recovered":339017728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":253,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.871571+0000","last_change":"2024-03-21T13:06:29.502310+0000","last_active":"2024-03-21T13:08:19.871571+0000","last_peered":"2024-03-21T13:08:19.871571+0000","last_clean":"2024-03-21T13:08:19.871571+0000","last_became_active":"2024-03-21T13:06:29.501983+0000","last_became_peered":"2024-03-21T13:06:29.501983+0000","last_unstale":"2024-03-21T13:08:19.871571+0000","last_undegraded":"2024-03-21T13:08:19.871571+0000","last_fullsized":"2024-03-21T13:08:19.871571+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:31.121913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:01:31.121913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":234,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872780+0000","last_change":"2024-03-21T13:06:29.511420+0000","last_active":"2024-03-21T13:08:19.872780+0000","last_peered":"2024-03-21T13:08:19.872780+0000","last_clean":"2024-03-21T13:08:19.872780+0000","last_became_active":"2024-03-21T13:06:29.510833+0000","last_became_peered":"2024-03-21T13:06:29.510833+0000","last_unstale":"2024-03-21T13:08:19.872780+0000","last_undegraded":"2024-03-21T13:08:19.872780+0000","last_fullsized":"2024-03-21T13:08:19.872780+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.324701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.324701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"86'15","reported_seq":312,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.876042+0000","last_change":"2024-03-21T13:06:29.513960+0000","last_active":"2024-03-21T13:08:19.876042+0000","last_peered":"2024-03-21T13:08:19.876042+0000","last_clean":"2024-03-21T13:08:19.876042+0000","last_became_active":"2024-03-21T13:06:29.492863+0000","last_became_peered":"2024-03-21T13:06:29.492863+0000","last_unstale":"2024-03-21T13:08:19.876042+0000","last_undegraded":"2024-03-21T13:08:19.876042+0000","last_fullsized":"2024-03-21T13:08:19.876042+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-03-21T13:05:42.308556+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.308556+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"125'12","reported_seq":377,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:18.149397+0000","last_change":"2024-03-21T13:08:18.149427+0000","last_active":"2024-03-21T13:08:18.149397+0000","last_peered":"2024-03-21T13:08:18.149397+0000","last_clean":"2024-03-21T13:08:18.149397+0000","last_became_active":"2024-03-21T13:02:26.253720+0000","last_became_peered":"2024-03-21T13:02:26.253720+0000","last_unstale":"2024-03-21T13:08:18.149397+0000","last_undegraded":"2024-03-21T13:08:18.149397+0000","last_fullsized":"2024-03-21T13:08:18.149397+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'12","last_scrub_stamp":"2024-03-21T13:08:18.149295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.149295+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:21.178979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"225'17194","reported_seq":31265,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005396+0000","last_change":"2024-03-21T13:07:08.359129+0000","last_active":"2024-03-21T13:08:16.005396+0000","last_peered":"2024-03-21T13:08:16.005396+0000","last_clean":"2024-03-21T13:08:16.005396+0000","last_became_active":"2024-03-21T13:07:08.338910+0000","last_became_peered":"2024-03-21T13:07:08.338910+0000","last_unstale":"2024-03-21T13:08:16.005396+0000","last_undegraded":"2024-03-21T13:08:16.005396+0000","last_fullsized":"2024-03-21T13:08:16.005396+0000","mapping_epoch":203,"log_start":"218'13769","ondisk_log_start":"218'13769","created":168,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17194,"num_write_kb":550145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6964,"num_bytes_recovered":456327192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":354,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005270+0000","last_change":"2024-03-21T13:06:29.489617+0000","last_active":"2024-03-21T13:08:16.005270+0000","last_peered":"2024-03-21T13:08:16.005270+0000","last_clean":"2024-03-21T13:08:16.005270+0000","last_became_active":"2024-03-21T13:06:29.489289+0000","last_became_peered":"2024-03-21T13:06:29.489289+0000","last_unstale":"2024-03-21T13:08:16.005270+0000","last_undegraded":"2024-03-21T13:08:16.005270+0000","last_fullsized":"2024-03-21T13:08:16.005270+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.951655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.951655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":398,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.880987+0000","last_change":"2024-03-21T13:07:55.868896+0000","last_active":"2024-03-21T13:08:19.880987+0000","last_peered":"2024-03-21T13:08:19.880987+0000","last_clean":"2024-03-21T13:08:19.880987+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:08:19.880987+0000","last_undegraded":"2024-03-21T13:08:19.880987+0000","last_fullsized":"2024-03-21T13:08:19.880987+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:07:55.868781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:55.868781+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:07.587595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"87'21","reported_seq":318,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.001880+0000","last_change":"2024-03-21T13:05:23.917855+0000","last_active":"2024-03-21T13:08:16.001880+0000","last_peered":"2024-03-21T13:08:16.001880+0000","last_clean":"2024-03-21T13:08:16.001880+0000","last_became_active":"2024-03-21T13:02:26.227165+0000","last_became_peered":"2024-03-21T13:02:26.227165+0000","last_unstale":"2024-03-21T13:08:16.001880+0000","last_undegraded":"2024-03-21T13:08:16.001880+0000","last_fullsized":"2024-03-21T13:08:16.001880+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'21","last_scrub_stamp":"2024-03-21T13:05:23.917691+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.917691+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"226'10","reported_seq":485,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872092+0000","last_change":"2024-03-21T13:08:16.739276+0000","last_active":"2024-03-21T13:08:19.872092+0000","last_peered":"2024-03-21T13:08:19.872092+0000","last_clean":"2024-03-21T13:08:19.872092+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:08:19.872092+0000","last_undegraded":"2024-03-21T13:08:19.872092+0000","last_fullsized":"2024-03-21T13:08:19.872092+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:08:16.739190+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:29.735435+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"225'16836","reported_seq":17081,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.879971+0000","last_change":"2024-03-21T13:07:57.366584+0000","last_active":"2024-03-21T13:08:19.879971+0000","last_peered":"2024-03-21T13:08:19.879971+0000","last_clean":"2024-03-21T13:08:19.879971+0000","last_became_active":"2024-03-21T13:04:46.260642+0000","last_became_peered":"2024-03-21T13:04:46.260642+0000","last_unstale":"2024-03-21T13:08:19.879971+0000","last_undegraded":"2024-03-21T13:08:19.879971+0000","last_fullsized":"2024-03-21T13:08:19.879971+0000","mapping_epoch":168,"log_start":"217'12959","ondisk_log_start":"217'12959","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'15076","last_scrub_stamp":"2024-03-21T13:07:57.366169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:07:57.366169+0000","objects_scrubbed":1776,"log_size":3877,"log_dups_size":0,"ondisk_log_size":3877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:01.505178+0000","scrub_duration":480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16836,"num_write_kb":538752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":275,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.872937+0000","last_change":"2024-03-21T13:06:29.491923+0000","last_active":"2024-03-21T13:08:19.872937+0000","last_peered":"2024-03-21T13:08:19.872937+0000","last_clean":"2024-03-21T13:08:19.872937+0000","last_became_active":"2024-03-21T13:06:29.491598+0000","last_became_peered":"2024-03-21T13:06:29.491598+0000","last_unstale":"2024-03-21T13:08:19.872937+0000","last_undegraded":"2024-03-21T13:08:19.872937+0000","last_fullsized":"2024-03-21T13:08:19.872937+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.416262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.416262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":356,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873971+0000","last_change":"2024-03-21T13:04:20.538204+0000","last_active":"2024-03-21T13:08:19.873971+0000","last_peered":"2024-03-21T13:08:19.873971+0000","last_clean":"2024-03-21T13:08:19.873971+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:08:19.873971+0000","last_undegraded":"2024-03-21T13:08:19.873971+0000","last_fullsized":"2024-03-21T13:08:19.873971+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:20.538115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:04:20.538115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"71'18","reported_seq":384,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.871941+0000","last_change":"2024-03-21T13:05:12.322522+0000","last_active":"2024-03-21T13:08:19.871941+0000","last_peered":"2024-03-21T13:08:19.871941+0000","last_clean":"2024-03-21T13:08:19.871941+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:08:19.871941+0000","last_undegraded":"2024-03-21T13:08:19.871941+0000","last_fullsized":"2024-03-21T13:08:19.871941+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T13:05:12.322284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:12.322284+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":356,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.875341+0000","last_change":"2024-03-21T13:08:04.095062+0000","last_active":"2024-03-21T13:08:19.875341+0000","last_peered":"2024-03-21T13:08:19.875341+0000","last_clean":"2024-03-21T13:08:19.875341+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:08:19.875341+0000","last_undegraded":"2024-03-21T13:08:19.875341+0000","last_fullsized":"2024-03-21T13:08:19.875341+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:04.094946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:04.094946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:23.528973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"225'16964","reported_seq":30926,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002027+0000","last_change":"2024-03-21T13:08:14.195085+0000","last_active":"2024-03-21T13:08:16.002027+0000","last_peered":"2024-03-21T13:08:16.002027+0000","last_clean":"2024-03-21T13:08:16.002027+0000","last_became_active":"2024-03-21T13:07:09.337338+0000","last_became_peered":"2024-03-21T13:07:09.337338+0000","last_unstale":"2024-03-21T13:08:16.002027+0000","last_undegraded":"2024-03-21T13:08:16.002027+0000","last_fullsized":"2024-03-21T13:08:16.002027+0000","mapping_epoch":204,"log_start":"218'13425","ondisk_log_start":"218'13425","created":168,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'16878","last_scrub_stamp":"2024-03-21T13:08:14.194740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:08:14.194740+0000","objects_scrubbed":88,"log_size":3539,"log_dups_size":0,"ondisk_log_size":3539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:43.859601+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16964,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6904,"num_bytes_recovered":452395008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":194,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002621+0000","last_change":"2024-03-21T13:06:29.489514+0000","last_active":"2024-03-21T13:08:16.002621+0000","last_peered":"2024-03-21T13:08:16.002621+0000","last_clean":"2024-03-21T13:08:16.002621+0000","last_became_active":"2024-03-21T13:06:29.488826+0000","last_became_peered":"2024-03-21T13:06:29.488826+0000","last_unstale":"2024-03-21T13:08:16.002621+0000","last_undegraded":"2024-03-21T13:08:16.002621+0000","last_fullsized":"2024-03-21T13:08:16.002621+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.328925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.328925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.e","version":"196'11","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.881176+0000","last_change":"2024-03-21T13:08:04.934320+0000","last_active":"2024-03-21T13:08:19.881176+0000","last_peered":"2024-03-21T13:08:19.881176+0000","last_clean":"2024-03-21T13:08:19.881176+0000","last_became_active":"2024-03-21T13:06:29.505444+0000","last_became_peered":"2024-03-21T13:06:29.505444+0000","last_unstale":"2024-03-21T13:08:19.881176+0000","last_undegraded":"2024-03-21T13:08:19.881176+0000","last_fullsized":"2024-03-21T13:08:19.881176+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'11","last_scrub_stamp":"2024-03-21T13:08:04.934196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:04.934196+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:16.986626+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":362,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.871938+0000","last_change":"2024-03-21T13:04:19.520964+0000","last_active":"2024-03-21T13:08:19.871938+0000","last_peered":"2024-03-21T13:08:19.871938+0000","last_clean":"2024-03-21T13:08:19.871938+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:08:19.871938+0000","last_undegraded":"2024-03-21T13:08:19.871938+0000","last_fullsized":"2024-03-21T13:08:19.871938+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:19.520824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:04:19.520824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":292,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.874052+0000","last_change":"2024-03-21T13:02:14.377274+0000","last_active":"2024-03-21T13:08:19.874052+0000","last_peered":"2024-03-21T13:08:19.874052+0000","last_clean":"2024-03-21T13:08:19.874052+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:08:19.874052+0000","last_undegraded":"2024-03-21T13:08:19.874052+0000","last_fullsized":"2024-03-21T13:08:19.874052+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:01:46.167736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.167736+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"225'16962","reported_seq":17125,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.875299+0000","last_change":"2024-03-21T13:07:50.397429+0000","last_active":"2024-03-21T13:08:19.875299+0000","last_peered":"2024-03-21T13:08:19.875299+0000","last_clean":"2024-03-21T13:08:19.875299+0000","last_became_active":"2024-03-21T13:04:46.260735+0000","last_became_peered":"2024-03-21T13:04:46.260735+0000","last_unstale":"2024-03-21T13:08:19.875299+0000","last_undegraded":"2024-03-21T13:08:19.875299+0000","last_fullsized":"2024-03-21T13:08:19.875299+0000","mapping_epoch":168,"log_start":"218'13614","ondisk_log_start":"218'13614","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'14622","last_scrub_stamp":"2024-03-21T13:07:50.396818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:07:50.396818+0000","objects_scrubbed":2393,"log_size":3348,"log_dups_size":0,"ondisk_log_size":3348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:08:54.874649+0000","scrub_duration":1311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16962,"num_write_kb":542784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":355,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873544+0000","last_change":"2024-03-21T13:05:39.518051+0000","last_active":"2024-03-21T13:08:19.873544+0000","last_peered":"2024-03-21T13:08:19.873544+0000","last_clean":"2024-03-21T13:08:19.873544+0000","last_became_active":"2024-03-21T13:02:26.243985+0000","last_became_peered":"2024-03-21T13:02:26.243985+0000","last_unstale":"2024-03-21T13:08:19.873544+0000","last_undegraded":"2024-03-21T13:08:19.873544+0000","last_fullsized":"2024-03-21T13:08:19.873544+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:39.517591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.517591+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":250,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.876619+0000","last_change":"2024-03-21T13:08:08.054195+0000","last_active":"2024-03-21T13:08:19.876619+0000","last_peered":"2024-03-21T13:08:19.876619+0000","last_clean":"2024-03-21T13:08:19.876619+0000","last_became_active":"2024-03-21T13:06:29.501230+0000","last_became_peered":"2024-03-21T13:06:29.501230+0000","last_unstale":"2024-03-21T13:08:19.876619+0000","last_undegraded":"2024-03-21T13:08:19.876619+0000","last_fullsized":"2024-03-21T13:08:19.876619+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:08.054077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:08.054077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:28.130473+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":378,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.875462+0000","last_change":"2024-03-21T13:07:55.115719+0000","last_active":"2024-03-21T13:08:19.875462+0000","last_peered":"2024-03-21T13:08:19.875462+0000","last_clean":"2024-03-21T13:08:19.875462+0000","last_became_active":"2024-03-21T13:06:29.499291+0000","last_became_peered":"2024-03-21T13:06:29.499291+0000","last_unstale":"2024-03-21T13:08:19.875462+0000","last_undegraded":"2024-03-21T13:08:19.875462+0000","last_fullsized":"2024-03-21T13:08:19.875462+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:55.115613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:55.115613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:23.200883+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'11","reported_seq":102,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002667+0000","last_change":"2024-03-21T13:06:45.412306+0000","last_active":"2024-03-21T13:08:16.002667+0000","last_peered":"2024-03-21T13:08:16.002667+0000","last_clean":"2024-03-21T13:08:16.002667+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:08:16.002667+0000","last_undegraded":"2024-03-21T13:08:16.002667+0000","last_fullsized":"2024-03-21T13:08:16.002667+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:26.195507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.195507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"225'17118","reported_seq":25927,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.871674+0000","last_change":"2024-03-21T13:07:44.899912+0000","last_active":"2024-03-21T13:08:19.871674+0000","last_peered":"2024-03-21T13:08:19.871674+0000","last_clean":"2024-03-21T13:08:19.871674+0000","last_became_active":"2024-03-21T13:07:44.885457+0000","last_became_peered":"2024-03-21T13:07:44.885457+0000","last_unstale":"2024-03-21T13:08:19.871674+0000","last_undegraded":"2024-03-21T13:08:19.871674+0000","last_fullsized":"2024-03-21T13:08:19.871674+0000","mapping_epoch":219,"log_start":"218'14065","ondisk_log_start":"218'14065","created":168,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3053,"log_dups_size":0,"ondisk_log_size":3053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17118,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4334,"num_bytes_recovered":284033024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":74,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873821+0000","last_change":"2024-03-21T13:06:29.508869+0000","last_active":"2024-03-21T13:08:19.873821+0000","last_peered":"2024-03-21T13:08:19.873821+0000","last_clean":"2024-03-21T13:08:19.873821+0000","last_became_active":"2024-03-21T13:06:29.508508+0000","last_became_peered":"2024-03-21T13:06:29.508508+0000","last_unstale":"2024-03-21T13:08:19.873821+0000","last_undegraded":"2024-03-21T13:08:19.873821+0000","last_fullsized":"2024-03-21T13:08:19.873821+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:36.362197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:05:36.362197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":256,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.876019+0000","last_change":"2024-03-21T13:08:05.143175+0000","last_active":"2024-03-21T13:08:19.876019+0000","last_peered":"2024-03-21T13:08:19.876019+0000","last_clean":"2024-03-21T13:08:19.876019+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:08:19.876019+0000","last_undegraded":"2024-03-21T13:08:19.876019+0000","last_fullsized":"2024-03-21T13:08:19.876019+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:05.142807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:05.142807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:17.163399+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":376,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002600+0000","last_change":"2024-03-21T13:06:30.274121+0000","last_active":"2024-03-21T13:08:16.002600+0000","last_peered":"2024-03-21T13:08:16.002600+0000","last_clean":"2024-03-21T13:08:16.002600+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:08:16.002600+0000","last_undegraded":"2024-03-21T13:08:16.002600+0000","last_fullsized":"2024-03-21T13:08:16.002600+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:30.274033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:30.274033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":174,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.002578+0000","last_change":"2024-03-21T13:06:29.531422+0000","last_active":"2024-03-21T13:08:16.002578+0000","last_peered":"2024-03-21T13:08:16.002578+0000","last_clean":"2024-03-21T13:08:16.002578+0000","last_became_active":"2024-03-21T13:06:29.489192+0000","last_became_peered":"2024-03-21T13:06:29.489192+0000","last_unstale":"2024-03-21T13:08:16.002578+0000","last_undegraded":"2024-03-21T13:08:16.002578+0000","last_fullsized":"2024-03-21T13:08:16.002578+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-21T13:05:38.215693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.215693+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.0","version":"225'16830","reported_seq":16959,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.871284+0000","last_change":"2024-03-21T13:04:46.258753+0000","last_active":"2024-03-21T13:08:19.871284+0000","last_peered":"2024-03-21T13:08:19.871284+0000","last_clean":"2024-03-21T13:08:19.871284+0000","last_became_active":"2024-03-21T13:04:46.258434+0000","last_became_peered":"2024-03-21T13:04:46.258434+0000","last_unstale":"2024-03-21T13:08:19.871284+0000","last_undegraded":"2024-03-21T13:08:19.871284+0000","last_fullsized":"2024-03-21T13:08:19.871284+0000","mapping_epoch":168,"log_start":"218'13721","ondisk_log_start":"218'13721","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:04:46.023690+0000","last_clean_scrub_stamp":"2024-03-21T13:04:46.023690+0000","objects_scrubbed":0,"log_size":3109,"log_dups_size":0,"ondisk_log_size":3109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16830,"num_write_kb":538560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":95,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005470+0000","last_change":"2024-03-21T13:06:29.497122+0000","last_active":"2024-03-21T13:08:16.005470+0000","last_peered":"2024-03-21T13:08:16.005470+0000","last_clean":"2024-03-21T13:08:16.005470+0000","last_became_active":"2024-03-21T13:06:29.496779+0000","last_became_peered":"2024-03-21T13:06:29.496779+0000","last_unstale":"2024-03-21T13:08:16.005470+0000","last_undegraded":"2024-03-21T13:08:16.005470+0000","last_fullsized":"2024-03-21T13:08:16.005470+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.291369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.291369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"196'11","reported_seq":305,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-21T13:08:16.005603+0000","last_change":"2024-03-21T13:06:29.522757+0000","last_active":"2024-03-21T13:08:16.005603+0000","last_peered":"2024-03-21T13:08:16.005603+0000","last_clean":"2024-03-21T13:08:16.005603+0000","last_became_active":"2024-03-21T13:06:29.501395+0000","last_became_peered":"2024-03-21T13:06:29.501395+0000","last_unstale":"2024-03-21T13:08:16.005603+0000","last_undegraded":"2024-03-21T13:08:16.005603+0000","last_fullsized":"2024-03-21T13:08:16.005603+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:05:30.300605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:30.300605+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":356,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873075+0000","last_change":"2024-03-21T13:05:38.492137+0000","last_active":"2024-03-21T13:08:19.873075+0000","last_peered":"2024-03-21T13:08:19.873075+0000","last_clean":"2024-03-21T13:08:19.873075+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:08:19.873075+0000","last_undegraded":"2024-03-21T13:08:19.873075+0000","last_fullsized":"2024-03-21T13:08:19.873075+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:38.492015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.492015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":303,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-21T13:08:19.873226+0000","last_change":"2024-03-21T13:05:22.406498+0000","last_active":"2024-03-21T13:08:19.873226+0000","last_peered":"2024-03-21T13:08:19.873226+0000","last_clean":"2024-03-21T13:08:19.873226+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:08:19.873226+0000","last_undegraded":"2024-03-21T13:08:19.873226+0000","last_fullsized":"2024-03-21T13:08:19.873226+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:22.406367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.406367+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":271742,"num_write_kb":8695681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49676,"num_bytes_recovered":3255369752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":66781184,"data_stored":66781184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53609,"ondisk_log_size":53609,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":572,"num_read_kb":567,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":5213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":682,"num_read_kb":439,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4428,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":15782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":433,"ondisk_log_size":433,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168697,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":379704,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93365448,"statfs":{"total":95995035648,"available":95606218752,"internally_reserved":0,"allocated":1490944,"data_stored":1031120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":387316170},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463039,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":433828,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93311324,"statfs":{"total":95995035648,"available":95550795776,"internally_reserved":0,"allocated":1470464,"data_stored":1030392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":442759622},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738525,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321372,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93423780,"statfs":{"total":95995035648,"available":95665950720,"internally_reserved":0,"allocated":1462272,"data_stored":1029591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411475,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182542,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":563784,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":93181368,"statfs":{"total":95995035648,"available":95417720832,"internally_reserved":0,"allocated":790528,"data_stored":378092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":576387527},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019016,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":515160,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93229992,"statfs":{"total":95995035648,"available":95467511808,"internally_reserved":0,"allocated":868352,"data_stored":439880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724675,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66781184,"data_stored":66781184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:08:26.806 INFO:tasks.thrashosds.thrasher:Setting 4.d to [3, 1, 0, 2, 4, 5] 2024-03-21T13:08:26.807 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.d', '3', '1', '0', '2', '4', '5'] 2024-03-21T13:08:26.807 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pg-upmap-items 4.d 3 1 0 2 4 5 2024-03-21T13:08:26.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:26.808 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:26.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:26.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:26.936 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:26.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:26.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:27.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:27.065 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:27.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:27.167 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:27.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:27.191 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:27.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:27.292 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:27.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:27.317 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:27.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:27.418 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:27.445 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:27.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:27.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:27.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:27.572 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:27.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:27.672 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:27.697 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:27.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:27.798 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:27.826 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:27.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:27.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:27.928 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:27.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:27.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:28.058 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:28.083 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:28.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:28.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:28.208 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:28.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:28.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:28.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:28.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:28.339 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:28.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:28.441 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:28.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:28.466 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:28.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:28.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:28.595 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:28.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:28.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:28 smithi163 ceph-mon[40675]: pgmap v427: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 738 B/s rd, 0 B/s wr, 3 op/s 2024-03-21T13:08:28.608 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:28 smithi163 ceph-mon[29621]: pgmap v427: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 738 B/s rd, 0 B/s wr, 3 op/s 2024-03-21T13:08:28.697 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:28.721 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:28.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:28.735 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:08:28.736 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:08:28.736 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-03-21T13:08:28.736 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-03-21T13:08:28.736 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:08:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:28 smithi204 ceph-mon[33153]: pgmap v427: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 738 B/s rd, 0 B/s wr, 3 op/s 2024-03-21T13:08:28.823 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:28.851 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:28.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:28.953 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:28.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:28.979 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:28.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:29.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:29.109 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:29.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:29.211 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:29.239 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:29.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:29.341 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:29.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:29.366 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:29.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:29.467 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:29.494 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:29.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:29.571 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-21T13:08:29.572 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:29.595 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:29.641 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:29.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:29.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:29.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:29.773 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:29.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:29.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:29.902 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:29.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:30.005 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:30.029 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:30.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:30.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:30.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:30.160 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:30.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:30.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:30.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:30.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:30.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:30.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:30.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:30.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:30.420 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:30.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:30.520 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:30 smithi163 ceph-mon[29621]: pgmap v428: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:08:30.521 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:30 smithi163 ceph-mon[40675]: pgmap v428: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:08:30.521 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:30.543 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:30.545 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:30.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:30.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:30.557 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:08:30.578 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:08:30.598 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:08:30.620 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:08:30.639 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:08:30.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:30.660 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:08:30.673 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:30.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:30.683 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:08:30.703 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:08:30.704 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:08:30.704 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:08:30.704 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:08:30.704 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:08:30.705 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:08:30.705 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:08:30.705 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:08:30.705 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:08:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:30 smithi204 ceph-mon[33153]: pgmap v428: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:08:30.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:30.801 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:30.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:30.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:30.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:30.930 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:30.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:31.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:31.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:31.055 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:31.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:31.157 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:31.185 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:31.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:31.213 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:31.287 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:31.312 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:31.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:31.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:31.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:31.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:31.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:31.553 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:31 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4015194676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:31.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:31 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4015194676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:31.571 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:31.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:31.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:31.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:31.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:31 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4015194676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:31.798 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:31.821 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:31.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:31.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:31.923 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:31.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:31.952 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:31.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:32.054 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:32.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:32.079 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:32.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:32.180 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:32.204 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:32.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:32.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:08:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:08:32.306 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:32.330 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:32.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:32.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:32.431 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:32.455 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:32.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:32.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:32 smithi163 ceph-mon[29621]: pgmap v429: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:08:32.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:32 smithi163 ceph-mon[40675]: pgmap v429: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:08:32.558 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:32.581 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:32.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:32.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:32.683 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:32.708 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:32.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:32 smithi204 ceph-mon[33153]: pgmap v429: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:08:32.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:32.837 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:32.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:32.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:32.939 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:32.964 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:32.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:33.065 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:33.089 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:33.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:33.190 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:33.214 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:33.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:33.318 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:33.344 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:33.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:33.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:33.473 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:33.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:33.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:33.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:33.603 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:33.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:33.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:33 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:33.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:33 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:08:33.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:33 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:08:33.640 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:33 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:33.640 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:33 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:08:33.640 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:33 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:08:33.705 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:33.729 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:33.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:33.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:33 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:33.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:33 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:08:33.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:33 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:08:33.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:33.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:33.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:33.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:33.966 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:33.993 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:33.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:34.094 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:34.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:34.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:34.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:34.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:34.251 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:34.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:34.353 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:34.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:34.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:34.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:34.500 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:08:34.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:34.507 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:34.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:34.608 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:34.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:34.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:34.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:34.713 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:34 smithi163 ceph-mon[40675]: pgmap v430: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:08:34.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:34 smithi163 ceph-mon[40675]: osdmap e230: 8 total, 8 up, 5 in 2024-03-21T13:08:34.714 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:34 smithi163 ceph-mon[29621]: pgmap v430: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:08:34.714 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:34 smithi163 ceph-mon[29621]: osdmap e230: 8 total, 8 up, 5 in 2024-03-21T13:08:34.738 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:34.741 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:34 smithi204 ceph-mon[33153]: pgmap v430: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:08:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:34 smithi204 ceph-mon[33153]: osdmap e230: 8 total, 8 up, 5 in 2024-03-21T13:08:34.764 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:34.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:34.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:34.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:34.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:34.895 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:34.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:34.997 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:35.022 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:35.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:35.123 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:35.152 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:35.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:35.253 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:35.278 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:35.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:35.345 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'unique_pool_1' removed 2024-03-21T13:08:35.379 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:35.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:35.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:35.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:35 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1693380798' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:08:35.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:35.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:08:35.454 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:35 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1693380798' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:08:35.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:35.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:08:35.504 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:35.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:35.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:35.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:35.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:35.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:35.661 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:35.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:35.728 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:08:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:08:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:08:35.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:35 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1693380798' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:08:35.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:35.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:08:35.762 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:35.811 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:35.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:35.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:35.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:35.938 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:35.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:36.039 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:36.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:36.064 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:36.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:36.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:36.191 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:36.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:36.292 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:36.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:36.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:36.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:36.417 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:36.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:36.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:36.543 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:36.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:36.568 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:36.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:36.669 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:36.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:36.694 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:36.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:36 smithi204 ceph-mon[33153]: pgmap v432: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:08:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:36 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1693380798' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:08:36.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:36 smithi204 ceph-mon[33153]: osdmap e231: 8 total, 8 up, 5 in 2024-03-21T13:08:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:36 smithi163 ceph-mon[29621]: pgmap v432: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:08:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:36 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1693380798' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:08:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:36 smithi163 ceph-mon[29621]: osdmap e231: 8 total, 8 up, 5 in 2024-03-21T13:08:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:36 smithi163 ceph-mon[40675]: pgmap v432: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:08:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:36 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1693380798' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:08:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:36 smithi163 ceph-mon[40675]: osdmap e231: 8 total, 8 up, 5 in 2024-03-21T13:08:36.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:36.820 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:36.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:36.921 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:36.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:36.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:36.951 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-21T13:08:36.952 INFO:tasks.radosbench:Beginning radosbench... 2024-03-21T13:08:36.952 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-21T13:08:36.953 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-21T13:08:37.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:37.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:37.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:37.178 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:37.203 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:37.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:37.304 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:37.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:37.330 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:37.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:37.362 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:08:37.431 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:37.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:37.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:37.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:37 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3543683604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:08:37.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:37 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3543683604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:08:37.562 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:37.587 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:37.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:37.675 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:37.689 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:37.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:37.718 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:37.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:37.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:37 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3543683604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:08:37.819 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:37.847 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:37.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:37.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:37.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:37.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:38.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:38.103 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:38.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:38.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:38.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:38.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:38.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:38.335 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:38.359 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:38.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:38.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:38.364 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:38 smithi163 ceph-mon[40675]: pgmap v434: 132 pgs: 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:08:38.364 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:38 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:08:38.364 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:38 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3543683604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:08:38.364 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:38 smithi163 ceph-mon[40675]: osdmap e232: 8 total, 8 up, 5 in 2024-03-21T13:08:38.364 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:38 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/679480984' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.d", "id": [3, 1, 0, 2, 4, 5]}]: dispatch 2024-03-21T13:08:38.364 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:38 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.d", "id": [3, 1, 0, 2, 4, 5]}]: dispatch 2024-03-21T13:08:38.365 INFO:teuthology.orchestra.run.smithi163.stderr:set 4.d pg_upmap_items mapping to [3->1,0->2,4->5] 2024-03-21T13:08:38.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:38.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:38.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:38.595 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:38.620 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:38.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:38.672 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:38 smithi163 ceph-mon[29621]: pgmap v434: 132 pgs: 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:08:38.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:38 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:08:38.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:38 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3543683604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:08:38.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:38 smithi163 ceph-mon[29621]: osdmap e232: 8 total, 8 up, 5 in 2024-03-21T13:08:38.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:38 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/679480984' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.d", "id": [3, 1, 0, 2, 4, 5]}]: dispatch 2024-03-21T13:08:38.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:38 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.d", "id": [3, 1, 0, 2, 4, 5]}]: dispatch 2024-03-21T13:08:38.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:38.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:38.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:38.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:38 smithi204 ceph-mon[33153]: pgmap v434: 132 pgs: 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:08:38.755 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:38 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:08:38.755 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:38 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3543683604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:08:38.755 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:38 smithi204 ceph-mon[33153]: osdmap e232: 8 total, 8 up, 5 in 2024-03-21T13:08:38.755 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:38 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/679480984' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.d", "id": [3, 1, 0, 2, 4, 5]}]: dispatch 2024-03-21T13:08:38.755 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:38 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.d", "id": [3, 1, 0, 2, 4, 5]}]: dispatch 2024-03-21T13:08:38.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:38.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:38.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:38.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:39.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:39.012 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:39.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:39.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:39.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:39.139 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:39.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:39.240 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:39.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:39.264 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:39.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:39.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:39.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:39.395 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:39.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:39.496 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:39.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:39 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.d", "id": [3, 1, 0, 2, 4, 5]}]': finished 2024-03-21T13:08:39.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:39 smithi163 ceph-mon[29621]: osdmap e233: 8 total, 8 up, 5 in 2024-03-21T13:08:39.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:39 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.d", "id": [3, 1, 0, 2, 4, 5]}]': finished 2024-03-21T13:08:39.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:39 smithi163 ceph-mon[40675]: osdmap e233: 8 total, 8 up, 5 in 2024-03-21T13:08:39.521 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:39.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:39.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:39.622 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:39.651 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:39.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:39.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:39 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.d", "id": [3, 1, 0, 2, 4, 5]}]': finished 2024-03-21T13:08:39.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:39 smithi204 ceph-mon[33153]: osdmap e233: 8 total, 8 up, 5 in 2024-03-21T13:08:39.753 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:39.776 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:39.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:39.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:39.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:39.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:39.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:39.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:40.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:40.035 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:40.037 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:40.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:40.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:40.139 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:40.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:40.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:40.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:40.296 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:40.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:40.398 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:40.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:40.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:40 smithi163 ceph-mon[29621]: pgmap v437: 132 pgs: 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 749 B/s rd, 0 op/s 2024-03-21T13:08:40.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:40 smithi163 ceph-mon[40675]: pgmap v437: 132 pgs: 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 749 B/s rd, 0 op/s 2024-03-21T13:08:40.528 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:40.553 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:40.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:40.654 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:40.678 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:40.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:40.730 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:08:40.730 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:08:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:40 smithi204 ceph-mon[33153]: pgmap v437: 132 pgs: 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 749 B/s rd, 0 op/s 2024-03-21T13:08:40.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:40.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:40.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:40.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:40.902 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:08:40.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:40.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:40.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:41.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:41.066 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:41.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:41.167 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:41.192 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:41.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:41.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:41.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:41.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:41.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:41.423 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:41.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:41.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:41.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:41.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:41.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:41.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:41.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:41.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:41.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:41.707 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:41.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:41.808 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:41.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:41.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:41.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:41.936 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:41.961 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:41.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:41.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:42.063 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:42.088 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:42.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:42.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:42.190 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:42.214 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:42.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:42.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:08:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:08:42.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:42.340 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:42.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:42.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:42.469 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:42.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:42.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:42.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:42 smithi163 ceph-mon[29621]: pgmap v438: 132 pgs: 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:08:42.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:42 smithi163 ceph-mon[40675]: pgmap v438: 132 pgs: 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:08:42.571 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:42.595 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:42.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:42.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:42.726 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:42.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:42 smithi204 ceph-mon[33153]: pgmap v438: 132 pgs: 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:08:42.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:42.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:42.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:42.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:42.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-21T13:08:42.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:42.986 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:42.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:43.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:43.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:43.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:43.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:43.216 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:43.241 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:43.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:43.342 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:43.366 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:43.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:43.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:43.432 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:43.433 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:43.433 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:08:43.454 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:08:43.468 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:43.474 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:08:43.494 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:08:43.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:43.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:43.514 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:08:43.535 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:08:43.566 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:08:43.594 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:08:43.595 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:08:43.595 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:08:43.595 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:08:43.595 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:08:43.595 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:08:43.595 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:08:43.595 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:08:43.595 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:08:43.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:43.626 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:43.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:43.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:43.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:43.756 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:43.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:43.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:43.858 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:43.883 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:43.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:43.984 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:44.009 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:44.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:44.110 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:44.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:44.135 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:44.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:44.236 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:44.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:44.265 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:44.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:44.367 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:44.392 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:44.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:44.493 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:44.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:44.549 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:08:44.549 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:08:44.614 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'unique_pool_2' created 2024-03-21T13:08:44.619 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:44.651 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:44.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:44.753 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:44.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:44.780 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:44.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:44.882 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:44.886 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:44.909 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:44.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:45.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:44 smithi204 ceph-mon[33153]: pgmap v439: 132 pgs: 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 686 B/s rd, 0 op/s 2024-03-21T13:08:45.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4160684929' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:45.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:44 smithi163 ceph-mon[29621]: pgmap v439: 132 pgs: 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 686 B/s rd, 0 op/s 2024-03-21T13:08:45.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4160684929' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:45.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:44 smithi163 ceph-mon[40675]: pgmap v439: 132 pgs: 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 686 B/s rd, 0 op/s 2024-03-21T13:08:45.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4160684929' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:45.011 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:45.035 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:45.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:45.137 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:45.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:45.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:45.263 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:45.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:45.287 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:45.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:45.330 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-21T13:08:45.356 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:45.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:45.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:45.414 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:45.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:45.515 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:45.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:45.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2897770815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-21T13:08:45.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-21T13:08:45.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:08:45.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:08:45.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-21T13:08:45.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[29621]: osdmap e234: 8 total, 8 up, 5 in 2024-03-21T13:08:45.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[29621]: pgmap v441: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 687 B/s rd, 0 op/s 2024-03-21T13:08:45.606 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2897770815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-21T13:08:45.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-21T13:08:45.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:08:45.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:08:45.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-21T13:08:45.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[40675]: osdmap e234: 8 total, 8 up, 5 in 2024-03-21T13:08:45.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-mon[40675]: pgmap v441: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 687 B/s rd, 0 op/s 2024-03-21T13:08:45.643 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:45.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:45.668 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:45.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:45.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:45.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:45.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:45.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:45.895 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:45.918 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:45.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:45.935 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:08:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:08:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:08:46.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:45 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2897770815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-21T13:08:46.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:45 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-21T13:08:46.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:45 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:08:46.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:45 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:08:46.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:45 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-21T13:08:46.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:45 smithi204 ceph-mon[33153]: osdmap e234: 8 total, 8 up, 5 in 2024-03-21T13:08:46.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:45 smithi204 ceph-mon[33153]: pgmap v441: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 687 B/s rd, 0 op/s 2024-03-21T13:08:46.020 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:46.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:46.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:46.146 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:46.172 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:46.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:46.273 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:46.297 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:46.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:46.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:46.399 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:46.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:46.424 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:46.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:46.525 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:46.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:46.549 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:46.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:46.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:46.653 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:08:46.678 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:46 smithi163 ceph-mon[29621]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:08:46.678 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:46 smithi163 ceph-mon[29621]: osdmap e235: 8 total, 8 up, 5 in 2024-03-21T13:08:46.679 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:46 smithi163 ceph-mon[40675]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:08:46.679 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:46 smithi163 ceph-mon[40675]: osdmap e235: 8 total, 8 up, 5 in 2024-03-21T13:08:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:46.679 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:46.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:46.780 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:46.806 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:46.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:46.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:46.908 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:46.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:46.932 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:46.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:47.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:46 smithi204 ceph-mon[33153]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:08:47.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:46 smithi204 ceph-mon[33153]: osdmap e235: 8 total, 8 up, 5 in 2024-03-21T13:08:47.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:47.061 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:47.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:47.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:47.155 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:47.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:47.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:47.192 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:47.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:47.293 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:47.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:47.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:47.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:47.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:47.447 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:47.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:47.548 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:47.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:47.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:47.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:47.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:47.700 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:47.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:47.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:47 smithi163 ceph-mon[40675]: osdmap e236: 8 total, 8 up, 5 in 2024-03-21T13:08:47.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:47 smithi163 ceph-mon[40675]: pgmap v444: 148 pgs: 2 creating+activating, 10 creating+peering, 136 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-21T13:08:47.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:47 smithi163 ceph-mon[29621]: osdmap e236: 8 total, 8 up, 5 in 2024-03-21T13:08:47.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:47 smithi163 ceph-mon[29621]: pgmap v444: 148 pgs: 2 creating+activating, 10 creating+peering, 136 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-21T13:08:47.801 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:47.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:47.826 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:47.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:47.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:47.955 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:47.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:48.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:47 smithi204 ceph-mon[33153]: osdmap e236: 8 total, 8 up, 5 in 2024-03-21T13:08:48.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:47 smithi204 ceph-mon[33153]: pgmap v444: 148 pgs: 2 creating+activating, 10 creating+peering, 136 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-21T13:08:48.057 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:48.082 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:48.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:48.183 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:48.207 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:48.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:48.308 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:48.333 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:48.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:48.438 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:48.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:48.467 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:48.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:48.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:48.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:48.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:48.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:48.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:48.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:48.827 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:48.854 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:48.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:48.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:48.956 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:48.980 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:48.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:48.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:48 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/681089840' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:08:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:48 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:08:49.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:48 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/681089840' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:08:49.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:48 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:08:49.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:48 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/681089840' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:08:49.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:48 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:08:49.081 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:49.110 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:49.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:49.199 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:08:49.211 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:49.237 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:49.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:49.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:49.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:49.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:49.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:49.548 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:49.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:49.650 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:49.677 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:49.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:49.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:49.776 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:49 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:08:49.776 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:49 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:08:49.777 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:49 smithi163 ceph-mon[29621]: osdmap e237: 8 total, 8 up, 5 in 2024-03-21T13:08:49.777 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:49 smithi163 ceph-mon[29621]: pgmap v446: 148 pgs: 2 creating+activating, 10 creating+peering, 136 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 103 KiB/s rd, 0 B/s wr, 170 op/s 2024-03-21T13:08:49.777 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:49 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:08:49.777 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:49 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:08:49.777 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:49 smithi163 ceph-mon[40675]: osdmap e237: 8 total, 8 up, 5 in 2024-03-21T13:08:49.777 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:49 smithi163 ceph-mon[40675]: pgmap v446: 148 pgs: 2 creating+activating, 10 creating+peering, 136 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 103 KiB/s rd, 0 B/s wr, 170 op/s 2024-03-21T13:08:49.778 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:49.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:49.803 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:49.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:49.904 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:49.933 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:49.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:49.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:50.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:49 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:08:50.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:49 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:08:50.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:49 smithi204 ceph-mon[33153]: osdmap e237: 8 total, 8 up, 5 in 2024-03-21T13:08:50.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:49 smithi204 ceph-mon[33153]: pgmap v446: 148 pgs: 2 creating+activating, 10 creating+peering, 136 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 103 KiB/s rd, 0 B/s wr, 170 op/s 2024-03-21T13:08:50.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:50.063 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:50.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:50.165 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:50.190 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:50.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:50.258 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:08:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:08:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-03-21T13:08:50.259 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:08:50.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:50.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:50.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:50.422 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:50.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:50.447 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:50.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:50.548 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:50.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:50.573 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:50.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:50.674 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:50.699 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:50.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:50.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:50.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:50.830 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:50.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:50.879 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:50.879 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:08:50.879 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:50.879 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:08:50.931 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:50.955 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:50.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:08:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:08:51.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:51.085 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:51.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:51.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:51.187 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:51.215 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:51.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:51.267 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:51.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:51.340 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:51.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:51.442 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:51.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:51.467 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:51.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:51.569 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:51.592 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:51.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:51.695 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:51.720 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:51.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:51.753 INFO:teuthology.orchestra.run.smithi163.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-21T13:08:51.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:51.825 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:51 smithi163 ceph-mon[29621]: pgmap v447: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-21T13:08:51.825 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:51 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/329851293' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:08:51.825 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:51 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:08:51.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:51 smithi163 ceph-mon[40675]: pgmap v447: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-21T13:08:51.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:51 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/329851293' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:08:51.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:51 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:08:51.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:51.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:51.852 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-21T13:08:51.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:51.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:51.980 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:51.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:52.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:08:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:08:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:51 smithi204 ceph-mon[33153]: pgmap v447: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-21T13:08:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:51 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/329851293' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:08:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:51 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:08:52.081 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:52.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:52.106 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:52.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:52.207 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:52.232 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:52.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:52.333 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:52.358 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:52.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:52.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:52.490 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:52.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:52.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:52.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:52.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:52.722 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:52.747 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:52.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:52.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:52.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:52.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:52.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:53.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:52 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:08:53.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:52 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:08:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:52 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:08:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:52 smithi204 ceph-mon[33153]: osdmap e238: 8 total, 8 up, 5 in 2024-03-21T13:08:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:52 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:53.007 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:53.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:53.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:53.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:52 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:08:53.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:52 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:08:53.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:52 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:08:53.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:52 smithi163 ceph-mon[29621]: osdmap e238: 8 total, 8 up, 5 in 2024-03-21T13:08:53.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:52 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:53.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:52 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:08:53.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:52 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:08:53.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:52 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:08:53.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:52 smithi163 ceph-mon[40675]: osdmap e238: 8 total, 8 up, 5 in 2024-03-21T13:08:53.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:52 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:08:53.109 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:53.135 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:53.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:53.231 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:53.237 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:53.263 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:53.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:53.280 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:08:53.364 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:53.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:53.389 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:53.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:53.464 DEBUG:teuthology.orchestra.run.smithi163:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-21T13:08:53.465 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_2 2024-03-21T13:08:53.465 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:08:53.468 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-21T13:08:53.468 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-21T13:08:53.490 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:53.519 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:hints = 1 2024-03-21T13:08:53.519 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-21T13:08:53.519 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Object prefix: benchmark_data_smithi163_131650 2024-03-21T13:08:53.520 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:08:53.520 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 0 0 0 0 0 0 - 0 2024-03-21T13:08:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:53.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:53.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:53.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:53.686 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:53.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:53.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:53.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:53.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:53.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:53.924 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:53.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:53 smithi163 ceph-mon[29621]: pgmap v449: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 991 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:08:53.935 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:53 smithi163 ceph-mon[40675]: pgmap v449: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 991 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:08:53.963 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:53.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:53.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:54.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:54.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:54.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:54.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:54.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:54.243 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:54.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:54.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:53 smithi204 ceph-mon[33153]: pgmap v449: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 991 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:08:54.344 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:54.381 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:54.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:54.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:54.520 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 1 15 3539 3524 220.236 220.25 0.00524374 0.00452477 2024-03-21T13:08:54.524 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:54.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:54.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:54.626 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:54.660 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:54.666 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:54.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:54.685 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:54.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:54.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:54 smithi163 ceph-mon[29621]: 3.19 scrub starts 2024-03-21T13:08:54.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:54 smithi163 ceph-mon[29621]: 3.19 scrub ok 2024-03-21T13:08:54.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:54 smithi163 ceph-mon[29621]: 5.9 scrub starts 2024-03-21T13:08:54.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:54 smithi163 ceph-mon[29621]: 5.9 scrub ok 2024-03-21T13:08:54.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:54 smithi163 ceph-mon[29621]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:08:54.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:54 smithi163 ceph-mon[40675]: 3.19 scrub starts 2024-03-21T13:08:54.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:54 smithi163 ceph-mon[40675]: 3.19 scrub ok 2024-03-21T13:08:54.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:54 smithi163 ceph-mon[40675]: 5.9 scrub starts 2024-03-21T13:08:54.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:54 smithi163 ceph-mon[40675]: 5.9 scrub ok 2024-03-21T13:08:54.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:54 smithi163 ceph-mon[40675]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:08:54.808 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:54.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:54.910 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:54.949 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:54.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:55.051 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:55.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:55.088 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:55.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:55.189 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:55.226 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:55.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:55.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:55.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:54 smithi204 ceph-mon[33153]: 3.19 scrub starts 2024-03-21T13:08:55.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:54 smithi204 ceph-mon[33153]: 3.19 scrub ok 2024-03-21T13:08:55.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:54 smithi204 ceph-mon[33153]: 5.9 scrub starts 2024-03-21T13:08:55.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:54 smithi204 ceph-mon[33153]: 5.9 scrub ok 2024-03-21T13:08:55.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:54 smithi204 ceph-mon[33153]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:08:55.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:55.365 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:55.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:55.467 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:55.512 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:55.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:55.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:55.520 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 2 16 6879 6863 214.453 208.688 0.00501213 0.00465517 2024-03-21T13:08:55.602 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:08:55.613 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:08:55.613 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:08:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:08:55.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:08:55.648 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:08:55.653 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:55.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:08:55.685 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:08:55.723 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:08:55.754 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:55.760 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:08:55.790 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:55.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:55.796 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:08:55.833 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:08:55.835 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:08:55.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:08:55.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:08:55.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:08:55.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:08:55.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:08:55.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:08:55.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:08:55.893 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:55.929 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:55.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:55.936 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:08:55.936 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:08:55.936 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[29621]: 4.1 scrub starts 2024-03-21T13:08:55.936 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[29621]: 4.1 scrub ok 2024-03-21T13:08:55.936 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[29621]: osdmap e239: 8 total, 8 up, 5 in 2024-03-21T13:08:55.937 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[29621]: pgmap v451: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 993 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:08:55.937 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4013839858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:55.937 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:08:55.937 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:08:55.937 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[40675]: 4.1 scrub starts 2024-03-21T13:08:55.937 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[40675]: 4.1 scrub ok 2024-03-21T13:08:55.937 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[40675]: osdmap e239: 8 total, 8 up, 5 in 2024-03-21T13:08:55.937 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[40675]: pgmap v451: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 993 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:08:55.937 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:55 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4013839858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:56.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:56.068 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:56.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:56.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:56.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:56.203 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:56.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:56.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:55 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:08:56.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:55 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:08:56.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:55 smithi204 ceph-mon[33153]: 4.1 scrub starts 2024-03-21T13:08:56.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:55 smithi204 ceph-mon[33153]: 4.1 scrub ok 2024-03-21T13:08:56.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:55 smithi204 ceph-mon[33153]: osdmap e239: 8 total, 8 up, 5 in 2024-03-21T13:08:56.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:55 smithi204 ceph-mon[33153]: pgmap v451: 148 pgs: 148 active+clean; 582 KiB data, 2.1 GiB used, 445 GiB / 447 GiB avail; 993 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:08:56.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:55 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4013839858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:08:56.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:56.344 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:56.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:56.445 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:08:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:56.480 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:56.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:08:56.520 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 3 16 9986 9970 207.692 194.188 0.00637742 0.00480123 2024-03-21T13:08:56.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:56.619 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:56.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:56.721 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:56.759 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:56.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:56.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:56.860 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:56.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:56.898 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:56.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:56.999 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:57.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:56 smithi163 ceph-mon[29621]: 6.0 scrub starts 2024-03-21T13:08:57.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:56 smithi163 ceph-mon[29621]: 6.0 scrub ok 2024-03-21T13:08:57.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:56 smithi163 ceph-mon[40675]: 6.0 scrub starts 2024-03-21T13:08:57.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:56 smithi163 ceph-mon[40675]: 6.0 scrub ok 2024-03-21T13:08:57.036 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:57.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:57.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:57.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:57.172 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:57.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:57.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:56 smithi204 ceph-mon[33153]: 6.0 scrub starts 2024-03-21T13:08:57.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:56 smithi204 ceph-mon[33153]: 6.0 scrub ok 2024-03-21T13:08:57.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:57.311 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:57.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:57.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:57.413 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:08:57.448 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:57.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:08:57.521 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 4 16 13158 13142 205.31 198.25 0.00338792 0.00486346 2024-03-21T13:08:57.550 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:57.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:57.588 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:57.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:57.689 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:57.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:57.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:57.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:57.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:57.867 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:57.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:57.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:58.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:58.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:58.106 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:58.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:58.143 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:58.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:58.152 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:57 smithi163 ceph-mon[29621]: 6.16 scrub starts 2024-03-21T13:08:58.152 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:57 smithi163 ceph-mon[29621]: 6.16 scrub ok 2024-03-21T13:08:58.152 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:57 smithi163 ceph-mon[29621]: 5.7 scrub starts 2024-03-21T13:08:58.152 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:57 smithi163 ceph-mon[29621]: 5.7 scrub ok 2024-03-21T13:08:58.152 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:57 smithi163 ceph-mon[29621]: pgmap v452: 148 pgs: 148 active+clean; 365 MiB data, 2.9 GiB used, 444 GiB / 447 GiB avail; 1.4 KiB/s rd, 45 MiB/s wr, 729 op/s 2024-03-21T13:08:58.153 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:57 smithi163 ceph-mon[40675]: 6.16 scrub starts 2024-03-21T13:08:58.153 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:57 smithi163 ceph-mon[40675]: 6.16 scrub ok 2024-03-21T13:08:58.153 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:57 smithi163 ceph-mon[40675]: 5.7 scrub starts 2024-03-21T13:08:58.153 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:57 smithi163 ceph-mon[40675]: 5.7 scrub ok 2024-03-21T13:08:58.153 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:57 smithi163 ceph-mon[40675]: pgmap v452: 148 pgs: 148 active+clean; 365 MiB data, 2.9 GiB used, 444 GiB / 447 GiB avail; 1.4 KiB/s rd, 45 MiB/s wr, 729 op/s 2024-03-21T13:08:58.244 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:58.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:57 smithi204 ceph-mon[33153]: 6.16 scrub starts 2024-03-21T13:08:58.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:57 smithi204 ceph-mon[33153]: 6.16 scrub ok 2024-03-21T13:08:58.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:57 smithi204 ceph-mon[33153]: 5.7 scrub starts 2024-03-21T13:08:58.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:57 smithi204 ceph-mon[33153]: 5.7 scrub ok 2024-03-21T13:08:58.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:57 smithi204 ceph-mon[33153]: pgmap v452: 148 pgs: 148 active+clean; 365 MiB data, 2.9 GiB used, 444 GiB / 447 GiB avail; 1.4 KiB/s rd, 45 MiB/s wr, 729 op/s 2024-03-21T13:08:58.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:58.268 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:58.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:58.369 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:58.392 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:58.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:58.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:58.494 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:58.521 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 5 16 15784 15768 197.07 164.125 0.00350721 0.00506745 2024-03-21T13:08:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:58.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:58.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:58.635 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:58.670 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:58.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:58.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:58.772 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:08:58.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:58.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:08:58.911 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:58.945 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:58.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:59.046 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:59.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:59.077 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:59.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:59.147 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:08:59.178 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:59.184 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[29621]: 3.c scrub starts 2024-03-21T13:08:59.184 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[29621]: 3.c scrub ok 2024-03-21T13:08:59.184 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[29621]: 6.18 scrub starts 2024-03-21T13:08:59.184 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[29621]: 6.18 scrub ok 2024-03-21T13:08:59.184 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[29621]: 6.1 scrub starts 2024-03-21T13:08:59.184 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[29621]: 6.1 scrub ok 2024-03-21T13:08:59.184 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[29621]: 4.2 scrub starts 2024-03-21T13:08:59.185 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[29621]: 4.2 scrub ok 2024-03-21T13:08:59.185 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[40675]: 3.c scrub starts 2024-03-21T13:08:59.185 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[40675]: 3.c scrub ok 2024-03-21T13:08:59.185 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[40675]: 6.18 scrub starts 2024-03-21T13:08:59.185 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[40675]: 6.18 scrub ok 2024-03-21T13:08:59.185 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[40675]: 6.1 scrub starts 2024-03-21T13:08:59.185 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[40675]: 6.1 scrub ok 2024-03-21T13:08:59.185 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[40675]: 4.2 scrub starts 2024-03-21T13:08:59.185 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:58 smithi163 ceph-mon[40675]: 4.2 scrub ok 2024-03-21T13:08:59.226 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:59.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:59.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:59.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:58 smithi204 ceph-mon[33153]: 3.c scrub starts 2024-03-21T13:08:59.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:58 smithi204 ceph-mon[33153]: 3.c scrub ok 2024-03-21T13:08:59.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:58 smithi204 ceph-mon[33153]: 6.18 scrub starts 2024-03-21T13:08:59.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:58 smithi204 ceph-mon[33153]: 6.18 scrub ok 2024-03-21T13:08:59.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:58 smithi204 ceph-mon[33153]: 6.1 scrub starts 2024-03-21T13:08:59.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:58 smithi204 ceph-mon[33153]: 6.1 scrub ok 2024-03-21T13:08:59.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:58 smithi204 ceph-mon[33153]: 4.2 scrub starts 2024-03-21T13:08:59.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:58 smithi204 ceph-mon[33153]: 4.2 scrub ok 2024-03-21T13:08:59.328 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:08:59.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:59.372 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:59.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:08:59.473 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:08:59.520 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 6 16 18004 17988 187.348 138.75 0.00652311 0.00533115 2024-03-21T13:08:59.523 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:08:59.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:59.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:08:59.627 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:59.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:59.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:08:59.768 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:08:59.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:59.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:59.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:08:59.904 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:08:59.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:08:59.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:08:59.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:00.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:00.077 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:00.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:00.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:00.203 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:00.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:00.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:59 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:09:00.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:59 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:09:00.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:59 smithi204 ceph-mon[33153]: 6.5 scrub starts 2024-03-21T13:09:00.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:59 smithi204 ceph-mon[33153]: 6.5 scrub ok 2024-03-21T13:09:00.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:59 smithi204 ceph-mon[33153]: 6.14 scrub starts 2024-03-21T13:09:00.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:59 smithi204 ceph-mon[33153]: 6.14 scrub ok 2024-03-21T13:09:00.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:59 smithi204 ceph-mon[33153]: osdmap e240: 8 total, 8 up, 5 in 2024-03-21T13:09:00.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:08:59 smithi204 ceph-mon[33153]: pgmap v454: 148 pgs: 148 active+clean; 365 MiB data, 2.9 GiB used, 444 GiB / 447 GiB avail; 725 B/s rd, 52 MiB/s wr, 825 op/s 2024-03-21T13:09:00.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:09:00.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:09:00.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[29621]: 6.5 scrub starts 2024-03-21T13:09:00.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[29621]: 6.5 scrub ok 2024-03-21T13:09:00.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[29621]: 6.14 scrub starts 2024-03-21T13:09:00.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[29621]: 6.14 scrub ok 2024-03-21T13:09:00.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[29621]: osdmap e240: 8 total, 8 up, 5 in 2024-03-21T13:09:00.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[29621]: pgmap v454: 148 pgs: 148 active+clean; 365 MiB data, 2.9 GiB used, 444 GiB / 447 GiB avail; 725 B/s rd, 52 MiB/s wr, 825 op/s 2024-03-21T13:09:00.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:09:00.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:09:00.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[40675]: 6.5 scrub starts 2024-03-21T13:09:00.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[40675]: 6.5 scrub ok 2024-03-21T13:09:00.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[40675]: 6.14 scrub starts 2024-03-21T13:09:00.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[40675]: 6.14 scrub ok 2024-03-21T13:09:00.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[40675]: osdmap e240: 8 total, 8 up, 5 in 2024-03-21T13:09:00.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:08:59 smithi163 ceph-mon[40675]: pgmap v454: 148 pgs: 148 active+clean; 365 MiB data, 2.9 GiB used, 444 GiB / 447 GiB avail; 725 B/s rd, 52 MiB/s wr, 825 op/s 2024-03-21T13:09:00.306 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:00.342 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:00.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:00.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:00.444 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:00.481 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:00.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:00.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:00.521 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 7 16 20270 20254 180.815 141.625 0.00523614 0.00552462 2024-03-21T13:09:00.583 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:00.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:00.616 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:00.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:00.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:00.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:00.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:00.854 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:00.891 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:00.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:00.926 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:09:00.926 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:09:00.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:01.029 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:01.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:01.130 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:01.169 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:01.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:01.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:01.306 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:01.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:01.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:01.408 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:01.443 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:01.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:01.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:01.521 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 8 16 22669 22653 176.953 149.938 0.00594027 0.00564555 2024-03-21T13:09:01.545 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:01.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:01.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:01.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:01.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:01.718 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:01.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:01.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:01.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:01.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:01.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:01.958 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:01.991 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:01.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:01.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:02.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:02.129 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:02.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:02.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:02.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:02.239 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-21T13:09:02.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:09:01 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:09:02.267 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:02.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:02.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:02.369 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:02.407 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:02.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:02.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:02.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:02 smithi163 ceph-mon[29621]: pgmap v455: 148 pgs: 148 active+clean; 911 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-21T13:09:02.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:02 smithi163 ceph-mon[29621]: osdmap e241: 8 total, 8 up, 5 in 2024-03-21T13:09:02.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:02 smithi163 ceph-mon[29621]: 6.1d scrub starts 2024-03-21T13:09:02.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:02 smithi163 ceph-mon[29621]: 6.1d scrub ok 2024-03-21T13:09:02.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:02 smithi163 ceph-mon[40675]: pgmap v455: 148 pgs: 148 active+clean; 911 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-21T13:09:02.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:02 smithi163 ceph-mon[40675]: osdmap e241: 8 total, 8 up, 5 in 2024-03-21T13:09:02.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:02 smithi163 ceph-mon[40675]: 6.1d scrub starts 2024-03-21T13:09:02.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:02 smithi163 ceph-mon[40675]: 6.1d scrub ok 2024-03-21T13:09:02.509 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:02.522 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 9 16 25118 25102 174.278 153.062 0.00507934 0.00573112 2024-03-21T13:09:02.547 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:02.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:02.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:02.649 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:02.702 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:02.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:02 smithi204 ceph-mon[33153]: pgmap v455: 148 pgs: 148 active+clean; 911 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-21T13:09:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:02 smithi204 ceph-mon[33153]: osdmap e241: 8 total, 8 up, 5 in 2024-03-21T13:09:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:02 smithi204 ceph-mon[33153]: 6.1d scrub starts 2024-03-21T13:09:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:02 smithi204 ceph-mon[33153]: 6.1d scrub ok 2024-03-21T13:09:02.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:02.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:02.841 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:02.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:02.942 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:02.981 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:02.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:02.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:03.083 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:03.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:03.123 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:03.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:03.224 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:03.274 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:03.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:03.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:03.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:03.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:03.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:03.514 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:03.522 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 10 16 27249 27233 170.157 133.188 0.00777788 0.00586991 2024-03-21T13:09:03.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:03.553 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:03.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:03.625 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:09:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:03 smithi163 ceph-mon[40675]: 4.1d scrub starts 2024-03-21T13:09:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:03 smithi163 ceph-mon[40675]: 4.1d scrub ok 2024-03-21T13:09:03.627 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:03.627 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:03 smithi163 ceph-mon[29621]: 4.1d scrub starts 2024-03-21T13:09:03.628 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:03 smithi163 ceph-mon[29621]: 4.1d scrub ok 2024-03-21T13:09:03.636 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:09:03.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:03.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:03.694 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:03.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:03.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:03 smithi204 ceph-mon[33153]: 4.1d scrub starts 2024-03-21T13:09:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:03 smithi204 ceph-mon[33153]: 4.1d scrub ok 2024-03-21T13:09:03.795 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:03.833 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:03.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:03.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:03.936 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:03.973 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:03.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:04.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:04.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:04.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:04.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:04.212 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:04.253 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:04.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:04.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:04.355 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:04.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:04.396 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:04.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:04.497 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:04.523 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 11 16 29457 29441 167.233 138 0.00632925 0.00597497 2024-03-21T13:09:04.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:04.538 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:04.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:04.639 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:04.675 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:04.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:04.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:04 smithi204 ceph-mon[33153]: pgmap v457: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.44k op/s 2024-03-21T13:09:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:04 smithi204 ceph-mon[33153]: 4.d scrub starts 2024-03-21T13:09:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:04 smithi204 ceph-mon[33153]: 4.d scrub ok 2024-03-21T13:09:04.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:04 smithi163 ceph-mon[29621]: pgmap v457: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.44k op/s 2024-03-21T13:09:04.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:04 smithi163 ceph-mon[29621]: 4.d scrub starts 2024-03-21T13:09:04.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:04 smithi163 ceph-mon[29621]: 4.d scrub ok 2024-03-21T13:09:04.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:04 smithi163 ceph-mon[40675]: pgmap v457: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.44k op/s 2024-03-21T13:09:04.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:04 smithi163 ceph-mon[40675]: 4.d scrub starts 2024-03-21T13:09:04.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:04 smithi163 ceph-mon[40675]: 4.d scrub ok 2024-03-21T13:09:04.778 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:04.823 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:04.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:04.924 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:04.971 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:04.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:04.973 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-03-21T13:09:02.808758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1277761960,"num_objects":19686,"num_object_clones":0,"num_object_copies":59058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19686,"num_whiteouts":0,"num_read":1959,"num_read_kb":1805,"num_write":20389,"num_write_kb":1249693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":1777081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20441,"ondisk_log_size":20441,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":5970196,"kb_used_data":3636068,"kb_used_omap":7,"kb_used_meta":2334072,"kb_avail":462755564,"statfs":{"total":479975178240,"available":473861697536,"internally_reserved":0,"allocated":3723333632,"data_stored":3720990731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7967,"internal_metadata":2390089953},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1277165568,"num_objects":19488,"num_object_clones":0,"num_object_copies":58464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19488,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":19488,"num_write_kb":1247232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003108"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":230,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807887+0000","last_change":"2024-03-21T13:06:45.404501+0000","last_active":"2024-03-21T13:08:58.807887+0000","last_peered":"2024-03-21T13:08:58.807887+0000","last_clean":"2024-03-21T13:08:58.807887+0000","last_became_active":"2024-03-21T13:06:45.404217+0000","last_became_peered":"2024-03-21T13:06:45.404217+0000","last_unstale":"2024-03-21T13:08:58.807887+0000","last_undegraded":"2024-03-21T13:08:58.807887+0000","last_fullsized":"2024-03-21T13:08:58.807887+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:20.083111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.083111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":276,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.805164+0000","last_change":"2024-03-21T13:08:20.891722+0000","last_active":"2024-03-21T13:08:58.805164+0000","last_peered":"2024-03-21T13:08:58.805164+0000","last_clean":"2024-03-21T13:08:58.805164+0000","last_became_active":"2024-03-21T13:08:20.891334+0000","last_became_peered":"2024-03-21T13:08:20.891334+0000","last_unstale":"2024-03-21T13:08:58.805164+0000","last_undegraded":"2024-03-21T13:08:58.805164+0000","last_fullsized":"2024-03-21T13:08:58.805164+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:54.360939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:07:54.360939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:11.125935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":105,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807523+0000","last_change":"2024-03-21T13:06:29.484964+0000","last_active":"2024-03-21T13:08:58.807523+0000","last_peered":"2024-03-21T13:08:58.807523+0000","last_clean":"2024-03-21T13:08:58.807523+0000","last_became_active":"2024-03-21T13:06:29.484528+0000","last_became_peered":"2024-03-21T13:06:29.484528+0000","last_unstale":"2024-03-21T13:08:58.807523+0000","last_undegraded":"2024-03-21T13:08:58.807523+0000","last_fullsized":"2024-03-21T13:08:58.807523+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:49.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.260693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"234'19","reported_seq":455,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.811628+0000","last_change":"2024-03-21T13:08:00.054633+0000","last_active":"2024-03-21T13:08:58.811628+0000","last_peered":"2024-03-21T13:08:58.811628+0000","last_clean":"2024-03-21T13:08:58.811628+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:08:58.811628+0000","last_undegraded":"2024-03-21T13:08:58.811628+0000","last_fullsized":"2024-03-21T13:08:58.811628+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-03-21T13:08:00.054482+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:08:00.054482+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:21.647462+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":370,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803379+0000","last_change":"2024-03-21T13:05:07.334438+0000","last_active":"2024-03-21T13:08:58.803379+0000","last_peered":"2024-03-21T13:08:58.803379+0000","last_clean":"2024-03-21T13:08:58.803379+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:08:58.803379+0000","last_undegraded":"2024-03-21T13:08:58.803379+0000","last_fullsized":"2024-03-21T13:08:58.803379+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.330296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.330296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":315,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807627+0000","last_change":"2024-03-21T13:06:29.490705+0000","last_active":"2024-03-21T13:08:58.807627+0000","last_peered":"2024-03-21T13:08:58.807627+0000","last_clean":"2024-03-21T13:08:58.807627+0000","last_became_active":"2024-03-21T13:06:29.490004+0000","last_became_peered":"2024-03-21T13:06:29.490004+0000","last_unstale":"2024-03-21T13:08:58.807627+0000","last_undegraded":"2024-03-21T13:08:58.807627+0000","last_fullsized":"2024-03-21T13:08:58.807627+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:50.991735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.991735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":392,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.802739+0000","last_change":"2024-03-21T13:06:29.542506+0000","last_active":"2024-03-21T13:08:58.802739+0000","last_peered":"2024-03-21T13:08:58.802739+0000","last_clean":"2024-03-21T13:08:58.802739+0000","last_became_active":"2024-03-21T13:06:29.511102+0000","last_became_peered":"2024-03-21T13:06:29.511102+0000","last_unstale":"2024-03-21T13:08:58.802739+0000","last_undegraded":"2024-03-21T13:08:58.802739+0000","last_fullsized":"2024-03-21T13:08:58.802739+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_clean_scrub_stamp":"2024-03-21T13:01:38.765896+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":443,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804204+0000","last_change":"2024-03-21T13:06:29.546224+0000","last_active":"2024-03-21T13:08:58.804204+0000","last_peered":"2024-03-21T13:08:58.804204+0000","last_clean":"2024-03-21T13:08:58.804204+0000","last_became_active":"2024-03-21T13:06:29.509791+0000","last_became_peered":"2024-03-21T13:06:29.509791+0000","last_unstale":"2024-03-21T13:08:58.804204+0000","last_undegraded":"2024-03-21T13:08:58.804204+0000","last_fullsized":"2024-03-21T13:08:58.804204+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:26.476043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.476043+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":282,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803922+0000","last_change":"2024-03-21T13:08:57.628140+0000","last_active":"2024-03-21T13:08:58.803922+0000","last_peered":"2024-03-21T13:08:58.803922+0000","last_clean":"2024-03-21T13:08:58.803922+0000","last_became_active":"2024-03-21T13:06:29.487372+0000","last_became_peered":"2024-03-21T13:06:29.487372+0000","last_unstale":"2024-03-21T13:08:58.803922+0000","last_undegraded":"2024-03-21T13:08:58.803922+0000","last_fullsized":"2024-03-21T13:08:58.803922+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:57.627967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.627967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:07.465261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":278,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807003+0000","last_change":"2024-03-21T13:08:08.963344+0000","last_active":"2024-03-21T13:08:58.807003+0000","last_peered":"2024-03-21T13:08:58.807003+0000","last_clean":"2024-03-21T13:08:58.807003+0000","last_became_active":"2024-03-21T13:02:26.236707+0000","last_became_peered":"2024-03-21T13:02:26.236707+0000","last_unstale":"2024-03-21T13:08:58.807003+0000","last_undegraded":"2024-03-21T13:08:58.807003+0000","last_fullsized":"2024-03-21T13:08:58.807003+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:08.963164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:08:08.963164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:30.848955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":254,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807764+0000","last_change":"2024-03-21T13:05:45.989199+0000","last_active":"2024-03-21T13:08:58.807764+0000","last_peered":"2024-03-21T13:08:58.807764+0000","last_clean":"2024-03-21T13:08:58.807764+0000","last_became_active":"2024-03-21T13:02:26.245402+0000","last_became_peered":"2024-03-21T13:02:26.245402+0000","last_unstale":"2024-03-21T13:08:58.807764+0000","last_undegraded":"2024-03-21T13:08:58.807764+0000","last_fullsized":"2024-03-21T13:08:58.807764+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:45.989114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:45.989114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":381,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803045+0000","last_change":"2024-03-21T13:05:51.580203+0000","last_active":"2024-03-21T13:08:58.803045+0000","last_peered":"2024-03-21T13:08:58.803045+0000","last_clean":"2024-03-21T13:08:58.803045+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:08:58.803045+0000","last_undegraded":"2024-03-21T13:08:58.803045+0000","last_fullsized":"2024-03-21T13:08:58.803045+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-21T13:05:51.580074+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:05:51.580074+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":380,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803143+0000","last_change":"2024-03-21T13:08:17.747461+0000","last_active":"2024-03-21T13:08:58.803143+0000","last_peered":"2024-03-21T13:08:58.803143+0000","last_clean":"2024-03-21T13:08:58.803143+0000","last_became_active":"2024-03-21T13:06:29.505469+0000","last_became_peered":"2024-03-21T13:06:29.505469+0000","last_unstale":"2024-03-21T13:08:58.803143+0000","last_undegraded":"2024-03-21T13:08:58.803143+0000","last_fullsized":"2024-03-21T13:08:58.803143+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:17.747360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:17.747360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:40.450392+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":394,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.808128+0000","last_change":"2024-03-21T13:08:09.443525+0000","last_active":"2024-03-21T13:08:58.808128+0000","last_peered":"2024-03-21T13:08:58.808128+0000","last_clean":"2024-03-21T13:08:58.808128+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:08:58.808128+0000","last_undegraded":"2024-03-21T13:08:58.808128+0000","last_fullsized":"2024-03-21T13:08:58.808128+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:09.443411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:09.443411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:10.311895+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":402,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.814120+0000","last_change":"2024-03-21T13:08:17.962862+0000","last_active":"2024-03-21T13:08:58.814120+0000","last_peered":"2024-03-21T13:08:58.814120+0000","last_clean":"2024-03-21T13:08:58.814120+0000","last_became_active":"2024-03-21T13:02:26.230057+0000","last_became_peered":"2024-03-21T13:02:26.230057+0000","last_unstale":"2024-03-21T13:08:58.814120+0000","last_undegraded":"2024-03-21T13:08:58.814120+0000","last_fullsized":"2024-03-21T13:08:58.814120+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:17.962759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:17.962759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:26.948935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"234'9","reported_seq":389,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803203+0000","last_change":"2024-03-21T13:06:29.517935+0000","last_active":"2024-03-21T13:08:58.803203+0000","last_peered":"2024-03-21T13:08:58.803203+0000","last_clean":"2024-03-21T13:08:58.803203+0000","last_became_active":"2024-03-21T13:06:29.506772+0000","last_became_peered":"2024-03-21T13:06:29.506772+0000","last_unstale":"2024-03-21T13:08:58.803203+0000","last_undegraded":"2024-03-21T13:08:58.803203+0000","last_fullsized":"2024-03-21T13:08:58.803203+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T13:05:16.384676+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:05:16.384676+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":401,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.814529+0000","last_change":"2024-03-21T13:07:47.084827+0000","last_active":"2024-03-21T13:08:58.814529+0000","last_peered":"2024-03-21T13:08:58.814529+0000","last_clean":"2024-03-21T13:08:58.814529+0000","last_became_active":"2024-03-21T13:02:26.226675+0000","last_became_peered":"2024-03-21T13:02:26.226675+0000","last_unstale":"2024-03-21T13:08:58.814529+0000","last_undegraded":"2024-03-21T13:08:58.814529+0000","last_fullsized":"2024-03-21T13:08:58.814529+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:47.084713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:47.084713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:14.317582+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":288,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.815043+0000","last_change":"2024-03-21T13:08:18.994766+0000","last_active":"2024-03-21T13:08:58.815043+0000","last_peered":"2024-03-21T13:08:58.815043+0000","last_clean":"2024-03-21T13:08:58.815043+0000","last_became_active":"2024-03-21T13:02:26.231879+0000","last_became_peered":"2024-03-21T13:02:26.231879+0000","last_unstale":"2024-03-21T13:08:58.815043+0000","last_undegraded":"2024-03-21T13:08:58.815043+0000","last_fullsized":"2024-03-21T13:08:58.815043+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:18.994687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.994687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:31.114182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":280,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804562+0000","last_change":"2024-03-21T13:06:28.985227+0000","last_active":"2024-03-21T13:08:58.804562+0000","last_peered":"2024-03-21T13:08:58.804562+0000","last_clean":"2024-03-21T13:08:58.804562+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:08:58.804562+0000","last_undegraded":"2024-03-21T13:08:58.804562+0000","last_fullsized":"2024-03-21T13:08:58.804562+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:28.985145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:28.985145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":179,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.805862+0000","last_change":"2024-03-21T13:06:29.540373+0000","last_active":"2024-03-21T13:08:58.805862+0000","last_peered":"2024-03-21T13:08:58.805862+0000","last_clean":"2024-03-21T13:08:58.805862+0000","last_became_active":"2024-03-21T13:06:29.504235+0000","last_became_peered":"2024-03-21T13:06:29.504235+0000","last_unstale":"2024-03-21T13:08:58.805862+0000","last_undegraded":"2024-03-21T13:08:58.805862+0000","last_fullsized":"2024-03-21T13:08:58.805862+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:20.158184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.158184+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":382,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.802880+0000","last_change":"2024-03-21T13:07:44.816849+0000","last_active":"2024-03-21T13:08:58.802880+0000","last_peered":"2024-03-21T13:08:58.802880+0000","last_clean":"2024-03-21T13:08:58.802880+0000","last_became_active":"2024-03-21T13:06:29.512081+0000","last_became_peered":"2024-03-21T13:06:29.512081+0000","last_unstale":"2024-03-21T13:08:58.802880+0000","last_undegraded":"2024-03-21T13:08:58.802880+0000","last_fullsized":"2024-03-21T13:08:58.802880+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:44.816722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:44.816722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:06.483247+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":301,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.805672+0000","last_change":"2024-03-21T13:06:29.500559+0000","last_active":"2024-03-21T13:08:58.805672+0000","last_peered":"2024-03-21T13:08:58.805672+0000","last_clean":"2024-03-21T13:08:58.805672+0000","last_became_active":"2024-03-21T13:06:29.498872+0000","last_became_peered":"2024-03-21T13:06:29.498872+0000","last_unstale":"2024-03-21T13:08:58.805672+0000","last_undegraded":"2024-03-21T13:08:58.805672+0000","last_fullsized":"2024-03-21T13:08:58.805672+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:29.464856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:04:29.464856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":418,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.812472+0000","last_change":"2024-03-21T13:08:20.956897+0000","last_active":"2024-03-21T13:08:58.812472+0000","last_peered":"2024-03-21T13:08:58.812472+0000","last_clean":"2024-03-21T13:08:58.812472+0000","last_became_active":"2024-03-21T13:06:29.487644+0000","last_became_peered":"2024-03-21T13:06:29.487644+0000","last_unstale":"2024-03-21T13:08:58.812472+0000","last_undegraded":"2024-03-21T13:08:58.812472+0000","last_fullsized":"2024-03-21T13:08:58.812472+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:20.956778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:20.956778+0000","last_clean_scrub_stamp":"2024-03-21T13:08:20.956778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:37.554104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":438,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803790+0000","last_change":"2024-03-21T13:02:14.362313+0000","last_active":"2024-03-21T13:08:58.803790+0000","last_peered":"2024-03-21T13:08:58.803790+0000","last_clean":"2024-03-21T13:08:58.803790+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:08:58.803790+0000","last_undegraded":"2024-03-21T13:08:58.803790+0000","last_fullsized":"2024-03-21T13:08:58.803790+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:01:39.755345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:39.755345+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":236,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804792+0000","last_change":"2024-03-21T13:06:29.537271+0000","last_active":"2024-03-21T13:08:58.804792+0000","last_peered":"2024-03-21T13:08:58.804792+0000","last_clean":"2024-03-21T13:08:58.804792+0000","last_became_active":"2024-03-21T13:06:29.504570+0000","last_became_peered":"2024-03-21T13:06:29.504570+0000","last_unstale":"2024-03-21T13:08:58.804792+0000","last_undegraded":"2024-03-21T13:08:58.804792+0000","last_fullsized":"2024-03-21T13:08:58.804792+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:50.295057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.295057+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":518,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807729+0000","last_change":"2024-03-21T13:05:38.022572+0000","last_active":"2024-03-21T13:08:58.807729+0000","last_peered":"2024-03-21T13:08:58.807729+0000","last_clean":"2024-03-21T13:08:58.807729+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:08:58.807729+0000","last_undegraded":"2024-03-21T13:08:58.807729+0000","last_fullsized":"2024-03-21T13:08:58.807729+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:05:38.022439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.022439+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":288,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803559+0000","last_change":"2024-03-21T13:08:53.540108+0000","last_active":"2024-03-21T13:08:58.803559+0000","last_peered":"2024-03-21T13:08:58.803559+0000","last_clean":"2024-03-21T13:08:58.803559+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:08:58.803559+0000","last_undegraded":"2024-03-21T13:08:58.803559+0000","last_fullsized":"2024-03-21T13:08:58.803559+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:08:53.539909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:53.539909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:03.095158+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":330,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.806392+0000","last_change":"2024-03-21T13:05:21.117502+0000","last_active":"2024-03-21T13:08:58.806392+0000","last_peered":"2024-03-21T13:08:58.806392+0000","last_clean":"2024-03-21T13:08:58.806392+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:08:58.806392+0000","last_undegraded":"2024-03-21T13:08:58.806392+0000","last_fullsized":"2024-03-21T13:08:58.806392+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:21.117370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.117370+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":270,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.814051+0000","last_change":"2024-03-21T13:06:29.499863+0000","last_active":"2024-03-21T13:08:58.814051+0000","last_peered":"2024-03-21T13:08:58.814051+0000","last_clean":"2024-03-21T13:08:58.814051+0000","last_became_active":"2024-03-21T13:06:29.499497+0000","last_became_peered":"2024-03-21T13:06:29.499497+0000","last_unstale":"2024-03-21T13:08:58.814051+0000","last_undegraded":"2024-03-21T13:08:58.814051+0000","last_fullsized":"2024-03-21T13:08:58.814051+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:26.498073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:06:26.498073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":287,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.806250+0000","last_change":"2024-03-21T13:06:29.501937+0000","last_active":"2024-03-21T13:08:58.806250+0000","last_peered":"2024-03-21T13:08:58.806250+0000","last_clean":"2024-03-21T13:08:58.806250+0000","last_became_active":"2024-03-21T13:06:29.501232+0000","last_became_peered":"2024-03-21T13:06:29.501232+0000","last_unstale":"2024-03-21T13:08:58.806250+0000","last_undegraded":"2024-03-21T13:08:58.806250+0000","last_fullsized":"2024-03-21T13:08:58.806250+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.088812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.088812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":394,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803776+0000","last_change":"2024-03-21T13:06:29.501866+0000","last_active":"2024-03-21T13:08:58.803776+0000","last_peered":"2024-03-21T13:08:58.803776+0000","last_clean":"2024-03-21T13:08:58.803776+0000","last_became_active":"2024-03-21T13:06:29.501471+0000","last_became_peered":"2024-03-21T13:06:29.501471+0000","last_unstale":"2024-03-21T13:08:58.803776+0000","last_undegraded":"2024-03-21T13:08:58.803776+0000","last_fullsized":"2024-03-21T13:08:58.803776+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:09.348646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:09.348646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"234'19","reported_seq":453,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807836+0000","last_change":"2024-03-21T13:05:49.982963+0000","last_active":"2024-03-21T13:08:58.807836+0000","last_peered":"2024-03-21T13:08:58.807836+0000","last_clean":"2024-03-21T13:08:58.807836+0000","last_became_active":"2024-03-21T13:02:26.254436+0000","last_became_peered":"2024-03-21T13:02:26.254436+0000","last_unstale":"2024-03-21T13:08:58.807836+0000","last_undegraded":"2024-03-21T13:08:58.807836+0000","last_fullsized":"2024-03-21T13:08:58.807836+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:49.982812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.982812+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":382,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.808313+0000","last_change":"2024-03-21T13:06:29.980827+0000","last_active":"2024-03-21T13:08:58.808313+0000","last_peered":"2024-03-21T13:08:58.808313+0000","last_clean":"2024-03-21T13:08:58.808313+0000","last_became_active":"2024-03-21T13:06:29.503679+0000","last_became_peered":"2024-03-21T13:06:29.503679+0000","last_unstale":"2024-03-21T13:08:58.808313+0000","last_undegraded":"2024-03-21T13:08:58.808313+0000","last_fullsized":"2024-03-21T13:08:58.808313+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:29.980750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:06:29.980750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":382,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.806828+0000","last_change":"2024-03-21T13:05:42.990138+0000","last_active":"2024-03-21T13:08:58.806828+0000","last_peered":"2024-03-21T13:08:58.806828+0000","last_clean":"2024-03-21T13:08:58.806828+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:08:58.806828+0000","last_undegraded":"2024-03-21T13:08:58.806828+0000","last_fullsized":"2024-03-21T13:08:58.806828+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:42.990022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.990022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":256,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804105+0000","last_change":"2024-03-21T13:05:28.499322+0000","last_active":"2024-03-21T13:08:58.804105+0000","last_peered":"2024-03-21T13:08:58.804105+0000","last_clean":"2024-03-21T13:08:58.804105+0000","last_became_active":"2024-03-21T13:02:26.242973+0000","last_became_peered":"2024-03-21T13:02:26.242973+0000","last_unstale":"2024-03-21T13:08:58.804105+0000","last_undegraded":"2024-03-21T13:08:58.804105+0000","last_fullsized":"2024-03-21T13:08:58.804105+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.499185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.499185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":402,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804074+0000","last_change":"2024-03-21T13:08:18.740907+0000","last_active":"2024-03-21T13:08:58.804074+0000","last_peered":"2024-03-21T13:08:58.804074+0000","last_clean":"2024-03-21T13:08:58.804074+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:08:58.804074+0000","last_undegraded":"2024-03-21T13:08:58.804074+0000","last_fullsized":"2024-03-21T13:08:58.804074+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-21T13:08:18.740815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.740815+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:26.851401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":370,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803588+0000","last_change":"2024-03-21T13:05:19.431337+0000","last_active":"2024-03-21T13:08:58.803588+0000","last_peered":"2024-03-21T13:08:58.803588+0000","last_clean":"2024-03-21T13:08:58.803588+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:08:58.803588+0000","last_undegraded":"2024-03-21T13:08:58.803588+0000","last_fullsized":"2024-03-21T13:08:58.803588+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:19.430947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:19.430947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":36,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807276+0000","last_change":"2024-03-21T13:08:20.955637+0000","last_active":"2024-03-21T13:08:58.807276+0000","last_peered":"2024-03-21T13:08:58.807276+0000","last_clean":"2024-03-21T13:08:58.807276+0000","last_became_active":"2024-03-21T13:08:20.955293+0000","last_became_peered":"2024-03-21T13:08:20.955293+0000","last_unstale":"2024-03-21T13:08:58.807276+0000","last_undegraded":"2024-03-21T13:08:58.807276+0000","last_fullsized":"2024-03-21T13:08:58.807276+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:03.434682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:03.434682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:12.144967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":384,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.806082+0000","last_change":"2024-03-21T13:07:50.428363+0000","last_active":"2024-03-21T13:08:58.806082+0000","last_peered":"2024-03-21T13:08:58.806082+0000","last_clean":"2024-03-21T13:08:58.806082+0000","last_became_active":"2024-03-21T13:06:29.501190+0000","last_became_peered":"2024-03-21T13:06:29.501190+0000","last_unstale":"2024-03-21T13:08:58.806082+0000","last_undegraded":"2024-03-21T13:08:58.806082+0000","last_fullsized":"2024-03-21T13:08:58.806082+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:50.428251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:50.428251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:18.954380+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":124,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803996+0000","last_change":"2024-03-21T13:06:29.532046+0000","last_active":"2024-03-21T13:08:58.803996+0000","last_peered":"2024-03-21T13:08:58.803996+0000","last_clean":"2024-03-21T13:08:58.803996+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:08:58.803996+0000","last_undegraded":"2024-03-21T13:08:58.803996+0000","last_fullsized":"2024-03-21T13:08:58.803996+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:40.262277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:40.262277+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":285,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807181+0000","last_change":"2024-03-21T13:08:20.069216+0000","last_active":"2024-03-21T13:08:58.807181+0000","last_peered":"2024-03-21T13:08:58.807181+0000","last_clean":"2024-03-21T13:08:58.807181+0000","last_became_active":"2024-03-21T13:02:26.252488+0000","last_became_peered":"2024-03-21T13:02:26.252488+0000","last_unstale":"2024-03-21T13:08:58.807181+0000","last_undegraded":"2024-03-21T13:08:58.807181+0000","last_fullsized":"2024-03-21T13:08:58.807181+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:20.069134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:20.069134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:34.216060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":386,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803280+0000","last_change":"2024-03-21T13:05:31.496412+0000","last_active":"2024-03-21T13:08:58.803280+0000","last_peered":"2024-03-21T13:08:58.803280+0000","last_clean":"2024-03-21T13:08:58.803280+0000","last_became_active":"2024-03-21T13:02:26.232682+0000","last_became_peered":"2024-03-21T13:02:26.232682+0000","last_unstale":"2024-03-21T13:08:58.803280+0000","last_undegraded":"2024-03-21T13:08:58.803280+0000","last_fullsized":"2024-03-21T13:08:58.803280+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.495639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.495639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":388,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804475+0000","last_change":"2024-03-21T13:07:48.347291+0000","last_active":"2024-03-21T13:08:58.804475+0000","last_peered":"2024-03-21T13:08:58.804475+0000","last_clean":"2024-03-21T13:08:58.804475+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:08:58.804475+0000","last_undegraded":"2024-03-21T13:08:58.804475+0000","last_fullsized":"2024-03-21T13:08:58.804475+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:48.347175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:48.347175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:16.886257+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":370,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807466+0000","last_change":"2024-03-21T13:06:29.518281+0000","last_active":"2024-03-21T13:08:58.807466+0000","last_peered":"2024-03-21T13:08:58.807466+0000","last_clean":"2024-03-21T13:08:58.807466+0000","last_became_active":"2024-03-21T13:06:29.496686+0000","last_became_peered":"2024-03-21T13:06:29.496686+0000","last_unstale":"2024-03-21T13:08:58.807466+0000","last_undegraded":"2024-03-21T13:08:58.807466+0000","last_fullsized":"2024-03-21T13:08:58.807466+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:05:34.036430+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.036430+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":215,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804632+0000","last_change":"2024-03-21T13:05:29.031235+0000","last_active":"2024-03-21T13:08:58.804632+0000","last_peered":"2024-03-21T13:08:58.804632+0000","last_clean":"2024-03-21T13:08:58.804632+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:08:58.804632+0000","last_undegraded":"2024-03-21T13:08:58.804632+0000","last_fullsized":"2024-03-21T13:08:58.804632+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.031062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.031062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":256,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803100+0000","last_change":"2024-03-21T13:08:05.697231+0000","last_active":"2024-03-21T13:08:58.803100+0000","last_peered":"2024-03-21T13:08:58.803100+0000","last_clean":"2024-03-21T13:08:58.803100+0000","last_became_active":"2024-03-21T13:02:26.242491+0000","last_became_peered":"2024-03-21T13:02:26.242491+0000","last_unstale":"2024-03-21T13:08:58.803100+0000","last_undegraded":"2024-03-21T13:08:58.803100+0000","last_fullsized":"2024-03-21T13:08:58.803100+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:05.697141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:05.697141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:28.363142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":100,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804760+0000","last_change":"2024-03-21T13:08:18.389543+0000","last_active":"2024-03-21T13:08:58.804760+0000","last_peered":"2024-03-21T13:08:58.804760+0000","last_clean":"2024-03-21T13:08:58.804760+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:08:58.804760+0000","last_undegraded":"2024-03-21T13:08:58.804760+0000","last_fullsized":"2024-03-21T13:08:58.804760+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:18.389423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.389423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:18.843144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":479,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.813310+0000","last_change":"2024-03-21T13:08:16.007543+0000","last_active":"2024-03-21T13:08:58.813310+0000","last_peered":"2024-03-21T13:08:58.813310+0000","last_clean":"2024-03-21T13:08:58.813310+0000","last_became_active":"2024-03-21T13:06:29.487725+0000","last_became_peered":"2024-03-21T13:06:29.487725+0000","last_unstale":"2024-03-21T13:08:58.813310+0000","last_undegraded":"2024-03-21T13:08:58.813310+0000","last_fullsized":"2024-03-21T13:08:58.813310+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:08:16.007433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:16.007433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:41.300649+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":288,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.805985+0000","last_change":"2024-03-21T13:08:56.597251+0000","last_active":"2024-03-21T13:08:58.805985+0000","last_peered":"2024-03-21T13:08:58.805985+0000","last_clean":"2024-03-21T13:08:58.805985+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:08:58.805985+0000","last_undegraded":"2024-03-21T13:08:58.805985+0000","last_fullsized":"2024-03-21T13:08:58.805985+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:56.597146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:56.597146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:02.490445+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"125'10","reported_seq":516,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.802701+0000","last_change":"2024-03-21T13:08:15.777606+0000","last_active":"2024-03-21T13:08:58.802701+0000","last_peered":"2024-03-21T13:08:58.802701+0000","last_clean":"2024-03-21T13:08:58.802701+0000","last_became_active":"2024-03-21T13:02:26.234494+0000","last_became_peered":"2024-03-21T13:02:26.234494+0000","last_unstale":"2024-03-21T13:08:58.802701+0000","last_undegraded":"2024-03-21T13:08:58.802701+0000","last_fullsized":"2024-03-21T13:08:58.802701+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:08:15.777522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:15.777522+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:35.180586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":394,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804441+0000","last_change":"2024-03-21T13:06:29.489915+0000","last_active":"2024-03-21T13:08:58.804441+0000","last_peered":"2024-03-21T13:08:58.804441+0000","last_clean":"2024-03-21T13:08:58.804441+0000","last_became_active":"2024-03-21T13:06:29.489392+0000","last_became_peered":"2024-03-21T13:06:29.489392+0000","last_unstale":"2024-03-21T13:08:58.804441+0000","last_undegraded":"2024-03-21T13:08:58.804441+0000","last_fullsized":"2024-03-21T13:08:58.804441+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.042372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.042372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'20","reported_seq":444,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.805818+0000","last_change":"2024-03-21T13:06:29.530165+0000","last_active":"2024-03-21T13:08:58.805818+0000","last_peered":"2024-03-21T13:08:58.805818+0000","last_clean":"2024-03-21T13:08:58.805818+0000","last_became_active":"2024-03-21T13:06:29.506163+0000","last_became_peered":"2024-03-21T13:06:29.506163+0000","last_unstale":"2024-03-21T13:08:58.805818+0000","last_undegraded":"2024-03-21T13:08:58.805818+0000","last_fullsized":"2024-03-21T13:08:58.805818+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:23.379004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.379004+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":384,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.806901+0000","last_change":"2024-03-21T13:06:45.402756+0000","last_active":"2024-03-21T13:08:58.806901+0000","last_peered":"2024-03-21T13:08:58.806901+0000","last_clean":"2024-03-21T13:08:58.806901+0000","last_became_active":"2024-03-21T13:06:45.402482+0000","last_became_peered":"2024-03-21T13:06:45.402482+0000","last_unstale":"2024-03-21T13:08:58.806901+0000","last_undegraded":"2024-03-21T13:08:58.806901+0000","last_fullsized":"2024-03-21T13:08:58.806901+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:21.157295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.157295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":518,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.806847+0000","last_change":"2024-03-21T13:07:52.673401+0000","last_active":"2024-03-21T13:08:58.806847+0000","last_peered":"2024-03-21T13:08:58.806847+0000","last_clean":"2024-03-21T13:08:58.806847+0000","last_became_active":"2024-03-21T13:06:29.507429+0000","last_became_peered":"2024-03-21T13:06:29.507429+0000","last_unstale":"2024-03-21T13:08:58.806847+0000","last_undegraded":"2024-03-21T13:08:58.806847+0000","last_fullsized":"2024-03-21T13:08:58.806847+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-03-21T13:07:52.673289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:07:52.673289+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":289,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807980+0000","last_change":"2024-03-21T13:08:21.365260+0000","last_active":"2024-03-21T13:08:58.807980+0000","last_peered":"2024-03-21T13:08:58.807980+0000","last_clean":"2024-03-21T13:08:58.807980+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:08:58.807980+0000","last_undegraded":"2024-03-21T13:08:58.807980+0000","last_fullsized":"2024-03-21T13:08:58.807980+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:08:21.365173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.335158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"234'15","reported_seq":316,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804745+0000","last_change":"2024-03-21T13:08:08.413361+0000","last_active":"2024-03-21T13:08:58.804745+0000","last_peered":"2024-03-21T13:08:58.804745+0000","last_clean":"2024-03-21T13:08:58.804745+0000","last_became_active":"2024-03-21T13:02:26.239161+0000","last_became_peered":"2024-03-21T13:02:26.239161+0000","last_unstale":"2024-03-21T13:08:58.804745+0000","last_undegraded":"2024-03-21T13:08:58.804745+0000","last_fullsized":"2024-03-21T13:08:58.804745+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:08:08.413234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:08.413234+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:37.899511+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":109,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.815270+0000","last_change":"2024-03-21T13:08:58.785566+0000","last_active":"2024-03-21T13:08:58.815270+0000","last_peered":"2024-03-21T13:08:58.815270+0000","last_clean":"2024-03-21T13:08:58.815270+0000","last_became_active":"2024-03-21T13:06:29.488783+0000","last_became_peered":"2024-03-21T13:06:29.488783+0000","last_unstale":"2024-03-21T13:08:58.815270+0000","last_undegraded":"2024-03-21T13:08:58.815270+0000","last_fullsized":"2024-03-21T13:08:58.815270+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:58.785444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:08:58.785444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:14.325341+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":380,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.805080+0000","last_change":"2024-03-21T13:05:17.347764+0000","last_active":"2024-03-21T13:08:58.805080+0000","last_peered":"2024-03-21T13:08:58.805080+0000","last_clean":"2024-03-21T13:08:58.805080+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:08:58.805080+0000","last_undegraded":"2024-03-21T13:08:58.805080+0000","last_fullsized":"2024-03-21T13:08:58.805080+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:17.347604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:17.347604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":394,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804982+0000","last_change":"2024-03-21T13:05:23.049951+0000","last_active":"2024-03-21T13:08:58.804982+0000","last_peered":"2024-03-21T13:08:58.804982+0000","last_clean":"2024-03-21T13:08:58.804982+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:08:58.804982+0000","last_undegraded":"2024-03-21T13:08:58.804982+0000","last_fullsized":"2024-03-21T13:08:58.804982+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:05:23.049811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.049811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'15","reported_seq":316,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.802360+0000","last_change":"2024-03-21T13:01:48.210983+0000","last_active":"2024-03-21T13:08:58.802360+0000","last_peered":"2024-03-21T13:08:58.802360+0000","last_clean":"2024-03-21T13:08:58.802360+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:08:58.802360+0000","last_undegraded":"2024-03-21T13:08:58.802360+0000","last_fullsized":"2024-03-21T13:08:58.802360+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:01:40.964615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:01:40.964615+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":376,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804577+0000","last_change":"2024-03-21T13:07:52.425295+0000","last_active":"2024-03-21T13:08:58.804577+0000","last_peered":"2024-03-21T13:08:58.804577+0000","last_clean":"2024-03-21T13:08:58.804577+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:08:58.804577+0000","last_undegraded":"2024-03-21T13:08:58.804577+0000","last_fullsized":"2024-03-21T13:08:58.804577+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:52.424928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:52.424928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:11.921996+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":370,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803464+0000","last_change":"2024-03-21T13:05:25.466812+0000","last_active":"2024-03-21T13:08:58.803464+0000","last_peered":"2024-03-21T13:08:58.803464+0000","last_clean":"2024-03-21T13:08:58.803464+0000","last_became_active":"2024-03-21T13:02:26.242502+0000","last_became_peered":"2024-03-21T13:02:26.242502+0000","last_unstale":"2024-03-21T13:08:58.803464+0000","last_undegraded":"2024-03-21T13:08:58.803464+0000","last_fullsized":"2024-03-21T13:08:58.803464+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:25.466695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:25.466695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":396,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807550+0000","last_change":"2024-03-21T13:05:41.004652+0000","last_active":"2024-03-21T13:08:58.807550+0000","last_peered":"2024-03-21T13:08:58.807550+0000","last_clean":"2024-03-21T13:08:58.807550+0000","last_became_active":"2024-03-21T13:02:26.252406+0000","last_became_peered":"2024-03-21T13:02:26.252406+0000","last_unstale":"2024-03-21T13:08:58.807550+0000","last_undegraded":"2024-03-21T13:08:58.807550+0000","last_fullsized":"2024-03-21T13:08:58.807550+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.004412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.004412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":424,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804809+0000","last_change":"2024-03-21T13:06:29.546967+0000","last_active":"2024-03-21T13:08:58.804809+0000","last_peered":"2024-03-21T13:08:58.804809+0000","last_clean":"2024-03-21T13:08:58.804809+0000","last_became_active":"2024-03-21T13:06:29.513331+0000","last_became_peered":"2024-03-21T13:06:29.513331+0000","last_unstale":"2024-03-21T13:08:58.804809+0000","last_undegraded":"2024-03-21T13:08:58.804809+0000","last_fullsized":"2024-03-21T13:08:58.804809+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'6","last_scrub_stamp":"2024-03-21T13:04:22.563861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:04:22.563861+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"240'1191","reported_seq":1213,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:59.809856+0000","last_change":"2024-03-21T13:08:45.883483+0000","last_active":"2024-03-21T13:08:59.809856+0000","last_peered":"2024-03-21T13:08:59.809856+0000","last_clean":"2024-03-21T13:08:59.809856+0000","last_became_active":"2024-03-21T13:08:45.883141+0000","last_became_peered":"2024-03-21T13:08:45.883141+0000","last_unstale":"2024-03-21T13:08:59.809856+0000","last_undegraded":"2024-03-21T13:08:59.809856+0000","last_fullsized":"2024-03-21T13:08:59.809856+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1191,"log_dups_size":0,"ondisk_log_size":1191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:09.921059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78053376,"num_objects":1191,"num_object_clones":0,"num_object_copies":3573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1191,"num_write_kb":76224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":261,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.806912+0000","last_change":"2024-03-21T13:06:29.509970+0000","last_active":"2024-03-21T13:08:58.806912+0000","last_peered":"2024-03-21T13:08:58.806912+0000","last_clean":"2024-03-21T13:08:58.806912+0000","last_became_active":"2024-03-21T13:06:29.509616+0000","last_became_peered":"2024-03-21T13:06:29.509616+0000","last_unstale":"2024-03-21T13:08:58.806912+0000","last_undegraded":"2024-03-21T13:08:58.806912+0000","last_fullsized":"2024-03-21T13:08:58.806912+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.973796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.973796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":424,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807356+0000","last_change":"2024-03-21T13:06:29.498660+0000","last_active":"2024-03-21T13:08:58.807356+0000","last_peered":"2024-03-21T13:08:58.807356+0000","last_clean":"2024-03-21T13:08:58.807356+0000","last_became_active":"2024-03-21T13:06:29.487439+0000","last_became_peered":"2024-03-21T13:06:29.487439+0000","last_unstale":"2024-03-21T13:08:58.807356+0000","last_undegraded":"2024-03-21T13:08:58.807356+0000","last_fullsized":"2024-03-21T13:08:58.807356+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:05:44.816353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:44.816353+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":341,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804355+0000","last_change":"2024-03-21T13:08:10.436112+0000","last_active":"2024-03-21T13:08:58.804355+0000","last_peered":"2024-03-21T13:08:58.804355+0000","last_clean":"2024-03-21T13:08:58.804355+0000","last_became_active":"2024-03-21T13:02:26.237800+0000","last_became_peered":"2024-03-21T13:02:26.237800+0000","last_unstale":"2024-03-21T13:08:58.804355+0000","last_undegraded":"2024-03-21T13:08:58.804355+0000","last_fullsized":"2024-03-21T13:08:58.804355+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:08:10.435742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:10.435742+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:37.629108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":517,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.805746+0000","last_change":"2024-03-21T13:08:53.619838+0000","last_active":"2024-03-21T13:08:58.805746+0000","last_peered":"2024-03-21T13:08:58.805746+0000","last_clean":"2024-03-21T13:08:58.805746+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:08:58.805746+0000","last_undegraded":"2024-03-21T13:08:58.805746+0000","last_fullsized":"2024-03-21T13:08:58.805746+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:08:53.619709+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:08:53.619709+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:57.309212+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"240'1232","reported_seq":1254,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:09:00.772451+0000","last_change":"2024-03-21T13:08:45.883554+0000","last_active":"2024-03-21T13:09:00.772451+0000","last_peered":"2024-03-21T13:09:00.772451+0000","last_clean":"2024-03-21T13:09:00.772451+0000","last_became_active":"2024-03-21T13:08:45.883255+0000","last_became_peered":"2024-03-21T13:08:45.883255+0000","last_unstale":"2024-03-21T13:09:00.772451+0000","last_undegraded":"2024-03-21T13:09:00.772451+0000","last_fullsized":"2024-03-21T13:09:00.772451+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1232,"log_dups_size":0,"ondisk_log_size":1232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:08.111005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80740352,"num_objects":1232,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1232,"num_write_kb":78848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":376,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803477+0000","last_change":"2024-03-21T13:05:43.596914+0000","last_active":"2024-03-21T13:08:58.803477+0000","last_peered":"2024-03-21T13:08:58.803477+0000","last_clean":"2024-03-21T13:08:58.803477+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:08:58.803477+0000","last_undegraded":"2024-03-21T13:08:58.803477+0000","last_fullsized":"2024-03-21T13:08:58.803477+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.596691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":394,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.806165+0000","last_change":"2024-03-21T13:05:09.138059+0000","last_active":"2024-03-21T13:08:58.806165+0000","last_peered":"2024-03-21T13:08:58.806165+0000","last_clean":"2024-03-21T13:08:58.806165+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:08:58.806165+0000","last_undegraded":"2024-03-21T13:08:58.806165+0000","last_fullsized":"2024-03-21T13:08:58.806165+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:09.137868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:09.137868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"234'18","reported_seq":330,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803887+0000","last_change":"2024-03-21T13:07:48.778302+0000","last_active":"2024-03-21T13:08:58.803887+0000","last_peered":"2024-03-21T13:08:58.803887+0000","last_clean":"2024-03-21T13:08:58.803887+0000","last_became_active":"2024-03-21T13:02:14.347102+0000","last_became_peered":"2024-03-21T13:02:14.347102+0000","last_unstale":"2024-03-21T13:08:58.803887+0000","last_undegraded":"2024-03-21T13:08:58.803887+0000","last_fullsized":"2024-03-21T13:08:58.803887+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:07:48.778145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:07:48.778145+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":252,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.811385+0000","last_change":"2024-03-21T13:07:59.084839+0000","last_active":"2024-03-21T13:08:58.811385+0000","last_peered":"2024-03-21T13:08:58.811385+0000","last_clean":"2024-03-21T13:08:58.811385+0000","last_became_active":"2024-03-21T13:06:29.486743+0000","last_became_peered":"2024-03-21T13:06:29.486743+0000","last_unstale":"2024-03-21T13:08:58.811385+0000","last_undegraded":"2024-03-21T13:08:58.811385+0000","last_fullsized":"2024-03-21T13:08:58.811385+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:59.082703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:07:59.082703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:09.013939+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"240'1154","reported_seq":1176,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:59.812760+0000","last_change":"2024-03-21T13:08:45.621486+0000","last_active":"2024-03-21T13:08:59.812760+0000","last_peered":"2024-03-21T13:08:59.812760+0000","last_clean":"2024-03-21T13:08:59.812760+0000","last_became_active":"2024-03-21T13:08:45.621197+0000","last_became_peered":"2024-03-21T13:08:45.621197+0000","last_unstale":"2024-03-21T13:08:59.812760+0000","last_undegraded":"2024-03-21T13:08:59.812760+0000","last_fullsized":"2024-03-21T13:08:59.812760+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1154,"log_dups_size":0,"ondisk_log_size":1154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:07.844936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75628544,"num_objects":1154,"num_object_clones":0,"num_object_copies":3462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1154,"num_write_kb":73856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":390,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803987+0000","last_change":"2024-03-21T13:08:14.371833+0000","last_active":"2024-03-21T13:08:58.803987+0000","last_peered":"2024-03-21T13:08:58.803987+0000","last_clean":"2024-03-21T13:08:58.803987+0000","last_became_active":"2024-03-21T13:06:29.503980+0000","last_became_peered":"2024-03-21T13:06:29.503980+0000","last_unstale":"2024-03-21T13:08:58.803987+0000","last_undegraded":"2024-03-21T13:08:58.803987+0000","last_fullsized":"2024-03-21T13:08:58.803987+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:08:14.371718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:15.534667+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":238,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804258+0000","last_change":"2024-03-21T13:06:29.503568+0000","last_active":"2024-03-21T13:08:58.804258+0000","last_peered":"2024-03-21T13:08:58.804258+0000","last_clean":"2024-03-21T13:08:58.804258+0000","last_became_active":"2024-03-21T13:06:29.503252+0000","last_became_peered":"2024-03-21T13:06:29.503252+0000","last_unstale":"2024-03-21T13:08:58.804258+0000","last_undegraded":"2024-03-21T13:08:58.804258+0000","last_fullsized":"2024-03-21T13:08:58.804258+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.216547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.216547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":394,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804722+0000","last_change":"2024-03-21T13:05:22.078652+0000","last_active":"2024-03-21T13:08:58.804722+0000","last_peered":"2024-03-21T13:08:58.804722+0000","last_clean":"2024-03-21T13:08:58.804722+0000","last_became_active":"2024-03-21T13:02:26.236780+0000","last_became_peered":"2024-03-21T13:02:26.236780+0000","last_unstale":"2024-03-21T13:08:58.804722+0000","last_undegraded":"2024-03-21T13:08:58.804722+0000","last_fullsized":"2024-03-21T13:08:58.804722+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:22.078546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.078546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'35","reported_seq":493,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807066+0000","last_change":"2024-03-21T13:05:29.992376+0000","last_active":"2024-03-21T13:08:58.807066+0000","last_peered":"2024-03-21T13:08:58.807066+0000","last_clean":"2024-03-21T13:08:58.807066+0000","last_became_active":"2024-03-21T13:02:26.240350+0000","last_became_peered":"2024-03-21T13:02:26.240350+0000","last_unstale":"2024-03-21T13:08:58.807066+0000","last_undegraded":"2024-03-21T13:08:58.807066+0000","last_fullsized":"2024-03-21T13:08:58.807066+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:05:29.992207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.992207+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"240'1154","reported_seq":1176,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:59.868656+0000","last_change":"2024-03-21T13:08:45.885456+0000","last_active":"2024-03-21T13:08:59.868656+0000","last_peered":"2024-03-21T13:08:59.868656+0000","last_clean":"2024-03-21T13:08:59.868656+0000","last_became_active":"2024-03-21T13:08:45.885006+0000","last_became_peered":"2024-03-21T13:08:45.885006+0000","last_unstale":"2024-03-21T13:08:59.868656+0000","last_undegraded":"2024-03-21T13:08:59.868656+0000","last_fullsized":"2024-03-21T13:08:59.868656+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1154,"log_dups_size":0,"ondisk_log_size":1154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:46.467430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75628544,"num_objects":1154,"num_object_clones":0,"num_object_copies":3462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1154,"num_write_kb":73856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":287,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804803+0000","last_change":"2024-03-21T13:06:29.501377+0000","last_active":"2024-03-21T13:08:58.804803+0000","last_peered":"2024-03-21T13:08:58.804803+0000","last_clean":"2024-03-21T13:08:58.804803+0000","last_became_active":"2024-03-21T13:06:29.500961+0000","last_became_peered":"2024-03-21T13:06:29.500961+0000","last_unstale":"2024-03-21T13:08:58.804803+0000","last_undegraded":"2024-03-21T13:08:58.804803+0000","last_fullsized":"2024-03-21T13:08:58.804803+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.926828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.926828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":109,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.812666+0000","last_change":"2024-03-21T13:08:12.031223+0000","last_active":"2024-03-21T13:08:58.812666+0000","last_peered":"2024-03-21T13:08:58.812666+0000","last_clean":"2024-03-21T13:08:58.812666+0000","last_became_active":"2024-03-21T13:06:29.502745+0000","last_became_peered":"2024-03-21T13:06:29.502745+0000","last_unstale":"2024-03-21T13:08:58.812666+0000","last_undegraded":"2024-03-21T13:08:58.812666+0000","last_fullsized":"2024-03-21T13:08:58.812666+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:12.031099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:12.031099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:13.044020+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":254,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.805728+0000","last_change":"2024-03-21T13:08:57.570445+0000","last_active":"2024-03-21T13:08:58.805728+0000","last_peered":"2024-03-21T13:08:58.805728+0000","last_clean":"2024-03-21T13:08:58.805728+0000","last_became_active":"2024-03-21T13:02:26.243974+0000","last_became_peered":"2024-03-21T13:02:26.243974+0000","last_unstale":"2024-03-21T13:08:58.805728+0000","last_undegraded":"2024-03-21T13:08:58.805728+0000","last_fullsized":"2024-03-21T13:08:58.805728+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:57.570303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.570303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:07.178439+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"234'19","reported_seq":362,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.813093+0000","last_change":"2024-03-21T13:07:58.117288+0000","last_active":"2024-03-21T13:08:58.813093+0000","last_peered":"2024-03-21T13:08:58.813093+0000","last_clean":"2024-03-21T13:08:58.813093+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:08:58.813093+0000","last_undegraded":"2024-03-21T13:08:58.813093+0000","last_fullsized":"2024-03-21T13:08:58.813093+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:07:58.117164+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:07:58.117164+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:18.674340+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"240'1137","reported_seq":1159,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:59.814605+0000","last_change":"2024-03-21T13:08:45.883467+0000","last_active":"2024-03-21T13:08:59.814605+0000","last_peered":"2024-03-21T13:08:59.814605+0000","last_clean":"2024-03-21T13:08:59.814605+0000","last_became_active":"2024-03-21T13:08:45.883117+0000","last_became_peered":"2024-03-21T13:08:45.883117+0000","last_unstale":"2024-03-21T13:08:59.814605+0000","last_undegraded":"2024-03-21T13:08:59.814605+0000","last_fullsized":"2024-03-21T13:08:59.814605+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1137,"log_dups_size":0,"ondisk_log_size":1137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:02.716893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74514432,"num_objects":1137,"num_object_clones":0,"num_object_copies":3411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1137,"num_write_kb":72768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":362,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803908+0000","last_change":"2024-03-21T13:01:46.738822+0000","last_active":"2024-03-21T13:08:58.803908+0000","last_peered":"2024-03-21T13:08:58.803908+0000","last_clean":"2024-03-21T13:08:58.803908+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:08:58.803908+0000","last_undegraded":"2024-03-21T13:08:58.803908+0000","last_fullsized":"2024-03-21T13:08:58.803908+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:01:46.738659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":286,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.806967+0000","last_change":"2024-03-21T13:05:43.961181+0000","last_active":"2024-03-21T13:08:58.806967+0000","last_peered":"2024-03-21T13:08:58.806967+0000","last_clean":"2024-03-21T13:08:58.806967+0000","last_became_active":"2024-03-21T13:02:26.243692+0000","last_became_peered":"2024-03-21T13:02:26.243692+0000","last_unstale":"2024-03-21T13:08:58.806967+0000","last_undegraded":"2024-03-21T13:08:58.806967+0000","last_fullsized":"2024-03-21T13:08:58.806967+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.961057+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":93,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.808287+0000","last_change":"2024-03-21T13:06:45.410372+0000","last_active":"2024-03-21T13:08:58.808287+0000","last_peered":"2024-03-21T13:08:58.808287+0000","last_clean":"2024-03-21T13:08:58.808287+0000","last_became_active":"2024-03-21T13:06:45.410017+0000","last_became_peered":"2024-03-21T13:06:45.410017+0000","last_unstale":"2024-03-21T13:08:58.808287+0000","last_undegraded":"2024-03-21T13:08:58.808287+0000","last_fullsized":"2024-03-21T13:08:58.808287+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:48.265982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:48.265982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"234'42","reported_seq":241,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.805953+0000","last_change":"2024-03-21T13:06:29.547664+0000","last_active":"2024-03-21T13:08:58.805953+0000","last_peered":"2024-03-21T13:08:58.805953+0000","last_clean":"2024-03-21T13:08:58.805953+0000","last_became_active":"2024-03-21T13:06:29.508620+0000","last_became_peered":"2024-03-21T13:06:29.508620+0000","last_unstale":"2024-03-21T13:08:58.805953+0000","last_undegraded":"2024-03-21T13:08:58.805953+0000","last_fullsized":"2024-03-21T13:08:58.805953+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'29","last_scrub_stamp":"2024-03-21T13:05:22.201879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.201879+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":97,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807666+0000","last_change":"2024-03-21T13:06:45.407111+0000","last_active":"2024-03-21T13:08:58.807666+0000","last_peered":"2024-03-21T13:08:58.807666+0000","last_clean":"2024-03-21T13:08:58.807666+0000","last_became_active":"2024-03-21T13:06:45.406734+0000","last_became_peered":"2024-03-21T13:06:45.406734+0000","last_unstale":"2024-03-21T13:08:58.807666+0000","last_undegraded":"2024-03-21T13:08:58.807666+0000","last_fullsized":"2024-03-21T13:08:58.807666+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.312583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.312583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.a","version":"240'1148","reported_seq":1169,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:59.822028+0000","last_change":"2024-03-21T13:08:45.620420+0000","last_active":"2024-03-21T13:08:59.822028+0000","last_peered":"2024-03-21T13:08:59.822028+0000","last_clean":"2024-03-21T13:08:59.822028+0000","last_became_active":"2024-03-21T13:08:45.620077+0000","last_became_peered":"2024-03-21T13:08:45.620077+0000","last_unstale":"2024-03-21T13:08:59.822028+0000","last_undegraded":"2024-03-21T13:08:59.822028+0000","last_fullsized":"2024-03-21T13:08:59.822028+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1148,"log_dups_size":0,"ondisk_log_size":1148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:49.619248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75235328,"num_objects":1148,"num_object_clones":0,"num_object_copies":3444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1148,"num_write_kb":73472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":384,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.805087+0000","last_change":"2024-03-21T13:08:58.723773+0000","last_active":"2024-03-21T13:08:58.805087+0000","last_peered":"2024-03-21T13:08:58.805087+0000","last_clean":"2024-03-21T13:08:58.805087+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:08:58.805087+0000","last_undegraded":"2024-03-21T13:08:58.805087+0000","last_fullsized":"2024-03-21T13:08:58.805087+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:58.723653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:58.723653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:14.781119+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":103,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807859+0000","last_change":"2024-03-21T13:06:29.510026+0000","last_active":"2024-03-21T13:08:58.807859+0000","last_peered":"2024-03-21T13:08:58.807859+0000","last_clean":"2024-03-21T13:08:58.807859+0000","last_became_active":"2024-03-21T13:06:29.509674+0000","last_became_peered":"2024-03-21T13:06:29.509674+0000","last_unstale":"2024-03-21T13:08:58.807859+0000","last_undegraded":"2024-03-21T13:08:58.807859+0000","last_fullsized":"2024-03-21T13:08:58.807859+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.247107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.247107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":101,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803461+0000","last_change":"2024-03-21T13:06:29.510171+0000","last_active":"2024-03-21T13:08:58.803461+0000","last_peered":"2024-03-21T13:08:58.803461+0000","last_clean":"2024-03-21T13:08:58.803461+0000","last_became_active":"2024-03-21T13:06:29.509315+0000","last_became_peered":"2024-03-21T13:06:29.509315+0000","last_unstale":"2024-03-21T13:08:58.803461+0000","last_undegraded":"2024-03-21T13:08:58.803461+0000","last_fullsized":"2024-03-21T13:08:58.803461+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:46.263644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:05:46.263644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"234'17","reported_seq":321,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807104+0000","last_change":"2024-03-21T13:05:15.119776+0000","last_active":"2024-03-21T13:08:58.807104+0000","last_peered":"2024-03-21T13:08:58.807104+0000","last_clean":"2024-03-21T13:08:58.807104+0000","last_became_active":"2024-03-21T13:02:26.234230+0000","last_became_peered":"2024-03-21T13:02:26.234230+0000","last_unstale":"2024-03-21T13:08:58.807104+0000","last_undegraded":"2024-03-21T13:08:58.807104+0000","last_fullsized":"2024-03-21T13:08:58.807104+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'13","last_scrub_stamp":"2024-03-21T13:05:15.119643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:15.119643+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":238,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.806487+0000","last_change":"2024-03-21T13:06:29.502194+0000","last_active":"2024-03-21T13:08:58.806487+0000","last_peered":"2024-03-21T13:08:58.806487+0000","last_clean":"2024-03-21T13:08:58.806487+0000","last_became_active":"2024-03-21T13:06:29.501625+0000","last_became_peered":"2024-03-21T13:06:29.501625+0000","last_unstale":"2024-03-21T13:08:58.806487+0000","last_undegraded":"2024-03-21T13:08:58.806487+0000","last_fullsized":"2024-03-21T13:08:58.806487+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:23.243468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.243468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"240'1305","reported_seq":1326,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:09:00.800988+0000","last_change":"2024-03-21T13:08:45.885447+0000","last_active":"2024-03-21T13:09:00.800988+0000","last_peered":"2024-03-21T13:09:00.800988+0000","last_clean":"2024-03-21T13:09:00.800988+0000","last_became_active":"2024-03-21T13:08:45.885134+0000","last_became_peered":"2024-03-21T13:08:45.885134+0000","last_unstale":"2024-03-21T13:09:00.800988+0000","last_undegraded":"2024-03-21T13:09:00.800988+0000","last_fullsized":"2024-03-21T13:09:00.800988+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1305,"log_dups_size":0,"ondisk_log_size":1305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:47.512515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85524480,"num_objects":1305,"num_object_clones":0,"num_object_copies":3915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1305,"num_write_kb":83520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":255,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.805099+0000","last_change":"2024-03-21T13:04:27.549179+0000","last_active":"2024-03-21T13:08:58.805099+0000","last_peered":"2024-03-21T13:08:58.805099+0000","last_clean":"2024-03-21T13:08:58.805099+0000","last_became_active":"2024-03-21T13:02:26.235640+0000","last_became_peered":"2024-03-21T13:02:26.235640+0000","last_unstale":"2024-03-21T13:08:58.805099+0000","last_undegraded":"2024-03-21T13:08:58.805099+0000","last_fullsized":"2024-03-21T13:08:58.805099+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:27.549086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:27.549086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":646,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804125+0000","last_change":"2024-03-21T13:08:07.454234+0000","last_active":"2024-03-21T13:08:58.804125+0000","last_peered":"2024-03-21T13:08:58.804125+0000","last_clean":"2024-03-21T13:08:58.804125+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:08:58.804125+0000","last_undegraded":"2024-03-21T13:08:58.804125+0000","last_fullsized":"2024-03-21T13:08:58.804125+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:08:07.454016+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:19.144146+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":388,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.802626+0000","last_change":"2024-03-21T13:07:49.758585+0000","last_active":"2024-03-21T13:08:58.802626+0000","last_peered":"2024-03-21T13:08:58.802626+0000","last_clean":"2024-03-21T13:08:58.802626+0000","last_became_active":"2024-03-21T13:02:26.242456+0000","last_became_peered":"2024-03-21T13:02:26.242456+0000","last_unstale":"2024-03-21T13:08:58.802626+0000","last_undegraded":"2024-03-21T13:08:58.802626+0000","last_fullsized":"2024-03-21T13:08:58.802626+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:49.758485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:49.758485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:07.636831+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"234'36","reported_seq":559,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807929+0000","last_change":"2024-03-21T13:05:42.033620+0000","last_active":"2024-03-21T13:08:58.807929+0000","last_peered":"2024-03-21T13:08:58.807929+0000","last_clean":"2024-03-21T13:08:58.807929+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:08:58.807929+0000","last_undegraded":"2024-03-21T13:08:58.807929+0000","last_fullsized":"2024-03-21T13:08:58.807929+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'24","last_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.033266+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":392,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804980+0000","last_change":"2024-03-21T13:06:29.504771+0000","last_active":"2024-03-21T13:08:58.804980+0000","last_peered":"2024-03-21T13:08:58.804980+0000","last_clean":"2024-03-21T13:08:58.804980+0000","last_became_active":"2024-03-21T13:06:29.504250+0000","last_became_peered":"2024-03-21T13:06:29.504250+0000","last_unstale":"2024-03-21T13:08:58.804980+0000","last_undegraded":"2024-03-21T13:08:58.804980+0000","last_fullsized":"2024-03-21T13:08:58.804980+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:20.061709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.061709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"240'1153","reported_seq":1175,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:59.855681+0000","last_change":"2024-03-21T13:08:45.883519+0000","last_active":"2024-03-21T13:08:59.855681+0000","last_peered":"2024-03-21T13:08:59.855681+0000","last_clean":"2024-03-21T13:08:59.855681+0000","last_became_active":"2024-03-21T13:08:45.883237+0000","last_became_peered":"2024-03-21T13:08:45.883237+0000","last_unstale":"2024-03-21T13:08:59.855681+0000","last_undegraded":"2024-03-21T13:08:59.855681+0000","last_fullsized":"2024-03-21T13:08:59.855681+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1153,"log_dups_size":0,"ondisk_log_size":1153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:45.476165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75563008,"num_objects":1153,"num_object_clones":0,"num_object_copies":3459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1153,"num_write_kb":73792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":257,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804857+0000","last_change":"2024-03-21T13:05:29.457708+0000","last_active":"2024-03-21T13:08:58.804857+0000","last_peered":"2024-03-21T13:08:58.804857+0000","last_clean":"2024-03-21T13:08:58.804857+0000","last_became_active":"2024-03-21T13:02:26.242708+0000","last_became_peered":"2024-03-21T13:02:26.242708+0000","last_unstale":"2024-03-21T13:08:58.804857+0000","last_undegraded":"2024-03-21T13:08:58.804857+0000","last_fullsized":"2024-03-21T13:08:58.804857+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.457597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.457597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":289,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.813252+0000","last_change":"2024-03-21T13:08:11.057903+0000","last_active":"2024-03-21T13:08:58.813252+0000","last_peered":"2024-03-21T13:08:58.813252+0000","last_clean":"2024-03-21T13:08:58.813252+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:08:58.813252+0000","last_undegraded":"2024-03-21T13:08:58.813252+0000","last_fullsized":"2024-03-21T13:08:58.813252+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:11.057788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:08:11.057788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:17.241879+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":1023,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804063+0000","last_change":"2024-03-21T13:08:05.448565+0000","last_active":"2024-03-21T13:08:58.804063+0000","last_peered":"2024-03-21T13:08:58.804063+0000","last_clean":"2024-03-21T13:08:58.804063+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:08:58.804063+0000","last_undegraded":"2024-03-21T13:08:58.804063+0000","last_fullsized":"2024-03-21T13:08:58.804063+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:08:05.448403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:08:05.448403+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:26.024762+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":612,"num_read_kb":607,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":485,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.808400+0000","last_change":"2024-03-21T13:07:50.821737+0000","last_active":"2024-03-21T13:08:58.808400+0000","last_peered":"2024-03-21T13:08:58.808400+0000","last_clean":"2024-03-21T13:08:58.808400+0000","last_became_active":"2024-03-21T13:06:29.526702+0000","last_became_peered":"2024-03-21T13:06:29.526702+0000","last_unstale":"2024-03-21T13:08:58.808400+0000","last_undegraded":"2024-03-21T13:08:58.808400+0000","last_fullsized":"2024-03-21T13:08:58.808400+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:07:50.821119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:07:50.821119+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:15.038128+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":43,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807376+0000","last_change":"2024-03-21T13:08:56.630611+0000","last_active":"2024-03-21T13:08:58.807376+0000","last_peered":"2024-03-21T13:08:58.807376+0000","last_clean":"2024-03-21T13:08:58.807376+0000","last_became_active":"2024-03-21T13:08:20.892193+0000","last_became_peered":"2024-03-21T13:08:20.892193+0000","last_unstale":"2024-03-21T13:08:58.807376+0000","last_undegraded":"2024-03-21T13:08:58.807376+0000","last_fullsized":"2024-03-21T13:08:58.807376+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:56.630492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:56.630492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:58.893366+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"240'1180","reported_seq":1202,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:59.812245+0000","last_change":"2024-03-21T13:08:45.885732+0000","last_active":"2024-03-21T13:08:59.812245+0000","last_peered":"2024-03-21T13:08:59.812245+0000","last_clean":"2024-03-21T13:08:59.812245+0000","last_became_active":"2024-03-21T13:08:45.885382+0000","last_became_peered":"2024-03-21T13:08:45.885382+0000","last_unstale":"2024-03-21T13:08:59.812245+0000","last_undegraded":"2024-03-21T13:08:59.812245+0000","last_fullsized":"2024-03-21T13:08:59.812245+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1180,"log_dups_size":0,"ondisk_log_size":1180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:04.140599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77332480,"num_objects":1180,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1180,"num_write_kb":75520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":281,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.806671+0000","last_change":"2024-03-21T13:08:57.641050+0000","last_active":"2024-03-21T13:08:58.806671+0000","last_peered":"2024-03-21T13:08:58.806671+0000","last_clean":"2024-03-21T13:08:58.806671+0000","last_became_active":"2024-03-21T13:06:29.489464+0000","last_became_peered":"2024-03-21T13:06:29.489464+0000","last_unstale":"2024-03-21T13:08:58.806671+0000","last_undegraded":"2024-03-21T13:08:58.806671+0000","last_fullsized":"2024-03-21T13:08:58.806671+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:57.640892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.640892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:20.853004+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":130,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.806409+0000","last_change":"2024-03-21T13:08:22.012978+0000","last_active":"2024-03-21T13:08:58.806409+0000","last_peered":"2024-03-21T13:08:58.806409+0000","last_clean":"2024-03-21T13:08:58.806409+0000","last_became_active":"2024-03-21T13:06:29.484174+0000","last_became_peered":"2024-03-21T13:06:29.484174+0000","last_unstale":"2024-03-21T13:08:58.806409+0000","last_undegraded":"2024-03-21T13:08:58.806409+0000","last_fullsized":"2024-03-21T13:08:58.806409+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:22.012892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:22.012892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:37.561153+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":367,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804661+0000","last_change":"2024-03-21T13:06:29.508182+0000","last_active":"2024-03-21T13:08:58.804661+0000","last_peered":"2024-03-21T13:08:58.804661+0000","last_clean":"2024-03-21T13:08:58.804661+0000","last_became_active":"2024-03-21T13:06:29.492612+0000","last_became_peered":"2024-03-21T13:06:29.492612+0000","last_unstale":"2024-03-21T13:08:58.804661+0000","last_undegraded":"2024-03-21T13:08:58.804661+0000","last_fullsized":"2024-03-21T13:08:58.804661+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:05:28.062237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.062237+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":394,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807169+0000","last_change":"2024-03-21T13:06:29.497191+0000","last_active":"2024-03-21T13:08:58.807169+0000","last_peered":"2024-03-21T13:08:58.807169+0000","last_clean":"2024-03-21T13:08:58.807169+0000","last_became_active":"2024-03-21T13:06:29.496679+0000","last_became_peered":"2024-03-21T13:06:29.496679+0000","last_unstale":"2024-03-21T13:08:58.807169+0000","last_undegraded":"2024-03-21T13:08:58.807169+0000","last_fullsized":"2024-03-21T13:08:58.807169+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:26.932188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.932188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"240'1327","reported_seq":1349,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:09:00.794353+0000","last_change":"2024-03-21T13:08:45.619068+0000","last_active":"2024-03-21T13:09:00.794353+0000","last_peered":"2024-03-21T13:09:00.794353+0000","last_clean":"2024-03-21T13:09:00.794353+0000","last_became_active":"2024-03-21T13:08:45.618800+0000","last_became_peered":"2024-03-21T13:08:45.618800+0000","last_unstale":"2024-03-21T13:09:00.794353+0000","last_undegraded":"2024-03-21T13:09:00.794353+0000","last_fullsized":"2024-03-21T13:09:00.794353+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1327,"log_dups_size":0,"ondisk_log_size":1327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:45.796314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86966272,"num_objects":1327,"num_object_clones":0,"num_object_copies":3981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1327,"num_write_kb":84928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":287,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803668+0000","last_change":"2024-03-21T13:08:55.488120+0000","last_active":"2024-03-21T13:08:58.803668+0000","last_peered":"2024-03-21T13:08:58.803668+0000","last_clean":"2024-03-21T13:08:58.803668+0000","last_became_active":"2024-03-21T13:06:29.501983+0000","last_became_peered":"2024-03-21T13:06:29.501983+0000","last_unstale":"2024-03-21T13:08:58.803668+0000","last_undegraded":"2024-03-21T13:08:58.803668+0000","last_fullsized":"2024-03-21T13:08:58.803668+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:55.487977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:55.487977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:57.490278+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":261,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803187+0000","last_change":"2024-03-21T13:06:29.511420+0000","last_active":"2024-03-21T13:08:58.803187+0000","last_peered":"2024-03-21T13:08:58.803187+0000","last_clean":"2024-03-21T13:08:58.803187+0000","last_became_active":"2024-03-21T13:06:29.510833+0000","last_became_peered":"2024-03-21T13:06:29.510833+0000","last_unstale":"2024-03-21T13:08:58.803187+0000","last_undegraded":"2024-03-21T13:08:58.803187+0000","last_fullsized":"2024-03-21T13:08:58.803187+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.324701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.324701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"234'25","reported_seq":362,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.812571+0000","last_change":"2024-03-21T13:08:57.835622+0000","last_active":"2024-03-21T13:08:58.812571+0000","last_peered":"2024-03-21T13:08:58.812571+0000","last_clean":"2024-03-21T13:08:58.812571+0000","last_became_active":"2024-03-21T13:06:29.492863+0000","last_became_peered":"2024-03-21T13:06:29.492863+0000","last_unstale":"2024-03-21T13:08:58.812571+0000","last_undegraded":"2024-03-21T13:08:58.812571+0000","last_fullsized":"2024-03-21T13:08:58.812571+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-21T13:08:57.835430+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.835430+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:22.275431+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"125'12","reported_seq":413,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.808438+0000","last_change":"2024-03-21T13:08:18.149427+0000","last_active":"2024-03-21T13:08:58.808438+0000","last_peered":"2024-03-21T13:08:58.808438+0000","last_clean":"2024-03-21T13:08:58.808438+0000","last_became_active":"2024-03-21T13:02:26.253720+0000","last_became_peered":"2024-03-21T13:02:26.253720+0000","last_unstale":"2024-03-21T13:08:58.808438+0000","last_undegraded":"2024-03-21T13:08:58.808438+0000","last_fullsized":"2024-03-21T13:08:58.808438+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'12","last_scrub_stamp":"2024-03-21T13:08:18.149295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.149295+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:21.178979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"240'1277","reported_seq":1299,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:09:00.791235+0000","last_change":"2024-03-21T13:08:45.618681+0000","last_active":"2024-03-21T13:09:00.791235+0000","last_peered":"2024-03-21T13:09:00.791235+0000","last_clean":"2024-03-21T13:09:00.791235+0000","last_became_active":"2024-03-21T13:08:45.618487+0000","last_became_peered":"2024-03-21T13:08:45.618487+0000","last_unstale":"2024-03-21T13:09:00.791235+0000","last_undegraded":"2024-03-21T13:09:00.791235+0000","last_fullsized":"2024-03-21T13:09:00.791235+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:08.493694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83689472,"num_objects":1277,"num_object_clones":0,"num_object_copies":3831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1277,"num_write_kb":81728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":382,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804790+0000","last_change":"2024-03-21T13:06:29.489617+0000","last_active":"2024-03-21T13:08:58.804790+0000","last_peered":"2024-03-21T13:08:58.804790+0000","last_clean":"2024-03-21T13:08:58.804790+0000","last_became_active":"2024-03-21T13:06:29.489289+0000","last_became_peered":"2024-03-21T13:06:29.489289+0000","last_unstale":"2024-03-21T13:08:58.804790+0000","last_undegraded":"2024-03-21T13:08:58.804790+0000","last_fullsized":"2024-03-21T13:08:58.804790+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.951655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.951655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":424,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.808439+0000","last_change":"2024-03-21T13:07:55.868896+0000","last_active":"2024-03-21T13:08:58.808439+0000","last_peered":"2024-03-21T13:08:58.808439+0000","last_clean":"2024-03-21T13:08:58.808439+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:08:58.808439+0000","last_undegraded":"2024-03-21T13:08:58.808439+0000","last_fullsized":"2024-03-21T13:08:58.808439+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:07:55.868781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:55.868781+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:07.587595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'35","reported_seq":376,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.806296+0000","last_change":"2024-03-21T13:08:54.645581+0000","last_active":"2024-03-21T13:08:58.806296+0000","last_peered":"2024-03-21T13:08:58.806296+0000","last_clean":"2024-03-21T13:08:58.806296+0000","last_became_active":"2024-03-21T13:02:26.227165+0000","last_became_peered":"2024-03-21T13:02:26.227165+0000","last_unstale":"2024-03-21T13:08:58.806296+0000","last_undegraded":"2024-03-21T13:08:58.806296+0000","last_fullsized":"2024-03-21T13:08:58.806296+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:08:54.645360+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:08:54.645360+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:11.752625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"226'10","reported_seq":520,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804651+0000","last_change":"2024-03-21T13:08:16.739276+0000","last_active":"2024-03-21T13:08:58.804651+0000","last_peered":"2024-03-21T13:08:58.804651+0000","last_clean":"2024-03-21T13:08:58.804651+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:08:58.804651+0000","last_undegraded":"2024-03-21T13:08:58.804651+0000","last_fullsized":"2024-03-21T13:08:58.804651+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:08:16.739190+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:29.735435+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"240'1148","reported_seq":1168,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:59.817669+0000","last_change":"2024-03-21T13:08:45.621307+0000","last_active":"2024-03-21T13:08:59.817669+0000","last_peered":"2024-03-21T13:08:59.817669+0000","last_clean":"2024-03-21T13:08:59.817669+0000","last_became_active":"2024-03-21T13:08:45.620972+0000","last_became_peered":"2024-03-21T13:08:45.620972+0000","last_unstale":"2024-03-21T13:08:59.817669+0000","last_undegraded":"2024-03-21T13:08:59.817669+0000","last_fullsized":"2024-03-21T13:08:59.817669+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1148,"log_dups_size":0,"ondisk_log_size":1148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:03.170614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75235328,"num_objects":1148,"num_object_clones":0,"num_object_copies":3444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1148,"num_write_kb":73472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":301,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803188+0000","last_change":"2024-03-21T13:06:29.491923+0000","last_active":"2024-03-21T13:08:58.803188+0000","last_peered":"2024-03-21T13:08:58.803188+0000","last_clean":"2024-03-21T13:08:58.803188+0000","last_became_active":"2024-03-21T13:06:29.491598+0000","last_became_peered":"2024-03-21T13:06:29.491598+0000","last_unstale":"2024-03-21T13:08:58.803188+0000","last_undegraded":"2024-03-21T13:08:58.803188+0000","last_fullsized":"2024-03-21T13:08:58.803188+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.416262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.416262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":382,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804294+0000","last_change":"2024-03-21T13:04:20.538204+0000","last_active":"2024-03-21T13:08:58.804294+0000","last_peered":"2024-03-21T13:08:58.804294+0000","last_clean":"2024-03-21T13:08:58.804294+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:08:58.804294+0000","last_undegraded":"2024-03-21T13:08:58.804294+0000","last_fullsized":"2024-03-21T13:08:58.804294+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:20.538115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:04:20.538115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"234'30","reported_seq":429,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804284+0000","last_change":"2024-03-21T13:05:12.322522+0000","last_active":"2024-03-21T13:08:58.804284+0000","last_peered":"2024-03-21T13:08:58.804284+0000","last_clean":"2024-03-21T13:08:58.804284+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:08:58.804284+0000","last_undegraded":"2024-03-21T13:08:58.804284+0000","last_fullsized":"2024-03-21T13:08:58.804284+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T13:05:12.322284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:12.322284+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":382,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.812623+0000","last_change":"2024-03-21T13:08:04.095062+0000","last_active":"2024-03-21T13:08:58.812623+0000","last_peered":"2024-03-21T13:08:58.812623+0000","last_clean":"2024-03-21T13:08:58.812623+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:08:58.812623+0000","last_undegraded":"2024-03-21T13:08:58.812623+0000","last_fullsized":"2024-03-21T13:08:58.812623+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:04.094946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:04.094946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:23.528973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"240'1377","reported_seq":1399,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:09:00.800724+0000","last_change":"2024-03-21T13:08:45.620216+0000","last_active":"2024-03-21T13:09:00.800724+0000","last_peered":"2024-03-21T13:09:00.800724+0000","last_clean":"2024-03-21T13:09:00.800724+0000","last_became_active":"2024-03-21T13:08:45.619950+0000","last_became_peered":"2024-03-21T13:08:45.619950+0000","last_unstale":"2024-03-21T13:09:00.800724+0000","last_undegraded":"2024-03-21T13:09:00.800724+0000","last_fullsized":"2024-03-21T13:09:00.800724+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1377,"log_dups_size":0,"ondisk_log_size":1377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:04.482513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90243072,"num_objects":1377,"num_object_clones":0,"num_object_copies":4131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1377,"num_write_kb":88128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":223,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807154+0000","last_change":"2024-03-21T13:06:29.489514+0000","last_active":"2024-03-21T13:08:58.807154+0000","last_peered":"2024-03-21T13:08:58.807154+0000","last_clean":"2024-03-21T13:08:58.807154+0000","last_became_active":"2024-03-21T13:06:29.488826+0000","last_became_peered":"2024-03-21T13:06:29.488826+0000","last_unstale":"2024-03-21T13:08:58.807154+0000","last_undegraded":"2024-03-21T13:08:58.807154+0000","last_fullsized":"2024-03-21T13:08:58.807154+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.328925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.328925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.e","version":"196'11","reported_seq":397,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.808573+0000","last_change":"2024-03-21T13:08:04.934320+0000","last_active":"2024-03-21T13:08:58.808573+0000","last_peered":"2024-03-21T13:08:58.808573+0000","last_clean":"2024-03-21T13:08:58.808573+0000","last_became_active":"2024-03-21T13:06:29.505444+0000","last_became_peered":"2024-03-21T13:06:29.505444+0000","last_unstale":"2024-03-21T13:08:58.808573+0000","last_undegraded":"2024-03-21T13:08:58.808573+0000","last_fullsized":"2024-03-21T13:08:58.808573+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'11","last_scrub_stamp":"2024-03-21T13:08:04.934196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:04.934196+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:16.986626+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":388,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804208+0000","last_change":"2024-03-21T13:04:19.520964+0000","last_active":"2024-03-21T13:08:58.804208+0000","last_peered":"2024-03-21T13:08:58.804208+0000","last_clean":"2024-03-21T13:08:58.804208+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:08:58.804208+0000","last_undegraded":"2024-03-21T13:08:58.804208+0000","last_fullsized":"2024-03-21T13:08:58.804208+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:19.520824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:04:19.520824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"234'34","reported_seq":352,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804752+0000","last_change":"2024-03-21T13:08:22.736499+0000","last_active":"2024-03-21T13:08:58.804752+0000","last_peered":"2024-03-21T13:08:58.804752+0000","last_clean":"2024-03-21T13:08:58.804752+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:08:58.804752+0000","last_undegraded":"2024-03-21T13:08:58.804752+0000","last_fullsized":"2024-03-21T13:08:58.804752+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:08:22.736371+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:52.513483+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"240'1168","reported_seq":1190,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:59.815816+0000","last_change":"2024-03-21T13:08:45.620469+0000","last_active":"2024-03-21T13:08:59.815816+0000","last_peered":"2024-03-21T13:08:59.815816+0000","last_clean":"2024-03-21T13:08:59.815816+0000","last_became_active":"2024-03-21T13:08:45.620089+0000","last_became_peered":"2024-03-21T13:08:45.620089+0000","last_unstale":"2024-03-21T13:08:59.815816+0000","last_undegraded":"2024-03-21T13:08:59.815816+0000","last_fullsized":"2024-03-21T13:08:59.815816+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1168,"log_dups_size":0,"ondisk_log_size":1168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:51.079941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76546048,"num_objects":1168,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1168,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":381,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.805142+0000","last_change":"2024-03-21T13:05:39.518051+0000","last_active":"2024-03-21T13:08:58.805142+0000","last_peered":"2024-03-21T13:08:58.805142+0000","last_clean":"2024-03-21T13:08:58.805142+0000","last_became_active":"2024-03-21T13:02:26.243985+0000","last_became_peered":"2024-03-21T13:02:26.243985+0000","last_unstale":"2024-03-21T13:08:58.805142+0000","last_undegraded":"2024-03-21T13:08:58.805142+0000","last_fullsized":"2024-03-21T13:08:58.805142+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:39.517591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.517591+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":276,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.814553+0000","last_change":"2024-03-21T13:08:08.054195+0000","last_active":"2024-03-21T13:08:58.814553+0000","last_peered":"2024-03-21T13:08:58.814553+0000","last_clean":"2024-03-21T13:08:58.814553+0000","last_became_active":"2024-03-21T13:06:29.501230+0000","last_became_peered":"2024-03-21T13:06:29.501230+0000","last_unstale":"2024-03-21T13:08:58.814553+0000","last_undegraded":"2024-03-21T13:08:58.814553+0000","last_fullsized":"2024-03-21T13:08:58.814553+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:08.054077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:08.054077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:28.130473+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":404,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.812747+0000","last_change":"2024-03-21T13:07:55.115719+0000","last_active":"2024-03-21T13:08:58.812747+0000","last_peered":"2024-03-21T13:08:58.812747+0000","last_clean":"2024-03-21T13:08:58.812747+0000","last_became_active":"2024-03-21T13:06:29.499291+0000","last_became_peered":"2024-03-21T13:06:29.499291+0000","last_unstale":"2024-03-21T13:08:58.812747+0000","last_undegraded":"2024-03-21T13:08:58.812747+0000","last_fullsized":"2024-03-21T13:08:58.812747+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:55.115613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:55.115613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:23.200883+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":143,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807314+0000","last_change":"2024-03-21T13:06:45.412306+0000","last_active":"2024-03-21T13:08:58.807314+0000","last_peered":"2024-03-21T13:08:58.807314+0000","last_clean":"2024-03-21T13:08:58.807314+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:08:58.807314+0000","last_undegraded":"2024-03-21T13:08:58.807314+0000","last_fullsized":"2024-03-21T13:08:58.807314+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:26.195507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.195507+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"240'1167","reported_seq":1189,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:59.860344+0000","last_change":"2024-03-21T13:08:45.885647+0000","last_active":"2024-03-21T13:08:59.860344+0000","last_peered":"2024-03-21T13:08:59.860344+0000","last_clean":"2024-03-21T13:08:59.860344+0000","last_became_active":"2024-03-21T13:08:45.884999+0000","last_became_peered":"2024-03-21T13:08:45.884999+0000","last_unstale":"2024-03-21T13:08:59.860344+0000","last_undegraded":"2024-03-21T13:08:59.860344+0000","last_fullsized":"2024-03-21T13:08:59.860344+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1167,"log_dups_size":0,"ondisk_log_size":1167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:53.386064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76480512,"num_objects":1167,"num_object_clones":0,"num_object_copies":3501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1167,"num_write_kb":74688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":101,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803339+0000","last_change":"2024-03-21T13:06:29.508869+0000","last_active":"2024-03-21T13:08:58.803339+0000","last_peered":"2024-03-21T13:08:58.803339+0000","last_clean":"2024-03-21T13:08:58.803339+0000","last_became_active":"2024-03-21T13:06:29.508508+0000","last_became_peered":"2024-03-21T13:06:29.508508+0000","last_unstale":"2024-03-21T13:08:58.803339+0000","last_undegraded":"2024-03-21T13:08:58.803339+0000","last_fullsized":"2024-03-21T13:08:58.803339+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:36.362197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:05:36.362197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":283,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.814280+0000","last_change":"2024-03-21T13:08:05.143175+0000","last_active":"2024-03-21T13:08:58.814280+0000","last_peered":"2024-03-21T13:08:58.814280+0000","last_clean":"2024-03-21T13:08:58.814280+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:08:58.814280+0000","last_undegraded":"2024-03-21T13:08:58.814280+0000","last_fullsized":"2024-03-21T13:08:58.814280+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:05.142807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:05.142807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:17.163399+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":404,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807257+0000","last_change":"2024-03-21T13:06:30.274121+0000","last_active":"2024-03-21T13:08:58.807257+0000","last_peered":"2024-03-21T13:08:58.807257+0000","last_clean":"2024-03-21T13:08:58.807257+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:08:58.807257+0000","last_undegraded":"2024-03-21T13:08:58.807257+0000","last_fullsized":"2024-03-21T13:08:58.807257+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:30.274033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:30.274033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"234'31","reported_seq":224,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.807470+0000","last_change":"2024-03-21T13:06:29.531422+0000","last_active":"2024-03-21T13:08:58.807470+0000","last_peered":"2024-03-21T13:08:58.807470+0000","last_clean":"2024-03-21T13:08:58.807470+0000","last_became_active":"2024-03-21T13:06:29.489192+0000","last_became_peered":"2024-03-21T13:06:29.489192+0000","last_unstale":"2024-03-21T13:08:58.807470+0000","last_undegraded":"2024-03-21T13:08:58.807470+0000","last_fullsized":"2024-03-21T13:08:58.807470+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-21T13:05:38.215693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.215693+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"240'1370","reported_seq":1390,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:09:00.795175+0000","last_change":"2024-03-21T13:08:45.620213+0000","last_active":"2024-03-21T13:09:00.795175+0000","last_peered":"2024-03-21T13:09:00.795175+0000","last_clean":"2024-03-21T13:09:00.795175+0000","last_became_active":"2024-03-21T13:08:45.619974+0000","last_became_peered":"2024-03-21T13:08:45.619974+0000","last_unstale":"2024-03-21T13:09:00.795175+0000","last_undegraded":"2024-03-21T13:09:00.795175+0000","last_fullsized":"2024-03-21T13:09:00.795175+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":1370,"log_dups_size":0,"ondisk_log_size":1370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:55.114760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89784320,"num_objects":1370,"num_object_clones":0,"num_object_copies":4110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1370,"num_write_kb":87680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":124,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804952+0000","last_change":"2024-03-21T13:06:29.497122+0000","last_active":"2024-03-21T13:08:58.804952+0000","last_peered":"2024-03-21T13:08:58.804952+0000","last_clean":"2024-03-21T13:08:58.804952+0000","last_became_active":"2024-03-21T13:06:29.496779+0000","last_became_peered":"2024-03-21T13:06:29.496779+0000","last_unstale":"2024-03-21T13:08:58.804952+0000","last_undegraded":"2024-03-21T13:08:58.804952+0000","last_fullsized":"2024-03-21T13:08:58.804952+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.291369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.291369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"196'11","reported_seq":342,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.804647+0000","last_change":"2024-03-21T13:06:29.522757+0000","last_active":"2024-03-21T13:08:58.804647+0000","last_peered":"2024-03-21T13:08:58.804647+0000","last_clean":"2024-03-21T13:08:58.804647+0000","last_became_active":"2024-03-21T13:06:29.501395+0000","last_became_peered":"2024-03-21T13:06:29.501395+0000","last_unstale":"2024-03-21T13:08:58.804647+0000","last_undegraded":"2024-03-21T13:08:58.804647+0000","last_fullsized":"2024-03-21T13:08:58.804647+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:05:30.300605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:30.300605+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":382,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.803498+0000","last_change":"2024-03-21T13:05:38.492137+0000","last_active":"2024-03-21T13:08:58.803498+0000","last_peered":"2024-03-21T13:08:58.803498+0000","last_clean":"2024-03-21T13:08:58.803498+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:08:58.803498+0000","last_undegraded":"2024-03-21T13:08:58.803498+0000","last_fullsized":"2024-03-21T13:08:58.803498+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:38.492015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.492015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"234'20","reported_seq":344,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-21T13:08:58.805620+0000","last_change":"2024-03-21T13:05:22.406498+0000","last_active":"2024-03-21T13:08:58.805620+0000","last_peered":"2024-03-21T13:08:58.805620+0000","last_clean":"2024-03-21T13:08:58.805620+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:08:58.805620+0000","last_undegraded":"2024-03-21T13:08:58.805620+0000","last_fullsized":"2024-03-21T13:08:58.805620+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:22.406367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.406367+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1277165568,"num_objects":19488,"num_object_clones":0,"num_object_copies":58464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19488,"num_write_kb":1247232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3802791936,"data_stored":3802791936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19488,"ondisk_log_size":19488,"up":48,"acting":48,"num_store_stats":5},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":612,"num_read_kb":607,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":5213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4428,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":15782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":726,"ondisk_log_size":726,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168705,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1354796,"kb_used_data":948772,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92390356,"statfs":{"total":95995035648,"available":94607724544,"internally_reserved":0,"allocated":971542528,"data_stored":971058482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":415758794},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463047,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1119388,"kb_used_data":650896,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92625764,"statfs":{"total":95995035648,"available":94848782336,"internally_reserved":0,"allocated":666517504,"data_stored":666053210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":479721926},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575007,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738533,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":966288,"kb_used_data":610248,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92778864,"statfs":{"total":95995035648,"available":95005556736,"internally_reserved":0,"allocated":624893952,"data_stored":624437049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411483,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182550,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1566828,"kb_used_data":985892,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92178324,"statfs":{"total":95995035648,"available":94390603776,"internally_reserved":0,"allocated":1009553408,"data_stored":1009068444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":594868679},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019024,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":962896,"kb_used_data":440260,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92782256,"statfs":{"total":95995035648,"available":95009030144,"internally_reserved":0,"allocated":450826240,"data_stored":450373546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724683,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":626786304,"data_stored":626786304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1024589824,"data_stored":1024589824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":460193792,"data_stored":460193792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":694353920,"data_stored":694353920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":996868096,"data_stored":996868096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:09:04.980 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-21T13:09:04.981 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool set unique_pool_2 pg_num 26 2024-03-21T13:09:05.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:05.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:05.121 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:05.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:05.218 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:05.221 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:05.222 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:05.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:05.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:05.350 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:05.399 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:05.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:05.500 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:05.523 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 12 16 31521 31505 164.047 129 0.0065944 0.00608925 2024-03-21T13:09:05.535 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:05.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:05.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:05 smithi163 ceph-mon[29621]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:09:05.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:05.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:09:05.594 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:05 smithi163 ceph-mon[40675]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:09:05.594 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:05.595 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:09:05.638 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:05.675 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:05.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:05.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:05 smithi204 ceph-mon[33153]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:09:05.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:05.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:09:05.777 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:05.815 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:05.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:05.910 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:09:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:09:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:09:05.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:05.954 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:05.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:06.055 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:06.089 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:06.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:06.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:06.191 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:06.226 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:06.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:06.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:06.324 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:09:06.328 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:06.358 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:06.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:06.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:06.494 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:06.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:06.523 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 13 16 33450 33434 160.702 120.562 0.0106135 0.00621756 2024-03-21T13:09:06.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:06.634 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:06.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:06.736 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:06 smithi204 ceph-mon[33153]: pgmap v458: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 639 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-03-21T13:09:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:06 smithi204 ceph-mon[33153]: 3.1e scrub starts 2024-03-21T13:09:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:06 smithi204 ceph-mon[33153]: 3.1e scrub ok 2024-03-21T13:09:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:06 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/621136307' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:09:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:06 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:09:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:06 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:09:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:06 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/621136307' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:09:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:06 smithi204 ceph-mon[33153]: osdmap e242: 8 total, 8 up, 5 in 2024-03-21T13:09:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[29621]: pgmap v458: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 639 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-03-21T13:09:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[29621]: 3.1e scrub starts 2024-03-21T13:09:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[29621]: 3.1e scrub ok 2024-03-21T13:09:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/621136307' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:09:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:09:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:09:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/621136307' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:09:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[29621]: osdmap e242: 8 total, 8 up, 5 in 2024-03-21T13:09:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[40675]: pgmap v458: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 639 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-03-21T13:09:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[40675]: 3.1e scrub starts 2024-03-21T13:09:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[40675]: 3.1e scrub ok 2024-03-21T13:09:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/621136307' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:09:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:09:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:09:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/621136307' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:09:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:06 smithi163 ceph-mon[40675]: osdmap e242: 8 total, 8 up, 5 in 2024-03-21T13:09:06.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:06.775 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:06.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:06.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:06.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:06.913 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:06.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:07.014 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:07.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:07.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:07.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:07.177 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:07.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:07.278 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:07.312 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:07.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:07.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:07.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:07.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:07.439 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:07.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:07.523 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 14 16 35259 35243 157.299 113.062 0.00603496 0.0063528 2024-03-21T13:09:07.540 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:07.574 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:07.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:07.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:07.708 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:07.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:07.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:07.845 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:07.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:07.947 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:07.986 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:07.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:08.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:08.125 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:08.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:08.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:08.226 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:08.261 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:08.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:08.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:08.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:08.399 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:08.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:08.500 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:08.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:08 smithi163 ceph-mon[29621]: pgmap v460: 148 pgs: 148 active+clean; 1.9 GiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 196 MiB/s wr, 3.13k op/s 2024-03-21T13:09:08.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:08 smithi163 ceph-mon[29621]: 5.5 deep-scrub starts 2024-03-21T13:09:08.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:08 smithi163 ceph-mon[29621]: 5.5 deep-scrub ok 2024-03-21T13:09:08.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:08 smithi163 ceph-mon[40675]: pgmap v460: 148 pgs: 148 active+clean; 1.9 GiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 196 MiB/s wr, 3.13k op/s 2024-03-21T13:09:08.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:08 smithi163 ceph-mon[40675]: 5.5 deep-scrub starts 2024-03-21T13:09:08.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:08 smithi163 ceph-mon[40675]: 5.5 deep-scrub ok 2024-03-21T13:09:08.523 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 15 15 37165 37150 154.758 119.188 0.00460995 0.00645724 2024-03-21T13:09:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:08.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:08.637 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:08.668 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:08.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:08 smithi204 ceph-mon[33153]: pgmap v460: 148 pgs: 148 active+clean; 1.9 GiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 196 MiB/s wr, 3.13k op/s 2024-03-21T13:09:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:08 smithi204 ceph-mon[33153]: 5.5 deep-scrub starts 2024-03-21T13:09:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:08 smithi204 ceph-mon[33153]: 5.5 deep-scrub ok 2024-03-21T13:09:08.770 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:08.806 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:08.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:08.908 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:08.943 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:08.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:08.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:09.045 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:09.082 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:09.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:09.183 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:09.219 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:09.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:09.321 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:09.357 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:09.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:09.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:09.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:09.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:09.499 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:09.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:09.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:09 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:09:09.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:09 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:09:09.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:09 smithi163 ceph-mon[29621]: osdmap e243: 8 total, 8 up, 5 in 2024-03-21T13:09:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:09 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:09:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:09 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:09:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:09 smithi163 ceph-mon[40675]: osdmap e243: 8 total, 8 up, 5 in 2024-03-21T13:09:09.524 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 16 16 39191 39175 152.995 126.562 0.00632472 0.00653056 2024-03-21T13:09:09.600 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:09.638 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:09.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:09.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:09.740 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:09.745 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:09.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:09 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:09:09.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:09 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:09:09.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:09 smithi204 ceph-mon[33153]: osdmap e243: 8 total, 8 up, 5 in 2024-03-21T13:09:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:09.785 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:09.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:09.886 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:09.923 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:09.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:09.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:10.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:10.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:10.065 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:10.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:10.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:10.204 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:10.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:10.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:10.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:10.343 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:10.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:10.445 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:10.482 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:10.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:10.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:10.523 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 17 16 41075 41059 150.921 117.75 0.0085153 0.00662115 2024-03-21T13:09:10.584 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:10.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:10.623 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:10.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:10.696 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:09:10.696 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:09:10.697 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:10 smithi163 ceph-mon[29621]: pgmap v462: 148 pgs: 148 active+clean; 1.9 GiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 684 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-21T13:09:10.697 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:10 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:09:10.697 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:10 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:09:10.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:10 smithi163 ceph-mon[40675]: pgmap v462: 148 pgs: 148 active+clean; 1.9 GiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 684 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-21T13:09:10.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:10 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:09:10.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:10 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:09:10.724 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:10.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:10 smithi204 ceph-mon[33153]: pgmap v462: 148 pgs: 148 active+clean; 1.9 GiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 684 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-21T13:09:10.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:10 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:09:10.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:10 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:09:10.764 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:10.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:10.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:10.866 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:10.900 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:10.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:10.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:11.002 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:11.037 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:11.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:11.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:11.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:11.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:11.276 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:11.279 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:09:11.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:11.310 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:11.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:11.412 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:11.457 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:11.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:11.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:11.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[29621]: 5.11 scrub starts 2024-03-21T13:09:11.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[29621]: 5.11 scrub ok 2024-03-21T13:09:11.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[29621]: 4.7 deep-scrub starts 2024-03-21T13:09:11.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[29621]: 4.7 deep-scrub ok 2024-03-21T13:09:11.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[29621]: 4.16 scrub starts 2024-03-21T13:09:11.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[29621]: 4.16 scrub ok 2024-03-21T13:09:11.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[29621]: 5.8 scrub starts 2024-03-21T13:09:11.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[29621]: 5.8 scrub ok 2024-03-21T13:09:11.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[29621]: osdmap e244: 8 total, 8 up, 5 in 2024-03-21T13:09:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[40675]: 5.11 scrub starts 2024-03-21T13:09:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[40675]: 5.11 scrub ok 2024-03-21T13:09:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[40675]: 4.7 deep-scrub starts 2024-03-21T13:09:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[40675]: 4.7 deep-scrub ok 2024-03-21T13:09:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[40675]: 4.16 scrub starts 2024-03-21T13:09:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[40675]: 4.16 scrub ok 2024-03-21T13:09:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[40675]: 5.8 scrub starts 2024-03-21T13:09:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[40675]: 5.8 scrub ok 2024-03-21T13:09:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:11 smithi163 ceph-mon[40675]: osdmap e244: 8 total, 8 up, 5 in 2024-03-21T13:09:11.523 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 18 16 42944 42928 149.026 116.812 0.00522299 0.00670414 2024-03-21T13:09:11.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:11.601 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:11.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:11.702 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:11.740 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:11.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:11.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:11.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:11 smithi204 ceph-mon[33153]: 5.11 scrub starts 2024-03-21T13:09:11.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:11 smithi204 ceph-mon[33153]: 5.11 scrub ok 2024-03-21T13:09:11.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:11 smithi204 ceph-mon[33153]: 4.7 deep-scrub starts 2024-03-21T13:09:11.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:11 smithi204 ceph-mon[33153]: 4.7 deep-scrub ok 2024-03-21T13:09:11.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:11 smithi204 ceph-mon[33153]: 4.16 scrub starts 2024-03-21T13:09:11.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:11 smithi204 ceph-mon[33153]: 4.16 scrub ok 2024-03-21T13:09:11.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:11 smithi204 ceph-mon[33153]: 5.8 scrub starts 2024-03-21T13:09:11.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:11 smithi204 ceph-mon[33153]: 5.8 scrub ok 2024-03-21T13:09:11.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:11 smithi204 ceph-mon[33153]: osdmap e244: 8 total, 8 up, 5 in 2024-03-21T13:09:11.842 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:11.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:11.878 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:11.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:11.979 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:12.017 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:12.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:12.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:12.153 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:12.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:12.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:09:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:09:12.255 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:12.287 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:12.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:12.389 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:12.424 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:12.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:12.523 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 19 16 44842 44826 147.426 118.625 0.00940186 0.00677795 2024-03-21T13:09:12.527 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:12.560 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:12.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:12.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:12.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:12 smithi163 ceph-mon[29621]: pgmap v463: 148 pgs: 148 active+clean; 2.3 GiB data, 9.2 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-21T13:09:12.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:12 smithi163 ceph-mon[29621]: 6.6 scrub starts 2024-03-21T13:09:12.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:12 smithi163 ceph-mon[29621]: 6.6 scrub ok 2024-03-21T13:09:12.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:12 smithi163 ceph-mon[29621]: 6.1a scrub starts 2024-03-21T13:09:12.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:12 smithi163 ceph-mon[29621]: 6.1a scrub ok 2024-03-21T13:09:12.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:12 smithi163 ceph-mon[40675]: pgmap v463: 148 pgs: 148 active+clean; 2.3 GiB data, 9.2 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-21T13:09:12.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:12 smithi163 ceph-mon[40675]: 6.6 scrub starts 2024-03-21T13:09:12.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:12 smithi163 ceph-mon[40675]: 6.6 scrub ok 2024-03-21T13:09:12.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:12 smithi163 ceph-mon[40675]: 6.1a scrub starts 2024-03-21T13:09:12.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:12 smithi163 ceph-mon[40675]: 6.1a scrub ok 2024-03-21T13:09:12.654 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-21T13:09:12.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:12.697 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:12 smithi204 ceph-mon[33153]: pgmap v463: 148 pgs: 148 active+clean; 2.3 GiB data, 9.2 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-21T13:09:12.697 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:12 smithi204 ceph-mon[33153]: 6.6 scrub starts 2024-03-21T13:09:12.697 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:12 smithi204 ceph-mon[33153]: 6.6 scrub ok 2024-03-21T13:09:12.697 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:12 smithi204 ceph-mon[33153]: 6.1a scrub starts 2024-03-21T13:09:12.697 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:12 smithi204 ceph-mon[33153]: 6.1a scrub ok 2024-03-21T13:09:12.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:12.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:12.841 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:12.871 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:12.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:12.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:12.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:13.007 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:13.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:13.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:13.043 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:13.109 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:13.153 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:13.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:13.254 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:13.297 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:13.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:13.305 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:09:13.337 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:09:13.370 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:09:13.399 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:13.405 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:09:13.437 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:13.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:13.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:13.449 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:09:13.482 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:09:13.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:13 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:13.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:13 smithi204 ceph-mon[33153]: 4.17 scrub starts 2024-03-21T13:09:13.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:13 smithi204 ceph-mon[33153]: 4.17 scrub ok 2024-03-21T13:09:13.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:13 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1963909175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:13.520 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:09:13.523 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:09:13.524245+0000 min lat: 0.00122048 max lat: 0.120711 avg lat: 0.00688824 2024-03-21T13:09:13.524 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:09:13.524 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 20 16 46449 46433 145.076 100.438 0.0106854 0.00688824 2024-03-21T13:09:13.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:13.559 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:09:13.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:13 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:13.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:13 smithi163 ceph-mon[29621]: 4.17 scrub starts 2024-03-21T13:09:13.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:13 smithi163 ceph-mon[29621]: 4.17 scrub ok 2024-03-21T13:09:13.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:13 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1963909175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:13.560 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:13 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:13.560 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:13 smithi163 ceph-mon[40675]: 4.17 scrub starts 2024-03-21T13:09:13.560 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:13 smithi163 ceph-mon[40675]: 4.17 scrub ok 2024-03-21T13:09:13.560 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:13 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1963909175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:09:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:09:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:09:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:09:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:09:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:09:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:09:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:09:13.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:13.576 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:13.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:13.677 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:13.717 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:13.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:13.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:13.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:13.859 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:13.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:13.960 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:13.998 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:13.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:14.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:14.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:14.139 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:14.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:14.240 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:14.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:14.277 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:14.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:14.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:14.393 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:14.415 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:14.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:14.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:14.523 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 21 15 48060 48045 142.965 100.75 0.0084144 0.00699028 2024-03-21T13:09:14.555 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:14.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:14.657 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:14.692 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:14.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:14.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:14 smithi163 ceph-mon[29621]: pgmap v465: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 789 B/s rd, 93 MiB/s wr, 1.48k op/s 2024-03-21T13:09:14.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:14 smithi163 ceph-mon[29621]: 3.11 scrub starts 2024-03-21T13:09:14.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:14 smithi163 ceph-mon[29621]: 3.11 scrub ok 2024-03-21T13:09:14.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:14 smithi163 ceph-mon[40675]: pgmap v465: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 789 B/s rd, 93 MiB/s wr, 1.48k op/s 2024-03-21T13:09:14.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:14 smithi163 ceph-mon[40675]: 3.11 scrub starts 2024-03-21T13:09:14.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:14 smithi163 ceph-mon[40675]: 3.11 scrub ok 2024-03-21T13:09:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:14 smithi204 ceph-mon[33153]: pgmap v465: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 789 B/s rd, 93 MiB/s wr, 1.48k op/s 2024-03-21T13:09:14.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:14 smithi204 ceph-mon[33153]: 3.11 scrub starts 2024-03-21T13:09:14.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:14 smithi204 ceph-mon[33153]: 3.11 scrub ok 2024-03-21T13:09:14.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:14.832 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:14.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:14.933 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:14.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:14.973 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:14.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:15.074 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:15.106 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:15.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:15.208 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:15.251 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:15.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:15.353 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:15.393 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:15.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:15.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:15.495 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:15.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:15 smithi163 ceph-mon[29621]: 3.e scrub starts 2024-03-21T13:09:15.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:15 smithi163 ceph-mon[29621]: 3.e scrub ok 2024-03-21T13:09:15.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:15 smithi163 ceph-mon[40675]: 3.e scrub starts 2024-03-21T13:09:15.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:15 smithi163 ceph-mon[40675]: 3.e scrub ok 2024-03-21T13:09:15.527 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 22 16 49610 49594 140.864 96.8125 0.0108885 0.00709334 2024-03-21T13:09:15.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:15.528 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:15.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:15.558 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:15 smithi204 ceph-mon[33153]: 3.e scrub starts 2024-03-21T13:09:15.558 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:15 smithi204 ceph-mon[33153]: 3.e scrub ok 2024-03-21T13:09:15.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:15.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:15.670 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:15.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:15.771 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:09:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:09:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25226.0:49978 9.c e244/234 9:3a82ff09:::benchmark_data_smithi163_131650_object49977:head v 244'3145, mlcod=244'3143)", 2024-03-21T13:09:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:15.763520+0000", 2024-03-21T13:09:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0062113410000000004, 2024-03-21T13:09:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062409020000000004, 2024-03-21T13:09:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.763520+0000", 2024-03-21T13:09:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.763520+0000", 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.763531+0000", 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1085e-05 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.763562+0000", 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0642999999999998e-05 2024-03-21T13:09:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.763574+0000", 2024-03-21T13:09:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.184e-05 2024-03-21T13:09:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.763581+0000", 2024-03-21T13:09:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4900000000000003e-06 2024-03-21T13:09:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.763638+0000", 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6372999999999997e-05 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.763662+0000", 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4808999999999999e-05 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.768747+0000", 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0050843490000000002 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25226.0:49979 9.0 e244/234 9:0eb6fd1b:::benchmark_data_smithi163_131650_object49978:head v 244'3119, mlcod=244'3115)", 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:15.765929+0000", 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0038021980000000001, 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039170699999999999, 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.765929+0000", 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.765929+0000", 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.765930+0000", 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9500000000000001e-07 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.765955+0000", 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4260000000000002e-05 2024-03-21T13:09:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.765967+0000", 2024-03-21T13:09:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2364000000000001e-05 2024-03-21T13:09:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.765974+0000", 2024-03-21T13:09:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0190000000000001e-06 2024-03-21T13:09:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.766067+0000", 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3152000000000005e-05 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.766089+0000", 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2107999999999999e-05 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25226.0:49983 9.c e244/234 9:37afa2b2:::benchmark_data_smithi163_131650_object49982:head v 244'3146, mlcod=244'3143)", 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:15.766913+0000", 2024-03-21T13:09:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0028183029999999999, 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002998386, 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.766913+0000", 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.766913+0000", 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.766914+0000", 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0800000000000004e-07 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.767011+0000", 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7159999999999998e-05 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.767024+0000", 2024-03-21T13:09:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2811e-05 2024-03-21T13:09:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.767031+0000", 2024-03-21T13:09:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.221e-06 2024-03-21T13:09:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.767106+0000", 2024-03-21T13:09:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4340999999999999e-05 2024-03-21T13:09:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.767124+0000", 2024-03-21T13:09:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8017e-05 2024-03-21T13:09:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25226.0:49980 9.d e244/234 9:be8a8e0d:::benchmark_data_smithi163_131650_object49979:head v 244'3109, mlcod=244'3107)", 2024-03-21T13:09:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:15.768086+0000", 2024-03-21T13:09:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0016451529999999999, 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0018906800000000001, 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.768086+0000", 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.768086+0000", 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.768087+0000", 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1800000000000005e-07 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.768114+0000", 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6857999999999999e-05 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.768129+0000", 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4756999999999999e-05 2024-03-21T13:09:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.768136+0000", 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0829999999999998e-06 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.768410+0000", 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027398800000000002 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:15.768427+0000", 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7558999999999999e-05 2024-03-21T13:09:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:15.788 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:15.788 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:15.788 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:15.788 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-03-21T13:09:15.788 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 4 2024-03-21T13:09:15.788 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:09:15.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:15.830 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:15.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:15.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:15.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:15.940 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:09:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:09:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:09:15.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:15.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:15.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:16.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:16.111 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:16.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:16.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:16.254 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:16.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:16.333 INFO:teuthology.orchestra.run.smithi163.stderr:set pool 9 pg_num to 26 2024-03-21T13:09:16.355 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:16.374 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:09:16.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:16.397 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:16.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:16.498 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[29621]: pgmap v466: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-21T13:09:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1838962635' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-03-21T13:09:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[29621]: 3.0 scrub starts 2024-03-21T13:09:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[29621]: 3.0 scrub ok 2024-03-21T13:09:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[29621]: 3.8 scrub starts 2024-03-21T13:09:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[29621]: 3.8 scrub ok 2024-03-21T13:09:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[29621]: 4.3 scrub starts 2024-03-21T13:09:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[29621]: 4.3 scrub ok 2024-03-21T13:09:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1838962635' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-03-21T13:09:16.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[29621]: osdmap e245: 8 total, 8 up, 5 in 2024-03-21T13:09:16.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[40675]: pgmap v466: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-21T13:09:16.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1838962635' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-03-21T13:09:16.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[40675]: 3.0 scrub starts 2024-03-21T13:09:16.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[40675]: 3.0 scrub ok 2024-03-21T13:09:16.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[40675]: 3.8 scrub starts 2024-03-21T13:09:16.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[40675]: 3.8 scrub ok 2024-03-21T13:09:16.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[40675]: 4.3 scrub starts 2024-03-21T13:09:16.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[40675]: 4.3 scrub ok 2024-03-21T13:09:16.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1838962635' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-03-21T13:09:16.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:16 smithi163 ceph-mon[40675]: osdmap e245: 8 total, 8 up, 5 in 2024-03-21T13:09:16.524 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 23 16 51142 51126 138.903 95.75 0.00673738 0.00719422 2024-03-21T13:09:16.532 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:16.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:16.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:16.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-21T13:09:16.634 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:16.667 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:16.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:16.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:16 smithi204 ceph-mon[33153]: pgmap v466: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-21T13:09:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:16 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1838962635' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-03-21T13:09:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:16 smithi204 ceph-mon[33153]: 3.0 scrub starts 2024-03-21T13:09:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:16 smithi204 ceph-mon[33153]: 3.0 scrub ok 2024-03-21T13:09:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:16 smithi204 ceph-mon[33153]: 3.8 scrub starts 2024-03-21T13:09:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:16 smithi204 ceph-mon[33153]: 3.8 scrub ok 2024-03-21T13:09:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:16 smithi204 ceph-mon[33153]: 4.3 scrub starts 2024-03-21T13:09:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:16 smithi204 ceph-mon[33153]: 4.3 scrub ok 2024-03-21T13:09:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:16 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1838962635' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-03-21T13:09:16.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:16 smithi204 ceph-mon[33153]: osdmap e245: 8 total, 8 up, 5 in 2024-03-21T13:09:16.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:16.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:16.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:16.910 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:16.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:16.959 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:16.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:17.001 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:17.060 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:17.110 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:17.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:17.212 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:17.249 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:17.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:17.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:17.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:17.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:17.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:17.447 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:09:17.453 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:17 smithi204 ceph-mon[33153]: 5.2 scrub starts 2024-03-21T13:09:17.453 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:17 smithi204 ceph-mon[33153]: 5.2 scrub ok 2024-03-21T13:09:17.453 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:17 smithi204 ceph-mon[33153]: 5.6 scrub starts 2024-03-21T13:09:17.453 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:17 smithi204 ceph-mon[33153]: 5.6 scrub ok 2024-03-21T13:09:17.453 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:17 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-21T13:09:17.453 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:17 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/878649652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:09:17.453 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:17 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:09:17.454 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:17.498 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:17.525 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 24 16 52334 52318 136.214 74.5 0.0535129 0.0073089 2024-03-21T13:09:17.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:17.556 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:17.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:17.657 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:17.682 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:17.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[29621]: 5.2 scrub starts 2024-03-21T13:09:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[29621]: 5.2 scrub ok 2024-03-21T13:09:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[29621]: 5.6 scrub starts 2024-03-21T13:09:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[29621]: 5.6 scrub ok 2024-03-21T13:09:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-21T13:09:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/878649652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:09:17.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:09:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[40675]: 5.2 scrub starts 2024-03-21T13:09:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[40675]: 5.2 scrub ok 2024-03-21T13:09:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[40675]: 5.6 scrub starts 2024-03-21T13:09:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[40675]: 5.6 scrub ok 2024-03-21T13:09:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-21T13:09:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/878649652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:09:17.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:17 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:09:17.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:17.810 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:17.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:17.911 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:17.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:17.936 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:17.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:18.038 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:18.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:18.063 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:18.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:18.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:18.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:18.190 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:18.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:18.297 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:18.322 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:18.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:18.424 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:18.459 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:18.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:18.525 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 25 15 52441 52426 131.036 6.75 0.00817927 0.00762663 2024-03-21T13:09:18.561 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:18.598 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:18.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:18.630 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:18 smithi204 ceph-mon[33153]: pgmap v468: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.28k op/s 2024-03-21T13:09:18.630 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:18 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:09:18.630 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:18 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-21T13:09:18.630 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:18 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:09:18.630 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:18 smithi204 ceph-mon[33153]: osdmap e246: 8 total, 8 up, 5 in 2024-03-21T13:09:18.703 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:18.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:18.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:18.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:18 smithi163 ceph-mon[40675]: pgmap v468: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.28k op/s 2024-03-21T13:09:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:18 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:09:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:18 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-21T13:09:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:18 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:09:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:18 smithi163 ceph-mon[40675]: osdmap e246: 8 total, 8 up, 5 in 2024-03-21T13:09:18.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:18 smithi163 ceph-mon[29621]: pgmap v468: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.28k op/s 2024-03-21T13:09:18.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:18 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:09:18.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:18 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-21T13:09:18.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:18 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:09:18.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:18 smithi163 ceph-mon[29621]: osdmap e246: 8 total, 8 up, 5 in 2024-03-21T13:09:18.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:18.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:18.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:18.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:18.980 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:18.986 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:09:18.986 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-03-21T13:09:18.987 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-03-21T13:09:18.987 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-03-21T13:09:18.987 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:09:19.015 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:19.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:19.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:19.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:19.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:19.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:19.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:19.294 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:19.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:19.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:19.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:19.431 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:19.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:19.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:19.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:19 smithi204 ceph-mon[33153]: osdmap e247: 8 total, 8 up, 5 in 2024-03-21T13:09:19.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:19 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-21T13:09:19.525 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 26 16 54276 54260 130.404 114.625 0.00203715 0.0076417 2024-03-21T13:09:19.533 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:19.557 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:19.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:19.658 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:19.684 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:19.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:19.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-21T13:09:19.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:19 smithi163 ceph-mon[29621]: osdmap e247: 8 total, 8 up, 5 in 2024-03-21T13:09:19.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:19 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-21T13:09:19.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:19 smithi163 ceph-mon[40675]: osdmap e247: 8 total, 8 up, 5 in 2024-03-21T13:09:19.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:19 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-21T13:09:19.784 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:19.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:19.810 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:19.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:19.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:19.934 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:19.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:19.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:20.039 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:20.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:20.063 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:20.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:20.164 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:20.189 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:20.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:20.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:20.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:20.398 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:20.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:20.473 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:09:20.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[29621]: pgmap v471: 158 pgs: 10 unknown, 148 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 853 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-21T13:09:20.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-21T13:09:20.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[29621]: osdmap e248: 8 total, 8 up, 5 in 2024-03-21T13:09:20.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:09:20.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:09:20.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:09:20.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[29621]: osdmap e249: 8 total, 8 up, 5 in 2024-03-21T13:09:20.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[40675]: pgmap v471: 158 pgs: 10 unknown, 148 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 853 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-21T13:09:20.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-21T13:09:20.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[40675]: osdmap e248: 8 total, 8 up, 5 in 2024-03-21T13:09:20.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:09:20.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:09:20.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:09:20.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:20 smithi163 ceph-mon[40675]: osdmap e249: 8 total, 8 up, 5 in 2024-03-21T13:09:20.492 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:09:20.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:20.511 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:09:20.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:20.524 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:20.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:20.525 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 27 16 54277 54261 125.577 0.0625 1.00973 0.00766017 2024-03-21T13:09:20.530 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:09:20.549 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:09:20.556 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:09:20.570 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:09:20.589 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:09:20.610 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:09:20.611 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:09:20.611 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:09:20.612 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:09:20.612 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:09:20.612 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:09:20.612 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:09:20.612 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:09:20.612 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:09:20.624 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:20.649 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:20.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:20.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:20.751 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:20 smithi204 ceph-mon[33153]: pgmap v471: 158 pgs: 10 unknown, 148 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 853 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-21T13:09:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-21T13:09:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:20 smithi204 ceph-mon[33153]: osdmap e248: 8 total, 8 up, 5 in 2024-03-21T13:09:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:20 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:09:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:20 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:09:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:09:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:20 smithi204 ceph-mon[33153]: osdmap e249: 8 total, 8 up, 5 in 2024-03-21T13:09:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:20.775 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:20.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:20.876 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:20.901 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:20.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:20.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:21.003 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:21.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:21.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:21.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:21.158 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:21.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:21.259 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:21.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:21.287 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:21.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:21.388 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:21.428 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:21.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:21.503 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:21.526 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 28 16 54398 54382 121.362 7.5625 0.00675462 0.00823549 2024-03-21T13:09:21.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:21.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:21.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:21.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:21.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:21.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:21 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1814343451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:21.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:21 smithi163 ceph-mon[40675]: osdmap e250: 8 total, 8 up, 5 in 2024-03-21T13:09:21.704 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:21 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1814343451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:21.704 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:21 smithi163 ceph-mon[29621]: osdmap e250: 8 total, 8 up, 5 in 2024-03-21T13:09:21.704 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:21 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1814343451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:21.704 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:21 smithi204 ceph-mon[33153]: osdmap e250: 8 total, 8 up, 5 in 2024-03-21T13:09:21.714 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:21.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:21.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:21.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:21.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:21.963 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:22.006 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:22.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:22.053 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:09:22.053 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:09:22.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:09:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:09:22.108 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:22.149 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:22.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:22.250 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:22.288 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:22.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:22.390 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:22.406 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:09:22.406 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:09:22.406 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-21T13:09:22.407 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:09:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:22.426 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:22.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:22.526 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 29 16 55873 55857 120.356 92.1875 0.00620155 0.00830337 2024-03-21T13:09:22.527 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:22.577 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:22.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:22.678 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:22.711 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:22 smithi163 ceph-mon[29621]: pgmap v474: 158 pgs: 1 peering, 9 unknown, 148 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 4.4 MiB/s wr, 70 op/s; 16 KiB/s, 0 objects/s recovering 2024-03-21T13:09:22.711 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:22 smithi163 ceph-mon[29621]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 247'3362 2024-03-21T13:09:22.711 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:22 smithi163 ceph-mon[29621]: 9.15 starting backfill to osd.4 from (0'0,0'0] MAX to 247'3384 2024-03-21T13:09:22.711 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:22 smithi163 ceph-mon[29621]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 247'3299 2024-03-21T13:09:22.711 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:22 smithi163 ceph-mon[29621]: 9.14 starting backfill to osd.6 from (0'0,0'0] MAX to 247'3266 2024-03-21T13:09:22.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:22 smithi163 ceph-mon[40675]: pgmap v474: 158 pgs: 1 peering, 9 unknown, 148 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 4.4 MiB/s wr, 70 op/s; 16 KiB/s, 0 objects/s recovering 2024-03-21T13:09:22.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:22 smithi163 ceph-mon[40675]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 247'3362 2024-03-21T13:09:22.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:22 smithi163 ceph-mon[40675]: 9.15 starting backfill to osd.4 from (0'0,0'0] MAX to 247'3384 2024-03-21T13:09:22.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:22 smithi163 ceph-mon[40675]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 247'3299 2024-03-21T13:09:22.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:22 smithi163 ceph-mon[40675]: 9.14 starting backfill to osd.6 from (0'0,0'0] MAX to 247'3266 2024-03-21T13:09:22.726 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:22.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:22.738 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:22 smithi204 ceph-mon[33153]: pgmap v474: 158 pgs: 1 peering, 9 unknown, 148 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 4.4 MiB/s wr, 70 op/s; 16 KiB/s, 0 objects/s recovering 2024-03-21T13:09:22.738 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:22 smithi204 ceph-mon[33153]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 247'3362 2024-03-21T13:09:22.738 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:22 smithi204 ceph-mon[33153]: 9.15 starting backfill to osd.4 from (0'0,0'0] MAX to 247'3384 2024-03-21T13:09:22.738 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:22 smithi204 ceph-mon[33153]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 247'3299 2024-03-21T13:09:22.738 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:22 smithi204 ceph-mon[33153]: 9.14 starting backfill to osd.6 from (0'0,0'0] MAX to 247'3266 2024-03-21T13:09:22.790 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-21T13:09:22.828 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:22.866 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:22.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:22.968 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:23.018 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:23.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:23.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:23.120 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:23.160 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:23.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:23.223 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:23.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:23.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:23.307 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:23.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:23.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:23.444 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:09:23.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:23.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:23.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:23.526 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 30 16 57098 57082 118.896 76.5625 0.00865339 0.00839784 2024-03-21T13:09:23.550 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:23.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:23.600 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:23.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:23.687 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:23 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:23.688 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:23 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:23.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:23.750 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:23.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:23 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:23 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:23 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:23 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:23.851 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:23.890 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:23.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:23.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:23.992 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:24.027 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:24.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:24.128 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:24.166 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:24.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:24.268 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:24.310 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:24.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:24.411 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:24.456 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:24.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:24.526 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 31 16 58297 58281 117.478 74.9375 0.00902441 0.00850682 2024-03-21T13:09:24.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:24.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:24.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:24.685 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:24.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:24 smithi204 ceph-mon[33153]: pgmap v476: 158 pgs: 3 remapped+peering, 1 peering, 6 unknown, 148 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.1 KiB/s rd, 1.5 MiB/s wr, 24 op/s; 29 KiB/s, 0 objects/s recovering 2024-03-21T13:09:24.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:24 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:24.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:24 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:24.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:24 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:24.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:24 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:24.709 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:24 smithi204 ceph-mon[33153]: Adjusting osd_memory_target on smithi204 to 3248M 2024-03-21T13:09:24.709 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:24 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:24.722 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:24.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:24.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[29621]: pgmap v476: 158 pgs: 3 remapped+peering, 1 peering, 6 unknown, 148 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.1 KiB/s rd, 1.5 MiB/s wr, 24 op/s; 29 KiB/s, 0 objects/s recovering 2024-03-21T13:09:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[29621]: Adjusting osd_memory_target on smithi204 to 3248M 2024-03-21T13:09:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:24.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[40675]: pgmap v476: 158 pgs: 3 remapped+peering, 1 peering, 6 unknown, 148 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.1 KiB/s rd, 1.5 MiB/s wr, 24 op/s; 29 KiB/s, 0 objects/s recovering 2024-03-21T13:09:24.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:24.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:24.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:24.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:24.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[40675]: Adjusting osd_memory_target on smithi204 to 3248M 2024-03-21T13:09:24.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:24 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:24.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:24.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:24.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:24.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:25.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:25.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:25.104 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:25.140 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:25.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:25.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:25.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:25.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:25.285 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:25.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:25.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:25.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:25.426 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:25.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:25.433 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:09:25.433 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:09:25.526 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 32 16 59524 59508 116.203 76.6875 0.02242 0.0085995 2024-03-21T13:09:25.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:25.559 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:25.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:25.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:25.661 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:25.688 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:25.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:25.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:25 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:25 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:25 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:25.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:25 smithi204 ceph-mon[33153]: pgmap v477: 158 pgs: 3 remapped+peering, 1 peering, 6 unknown, 148 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 1.1 MiB/s wr, 18 op/s; 21 KiB/s, 0 objects/s recovering 2024-03-21T13:09:25.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:25 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:25.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:25 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:25.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:25 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:25.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:25 smithi163 ceph-mon[29621]: pgmap v477: 158 pgs: 3 remapped+peering, 1 peering, 6 unknown, 148 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 1.1 MiB/s wr, 18 op/s; 21 KiB/s, 0 objects/s recovering 2024-03-21T13:09:25.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:09:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:09:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:09:25.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:25 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:25 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:25 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:25.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:25 smithi163 ceph-mon[40675]: pgmap v477: 158 pgs: 3 remapped+peering, 1 peering, 6 unknown, 148 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 1.1 MiB/s wr, 18 op/s; 21 KiB/s, 0 objects/s recovering 2024-03-21T13:09:25.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:25.827 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:25.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:25.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:25.919 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:25.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:25.963 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:25.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:26.065 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:26.106 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:26.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:26.207 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:26.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-21T13:09:26.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:26.250 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:26.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:26.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:26.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:26.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:26.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:26.489 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:26.526 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 33 16 60851 60835 115.195 82.9375 0.0183244 0.00867641 2024-03-21T13:09:26.528 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:26.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:26.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:26.603 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:09:26.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:26.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:26.694 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:26.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:26.724 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:26.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:26.837 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:26.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:26.868 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:26 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2423149265' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:09:26.868 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:26 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:09:26.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:26.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:26.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:26.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:26 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2423149265' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:09:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:26 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:09:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:26 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2423149265' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:09:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:26 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:09:27.080 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:27.127 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:27.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:27.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:27.268 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:27.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:27.370 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:27.414 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:27.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:27.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:27.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:27.526 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 34 16 62281 62265 114.435 89.375 0.0121108 0.00873287 2024-03-21T13:09:27.539 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:27.550 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:27.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:27.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:27.652 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:27.698 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:27.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:27.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:27 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:09:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:27 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:09:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:27 smithi163 ceph-mon[29621]: osdmap e251: 8 total, 8 up, 5 in 2024-03-21T13:09:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:27 smithi163 ceph-mon[29621]: pgmap v479: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 3.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 802 B/s rd, 95 MiB/s wr, 1.52k op/s; 3352/177849 objects degraded (1.885%); 13900/177849 objects misplaced (7.816%); 17 MiB/s, 274 objects/s recovering 2024-03-21T13:09:27.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:27 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:09:27.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:27 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:09:27.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:27 smithi163 ceph-mon[40675]: osdmap e251: 8 total, 8 up, 5 in 2024-03-21T13:09:27.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:27 smithi163 ceph-mon[40675]: pgmap v479: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 3.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 802 B/s rd, 95 MiB/s wr, 1.52k op/s; 3352/177849 objects degraded (1.885%); 13900/177849 objects misplaced (7.816%); 17 MiB/s, 274 objects/s recovering 2024-03-21T13:09:27.800 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:27.839 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:27.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:27.941 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:27.978 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:27.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:27 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:09:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:27 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:09:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:27 smithi204 ceph-mon[33153]: osdmap e251: 8 total, 8 up, 5 in 2024-03-21T13:09:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:27 smithi204 ceph-mon[33153]: pgmap v479: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 3.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 802 B/s rd, 95 MiB/s wr, 1.52k op/s; 3352/177849 objects degraded (1.885%); 13900/177849 objects misplaced (7.816%); 17 MiB/s, 274 objects/s recovering 2024-03-21T13:09:28.079 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:28.126 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:28.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:28.227 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:28.272 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:28.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:28.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:28.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:28.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:28.515 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:28.526 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 35 16 63741 63725 113.773 91.25 0.0239834 0.00878219 2024-03-21T13:09:28.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:28.552 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:28.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:28.653 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:28.677 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:28 smithi163 ceph-mon[29621]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:09:28.677 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:28 smithi163 ceph-mon[29621]: Health check failed: Degraded data redundancy: 3352/177849 objects degraded (1.885%), 1 pg degraded (PG_DEGRADED) 2024-03-21T13:09:28.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:28 smithi163 ceph-mon[40675]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:09:28.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:28 smithi163 ceph-mon[40675]: Health check failed: Degraded data redundancy: 3352/177849 objects degraded (1.885%), 1 pg degraded (PG_DEGRADED) 2024-03-21T13:09:28.690 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:28.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:28.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:28.804 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:09:28.804 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:09:28.828 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:28.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:28.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:28.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:28.966 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:28.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:29.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:28 smithi204 ceph-mon[33153]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:09:29.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:28 smithi204 ceph-mon[33153]: Health check failed: Degraded data redundancy: 3352/177849 objects degraded (1.885%), 1 pg degraded (PG_DEGRADED) 2024-03-21T13:09:29.067 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:29.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:29.100 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:29.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:29.201 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:29.240 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:29.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:29.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:29.381 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:29.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:29.482 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:29.519 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:29.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:29.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:29.526 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 36 16 64869 64853 112.571 70.5 0.00574144 0.00887828 2024-03-21T13:09:29.548 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:09:29.622 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:29.668 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:29.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:29.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:29.801 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:29.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:29.902 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:29.941 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:29.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:30.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:29 smithi204 ceph-mon[33153]: pgmap v480: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 3.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.21k op/s; 3352/177849 objects degraded (1.885%); 13900/177849 objects misplaced (7.816%); 14 MiB/s, 218 objects/s recovering 2024-03-21T13:09:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:29 smithi163 ceph-mon[29621]: pgmap v480: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 3.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.21k op/s; 3352/177849 objects degraded (1.885%); 13900/177849 objects misplaced (7.816%); 14 MiB/s, 218 objects/s recovering 2024-03-21T13:09:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:29 smithi163 ceph-mon[40675]: pgmap v480: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 3.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.21k op/s; 3352/177849 objects degraded (1.885%); 13900/177849 objects misplaced (7.816%); 14 MiB/s, 218 objects/s recovering 2024-03-21T13:09:30.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:30.068 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:30.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:30.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:30.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:30.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:30.311 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:30.336 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:30.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:30.438 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:30.477 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:30.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:30.527 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 37 16 65850 65834 111.185 61.3125 0.0088596 0.00898907 2024-03-21T13:09:30.578 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:30.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:30.620 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:30.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:30.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:30.769 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:30.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:30.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:30.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:30.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:31.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:31.037 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:31.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:31.139 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:31.170 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:31.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:31.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:31.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:31.402 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:31.428 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:31.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:31.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:31.527 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 38 16 66578 66562 109.457 45.5 0.00293391 0.00899201 2024-03-21T13:09:31.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:31.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:31.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:31.658 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:31.689 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:31.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:31.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:31.750 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:09:31.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:31 smithi163 ceph-mon[29621]: osdmap e252: 8 total, 8 up, 5 in 2024-03-21T13:09:31.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:31 smithi163 ceph-mon[29621]: pgmap v482: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 3.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s; 3352/190770 objects degraded (1.757%); 13138/190770 objects misplaced (6.887%); 23 MiB/s, 362 objects/s recovering 2024-03-21T13:09:31.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:31 smithi163 ceph-mon[29621]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:09:31.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:31 smithi163 ceph-mon[40675]: osdmap e252: 8 total, 8 up, 5 in 2024-03-21T13:09:31.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:31 smithi163 ceph-mon[40675]: pgmap v482: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 3.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s; 3352/190770 objects degraded (1.757%); 13138/190770 objects misplaced (6.887%); 23 MiB/s, 362 objects/s recovering 2024-03-21T13:09:31.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:31 smithi163 ceph-mon[40675]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:09:31.759 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:09:31.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:31.830 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:31.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:31.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:31.932 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:31.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:31.969 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:31.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:31 smithi204 ceph-mon[33153]: osdmap e252: 8 total, 8 up, 5 in 2024-03-21T13:09:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:31 smithi204 ceph-mon[33153]: pgmap v482: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 3.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s; 3352/190770 objects degraded (1.757%); 13138/190770 objects misplaced (6.887%); 23 MiB/s, 362 objects/s recovering 2024-03-21T13:09:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:31 smithi204 ceph-mon[33153]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:09:32.003 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:09:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:09:32.070 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:32.107 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:32.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:32.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:32.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:32.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:32.235 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:32.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:32.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:32.361 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:32.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:32.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:32.500 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:32.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:32.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:32.527 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 39 16 67548 67532 108.205 60.625 0.0129166 0.00923718 2024-03-21T13:09:32.603 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:32.641 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:32.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:32.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:32.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:32.885 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:32.912 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:32.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:33.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:32 smithi204 ceph-mon[33153]: osdmap e253: 8 total, 8 up, 5 in 2024-03-21T13:09:33.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:32 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:33.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:32 smithi163 ceph-mon[29621]: osdmap e253: 8 total, 8 up, 5 in 2024-03-21T13:09:33.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:32 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:33.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:32 smithi163 ceph-mon[40675]: osdmap e253: 8 total, 8 up, 5 in 2024-03-21T13:09:33.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:32 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:33.012 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:33.048 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:33.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:33.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:33.172 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:33.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:33.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:33.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:33.311 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:33.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:33.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:33.413 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:33.440 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:33.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:33.527 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:09:33.527721+0000 min lat: 0.00122048 max lat: 2.05061 avg lat: 0.00930924 2024-03-21T13:09:33.527 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:09:33.527 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 40 16 68743 68727 107.367 74.6875 0.0101608 0.00930924 2024-03-21T13:09:33.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:33.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:33.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:33.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:33.683 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:33.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:33.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:33.825 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:33.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:33.865 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:33.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:33.887 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:33 smithi163 ceph-mon[29621]: pgmap v484: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 4.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 822 B/s rd, 63 MiB/s wr, 1.01k op/s; 3352/196758 objects degraded (1.704%); 12750/196758 objects misplaced (6.480%); 15 MiB/s, 232 objects/s recovering 2024-03-21T13:09:33.888 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:33 smithi163 ceph-mon[40675]: pgmap v484: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 4.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 822 B/s rd, 63 MiB/s wr, 1.01k op/s; 3352/196758 objects degraded (1.704%); 12750/196758 objects misplaced (6.480%); 15 MiB/s, 232 objects/s recovering 2024-03-21T13:09:33.965 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:33 smithi204 ceph-mon[33153]: pgmap v484: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 4.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 822 B/s rd, 63 MiB/s wr, 1.01k op/s; 3352/196758 objects degraded (1.704%); 12750/196758 objects misplaced (6.480%); 15 MiB/s, 232 objects/s recovering 2024-03-21T13:09:34.004 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:34.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:34.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:34.106 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:34.144 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:34.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:34.245 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:34.283 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:34.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:34.384 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:34.426 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:34.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:34.527 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:34.528 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 41 16 70026 70010 106.704 80.1875 0.010147 0.00936226 2024-03-21T13:09:34.568 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:34.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:34.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:34.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:34.694 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:34.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:34.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:34.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:34.831 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:34.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:34.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:34.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:34.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:35.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:34 smithi204 ceph-mon[33153]: Health check update: Degraded data redundancy: 3352/196758 objects degraded (1.704%), 1 pg degraded (PG_DEGRADED) 2024-03-21T13:09:35.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:34 smithi163 ceph-mon[29621]: Health check update: Degraded data redundancy: 3352/196758 objects degraded (1.704%), 1 pg degraded (PG_DEGRADED) 2024-03-21T13:09:35.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:34 smithi163 ceph-mon[40675]: Health check update: Degraded data redundancy: 3352/196758 objects degraded (1.704%), 1 pg degraded (PG_DEGRADED) 2024-03-21T13:09:35.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:35.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-21T13:09:35.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:35.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:35.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:35.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:35.271 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:35.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:35.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:35.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:35.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:35.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:35.514 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:35.527 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 42 16 71039 71023 105.67 63.3125 0.0128912 0.0094588 2024-03-21T13:09:35.527 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:35.567 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:35.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:35.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:35.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:35.708 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:35.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:35.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:35.847 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:35.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:35.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:35.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:35.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:35.973 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:35.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:36.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:09:36.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:35 smithi204 ceph-mon[33153]: pgmap v485: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 4.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 639 B/s rd, 49 MiB/s wr, 788 op/s; 3352/196758 objects degraded (1.704%); 12750/196758 objects misplaced (6.480%); 11 MiB/s, 180 objects/s recovering 2024-03-21T13:09:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:09:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:35 smithi163 ceph-mon[29621]: pgmap v485: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 4.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 639 B/s rd, 49 MiB/s wr, 788 op/s; 3352/196758 objects degraded (1.704%); 12750/196758 objects misplaced (6.480%); 11 MiB/s, 180 objects/s recovering 2024-03-21T13:09:36.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:09:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:09:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:09:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:36.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:09:36.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:35 smithi163 ceph-mon[40675]: pgmap v485: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 149 active+clean, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 4.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 639 B/s rd, 49 MiB/s wr, 788 op/s; 3352/196758 objects degraded (1.704%); 12750/196758 objects misplaced (6.480%); 11 MiB/s, 180 objects/s recovering 2024-03-21T13:09:36.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:36.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:36.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:36.212 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:36.236 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:36.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:36.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:36.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:36.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:36.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:36.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:36.501 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:36.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:36.527 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 43 16 72040 72024 104.668 62.5625 0.00518647 0.00954692 2024-03-21T13:09:36.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:36.639 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:36.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:36.741 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:36.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:36.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:36.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:36.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:36.923 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:36.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:36.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:37.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:37.064 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:37.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:37.166 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:37.204 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:37.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:37.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:37.308 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:37.333 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:37.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:37.434 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:37.475 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:37.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:37.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:37.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:37 smithi163 ceph-mon[29621]: osdmap e254: 8 total, 8 up, 5 in 2024-03-21T13:09:37.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:37 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2575165599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:37.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:37 smithi163 ceph-mon[40675]: osdmap e254: 8 total, 8 up, 5 in 2024-03-21T13:09:37.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:37 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2575165599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:37.527 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 44 16 73069 73053 103.751 64.3125 0.00664275 0.00962939 2024-03-21T13:09:37.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:37.614 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:37.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:37.717 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:37.745 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:37.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:37.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:37.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:37 smithi204 ceph-mon[33153]: osdmap e254: 8 total, 8 up, 5 in 2024-03-21T13:09:37.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:37 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2575165599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:37.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:37.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:37.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:37.985 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:38.017 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:38.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:38.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:38.119 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:38.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:38.145 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:38.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:38.246 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:38.268 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:09:38.268 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:09:38.268 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:09:38.268 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:09:38.268 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:09:38.268 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:09:38.268 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:09:38.268 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:09:38.269 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:09:38.269 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:09:38.270 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:09:38.270 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:09:38.270 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:09:38.270 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:09:38.270 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:09:38.270 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:09:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:38.278 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:38.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:38.379 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:38.410 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:38.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:38.511 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:38.527 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 45 15 73757 73742 102.402 43.0625 0.0122965 0.00976085 2024-03-21T13:09:38.543 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:38.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:38.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:38.646 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:38.679 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:38.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:38.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:38.693 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:38 smithi204 ceph-mon[33153]: pgmap v487: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 150 active+clean, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 4.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 823 B/s rd, 74 MiB/s wr, 1.18k op/s; 3352/212733 objects degraded (1.576%); 9834/212733 objects misplaced (4.623%); 34 MiB/s, 541 objects/s recovering 2024-03-21T13:09:38.694 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:38 smithi204 ceph-mon[33153]: osdmap e255: 8 total, 8 up, 5 in 2024-03-21T13:09:38.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:38 smithi163 ceph-mon[29621]: pgmap v487: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 150 active+clean, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 4.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 823 B/s rd, 74 MiB/s wr, 1.18k op/s; 3352/212733 objects degraded (1.576%); 9834/212733 objects misplaced (4.623%); 34 MiB/s, 541 objects/s recovering 2024-03-21T13:09:38.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:38 smithi163 ceph-mon[29621]: osdmap e255: 8 total, 8 up, 5 in 2024-03-21T13:09:38.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:38 smithi163 ceph-mon[40675]: pgmap v487: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 150 active+clean, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 4.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 823 B/s rd, 74 MiB/s wr, 1.18k op/s; 3352/212733 objects degraded (1.576%); 9834/212733 objects misplaced (4.623%); 34 MiB/s, 541 objects/s recovering 2024-03-21T13:09:38.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:38 smithi163 ceph-mon[40675]: osdmap e255: 8 total, 8 up, 5 in 2024-03-21T13:09:38.782 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:38.818 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:38.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:38.916 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:09:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-03-21T13:09:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-03-21T13:09:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:09:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:5937 7.0 7:0cfef427:::benchmark_data_smithi163_80312_object5936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:09:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:00:56.236353+0000", 2024-03-21T13:09:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 522.67579816299997, 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.080287529999999996, 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5937 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236353+0000", 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236353+0000", 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236354+0000", 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3859999999999999e-06 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236405+0000", 2024-03-21T13:09:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1201e-05 2024-03-21T13:09:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236424+0000", 2024-03-21T13:09:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8842e-05 2024-03-21T13:09:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236432+0000", 2024-03-21T13:09:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9990000000000008e-06 2024-03-21T13:09:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236490+0000", 2024-03-21T13:09:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7497999999999997e-05 2024-03-21T13:09:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236613+0000", 2024-03-21T13:09:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012285099999999999 2024-03-21T13:09:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:09:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.236761+0000", 2024-03-21T13:09:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014846799999999999 2024-03-21T13:09:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.237277+0000", 2024-03-21T13:09:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000516195 2024-03-21T13:09:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.238004+0000", 2024-03-21T13:09:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00072655799999999998 2024-03-21T13:09:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.316548+0000", 2024-03-21T13:09:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.078544055000000002 2024-03-21T13:09:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.316590+0000", 2024-03-21T13:09:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2398000000000003e-05 2024-03-21T13:09:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.316640+0000", 2024-03-21T13:09:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0078999999999999e-05 2024-03-21T13:09:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:5965 7.0 7:03da52ce:::benchmark_data_smithi163_80312_object5964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:09:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:00:56.326347+0000", 2024-03-21T13:09:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 522.58580454599996, 2024-03-21T13:09:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.10961245, 2024-03-21T13:09:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5965 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326347+0000", 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326347+0000", 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326348+0000", 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.451e-06 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326386+0000", 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8075999999999999e-05 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326398+0000", 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2306999999999999e-05 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326408+0000", 2024-03-21T13:09:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4299999999999995e-06 2024-03-21T13:09:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326474+0000", 2024-03-21T13:09:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6591e-05 2024-03-21T13:09:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326610+0000", 2024-03-21T13:09:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013582700000000001 2024-03-21T13:09:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:09:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.326782+0000", 2024-03-21T13:09:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017180799999999999 2024-03-21T13:09:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.327361+0000", 2024-03-21T13:09:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00057854599999999999 2024-03-21T13:09:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.328867+0000", 2024-03-21T13:09:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001506082 2024-03-21T13:09:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.435844+0000", 2024-03-21T13:09:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.10697730799999999 2024-03-21T13:09:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.435914+0000", 2024-03-21T13:09:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9698999999999998e-05 2024-03-21T13:09:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.435959+0000", 2024-03-21T13:09:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5324999999999997e-05 2024-03-21T13:09:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:5979 7.0 7:060be99b:::benchmark_data_smithi163_80312_object5978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T13:09:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:00:56.330543+0000", 2024-03-21T13:09:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 522.58160845299994, 2024-03-21T13:09:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.10700628399999999, 2024-03-21T13:09:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:38.931 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5979 2024-03-21T13:09:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330543+0000", 2024-03-21T13:09:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330543+0000", 2024-03-21T13:09:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330544+0000", 2024-03-21T13:09:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5209999999999999e-06 2024-03-21T13:09:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330628+0000", 2024-03-21T13:09:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3535000000000006e-05 2024-03-21T13:09:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330637+0000", 2024-03-21T13:09:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7720000000000006e-06 2024-03-21T13:09:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330644+0000", 2024-03-21T13:09:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6730000000000003e-06 2024-03-21T13:09:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330706+0000", 2024-03-21T13:09:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2160000000000001e-05 2024-03-21T13:09:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330804+0000", 2024-03-21T13:09:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7522999999999994e-05 2024-03-21T13:09:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:09:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.330931+0000", 2024-03-21T13:09:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000127584 2024-03-21T13:09:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.331447+0000", 2024-03-21T13:09:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00051556400000000004 2024-03-21T13:09:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.331956+0000", 2024-03-21T13:09:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000509302 2024-03-21T13:09:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.437417+0000", 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.105460939 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.437501+0000", 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4078000000000004e-05 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:00:56.437549+0000", 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7633000000000003e-05 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:66554 7.0 7:00032620:::benchmark_data_smithi163_80312_object66553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:01:32.675052+0000", 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 486.23709951000001, 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.094370187999999994, 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 66554 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675052+0000", 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675052+0000", 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675053+0000", 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.178e-06 2024-03-21T13:09:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675115+0000", 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2138999999999996e-05 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675125+0000", 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0362e-05 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675131+0000", 2024-03-21T13:09:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2519999999999996e-06 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675182+0000", 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0618000000000002e-05 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675272+0000", 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0174999999999996e-05 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:09:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.675363+0000", 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0754999999999994e-05 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.676991+0000", 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016278849999999999 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.677307+0000", 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031613000000000002 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.769314+0000", 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.092007163000000003 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.769377+0000", 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2644000000000005e-05 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:32.769422+0000", 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4886999999999999e-05 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:94688 7.0 7:0e49193b:::benchmark_data_smithi163_80312_object94687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:01:59.258336+0000", 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 459.65381543500001, 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.099394761999999998, 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 94688 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258336+0000", 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258336+0000", 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258337+0000", 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2440000000000001e-06 2024-03-21T13:09:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258393+0000", 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6415000000000001e-05 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258405+0000", 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.135e-05 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258411+0000", 2024-03-21T13:09:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.993e-06 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258461+0000", 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9886000000000003e-05 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258555+0000", 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4494999999999996e-05 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:09:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.258650+0000", 2024-03-21T13:09:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4661999999999996e-05 2024-03-21T13:09:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.264002+0000", 2024-03-21T13:09:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053522109999999999 2024-03-21T13:09:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.264051+0000", 2024-03-21T13:09:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8980999999999997e-05 2024-03-21T13:09:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.357625+0000", 2024-03-21T13:09:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.093574288000000005 2024-03-21T13:09:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.357687+0000", 2024-03-21T13:09:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1737999999999997e-05 2024-03-21T13:09:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.357730+0000", 2024-03-21T13:09:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3498999999999998e-05 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:94691 7.0 7:0868e98a:::benchmark_data_smithi163_80312_object94690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:01:59.260462+0000", 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 459.65168896, 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.097889528000000003, 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 94691 2024-03-21T13:09:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260462+0000", 2024-03-21T13:09:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260462+0000", 2024-03-21T13:09:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260463+0000", 2024-03-21T13:09:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6899999999999996e-07 2024-03-21T13:09:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260528+0000", 2024-03-21T13:09:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5376000000000006e-05 2024-03-21T13:09:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260538+0000", 2024-03-21T13:09:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6600000000000007e-06 2024-03-21T13:09:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260555+0000", 2024-03-21T13:09:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7503000000000002e-05 2024-03-21T13:09:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.260943+0000", 2024-03-21T13:09:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038779500000000002 2024-03-21T13:09:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.261055+0000", 2024-03-21T13:09:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112186 2024-03-21T13:09:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:09:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.261202+0000", 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014699900000000001 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.264043+0000", 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028405769999999999 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.264118+0000", 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4924000000000003e-05 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.358274+0000", 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.09415635 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.358311+0000", 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7169000000000003e-05 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.358352+0000", 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0219999999999998e-05 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:95064 7.0 7:0c9f881e:::benchmark_data_smithi163_80312_object95063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:01:59.727240+0000", 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 459.18491120300001, 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.080850353999999999, 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 95064 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727240+0000", 2024-03-21T13:09:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727240+0000", 2024-03-21T13:09:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727241+0000", 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.167e-06 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727287+0000", 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5596999999999997e-05 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727298+0000", 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1144e-05 2024-03-21T13:09:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727304+0000", 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3470000000000004e-06 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727352+0000", 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8359000000000002e-05 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727438+0000", 2024-03-21T13:09:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5470000000000007e-05 2024-03-21T13:09:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-03-21T13:09:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.727545+0000", 2024-03-21T13:09:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107428 2024-03-21T13:09:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.730267+0000", 2024-03-21T13:09:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027215310000000001 2024-03-21T13:09:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.731653+0000", 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001386034 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.807949+0000", 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.076295899 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.808042+0000", 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3126000000000003e-05 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:01:59.808090+0000", 2024-03-21T13:09:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8251999999999998e-05 2024-03-21T13:09:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116056 7.b 7:df017f68:::benchmark_data_smithi163_80312_object116055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:09:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.467833+0000", 2024-03-21T13:09:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 436.44431841099998, 2024-03-21T13:09:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.085221341000000006, 2024-03-21T13:09:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116056 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467833+0000", 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467833+0000", 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467844+0000", 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1499e-05 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467906+0000", 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1583999999999999e-05 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467918+0000", 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2690999999999999e-05 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467926+0000", 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0190000000000006e-06 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.467984+0000", 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8004000000000002e-05 2024-03-21T13:09:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.468133+0000", 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014849500000000001 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,6", 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.468321+0000", 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018788600000000001 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.471683+0000", 2024-03-21T13:09:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003362659 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.471770+0000", 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6512000000000001e-05 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.475037+0000", 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032672109999999999 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.552843+0000", 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.077806288000000001 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.552926+0000", 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2328000000000002e-05 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.553054+0000", 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012816499999999999 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116068 7.0 7:09402460:::benchmark_data_smithi163_80312_object116067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.483157+0000", 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 436.42899380400002, 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.081778124999999993, 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116068 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483157+0000", 2024-03-21T13:09:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483157+0000", 2024-03-21T13:09:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483159+0000", 2024-03-21T13:09:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.407e-06 2024-03-21T13:09:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483192+0000", 2024-03-21T13:09:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3175e-05 2024-03-21T13:09:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483203+0000", 2024-03-21T13:09:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1457999999999999e-05 2024-03-21T13:09:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483211+0000", 2024-03-21T13:09:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6620000000000005e-06 2024-03-21T13:09:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483281+0000", 2024-03-21T13:09:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9734999999999996e-05 2024-03-21T13:09:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483422+0000", 2024-03-21T13:09:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014167800000000001 2024-03-21T13:09:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:09:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.483585+0000", 2024-03-21T13:09:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016310599999999999 2024-03-21T13:09:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.486348+0000", 2024-03-21T13:09:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027625190000000002 2024-03-21T13:09:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487663+0000", 2024-03-21T13:09:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001315309 2024-03-21T13:09:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.563548+0000", 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.075884518999999998 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.564765+0000", 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001216983 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.564819+0000", 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4404999999999998e-05 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.564935+0000", 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011616900000000001 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116075 7.0 7:08926736:::benchmark_data_smithi163_80312_object116074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.487709+0000", 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 436.42444234999999, 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.078736329999999993, 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116075 2024-03-21T13:09:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487709+0000", 2024-03-21T13:09:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487709+0000", 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487709+0000", 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6599999999999995e-07 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487766+0000", 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.681e-05 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487775+0000", 2024-03-21T13:09:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5820000000000002e-06 2024-03-21T13:09:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.487782+0000", 2024-03-21T13:09:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2169999999999997e-06 2024-03-21T13:09:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.490331+0000", 2024-03-21T13:09:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025494060000000002 2024-03-21T13:09:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.490445+0000", 2024-03-21T13:09:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000113776 2024-03-21T13:09:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:09:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.490581+0000", 2024-03-21T13:09:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000136224 2024-03-21T13:09:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.492567+0000", 2024-03-21T13:09:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001985503 2024-03-21T13:09:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.493189+0000", 2024-03-21T13:09:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00062200299999999999 2024-03-21T13:09:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.565262+0000", 2024-03-21T13:09:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072073034999999994 2024-03-21T13:09:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.566264+0000", 2024-03-21T13:09:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010021769999999999 2024-03-21T13:09:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.566331+0000", 2024-03-21T13:09:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6910999999999997e-05 2024-03-21T13:09:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.566445+0000", 2024-03-21T13:09:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011391999999999999 2024-03-21T13:09:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116240 7.b 7:d015c35c:::benchmark_data_smithi163_80312_object116239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:09:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.720013+0000", 2024-03-21T13:09:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 436.192138215, 2024-03-21T13:09:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.130214526, 2024-03-21T13:09:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116240 2024-03-21T13:09:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720013+0000", 2024-03-21T13:09:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720013+0000", 2024-03-21T13:09:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720014+0000", 2024-03-21T13:09:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4440000000000001e-06 2024-03-21T13:09:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720050+0000", 2024-03-21T13:09:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5490000000000001e-05 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720064+0000", 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3773e-05 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720072+0000", 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2290000000000003e-06 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720132+0000", 2024-03-21T13:09:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9831999999999999e-05 2024-03-21T13:09:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720279+0000", 2024-03-21T13:09:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014782799999999999 2024-03-21T13:09:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,6", 2024-03-21T13:09:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.720455+0000", 2024-03-21T13:09:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017585499999999999 2024-03-21T13:09:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.723242+0000", 2024-03-21T13:09:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027871419999999998 2024-03-21T13:09:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.723669+0000", 2024-03-21T13:09:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000426933 2024-03-21T13:09:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.723963+0000", 2024-03-21T13:09:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029338200000000002 2024-03-21T13:09:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.850012+0000", 2024-03-21T13:09:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.126049095 2024-03-21T13:09:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.850083+0000", 2024-03-21T13:09:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1653e-05 2024-03-21T13:09:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.850227+0000", 2024-03-21T13:09:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014386999999999999 2024-03-21T13:09:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116247 7.0 7:0a4d72c9:::benchmark_data_smithi163_80312_object116246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:09:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.726834+0000", 2024-03-21T13:09:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 436.185317435, 2024-03-21T13:09:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.130439006, 2024-03-21T13:09:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116247 2024-03-21T13:09:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726834+0000", 2024-03-21T13:09:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726834+0000", 2024-03-21T13:09:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726835+0000", 2024-03-21T13:09:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4959999999999999e-06 2024-03-21T13:09:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726865+0000", 2024-03-21T13:09:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0080999999999999e-05 2024-03-21T13:09:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726886+0000", 2024-03-21T13:09:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1182000000000001e-05 2024-03-21T13:09:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726895+0000", 2024-03-21T13:09:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4719999999999995e-06 2024-03-21T13:09:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.726971+0000", 2024-03-21T13:09:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5972999999999999e-05 2024-03-21T13:09:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.727099+0000", 2024-03-21T13:09:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000128162 2024-03-21T13:09:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:09:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.727267+0000", 2024-03-21T13:09:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016778600000000001 2024-03-21T13:09:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730152+0000", 2024-03-21T13:09:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028849560000000002 2024-03-21T13:09:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.732900+0000", 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027485600000000002 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.848690+0000", 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.11578986600000001 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.857079+0000", 2024-03-21T13:09:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0083888729999999998 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.857149+0000", 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.991e-05 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.857273+0000", 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012368899999999999 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116251 7.0 7:017df485:::benchmark_data_smithi163_80312_object116250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:22.730529+0000", 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 436.18162164699999, 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12804560300000001, 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116251 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730529+0000", 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730529+0000", 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730530+0000", 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0809999999999999e-06 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730582+0000", 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1319000000000001e-05 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730592+0000", 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0332e-05 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730598+0000", 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3289999999999999e-06 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730645+0000", 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6916000000000003e-05 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730750+0000", 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104949 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.730879+0000", 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012888 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.732868+0000", 2024-03-21T13:09:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001989237 2024-03-21T13:09:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.733022+0000", 2024-03-21T13:09:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000153108 2024-03-21T13:09:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.858130+0000", 2024-03-21T13:09:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.125108673 2024-03-21T13:09:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.858342+0000", 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000211428 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.858408+0000", 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5899e-05 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:22.858575+0000", 2024-03-21T13:09:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016745200000000001 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116535 7.b 7:db9c5de4:::benchmark_data_smithi163_80312_object116534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:23.124212+0000", 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 435.78793896500002, 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.092375593000000006, 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116535 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124212+0000", 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124212+0000", 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124213+0000", 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3700000000000001e-07 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124242+0000", 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9159e-05 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124253+0000", 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0869e-05 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.124259+0000", 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9410000000000001e-06 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.127981+0000", 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003721673 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.130206+0000", 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022252589999999998 2024-03-21T13:09:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,6", 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.130421+0000", 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000214828 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.136255+0000", 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058337229999999999 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.137633+0000", 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013781570000000001 2024-03-21T13:09:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.138730+0000", 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010970929999999999 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.216445+0000", 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.077715203999999996 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.216500+0000", 2024-03-21T13:09:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4321999999999999e-05 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.216588+0000", 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7928000000000002e-05 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:116566 7.b 7:ddb81152:::benchmark_data_smithi163_80312_object116565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:23.152336+0000", 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 435.75981471599999, 2024-03-21T13:09:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.089229614999999998, 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116566 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152336+0000", 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152336+0000", 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152338+0000", 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.57e-06 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152379+0000", 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1118000000000002e-05 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152399+0000", 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0415e-05 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152423+0000", 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3938999999999999e-05 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.152487+0000", 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3590000000000006e-05 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.153143+0000", 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00065597699999999997 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,6", 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.153343+0000", 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020033 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.156773+0000", 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034299539999999998 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.157158+0000", 2024-03-21T13:09:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000385153 2024-03-21T13:09:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:39.023 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:39.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.157320+0000", 2024-03-21T13:09:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016143900000000001 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.241425+0000", 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.084105503999999998 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.241470+0000", 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5158999999999999e-05 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:23.241566+0000", 2024-03-21T13:09:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5466999999999999e-05 2024-03-21T13:09:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118902 7.0 7:007d3f07:::benchmark_data_smithi163_80312_object1461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-21T13:09:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.702328+0000", 2024-03-21T13:09:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 433.20982271399998, 2024-03-21T13:09:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.511295948, 2024-03-21T13:09:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:09:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118902 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702328+0000", 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702328+0000", 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702329+0000", 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1510000000000001e-06 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702334+0000", 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0300000000000004e-06 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702343+0000", 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2029999999999998e-06 2024-03-21T13:09:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702350+0000", 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.182e-06 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702410+0000", 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.05e-05 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702498+0000", 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7113000000000004e-05 2024-03-21T13:09:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2,3", 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.702603+0000", 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105567 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.705654+0000", 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030509489999999998 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.709103+0000", 2024-03-21T13:09:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003448914 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.712195+0000", 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030919419999999999 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.750528+0000", 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.038333000999999998 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:09:39.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.750541+0000", 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3454e-05 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213611+0000", 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46306928400000003 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213624+0000", 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3658e-05 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118942 7.0 7.1f908400 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.753006+0000", 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 433.15914457000002, 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46065019600000001, 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118942 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753006+0000", 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753006+0000", 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753008+0000", 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3999999999999999e-06 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753011+0000", 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5899999999999999e-06 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753020+0000", 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.242e-06 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.753027+0000", 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3220000000000004e-06 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.754026+0000", 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00099910600000000004 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.754036+0000", 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7110000000000007e-06 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213649+0000", 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45961280399999999 2024-03-21T13:09:39.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213657+0000", 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0209999999999999e-06 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118952 7.0 7.3905ab10 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.767903+0000", 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 433.14424769599998, 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44580361899999998, 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:39.037 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:09:39.038 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:39.038 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:39.038 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:39.038 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118952 2024-03-21T13:09:39.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.038 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:39.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:39.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767903+0000", 2024-03-21T13:09:39.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767903+0000", 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767905+0000", 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5370000000000001e-06 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767914+0000", 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0880000000000001e-06 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767925+0000", 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1231e-05 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767934+0000", 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4889999999999995e-06 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.768019+0000", 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4860999999999999e-05 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.768029+0000", 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0558e-05 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213698+0000", 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44566897700000002 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213707+0000", 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8780000000000004e-06 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.041 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118953 7.0 7.83d475e0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.767956+0000", 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 433.14419534199999, 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44578293800000002, 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118953 2024-03-21T13:09:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767956+0000", 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767956+0000", 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:39.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767956+0000", 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7199999999999999e-07 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767959+0000", 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9289999999999998e-06 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767966+0000", 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3280000000000002e-06 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.767971+0000", 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6180000000000003e-06 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.768055+0000", 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3486999999999995e-05 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.768062+0000", 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9040000000000003e-06 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213730+0000", 2024-03-21T13:09:39.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44566886999999999 2024-03-21T13:09:39.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:39.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.213739+0000", 2024-03-21T13:09:39.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2300000000000008e-06 2024-03-21T13:09:39.046 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.046 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:39.046 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.046 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14820.0:118970 7.b 7.e8b90beb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:02:25.796717+0000", 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 433.11543408199998, 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42020922900000002, 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14820", 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 118970 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:39.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796717+0000", 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796717+0000", 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796719+0000", 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0040000000000002e-06 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:39.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796728+0000", 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9570000000000008e-06 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796748+0000", 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9741e-05 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796758+0000", 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9569999999999997e-06 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796848+0000", 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0754000000000006e-05 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:25.796860+0000", 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1265e-05 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.216916+0000", 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42005610100000002 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:39.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:09:39.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:02:26.216926+0000", 2024-03-21T13:09:39.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.045e-05 2024-03-21T13:09:39.051 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.051 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:39.051 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.051 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:39.051 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:39.051 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:09:39.124 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:39.159 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:39.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:39.260 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:39.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:39.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:39.402 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:39.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:39.439 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:39.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:39.528 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 46 16 74983 74967 101.84 76.5625 0.00431931 0.00981419 2024-03-21T13:09:39.540 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:39.554 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-03-21T13:09:30.815754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4155054504,"num_objects":63590,"num_object_clones":0,"num_object_copies":190770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3352,"num_objects_misplaced":13138,"num_objects_unfound":0,"num_objects_dirty":63590,"num_whiteouts":0,"num_read":1989,"num_read_kb":1835,"num_write":64293,"num_write_kb":4059549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3145,"num_bytes_recovered":192093625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88714,"ondisk_log_size":88714,"up":474,"acting":470,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":485,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":15199892,"kb_used_data":12589484,"kb_used_omap":7,"kb_used_meta":2610360,"kb_avail":453525868,"statfs":{"total":479975178240,"available":464410488832,"internally_reserved":0,"allocated":12891631616,"data_stored":12889013502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7967,"internal_metadata":2673008865},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":927956992,"num_objects":14160,"num_object_clones":0,"num_object_copies":42480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3352,"num_objects_misplaced":13138,"num_objects_unfound":0,"num_objects_dirty":14160,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":14160,"num_write_kb":906208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":190185472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002771"},"pg_stats":[{"pgid":"9.15","version":"251'3730","reported_seq":3814,"reported_epoch":251,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-21T13:09:29.784955+0000","last_change":"2024-03-21T13:09:21.479086+0000","last_active":"2024-03-21T13:09:29.784955+0000","last_peered":"2024-03-21T13:09:29.784955+0000","last_clean":"2024-03-21T13:09:19.415800+0000","last_became_active":"2024-03-21T13:09:21.396050+0000","last_became_peered":"2024-03-21T13:09:21.396050+0000","last_unstale":"2024-03-21T13:09:29.784955+0000","last_undegraded":"2024-03-21T13:09:29.784955+0000","last_fullsized":"2024-03-21T13:09:29.784955+0000","mapping_epoch":249,"log_start":"238'100","ondisk_log_start":"238'100","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3630,"log_dups_size":0,"ondisk_log_size":3630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135102464,"num_objects":2062,"num_object_clones":0,"num_object_copies":6186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2046,"num_objects_unfound":0,"num_objects_dirty":2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2062,"num_write_kb":131936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,0],"avail_no_missing":["7","0","3"],"object_location_counts":[{"shards":"0,3,7","objects":2062}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":261,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.605148+0000","last_change":"2024-03-21T13:09:11.718192+0000","last_active":"2024-03-21T13:09:26.605148+0000","last_peered":"2024-03-21T13:09:26.605148+0000","last_clean":"2024-03-21T13:09:26.605148+0000","last_became_active":"2024-03-21T13:06:45.404217+0000","last_became_peered":"2024-03-21T13:06:45.404217+0000","last_unstale":"2024-03-21T13:09:26.605148+0000","last_undegraded":"2024-03-21T13:09:26.605148+0000","last_fullsized":"2024-03-21T13:09:26.605148+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:11.717992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:09:11.717992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:21.823934+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":298,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.600689+0000","last_change":"2024-03-21T13:08:20.891722+0000","last_active":"2024-03-21T13:09:26.600689+0000","last_peered":"2024-03-21T13:09:26.600689+0000","last_clean":"2024-03-21T13:09:26.600689+0000","last_became_active":"2024-03-21T13:08:20.891334+0000","last_became_peered":"2024-03-21T13:08:20.891334+0000","last_unstale":"2024-03-21T13:09:26.600689+0000","last_undegraded":"2024-03-21T13:09:26.600689+0000","last_fullsized":"2024-03-21T13:09:26.600689+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:54.360939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:07:54.360939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":127,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.604692+0000","last_change":"2024-03-21T13:06:29.484964+0000","last_active":"2024-03-21T13:09:26.604692+0000","last_peered":"2024-03-21T13:09:26.604692+0000","last_clean":"2024-03-21T13:09:26.604692+0000","last_became_active":"2024-03-21T13:06:29.484528+0000","last_became_peered":"2024-03-21T13:06:29.484528+0000","last_unstale":"2024-03-21T13:09:26.604692+0000","last_undegraded":"2024-03-21T13:09:26.604692+0000","last_fullsized":"2024-03-21T13:09:26.604692+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:49.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.260693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"234'19","reported_seq":477,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.597827+0000","last_change":"2024-03-21T13:08:00.054633+0000","last_active":"2024-03-21T13:09:26.597827+0000","last_peered":"2024-03-21T13:09:26.597827+0000","last_clean":"2024-03-21T13:09:26.597827+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:09:26.597827+0000","last_undegraded":"2024-03-21T13:09:26.597827+0000","last_fullsized":"2024-03-21T13:09:26.597827+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-03-21T13:08:00.054482+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:08:00.054482+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"250'3458","reported_seq":3543,"reported_epoch":250,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-21T13:09:25.795950+0000","last_change":"2024-03-21T13:09:21.484383+0000","last_active":"2024-03-21T13:09:25.795950+0000","last_peered":"2024-03-21T13:09:25.795950+0000","last_clean":"2024-03-21T13:09:19.430035+0000","last_became_active":"2024-03-21T13:09:21.389000+0000","last_became_peered":"2024-03-21T13:09:21.389000+0000","last_unstale":"2024-03-21T13:09:25.795950+0000","last_undegraded":"2024-03-21T13:09:25.795950+0000","last_fullsized":"2024-03-21T13:09:25.795950+0000","mapping_epoch":249,"log_start":"238'200","ondisk_log_start":"238'200","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3258,"log_dups_size":0,"ondisk_log_size":3258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121470976,"num_objects":1854,"num_object_clones":0,"num_object_copies":5562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1841,"num_objects_unfound":0,"num_objects_dirty":1854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1854,"num_write_kb":118624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,0],"avail_no_missing":["3","0","4"],"object_location_counts":[{"shards":"0,3,4","objects":1854}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":390,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.332418+0000","last_change":"2024-03-21T13:05:07.334438+0000","last_active":"2024-03-21T13:09:21.332418+0000","last_peered":"2024-03-21T13:09:21.332418+0000","last_clean":"2024-03-21T13:09:21.332418+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:09:21.332418+0000","last_undegraded":"2024-03-21T13:09:21.332418+0000","last_fullsized":"2024-03-21T13:09:21.332418+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:07.330296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:05:07.330296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678579+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":337,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.602002+0000","last_change":"2024-03-21T13:06:29.490705+0000","last_active":"2024-03-21T13:09:26.602002+0000","last_peered":"2024-03-21T13:09:26.602002+0000","last_clean":"2024-03-21T13:09:26.602002+0000","last_became_active":"2024-03-21T13:06:29.490004+0000","last_became_peered":"2024-03-21T13:06:29.490004+0000","last_unstale":"2024-03-21T13:09:26.602002+0000","last_undegraded":"2024-03-21T13:09:26.602002+0000","last_fullsized":"2024-03-21T13:09:26.602002+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:50.991735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.991735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":420,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.331609+0000","last_change":"2024-03-21T13:09:05.740763+0000","last_active":"2024-03-21T13:09:21.331609+0000","last_peered":"2024-03-21T13:09:21.331609+0000","last_clean":"2024-03-21T13:09:21.331609+0000","last_became_active":"2024-03-21T13:06:29.511102+0000","last_became_peered":"2024-03-21T13:06:29.511102+0000","last_unstale":"2024-03-21T13:09:21.331609+0000","last_undegraded":"2024-03-21T13:09:21.331609+0000","last_fullsized":"2024-03-21T13:09:21.331609+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:09:05.740634+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_clean_scrub_stamp":"2024-03-21T13:09:05.740634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:25.969057+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":463,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.330857+0000","last_change":"2024-03-21T13:06:29.546224+0000","last_active":"2024-03-21T13:09:21.330857+0000","last_peered":"2024-03-21T13:09:21.330857+0000","last_clean":"2024-03-21T13:09:21.330857+0000","last_became_active":"2024-03-21T13:06:29.509791+0000","last_became_peered":"2024-03-21T13:06:29.509791+0000","last_unstale":"2024-03-21T13:09:21.330857+0000","last_undegraded":"2024-03-21T13:09:21.330857+0000","last_fullsized":"2024-03-21T13:09:21.330857+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:26.476043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.476043+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678734+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"251'3605","reported_seq":3656,"reported_epoch":251,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-21T13:09:29.813353+0000","last_change":"2024-03-21T13:09:21.474051+0000","last_active":"2024-03-21T13:09:29.813353+0000","last_peered":"2024-03-21T13:09:29.813353+0000","last_clean":"2024-03-21T13:09:19.415570+0000","last_became_active":"2024-03-21T13:09:21.466895+0000","last_became_peered":"2024-03-21T13:09:21.466895+0000","last_unstale":"2024-03-21T13:09:29.813353+0000","last_undegraded":"2024-03-21T13:09:21.410849+0000","last_fullsized":"2024-03-21T13:09:21.332328+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3605,"log_dups_size":0,"ondisk_log_size":3605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130252800,"num_objects":1988,"num_object_clones":0,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1988,"num_write_kb":127200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0],"avail_no_missing":["7","3","6"],"object_location_counts":[{"shards":"0,7","objects":312},{"shards":"3,6,7","objects":1676}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":305,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.598205+0000","last_change":"2024-03-21T13:08:57.628140+0000","last_active":"2024-03-21T13:09:26.598205+0000","last_peered":"2024-03-21T13:09:26.598205+0000","last_clean":"2024-03-21T13:09:26.598205+0000","last_became_active":"2024-03-21T13:06:29.487372+0000","last_became_peered":"2024-03-21T13:06:29.487372+0000","last_unstale":"2024-03-21T13:09:26.598205+0000","last_undegraded":"2024-03-21T13:09:26.598205+0000","last_fullsized":"2024-03-21T13:09:26.598205+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:57.627967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.627967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:07.465261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":300,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.592782+0000","last_change":"2024-03-21T13:08:08.963344+0000","last_active":"2024-03-21T13:09:26.592782+0000","last_peered":"2024-03-21T13:09:26.592782+0000","last_clean":"2024-03-21T13:09:26.592782+0000","last_became_active":"2024-03-21T13:02:26.236707+0000","last_became_peered":"2024-03-21T13:02:26.236707+0000","last_unstale":"2024-03-21T13:09:26.592782+0000","last_undegraded":"2024-03-21T13:09:26.592782+0000","last_fullsized":"2024-03-21T13:09:26.592782+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:08.963164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:08:08.963164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:30.848955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":276,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.605261+0000","last_change":"2024-03-21T13:05:45.989199+0000","last_active":"2024-03-21T13:09:26.605261+0000","last_peered":"2024-03-21T13:09:26.605261+0000","last_clean":"2024-03-21T13:09:26.605261+0000","last_became_active":"2024-03-21T13:02:26.245402+0000","last_became_peered":"2024-03-21T13:02:26.245402+0000","last_unstale":"2024-03-21T13:09:26.605261+0000","last_undegraded":"2024-03-21T13:09:26.605261+0000","last_fullsized":"2024-03-21T13:09:26.605261+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:45.989114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:45.989114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":401,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.331266+0000","last_change":"2024-03-21T13:05:51.580203+0000","last_active":"2024-03-21T13:09:21.331266+0000","last_peered":"2024-03-21T13:09:21.331266+0000","last_clean":"2024-03-21T13:09:21.331266+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:09:21.331266+0000","last_undegraded":"2024-03-21T13:09:21.331266+0000","last_fullsized":"2024-03-21T13:09:21.331266+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-21T13:05:51.580074+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:05:51.580074+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678920+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"251'3615","reported_seq":5175,"reported_epoch":251,"state":"active+recovering+remapped","last_fresh":"2024-03-21T13:09:30.078636+0000","last_change":"2024-03-21T13:09:21.482380+0000","last_active":"2024-03-21T13:09:30.078636+0000","last_peered":"2024-03-21T13:09:30.078636+0000","last_clean":"2024-03-21T13:09:19.415088+0000","last_became_active":"2024-03-21T13:09:21.467792+0000","last_became_peered":"2024-03-21T13:09:21.467792+0000","last_unstale":"2024-03-21T13:09:30.078636+0000","last_undegraded":"2024-03-21T13:09:30.078636+0000","last_fullsized":"2024-03-21T13:09:30.078636+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3615,"log_dups_size":0,"ondisk_log_size":3615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132743168,"num_objects":2026,"num_object_clones":0,"num_object_copies":6078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":927,"num_objects_unfound":0,"num_objects_dirty":2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":129632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":753,"num_bytes_recovered":49348608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[0,6,7],"avail_no_missing":["0","6","7"],"object_location_counts":[{"shards":"0,6,7","objects":1099}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":400,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.329832+0000","last_change":"2024-03-21T13:08:17.747461+0000","last_active":"2024-03-21T13:09:21.329832+0000","last_peered":"2024-03-21T13:09:21.329832+0000","last_clean":"2024-03-21T13:09:21.329832+0000","last_became_active":"2024-03-21T13:06:29.505469+0000","last_became_peered":"2024-03-21T13:06:29.505469+0000","last_unstale":"2024-03-21T13:09:21.329832+0000","last_undegraded":"2024-03-21T13:09:21.329832+0000","last_fullsized":"2024-03-21T13:09:21.329832+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:17.747360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:17.747360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:40.450392+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":416,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.605367+0000","last_change":"2024-03-21T13:08:09.443525+0000","last_active":"2024-03-21T13:09:26.605367+0000","last_peered":"2024-03-21T13:09:26.605367+0000","last_clean":"2024-03-21T13:09:26.605367+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:09:26.605367+0000","last_undegraded":"2024-03-21T13:09:26.605367+0000","last_fullsized":"2024-03-21T13:09:26.605367+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:09.443411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:09.443411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":424,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.596839+0000","last_change":"2024-03-21T13:08:17.962862+0000","last_active":"2024-03-21T13:09:26.596839+0000","last_peered":"2024-03-21T13:09:26.596839+0000","last_clean":"2024-03-21T13:09:26.596839+0000","last_became_active":"2024-03-21T13:02:26.230057+0000","last_became_peered":"2024-03-21T13:02:26.230057+0000","last_unstale":"2024-03-21T13:09:26.596839+0000","last_undegraded":"2024-03-21T13:09:26.596839+0000","last_fullsized":"2024-03-21T13:09:26.596839+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:17.962759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:17.962759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:26.948935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"234'9","reported_seq":409,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.331970+0000","last_change":"2024-03-21T13:06:29.517935+0000","last_active":"2024-03-21T13:09:21.331970+0000","last_peered":"2024-03-21T13:09:21.331970+0000","last_clean":"2024-03-21T13:09:21.331970+0000","last_became_active":"2024-03-21T13:06:29.506772+0000","last_became_peered":"2024-03-21T13:06:29.506772+0000","last_unstale":"2024-03-21T13:09:21.331970+0000","last_undegraded":"2024-03-21T13:09:21.331970+0000","last_fullsized":"2024-03-21T13:09:21.331970+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-21T13:05:16.384676+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:05:16.384676+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"251'3764","reported_seq":3821,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:29.826650+0000","last_change":"2024-03-21T13:09:20.447059+0000","last_active":"2024-03-21T13:09:29.826650+0000","last_peered":"2024-03-21T13:09:29.826650+0000","last_clean":"2024-03-21T13:09:29.826650+0000","last_became_active":"2024-03-21T13:09:20.445112+0000","last_became_peered":"2024-03-21T13:09:20.445112+0000","last_unstale":"2024-03-21T13:09:29.826650+0000","last_undegraded":"2024-03-21T13:09:29.826650+0000","last_fullsized":"2024-03-21T13:09:29.826650+0000","mapping_epoch":248,"log_start":"239'300","ondisk_log_start":"239'300","created":246,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3464,"log_dups_size":0,"ondisk_log_size":3464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:31.633948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136609792,"num_objects":2085,"num_object_clones":0,"num_object_copies":6255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2085,"num_write_kb":133408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":423,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.595598+0000","last_change":"2024-03-21T13:07:47.084827+0000","last_active":"2024-03-21T13:09:26.595598+0000","last_peered":"2024-03-21T13:09:26.595598+0000","last_clean":"2024-03-21T13:09:26.595598+0000","last_became_active":"2024-03-21T13:02:26.226675+0000","last_became_peered":"2024-03-21T13:02:26.226675+0000","last_unstale":"2024-03-21T13:09:26.595598+0000","last_undegraded":"2024-03-21T13:09:26.595598+0000","last_fullsized":"2024-03-21T13:09:26.595598+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:47.084713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:47.084713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":310,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.599396+0000","last_change":"2024-03-21T13:08:18.994766+0000","last_active":"2024-03-21T13:09:26.599396+0000","last_peered":"2024-03-21T13:09:26.599396+0000","last_clean":"2024-03-21T13:09:26.599396+0000","last_became_active":"2024-03-21T13:02:26.231879+0000","last_became_peered":"2024-03-21T13:02:26.231879+0000","last_unstale":"2024-03-21T13:09:26.599396+0000","last_undegraded":"2024-03-21T13:09:26.599396+0000","last_fullsized":"2024-03-21T13:09:26.599396+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:18.994687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.994687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:31.114182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":303,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.612216+0000","last_change":"2024-03-21T13:06:28.985227+0000","last_active":"2024-03-21T13:09:26.612216+0000","last_peered":"2024-03-21T13:09:26.612216+0000","last_clean":"2024-03-21T13:09:26.612216+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:09:26.612216+0000","last_undegraded":"2024-03-21T13:09:26.612216+0000","last_fullsized":"2024-03-21T13:09:26.612216+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:28.985145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:28.985145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":199,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.350879+0000","last_change":"2024-03-21T13:06:29.540373+0000","last_active":"2024-03-21T13:09:21.350879+0000","last_peered":"2024-03-21T13:09:21.350879+0000","last_clean":"2024-03-21T13:09:21.350879+0000","last_became_active":"2024-03-21T13:06:29.504235+0000","last_became_peered":"2024-03-21T13:06:29.504235+0000","last_unstale":"2024-03-21T13:09:21.350879+0000","last_undegraded":"2024-03-21T13:09:21.350879+0000","last_fullsized":"2024-03-21T13:09:21.350879+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:20.158184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.158184+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"251'3700","reported_seq":4898,"reported_epoch":251,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-21T13:09:30.013447+0000","last_change":"2024-03-21T13:09:21.452313+0000","last_active":"2024-03-21T13:09:30.013447+0000","last_peered":"2024-03-21T13:09:30.013447+0000","last_clean":"2024-03-21T13:09:19.411088+0000","last_became_active":"2024-03-21T13:09:21.427199+0000","last_became_peered":"2024-03-21T13:09:21.427199+0000","last_unstale":"2024-03-21T13:09:30.013447+0000","last_undegraded":"2024-03-21T13:09:30.013447+0000","last_fullsized":"2024-03-21T13:09:21.333198+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135856128,"num_objects":2073,"num_object_clones":0,"num_object_copies":6219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1093,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2073,"num_write_kb":132672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":37552128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[0,4],"avail_no_missing":["0","4","6"],"object_location_counts":[{"shards":"0,4","objects":980},{"shards":"0,4,6","objects":1093}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":403,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.331511+0000","last_change":"2024-03-21T13:07:44.816849+0000","last_active":"2024-03-21T13:09:21.331511+0000","last_peered":"2024-03-21T13:09:21.331511+0000","last_clean":"2024-03-21T13:09:21.331511+0000","last_became_active":"2024-03-21T13:06:29.512081+0000","last_became_peered":"2024-03-21T13:06:29.512081+0000","last_unstale":"2024-03-21T13:09:21.331511+0000","last_undegraded":"2024-03-21T13:09:21.331511+0000","last_fullsized":"2024-03-21T13:09:21.331511+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:44.816722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:44.816722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678987+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":321,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.337910+0000","last_change":"2024-03-21T13:06:29.500559+0000","last_active":"2024-03-21T13:09:21.337910+0000","last_peered":"2024-03-21T13:09:21.337910+0000","last_clean":"2024-03-21T13:09:21.337910+0000","last_became_active":"2024-03-21T13:06:29.498872+0000","last_became_peered":"2024-03-21T13:06:29.498872+0000","last_unstale":"2024-03-21T13:09:21.337910+0000","last_undegraded":"2024-03-21T13:09:21.337910+0000","last_fullsized":"2024-03-21T13:09:21.337910+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:29.464856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:04:29.464856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":440,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.594839+0000","last_change":"2024-03-21T13:08:20.956897+0000","last_active":"2024-03-21T13:09:26.594839+0000","last_peered":"2024-03-21T13:09:26.594839+0000","last_clean":"2024-03-21T13:09:26.594839+0000","last_became_active":"2024-03-21T13:06:29.487644+0000","last_became_peered":"2024-03-21T13:06:29.487644+0000","last_unstale":"2024-03-21T13:09:26.594839+0000","last_undegraded":"2024-03-21T13:09:26.594839+0000","last_fullsized":"2024-03-21T13:09:26.594839+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:20.956778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:20.956778+0000","last_clean_scrub_stamp":"2024-03-21T13:08:20.956778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:37.554104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":466,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.331509+0000","last_change":"2024-03-21T13:09:02.714077+0000","last_active":"2024-03-21T13:09:21.331509+0000","last_peered":"2024-03-21T13:09:21.331509+0000","last_clean":"2024-03-21T13:09:21.331509+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:09:21.331509+0000","last_undegraded":"2024-03-21T13:09:21.331509+0000","last_fullsized":"2024-03-21T13:09:21.331509+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-21T13:09:02.713898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:02.713898+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:07.999364+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"251'3693","reported_seq":4816,"reported_epoch":251,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-21T13:09:29.850883+0000","last_change":"2024-03-21T13:09:21.444142+0000","last_active":"2024-03-21T13:09:29.850883+0000","last_peered":"2024-03-21T13:09:29.850883+0000","last_clean":"2024-03-21T13:09:19.430239+0000","last_became_active":"2024-03-21T13:09:21.411216+0000","last_became_peered":"2024-03-21T13:09:21.411216+0000","last_unstale":"2024-03-21T13:09:29.850883+0000","last_undegraded":"2024-03-21T13:09:29.850883+0000","last_fullsized":"2024-03-21T13:09:21.331513+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3693,"log_dups_size":0,"ondisk_log_size":3693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134414336,"num_objects":2051,"num_object_clones":0,"num_object_copies":6153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1178,"num_objects_unfound":0,"num_objects_dirty":2051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2051,"num_write_kb":131264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":535,"num_bytes_recovered":35061760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6],"avail_no_missing":["7","3","6"],"object_location_counts":[{"shards":"3,6,7","objects":1178},{"shards":"6,7","objects":873}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":256,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.337535+0000","last_change":"2024-03-21T13:06:29.537271+0000","last_active":"2024-03-21T13:09:21.337535+0000","last_peered":"2024-03-21T13:09:21.337535+0000","last_clean":"2024-03-21T13:09:21.337535+0000","last_became_active":"2024-03-21T13:06:29.504570+0000","last_became_peered":"2024-03-21T13:06:29.504570+0000","last_unstale":"2024-03-21T13:09:21.337535+0000","last_undegraded":"2024-03-21T13:09:21.337535+0000","last_fullsized":"2024-03-21T13:09:21.337535+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:50.295057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.295057+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":547,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.604935+0000","last_change":"2024-03-21T13:05:38.022572+0000","last_active":"2024-03-21T13:09:26.604935+0000","last_peered":"2024-03-21T13:09:26.604935+0000","last_clean":"2024-03-21T13:09:26.604935+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:09:26.604935+0000","last_undegraded":"2024-03-21T13:09:26.604935+0000","last_fullsized":"2024-03-21T13:09:26.604935+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:05:38.022439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.022439+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":308,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.330425+0000","last_change":"2024-03-21T13:08:53.540108+0000","last_active":"2024-03-21T13:09:21.330425+0000","last_peered":"2024-03-21T13:09:21.330425+0000","last_clean":"2024-03-21T13:09:21.330425+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:09:21.330425+0000","last_undegraded":"2024-03-21T13:09:21.330425+0000","last_fullsized":"2024-03-21T13:09:21.330425+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:08:53.539909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:53.539909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:03.095158+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":352,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.603475+0000","last_change":"2024-03-21T13:05:21.117502+0000","last_active":"2024-03-21T13:09:26.603475+0000","last_peered":"2024-03-21T13:09:26.603475+0000","last_clean":"2024-03-21T13:09:26.603475+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:09:26.603475+0000","last_undegraded":"2024-03-21T13:09:26.603475+0000","last_fullsized":"2024-03-21T13:09:26.603475+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:21.117370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.117370+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"250'3538","reported_seq":5233,"reported_epoch":250,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:09:25.828840+0000","last_change":"2024-03-21T13:09:21.378881+0000","last_active":"2024-03-21T13:09:25.828840+0000","last_peered":"2024-03-21T13:09:25.828840+0000","last_clean":"2024-03-21T13:09:19.414236+0000","last_became_active":"2024-03-21T13:09:21.374014+0000","last_became_peered":"2024-03-21T13:09:21.374014+0000","last_unstale":"2024-03-21T13:09:25.828840+0000","last_undegraded":"2024-03-21T13:09:25.828840+0000","last_fullsized":"2024-03-21T13:09:25.828840+0000","mapping_epoch":249,"log_start":"239'300","ondisk_log_start":"239'300","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3238,"log_dups_size":0,"ondisk_log_size":3238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123633664,"num_objects":1887,"num_object_clones":0,"num_object_copies":5661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1025,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1887,"num_write_kb":120736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":820,"num_bytes_recovered":53739520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[3,7,6],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6,7","objects":1887}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":300,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.598981+0000","last_change":"2024-03-21T13:09:01.881269+0000","last_active":"2024-03-21T13:09:26.598981+0000","last_peered":"2024-03-21T13:09:26.598981+0000","last_clean":"2024-03-21T13:09:26.598981+0000","last_became_active":"2024-03-21T13:06:29.499497+0000","last_became_peered":"2024-03-21T13:06:29.499497+0000","last_unstale":"2024-03-21T13:09:26.598981+0000","last_undegraded":"2024-03-21T13:09:26.598981+0000","last_fullsized":"2024-03-21T13:09:26.598981+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:01.881154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:09:01.881154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:16.397551+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":309,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.603341+0000","last_change":"2024-03-21T13:06:29.501937+0000","last_active":"2024-03-21T13:09:26.603341+0000","last_peered":"2024-03-21T13:09:26.603341+0000","last_clean":"2024-03-21T13:09:26.603341+0000","last_became_active":"2024-03-21T13:06:29.501232+0000","last_became_peered":"2024-03-21T13:06:29.501232+0000","last_unstale":"2024-03-21T13:09:26.603341+0000","last_undegraded":"2024-03-21T13:09:26.603341+0000","last_fullsized":"2024-03-21T13:09:26.603341+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.088812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.088812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":414,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.329531+0000","last_change":"2024-03-21T13:06:29.501866+0000","last_active":"2024-03-21T13:09:21.329531+0000","last_peered":"2024-03-21T13:09:21.329531+0000","last_clean":"2024-03-21T13:09:21.329531+0000","last_became_active":"2024-03-21T13:06:29.501471+0000","last_became_peered":"2024-03-21T13:06:29.501471+0000","last_unstale":"2024-03-21T13:09:21.329531+0000","last_undegraded":"2024-03-21T13:09:21.329531+0000","last_fullsized":"2024-03-21T13:09:21.329531+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:09.348646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:09.348646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678528+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"234'19","reported_seq":476,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.605001+0000","last_change":"2024-03-21T13:05:49.982963+0000","last_active":"2024-03-21T13:09:26.605001+0000","last_peered":"2024-03-21T13:09:26.605001+0000","last_clean":"2024-03-21T13:09:26.605001+0000","last_became_active":"2024-03-21T13:02:26.254436+0000","last_became_peered":"2024-03-21T13:02:26.254436+0000","last_unstale":"2024-03-21T13:09:26.605001+0000","last_undegraded":"2024-03-21T13:09:26.605001+0000","last_fullsized":"2024-03-21T13:09:26.605001+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:49.982812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.982812+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":404,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.606064+0000","last_change":"2024-03-21T13:06:29.980827+0000","last_active":"2024-03-21T13:09:26.606064+0000","last_peered":"2024-03-21T13:09:26.606064+0000","last_clean":"2024-03-21T13:09:26.606064+0000","last_became_active":"2024-03-21T13:06:29.503679+0000","last_became_peered":"2024-03-21T13:06:29.503679+0000","last_unstale":"2024-03-21T13:09:26.606064+0000","last_undegraded":"2024-03-21T13:09:26.606064+0000","last_fullsized":"2024-03-21T13:09:26.606064+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:29.980750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:06:29.980750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":412,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.605161+0000","last_change":"2024-03-21T13:09:10.565138+0000","last_active":"2024-03-21T13:09:26.605161+0000","last_peered":"2024-03-21T13:09:26.605161+0000","last_clean":"2024-03-21T13:09:26.605161+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:09:26.605161+0000","last_undegraded":"2024-03-21T13:09:26.605161+0000","last_fullsized":"2024-03-21T13:09:26.605161+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:10.565030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.565030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:13.630323+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":276,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.331783+0000","last_change":"2024-03-21T13:05:28.499322+0000","last_active":"2024-03-21T13:09:21.331783+0000","last_peered":"2024-03-21T13:09:21.331783+0000","last_clean":"2024-03-21T13:09:21.331783+0000","last_became_active":"2024-03-21T13:02:26.242973+0000","last_became_peered":"2024-03-21T13:02:26.242973+0000","last_unstale":"2024-03-21T13:09:21.331783+0000","last_undegraded":"2024-03-21T13:09:21.331783+0000","last_fullsized":"2024-03-21T13:09:21.331783+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.499185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.499185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678666+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":422,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.329689+0000","last_change":"2024-03-21T13:08:18.740907+0000","last_active":"2024-03-21T13:09:21.329689+0000","last_peered":"2024-03-21T13:09:21.329689+0000","last_clean":"2024-03-21T13:09:21.329689+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:09:21.329689+0000","last_undegraded":"2024-03-21T13:09:21.329689+0000","last_fullsized":"2024-03-21T13:09:21.329689+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-21T13:08:18.740815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.740815+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:26.851401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":390,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.332674+0000","last_change":"2024-03-21T13:05:19.431337+0000","last_active":"2024-03-21T13:09:21.332674+0000","last_peered":"2024-03-21T13:09:21.332674+0000","last_clean":"2024-03-21T13:09:21.332674+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:09:21.332674+0000","last_undegraded":"2024-03-21T13:09:21.332674+0000","last_fullsized":"2024-03-21T13:09:21.332674+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:19.430947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:19.430947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678614+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":58,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.601731+0000","last_change":"2024-03-21T13:08:20.955637+0000","last_active":"2024-03-21T13:09:26.601731+0000","last_peered":"2024-03-21T13:09:26.601731+0000","last_clean":"2024-03-21T13:09:26.601731+0000","last_became_active":"2024-03-21T13:08:20.955293+0000","last_became_peered":"2024-03-21T13:08:20.955293+0000","last_unstale":"2024-03-21T13:09:26.601731+0000","last_undegraded":"2024-03-21T13:09:26.601731+0000","last_fullsized":"2024-03-21T13:09:26.601731+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:03.434682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:03.434682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":406,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.604265+0000","last_change":"2024-03-21T13:07:50.428363+0000","last_active":"2024-03-21T13:09:26.604265+0000","last_peered":"2024-03-21T13:09:26.604265+0000","last_clean":"2024-03-21T13:09:26.604265+0000","last_became_active":"2024-03-21T13:06:29.501190+0000","last_became_peered":"2024-03-21T13:06:29.501190+0000","last_unstale":"2024-03-21T13:09:26.604265+0000","last_undegraded":"2024-03-21T13:09:26.604265+0000","last_fullsized":"2024-03-21T13:09:26.604265+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:50.428251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:50.428251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":144,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.330692+0000","last_change":"2024-03-21T13:06:29.532046+0000","last_active":"2024-03-21T13:09:21.330692+0000","last_peered":"2024-03-21T13:09:21.330692+0000","last_clean":"2024-03-21T13:09:21.330692+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:09:21.330692+0000","last_undegraded":"2024-03-21T13:09:21.330692+0000","last_fullsized":"2024-03-21T13:09:21.330692+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:40.262277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:40.262277+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678854+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":308,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.601680+0000","last_change":"2024-03-21T13:08:20.069216+0000","last_active":"2024-03-21T13:09:26.601680+0000","last_peered":"2024-03-21T13:09:26.601680+0000","last_clean":"2024-03-21T13:09:26.601680+0000","last_became_active":"2024-03-21T13:02:26.252488+0000","last_became_peered":"2024-03-21T13:02:26.252488+0000","last_unstale":"2024-03-21T13:09:26.601680+0000","last_undegraded":"2024-03-21T13:09:26.601680+0000","last_fullsized":"2024-03-21T13:09:26.601680+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:20.069134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:20.069134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:34.216060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":406,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.332171+0000","last_change":"2024-03-21T13:05:31.496412+0000","last_active":"2024-03-21T13:09:21.332171+0000","last_peered":"2024-03-21T13:09:21.332171+0000","last_clean":"2024-03-21T13:09:21.332171+0000","last_became_active":"2024-03-21T13:02:26.232682+0000","last_became_peered":"2024-03-21T13:02:26.232682+0000","last_unstale":"2024-03-21T13:09:21.332171+0000","last_undegraded":"2024-03-21T13:09:21.332171+0000","last_fullsized":"2024-03-21T13:09:21.332171+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.495639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.495639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678835+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":410,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.601020+0000","last_change":"2024-03-21T13:07:48.347291+0000","last_active":"2024-03-21T13:09:26.601020+0000","last_peered":"2024-03-21T13:09:26.601020+0000","last_clean":"2024-03-21T13:09:26.601020+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:09:26.601020+0000","last_undegraded":"2024-03-21T13:09:26.601020+0000","last_fullsized":"2024-03-21T13:09:26.601020+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:48.347175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:48.347175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":393,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.602458+0000","last_change":"2024-03-21T13:06:29.518281+0000","last_active":"2024-03-21T13:09:26.602458+0000","last_peered":"2024-03-21T13:09:26.602458+0000","last_clean":"2024-03-21T13:09:26.602458+0000","last_became_active":"2024-03-21T13:06:29.496686+0000","last_became_peered":"2024-03-21T13:06:29.496686+0000","last_unstale":"2024-03-21T13:09:26.602458+0000","last_undegraded":"2024-03-21T13:09:26.602458+0000","last_fullsized":"2024-03-21T13:09:26.602458+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:05:34.036430+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.036430+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":237,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.600174+0000","last_change":"2024-03-21T13:05:29.031235+0000","last_active":"2024-03-21T13:09:26.600174+0000","last_peered":"2024-03-21T13:09:26.600174+0000","last_clean":"2024-03-21T13:09:26.600174+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:09:26.600174+0000","last_undegraded":"2024-03-21T13:09:26.600174+0000","last_fullsized":"2024-03-21T13:09:26.600174+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.031062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.031062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":276,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.331425+0000","last_change":"2024-03-21T13:08:05.697231+0000","last_active":"2024-03-21T13:09:21.331425+0000","last_peered":"2024-03-21T13:09:21.331425+0000","last_clean":"2024-03-21T13:09:21.331425+0000","last_became_active":"2024-03-21T13:02:26.242491+0000","last_became_peered":"2024-03-21T13:02:26.242491+0000","last_unstale":"2024-03-21T13:09:21.331425+0000","last_undegraded":"2024-03-21T13:09:21.331425+0000","last_fullsized":"2024-03-21T13:09:21.331425+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:05.697141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:05.697141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:28.363142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":123,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.598461+0000","last_change":"2024-03-21T13:08:18.389543+0000","last_active":"2024-03-21T13:09:26.598461+0000","last_peered":"2024-03-21T13:09:26.598461+0000","last_clean":"2024-03-21T13:09:26.598461+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:09:26.598461+0000","last_undegraded":"2024-03-21T13:09:26.598461+0000","last_fullsized":"2024-03-21T13:09:26.598461+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:18.389423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.389423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":501,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.595591+0000","last_change":"2024-03-21T13:08:16.007543+0000","last_active":"2024-03-21T13:09:26.595591+0000","last_peered":"2024-03-21T13:09:26.595591+0000","last_clean":"2024-03-21T13:09:26.595591+0000","last_became_active":"2024-03-21T13:06:29.487725+0000","last_became_peered":"2024-03-21T13:06:29.487725+0000","last_unstale":"2024-03-21T13:09:26.595591+0000","last_undegraded":"2024-03-21T13:09:26.595591+0000","last_fullsized":"2024-03-21T13:09:26.595591+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:08:16.007433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:16.007433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:41.300649+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.19","version":"250'3477","reported_seq":4119,"reported_epoch":250,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-21T13:09:25.828062+0000","last_change":"2024-03-21T13:09:21.482476+0000","last_active":"2024-03-21T13:09:25.828062+0000","last_peered":"2024-03-21T13:09:25.828062+0000","last_clean":"2024-03-21T13:09:19.411404+0000","last_became_active":"2024-03-21T13:09:21.448849+0000","last_became_peered":"2024-03-21T13:09:21.448849+0000","last_unstale":"2024-03-21T13:09:25.828062+0000","last_undegraded":"2024-03-21T13:09:25.828062+0000","last_fullsized":"2024-03-21T13:09:21.329381+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3477,"log_dups_size":0,"ondisk_log_size":3477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121733120,"num_objects":1858,"num_object_clones":0,"num_object_copies":5574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2996,"num_objects_unfound":0,"num_objects_dirty":1858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1858,"num_write_kb":118880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":12910592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,4],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3,7","objects":1498},{"shards":"3,4","objects":360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":311,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.601326+0000","last_change":"2024-03-21T13:08:56.597251+0000","last_active":"2024-03-21T13:09:26.601326+0000","last_peered":"2024-03-21T13:09:26.601326+0000","last_clean":"2024-03-21T13:09:26.601326+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:09:26.601326+0000","last_undegraded":"2024-03-21T13:09:26.601326+0000","last_fullsized":"2024-03-21T13:09:26.601326+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:56.597146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:56.597146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:02.490445+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"125'10","reported_seq":541,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.330249+0000","last_change":"2024-03-21T13:08:15.777606+0000","last_active":"2024-03-21T13:09:21.330249+0000","last_peered":"2024-03-21T13:09:21.330249+0000","last_clean":"2024-03-21T13:09:21.330249+0000","last_became_active":"2024-03-21T13:02:26.234494+0000","last_became_peered":"2024-03-21T13:02:26.234494+0000","last_unstale":"2024-03-21T13:09:21.330249+0000","last_undegraded":"2024-03-21T13:09:21.330249+0000","last_fullsized":"2024-03-21T13:09:21.330249+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:08:15.777522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:15.777522+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:35.180586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":416,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.612022+0000","last_change":"2024-03-21T13:06:29.489915+0000","last_active":"2024-03-21T13:09:26.612022+0000","last_peered":"2024-03-21T13:09:26.612022+0000","last_clean":"2024-03-21T13:09:26.612022+0000","last_became_active":"2024-03-21T13:06:29.489392+0000","last_became_peered":"2024-03-21T13:06:29.489392+0000","last_unstale":"2024-03-21T13:09:26.612022+0000","last_undegraded":"2024-03-21T13:09:26.612022+0000","last_fullsized":"2024-03-21T13:09:26.612022+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.042372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.042372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'20","reported_seq":464,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.350937+0000","last_change":"2024-03-21T13:06:29.530165+0000","last_active":"2024-03-21T13:09:21.350937+0000","last_peered":"2024-03-21T13:09:21.350937+0000","last_clean":"2024-03-21T13:09:21.350937+0000","last_became_active":"2024-03-21T13:06:29.506163+0000","last_became_peered":"2024-03-21T13:06:29.506163+0000","last_unstale":"2024-03-21T13:09:21.350937+0000","last_undegraded":"2024-03-21T13:09:21.350937+0000","last_fullsized":"2024-03-21T13:09:21.350937+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:23.379004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.379004+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678752+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"251'3652","reported_seq":3708,"reported_epoch":251,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-21T13:09:29.978412+0000","last_change":"2024-03-21T13:09:21.396302+0000","last_active":"2024-03-21T13:09:29.978412+0000","last_peered":"2024-03-21T13:09:29.978412+0000","last_clean":"2024-03-21T13:09:19.420348+0000","last_became_active":"2024-03-21T13:09:21.388409+0000","last_became_peered":"2024-03-21T13:09:21.388409+0000","last_unstale":"2024-03-21T13:09:29.978412+0000","last_undegraded":"2024-03-21T13:09:29.978412+0000","last_fullsized":"2024-03-21T13:09:29.978412+0000","mapping_epoch":249,"log_start":"238'200","ondisk_log_start":"238'200","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3452,"log_dups_size":0,"ondisk_log_size":3452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133169152,"num_objects":2032,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2032,"num_objects_unfound":0,"num_objects_dirty":2032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2032,"num_write_kb":130048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":2032}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":407,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.600104+0000","last_change":"2024-03-21T13:06:45.402756+0000","last_active":"2024-03-21T13:09:26.600104+0000","last_peered":"2024-03-21T13:09:26.600104+0000","last_clean":"2024-03-21T13:09:26.600104+0000","last_became_active":"2024-03-21T13:06:45.402482+0000","last_became_peered":"2024-03-21T13:06:45.402482+0000","last_unstale":"2024-03-21T13:09:26.600104+0000","last_undegraded":"2024-03-21T13:09:26.600104+0000","last_fullsized":"2024-03-21T13:09:26.600104+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:21.157295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.157295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":543,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.351069+0000","last_change":"2024-03-21T13:07:52.673401+0000","last_active":"2024-03-21T13:09:21.351069+0000","last_peered":"2024-03-21T13:09:21.351069+0000","last_clean":"2024-03-21T13:09:21.351069+0000","last_became_active":"2024-03-21T13:06:29.507429+0000","last_became_peered":"2024-03-21T13:06:29.507429+0000","last_unstale":"2024-03-21T13:09:21.351069+0000","last_undegraded":"2024-03-21T13:09:21.351069+0000","last_fullsized":"2024-03-21T13:09:21.351069+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-03-21T13:07:52.673289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:07:52.673289+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":311,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.605232+0000","last_change":"2024-03-21T13:08:21.365260+0000","last_active":"2024-03-21T13:09:26.605232+0000","last_peered":"2024-03-21T13:09:26.605232+0000","last_clean":"2024-03-21T13:09:26.605232+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:09:26.605232+0000","last_undegraded":"2024-03-21T13:09:26.605232+0000","last_fullsized":"2024-03-21T13:09:26.605232+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:08:21.365173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"234'15","reported_seq":338,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.601341+0000","last_change":"2024-03-21T13:08:08.413361+0000","last_active":"2024-03-21T13:09:26.601341+0000","last_peered":"2024-03-21T13:09:26.601341+0000","last_clean":"2024-03-21T13:09:26.601341+0000","last_became_active":"2024-03-21T13:02:26.239161+0000","last_became_peered":"2024-03-21T13:02:26.239161+0000","last_unstale":"2024-03-21T13:09:26.601341+0000","last_undegraded":"2024-03-21T13:09:26.601341+0000","last_fullsized":"2024-03-21T13:09:26.601341+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:08:08.413234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:08.413234+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:37.899511+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":131,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.597223+0000","last_change":"2024-03-21T13:08:58.785566+0000","last_active":"2024-03-21T13:09:26.597223+0000","last_peered":"2024-03-21T13:09:26.597223+0000","last_clean":"2024-03-21T13:09:26.597223+0000","last_became_active":"2024-03-21T13:06:29.488783+0000","last_became_peered":"2024-03-21T13:06:29.488783+0000","last_unstale":"2024-03-21T13:09:26.597223+0000","last_undegraded":"2024-03-21T13:09:26.597223+0000","last_fullsized":"2024-03-21T13:09:26.597223+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:58.785444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:08:58.785444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:14.325341+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":400,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.332592+0000","last_change":"2024-03-21T13:05:17.347764+0000","last_active":"2024-03-21T13:09:21.332592+0000","last_peered":"2024-03-21T13:09:21.332592+0000","last_clean":"2024-03-21T13:09:21.332592+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:09:21.332592+0000","last_undegraded":"2024-03-21T13:09:21.332592+0000","last_fullsized":"2024-03-21T13:09:21.332592+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:17.347604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:17.347604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678563+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":424,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.600486+0000","last_change":"2024-03-21T13:09:13.694824+0000","last_active":"2024-03-21T13:09:26.600486+0000","last_peered":"2024-03-21T13:09:26.600486+0000","last_clean":"2024-03-21T13:09:26.600486+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:09:26.600486+0000","last_undegraded":"2024-03-21T13:09:26.600486+0000","last_fullsized":"2024-03-21T13:09:26.600486+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:09:13.694702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:13.694702+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:33.103790+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'15","reported_seq":345,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.331057+0000","last_change":"2024-03-21T13:09:10.698886+0000","last_active":"2024-03-21T13:09:21.331057+0000","last_peered":"2024-03-21T13:09:21.331057+0000","last_clean":"2024-03-21T13:09:21.331057+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:09:21.331057+0000","last_undegraded":"2024-03-21T13:09:21.331057+0000","last_fullsized":"2024-03-21T13:09:21.331057+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:09:10.698706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.698706+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:33.637181+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":398,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.600933+0000","last_change":"2024-03-21T13:07:52.425295+0000","last_active":"2024-03-21T13:09:26.600933+0000","last_peered":"2024-03-21T13:09:26.600933+0000","last_clean":"2024-03-21T13:09:26.600933+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:09:26.600933+0000","last_undegraded":"2024-03-21T13:09:26.600933+0000","last_fullsized":"2024-03-21T13:09:26.600933+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:52.424928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:52.424928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":390,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.331784+0000","last_change":"2024-03-21T13:05:25.466812+0000","last_active":"2024-03-21T13:09:21.331784+0000","last_peered":"2024-03-21T13:09:21.331784+0000","last_clean":"2024-03-21T13:09:21.331784+0000","last_became_active":"2024-03-21T13:02:26.242502+0000","last_became_peered":"2024-03-21T13:02:26.242502+0000","last_unstale":"2024-03-21T13:09:21.331784+0000","last_undegraded":"2024-03-21T13:09:21.331784+0000","last_fullsized":"2024-03-21T13:09:21.331784+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:25.466695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:25.466695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":418,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.601821+0000","last_change":"2024-03-21T13:05:41.004652+0000","last_active":"2024-03-21T13:09:26.601821+0000","last_peered":"2024-03-21T13:09:26.601821+0000","last_clean":"2024-03-21T13:09:26.601821+0000","last_became_active":"2024-03-21T13:02:26.252406+0000","last_became_peered":"2024-03-21T13:02:26.252406+0000","last_unstale":"2024-03-21T13:09:26.601821+0000","last_undegraded":"2024-03-21T13:09:26.601821+0000","last_fullsized":"2024-03-21T13:09:26.601821+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.004412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.004412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":452,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.332410+0000","last_change":"2024-03-21T13:09:12.713079+0000","last_active":"2024-03-21T13:09:21.332410+0000","last_peered":"2024-03-21T13:09:21.332410+0000","last_clean":"2024-03-21T13:09:21.332410+0000","last_became_active":"2024-03-21T13:06:29.513331+0000","last_became_peered":"2024-03-21T13:06:29.513331+0000","last_unstale":"2024-03-21T13:09:21.332410+0000","last_undegraded":"2024-03-21T13:09:21.332410+0000","last_fullsized":"2024-03-21T13:09:21.332410+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-21T13:09:12.712840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:12.712840+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:25.402649+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"251'3737","reported_seq":3790,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:29.840615+0000","last_change":"2024-03-21T13:09:18.454946+0000","last_active":"2024-03-21T13:09:29.840615+0000","last_peered":"2024-03-21T13:09:29.840615+0000","last_clean":"2024-03-21T13:09:29.840615+0000","last_became_active":"2024-03-21T13:09:18.453217+0000","last_became_peered":"2024-03-21T13:09:18.453217+0000","last_unstale":"2024-03-21T13:09:29.840615+0000","last_undegraded":"2024-03-21T13:09:29.840615+0000","last_fullsized":"2024-03-21T13:09:29.840615+0000","mapping_epoch":234,"log_start":"238'100","ondisk_log_start":"238'100","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3637,"log_dups_size":0,"ondisk_log_size":3637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:30.621369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135561216,"num_objects":2068,"num_object_clones":0,"num_object_copies":6204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2068,"num_write_kb":132384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":283,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.601856+0000","last_change":"2024-03-21T13:06:29.509970+0000","last_active":"2024-03-21T13:09:26.601856+0000","last_peered":"2024-03-21T13:09:26.601856+0000","last_clean":"2024-03-21T13:09:26.601856+0000","last_became_active":"2024-03-21T13:06:29.509616+0000","last_became_peered":"2024-03-21T13:06:29.509616+0000","last_unstale":"2024-03-21T13:09:26.601856+0000","last_undegraded":"2024-03-21T13:09:26.601856+0000","last_fullsized":"2024-03-21T13:09:26.601856+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.973796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.973796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":446,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.600537+0000","last_change":"2024-03-21T13:06:29.498660+0000","last_active":"2024-03-21T13:09:26.600537+0000","last_peered":"2024-03-21T13:09:26.600537+0000","last_clean":"2024-03-21T13:09:26.600537+0000","last_became_active":"2024-03-21T13:06:29.487439+0000","last_became_peered":"2024-03-21T13:06:29.487439+0000","last_unstale":"2024-03-21T13:09:26.600537+0000","last_undegraded":"2024-03-21T13:09:26.600537+0000","last_fullsized":"2024-03-21T13:09:26.600537+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:05:44.816353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:44.816353+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":363,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.604830+0000","last_change":"2024-03-21T13:08:10.436112+0000","last_active":"2024-03-21T13:09:26.604830+0000","last_peered":"2024-03-21T13:09:26.604830+0000","last_clean":"2024-03-21T13:09:26.604830+0000","last_became_active":"2024-03-21T13:02:26.237800+0000","last_became_peered":"2024-03-21T13:02:26.237800+0000","last_unstale":"2024-03-21T13:09:26.604830+0000","last_undegraded":"2024-03-21T13:09:26.604830+0000","last_fullsized":"2024-03-21T13:09:26.604830+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:08:10.435742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:10.435742+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:37.629108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":545,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.600813+0000","last_change":"2024-03-21T13:08:53.619838+0000","last_active":"2024-03-21T13:09:26.600813+0000","last_peered":"2024-03-21T13:09:26.600813+0000","last_clean":"2024-03-21T13:09:26.600813+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:09:26.600813+0000","last_undegraded":"2024-03-21T13:09:26.600813+0000","last_fullsized":"2024-03-21T13:09:26.600813+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:08:53.619709+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:08:53.619709+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:57.309212+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"250'3474","reported_seq":3525,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:25.782857+0000","last_change":"2024-03-21T13:09:18.436184+0000","last_active":"2024-03-21T13:09:25.782857+0000","last_peered":"2024-03-21T13:09:25.782857+0000","last_clean":"2024-03-21T13:09:25.782857+0000","last_became_active":"2024-03-21T13:09:18.434636+0000","last_became_peered":"2024-03-21T13:09:18.434636+0000","last_unstale":"2024-03-21T13:09:25.782857+0000","last_undegraded":"2024-03-21T13:09:25.782857+0000","last_fullsized":"2024-03-21T13:09:25.782857+0000","mapping_epoch":234,"log_start":"238'200","ondisk_log_start":"238'200","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3274,"log_dups_size":0,"ondisk_log_size":3274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:29.812002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122519552,"num_objects":1869,"num_object_clones":0,"num_object_copies":5607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1869,"num_write_kb":119648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":396,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.332575+0000","last_change":"2024-03-21T13:05:43.596914+0000","last_active":"2024-03-21T13:09:21.332575+0000","last_peered":"2024-03-21T13:09:21.332575+0000","last_clean":"2024-03-21T13:09:21.332575+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:09:21.332575+0000","last_undegraded":"2024-03-21T13:09:21.332575+0000","last_fullsized":"2024-03-21T13:09:21.332575+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.596691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678935+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":424,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.603244+0000","last_change":"2024-03-21T13:09:14.724844+0000","last_active":"2024-03-21T13:09:26.603244+0000","last_peered":"2024-03-21T13:09:26.603244+0000","last_clean":"2024-03-21T13:09:26.603244+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:09:26.603244+0000","last_undegraded":"2024-03-21T13:09:26.603244+0000","last_fullsized":"2024-03-21T13:09:26.603244+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:14.724681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:14.724681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:19.151411+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"234'18","reported_seq":350,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.330589+0000","last_change":"2024-03-21T13:07:48.778302+0000","last_active":"2024-03-21T13:09:21.330589+0000","last_peered":"2024-03-21T13:09:21.330589+0000","last_clean":"2024-03-21T13:09:21.330589+0000","last_became_active":"2024-03-21T13:02:14.347102+0000","last_became_peered":"2024-03-21T13:02:14.347102+0000","last_unstale":"2024-03-21T13:09:21.330589+0000","last_undegraded":"2024-03-21T13:09:21.330589+0000","last_fullsized":"2024-03-21T13:09:21.330589+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:07:48.778145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:07:48.778145+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678971+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":282,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.597548+0000","last_change":"2024-03-21T13:09:10.712435+0000","last_active":"2024-03-21T13:09:26.597548+0000","last_peered":"2024-03-21T13:09:26.597548+0000","last_clean":"2024-03-21T13:09:26.597548+0000","last_became_active":"2024-03-21T13:06:29.486743+0000","last_became_peered":"2024-03-21T13:06:29.486743+0000","last_unstale":"2024-03-21T13:09:26.597548+0000","last_undegraded":"2024-03-21T13:09:26.597548+0000","last_fullsized":"2024-03-21T13:09:26.597548+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:10.712191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.712191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:31.517169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"251'3618","reported_seq":3667,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:29.841546+0000","last_change":"2024-03-21T13:09:18.455258+0000","last_active":"2024-03-21T13:09:29.841546+0000","last_peered":"2024-03-21T13:09:29.841546+0000","last_clean":"2024-03-21T13:09:29.841546+0000","last_became_active":"2024-03-21T13:09:18.453275+0000","last_became_peered":"2024-03-21T13:09:18.453275+0000","last_unstale":"2024-03-21T13:09:29.841546+0000","last_undegraded":"2024-03-21T13:09:29.841546+0000","last_fullsized":"2024-03-21T13:09:29.841546+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3618,"log_dups_size":0,"ondisk_log_size":3618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:30.621348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131104768,"num_objects":2000,"num_object_clones":0,"num_object_copies":6000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2000,"num_write_kb":128032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":412,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.598255+0000","last_change":"2024-03-21T13:08:14.371833+0000","last_active":"2024-03-21T13:09:26.598255+0000","last_peered":"2024-03-21T13:09:26.598255+0000","last_clean":"2024-03-21T13:09:26.598255+0000","last_became_active":"2024-03-21T13:06:29.503980+0000","last_became_peered":"2024-03-21T13:06:29.503980+0000","last_unstale":"2024-03-21T13:09:26.598255+0000","last_undegraded":"2024-03-21T13:09:26.598255+0000","last_fullsized":"2024-03-21T13:09:26.598255+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:08:14.371718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":260,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.611770+0000","last_change":"2024-03-21T13:06:29.503568+0000","last_active":"2024-03-21T13:09:26.611770+0000","last_peered":"2024-03-21T13:09:26.611770+0000","last_clean":"2024-03-21T13:09:26.611770+0000","last_became_active":"2024-03-21T13:06:29.503252+0000","last_became_peered":"2024-03-21T13:06:29.503252+0000","last_unstale":"2024-03-21T13:09:26.611770+0000","last_undegraded":"2024-03-21T13:09:26.611770+0000","last_fullsized":"2024-03-21T13:09:26.611770+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.216547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.216547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":416,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.601256+0000","last_change":"2024-03-21T13:05:22.078652+0000","last_active":"2024-03-21T13:09:26.601256+0000","last_peered":"2024-03-21T13:09:26.601256+0000","last_clean":"2024-03-21T13:09:26.601256+0000","last_became_active":"2024-03-21T13:02:26.236780+0000","last_became_peered":"2024-03-21T13:02:26.236780+0000","last_unstale":"2024-03-21T13:09:26.601256+0000","last_undegraded":"2024-03-21T13:09:26.601256+0000","last_fullsized":"2024-03-21T13:09:26.601256+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:22.078546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.078546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'35","reported_seq":516,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.602121+0000","last_change":"2024-03-21T13:05:29.992376+0000","last_active":"2024-03-21T13:09:26.602121+0000","last_peered":"2024-03-21T13:09:26.602121+0000","last_clean":"2024-03-21T13:09:26.602121+0000","last_became_active":"2024-03-21T13:02:26.240350+0000","last_became_peered":"2024-03-21T13:02:26.240350+0000","last_unstale":"2024-03-21T13:09:26.602121+0000","last_undegraded":"2024-03-21T13:09:26.602121+0000","last_fullsized":"2024-03-21T13:09:26.602121+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-21T13:05:29.992207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.992207+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"251'3546","reported_seq":3599,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:29.855385+0000","last_change":"2024-03-21T13:09:18.454626+0000","last_active":"2024-03-21T13:09:29.855385+0000","last_peered":"2024-03-21T13:09:29.855385+0000","last_clean":"2024-03-21T13:09:29.855385+0000","last_became_active":"2024-03-21T13:09:18.452863+0000","last_became_peered":"2024-03-21T13:09:18.452863+0000","last_unstale":"2024-03-21T13:09:29.855385+0000","last_undegraded":"2024-03-21T13:09:29.855385+0000","last_fullsized":"2024-03-21T13:09:29.855385+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3546,"log_dups_size":0,"ondisk_log_size":3546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:30.497940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128221184,"num_objects":1956,"num_object_clones":0,"num_object_copies":5868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1956,"num_write_kb":125216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":309,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.600303+0000","last_change":"2024-03-21T13:06:29.501377+0000","last_active":"2024-03-21T13:09:26.600303+0000","last_peered":"2024-03-21T13:09:26.600303+0000","last_clean":"2024-03-21T13:09:26.600303+0000","last_became_active":"2024-03-21T13:06:29.500961+0000","last_became_peered":"2024-03-21T13:06:29.500961+0000","last_unstale":"2024-03-21T13:09:26.600303+0000","last_undegraded":"2024-03-21T13:09:26.600303+0000","last_fullsized":"2024-03-21T13:09:26.600303+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:39.926828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.926828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":131,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.595057+0000","last_change":"2024-03-21T13:08:12.031223+0000","last_active":"2024-03-21T13:09:26.595057+0000","last_peered":"2024-03-21T13:09:26.595057+0000","last_clean":"2024-03-21T13:09:26.595057+0000","last_became_active":"2024-03-21T13:06:29.502745+0000","last_became_peered":"2024-03-21T13:06:29.502745+0000","last_unstale":"2024-03-21T13:09:26.595057+0000","last_undegraded":"2024-03-21T13:09:26.595057+0000","last_fullsized":"2024-03-21T13:09:26.595057+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:12.031099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:12.031099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":275,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.350819+0000","last_change":"2024-03-21T13:08:57.570445+0000","last_active":"2024-03-21T13:09:21.350819+0000","last_peered":"2024-03-21T13:09:21.350819+0000","last_clean":"2024-03-21T13:09:21.350819+0000","last_became_active":"2024-03-21T13:02:26.243974+0000","last_became_peered":"2024-03-21T13:02:26.243974+0000","last_unstale":"2024-03-21T13:09:21.350819+0000","last_undegraded":"2024-03-21T13:09:21.350819+0000","last_fullsized":"2024-03-21T13:09:21.350819+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:57.570303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.570303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:07.178439+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"234'19","reported_seq":384,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.595327+0000","last_change":"2024-03-21T13:07:58.117288+0000","last_active":"2024-03-21T13:09:26.595327+0000","last_peered":"2024-03-21T13:09:26.595327+0000","last_clean":"2024-03-21T13:09:26.595327+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:09:26.595327+0000","last_undegraded":"2024-03-21T13:09:26.595327+0000","last_fullsized":"2024-03-21T13:09:26.595327+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-21T13:07:58.117164+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:07:58.117164+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"251'3650","reported_seq":3703,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:29.827712+0000","last_change":"2024-03-21T13:09:18.468532+0000","last_active":"2024-03-21T13:09:29.827712+0000","last_peered":"2024-03-21T13:09:29.827712+0000","last_clean":"2024-03-21T13:09:29.827712+0000","last_became_active":"2024-03-21T13:09:18.465669+0000","last_became_peered":"2024-03-21T13:09:18.465669+0000","last_unstale":"2024-03-21T13:09:29.827712+0000","last_undegraded":"2024-03-21T13:09:29.827712+0000","last_fullsized":"2024-03-21T13:09:29.827712+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3650,"log_dups_size":0,"ondisk_log_size":3650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:30.621327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133070848,"num_objects":2030,"num_object_clones":0,"num_object_copies":6090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2030,"num_write_kb":129952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":390,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.331890+0000","last_change":"2024-03-21T13:09:11.661260+0000","last_active":"2024-03-21T13:09:21.331890+0000","last_peered":"2024-03-21T13:09:21.331890+0000","last_clean":"2024-03-21T13:09:21.331890+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:09:21.331890+0000","last_undegraded":"2024-03-21T13:09:21.331890+0000","last_fullsized":"2024-03-21T13:09:21.331890+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:11.661130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:09:11.661130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:30.086420+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":308,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.601974+0000","last_change":"2024-03-21T13:05:43.961181+0000","last_active":"2024-03-21T13:09:26.601974+0000","last_peered":"2024-03-21T13:09:26.601974+0000","last_clean":"2024-03-21T13:09:26.601974+0000","last_became_active":"2024-03-21T13:02:26.243692+0000","last_became_peered":"2024-03-21T13:02:26.243692+0000","last_unstale":"2024-03-21T13:09:26.601974+0000","last_undegraded":"2024-03-21T13:09:26.601974+0000","last_fullsized":"2024-03-21T13:09:26.601974+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.961057+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":115,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.602491+0000","last_change":"2024-03-21T13:06:45.410372+0000","last_active":"2024-03-21T13:09:26.602491+0000","last_peered":"2024-03-21T13:09:26.602491+0000","last_clean":"2024-03-21T13:09:26.602491+0000","last_became_active":"2024-03-21T13:06:45.410017+0000","last_became_peered":"2024-03-21T13:06:45.410017+0000","last_unstale":"2024-03-21T13:09:26.602491+0000","last_undegraded":"2024-03-21T13:09:26.602491+0000","last_fullsized":"2024-03-21T13:09:26.602491+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:48.265982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:48.265982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"234'42","reported_seq":266,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.351166+0000","last_change":"2024-03-21T13:06:29.547664+0000","last_active":"2024-03-21T13:09:21.351166+0000","last_peered":"2024-03-21T13:09:21.351166+0000","last_clean":"2024-03-21T13:09:21.351166+0000","last_became_active":"2024-03-21T13:06:29.508620+0000","last_became_peered":"2024-03-21T13:06:29.508620+0000","last_unstale":"2024-03-21T13:09:21.351166+0000","last_undegraded":"2024-03-21T13:09:21.351166+0000","last_fullsized":"2024-03-21T13:09:21.351166+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'29","last_scrub_stamp":"2024-03-21T13:05:22.201879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.201879+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":127,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.604928+0000","last_change":"2024-03-21T13:09:07.643588+0000","last_active":"2024-03-21T13:09:26.604928+0000","last_peered":"2024-03-21T13:09:26.604928+0000","last_clean":"2024-03-21T13:09:26.604928+0000","last_became_active":"2024-03-21T13:06:45.406734+0000","last_became_peered":"2024-03-21T13:06:45.406734+0000","last_unstale":"2024-03-21T13:09:26.604928+0000","last_undegraded":"2024-03-21T13:09:26.604928+0000","last_fullsized":"2024-03-21T13:09:26.604928+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:07.643482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:09:07.643482+0000","last_clean_scrub_stamp":"2024-03-21T13:09:07.643482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:33.867868+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.a","version":"251'4105","reported_seq":4152,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:29.842904+0000","last_change":"2024-03-21T13:08:45.620420+0000","last_active":"2024-03-21T13:09:29.842904+0000","last_peered":"2024-03-21T13:09:29.842904+0000","last_clean":"2024-03-21T13:09:29.842904+0000","last_became_active":"2024-03-21T13:08:45.620077+0000","last_became_peered":"2024-03-21T13:08:45.620077+0000","last_unstale":"2024-03-21T13:09:29.842904+0000","last_undegraded":"2024-03-21T13:09:29.842904+0000","last_fullsized":"2024-03-21T13:09:29.842904+0000","mapping_epoch":234,"log_start":"240'1147","ondisk_log_start":"240'1147","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":2958,"log_dups_size":725,"ondisk_log_size":2958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:51.646858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269025280,"num_objects":4105,"num_object_clones":0,"num_object_copies":12315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4105,"num_write_kb":262720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":407,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.603670+0000","last_change":"2024-03-21T13:08:58.723773+0000","last_active":"2024-03-21T13:09:26.603670+0000","last_peered":"2024-03-21T13:09:26.603670+0000","last_clean":"2024-03-21T13:09:26.603670+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:09:26.603670+0000","last_undegraded":"2024-03-21T13:09:26.603670+0000","last_fullsized":"2024-03-21T13:09:26.603670+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:58.723653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:58.723653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:14.781119+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":133,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.602258+0000","last_change":"2024-03-21T13:09:15.591078+0000","last_active":"2024-03-21T13:09:26.602258+0000","last_peered":"2024-03-21T13:09:26.602258+0000","last_clean":"2024-03-21T13:09:26.602258+0000","last_became_active":"2024-03-21T13:06:29.509674+0000","last_became_peered":"2024-03-21T13:06:29.509674+0000","last_unstale":"2024-03-21T13:09:26.602258+0000","last_undegraded":"2024-03-21T13:09:26.602258+0000","last_fullsized":"2024-03-21T13:09:26.602258+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:15.590828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:15.590828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:15.640211+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":121,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.330004+0000","last_change":"2024-03-21T13:06:29.510171+0000","last_active":"2024-03-21T13:09:21.330004+0000","last_peered":"2024-03-21T13:09:21.330004+0000","last_clean":"2024-03-21T13:09:21.330004+0000","last_became_active":"2024-03-21T13:06:29.509315+0000","last_became_peered":"2024-03-21T13:06:29.509315+0000","last_unstale":"2024-03-21T13:09:21.330004+0000","last_undegraded":"2024-03-21T13:09:21.330004+0000","last_fullsized":"2024-03-21T13:09:21.330004+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:46.263644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:05:46.263644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"243'20","reported_seq":354,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.600268+0000","last_change":"2024-03-21T13:09:10.683920+0000","last_active":"2024-03-21T13:09:26.600268+0000","last_peered":"2024-03-21T13:09:26.600268+0000","last_clean":"2024-03-21T13:09:26.600268+0000","last_became_active":"2024-03-21T13:02:26.234230+0000","last_became_peered":"2024-03-21T13:02:26.234230+0000","last_unstale":"2024-03-21T13:09:26.600268+0000","last_undegraded":"2024-03-21T13:09:26.600268+0000","last_fullsized":"2024-03-21T13:09:26.600268+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'20","last_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_deep_scrub":"243'20","last_deep_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.683382+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:34.770339+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":268,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.603564+0000","last_change":"2024-03-21T13:09:16.750329+0000","last_active":"2024-03-21T13:09:26.603564+0000","last_peered":"2024-03-21T13:09:26.603564+0000","last_clean":"2024-03-21T13:09:26.603564+0000","last_became_active":"2024-03-21T13:06:29.501625+0000","last_became_peered":"2024-03-21T13:06:29.501625+0000","last_unstale":"2024-03-21T13:09:26.603564+0000","last_undegraded":"2024-03-21T13:09:26.603564+0000","last_fullsized":"2024-03-21T13:09:26.603564+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:16.750077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:16.750077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:45.379145+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"250'3480","reported_seq":3535,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:25.748822+0000","last_change":"2024-03-21T13:09:18.477598+0000","last_active":"2024-03-21T13:09:25.748822+0000","last_peered":"2024-03-21T13:09:25.748822+0000","last_clean":"2024-03-21T13:09:25.748822+0000","last_became_active":"2024-03-21T13:09:18.458922+0000","last_became_peered":"2024-03-21T13:09:18.458922+0000","last_unstale":"2024-03-21T13:09:25.748822+0000","last_undegraded":"2024-03-21T13:09:25.748822+0000","last_fullsized":"2024-03-21T13:09:25.748822+0000","mapping_epoch":234,"log_start":"238'200","ondisk_log_start":"238'200","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3280,"log_dups_size":0,"ondisk_log_size":3280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:29.811930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121896960,"num_objects":1860,"num_object_clones":0,"num_object_copies":5580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1860,"num_write_kb":119040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":275,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.332506+0000","last_change":"2024-03-21T13:04:27.549179+0000","last_active":"2024-03-21T13:09:21.332506+0000","last_peered":"2024-03-21T13:09:21.332506+0000","last_clean":"2024-03-21T13:09:21.332506+0000","last_became_active":"2024-03-21T13:02:26.235640+0000","last_became_peered":"2024-03-21T13:02:26.235640+0000","last_unstale":"2024-03-21T13:09:21.332506+0000","last_undegraded":"2024-03-21T13:09:21.332506+0000","last_fullsized":"2024-03-21T13:09:21.332506+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:27.549086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:04:27.549086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678513+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":668,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.604596+0000","last_change":"2024-03-21T13:08:07.454234+0000","last_active":"2024-03-21T13:09:26.604596+0000","last_peered":"2024-03-21T13:09:26.604596+0000","last_clean":"2024-03-21T13:09:26.604596+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:09:26.604596+0000","last_undegraded":"2024-03-21T13:09:26.604596+0000","last_fullsized":"2024-03-21T13:09:26.604596+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:08:07.454016+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":409,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.330948+0000","last_change":"2024-03-21T13:07:49.758585+0000","last_active":"2024-03-21T13:09:21.330948+0000","last_peered":"2024-03-21T13:09:21.330948+0000","last_clean":"2024-03-21T13:09:21.330948+0000","last_became_active":"2024-03-21T13:02:26.242456+0000","last_became_peered":"2024-03-21T13:02:26.242456+0000","last_unstale":"2024-03-21T13:09:21.330948+0000","last_undegraded":"2024-03-21T13:09:21.330948+0000","last_fullsized":"2024-03-21T13:09:21.330948+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:49.758485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:49.758485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.679002+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"234'36","reported_seq":588,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.605453+0000","last_change":"2024-03-21T13:05:42.033620+0000","last_active":"2024-03-21T13:09:26.605453+0000","last_peered":"2024-03-21T13:09:26.605453+0000","last_clean":"2024-03-21T13:09:26.605453+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:09:26.605453+0000","last_undegraded":"2024-03-21T13:09:26.605453+0000","last_fullsized":"2024-03-21T13:09:26.605453+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'24","last_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.033266+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":415,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.599943+0000","last_change":"2024-03-21T13:06:29.504771+0000","last_active":"2024-03-21T13:09:26.599943+0000","last_peered":"2024-03-21T13:09:26.599943+0000","last_clean":"2024-03-21T13:09:26.599943+0000","last_became_active":"2024-03-21T13:06:29.504250+0000","last_became_peered":"2024-03-21T13:06:29.504250+0000","last_unstale":"2024-03-21T13:09:26.599943+0000","last_undegraded":"2024-03-21T13:09:26.599943+0000","last_fullsized":"2024-03-21T13:09:26.599943+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:20.061709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.061709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"251'4052","reported_seq":4099,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:29.884640+0000","last_change":"2024-03-21T13:08:45.883519+0000","last_active":"2024-03-21T13:09:29.884640+0000","last_peered":"2024-03-21T13:09:29.884640+0000","last_clean":"2024-03-21T13:09:29.884640+0000","last_became_active":"2024-03-21T13:08:45.883237+0000","last_became_peered":"2024-03-21T13:08:45.883237+0000","last_unstale":"2024-03-21T13:09:29.884640+0000","last_undegraded":"2024-03-21T13:09:29.884640+0000","last_fullsized":"2024-03-21T13:09:29.884640+0000","mapping_epoch":234,"log_start":"239'806","ondisk_log_start":"239'806","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3246,"log_dups_size":0,"ondisk_log_size":3246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:49.722278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265551872,"num_objects":4052,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4052,"num_write_kb":259328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":277,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.337688+0000","last_change":"2024-03-21T13:05:29.457708+0000","last_active":"2024-03-21T13:09:21.337688+0000","last_peered":"2024-03-21T13:09:21.337688+0000","last_clean":"2024-03-21T13:09:21.337688+0000","last_became_active":"2024-03-21T13:02:26.242708+0000","last_became_peered":"2024-03-21T13:02:26.242708+0000","last_unstale":"2024-03-21T13:09:21.337688+0000","last_undegraded":"2024-03-21T13:09:21.337688+0000","last_fullsized":"2024-03-21T13:09:21.337688+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:29.457597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:29.457597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678630+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":311,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.598592+0000","last_change":"2024-03-21T13:08:11.057903+0000","last_active":"2024-03-21T13:09:26.598592+0000","last_peered":"2024-03-21T13:09:26.598592+0000","last_clean":"2024-03-21T13:09:26.598592+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:09:26.598592+0000","last_undegraded":"2024-03-21T13:09:26.598592+0000","last_fullsized":"2024-03-21T13:09:26.598592+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:11.057788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:08:11.057788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":1075,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:28.050640+0000","last_change":"2024-03-21T13:08:05.448565+0000","last_active":"2024-03-21T13:09:28.050640+0000","last_peered":"2024-03-21T13:09:28.050640+0000","last_clean":"2024-03-21T13:09:28.050640+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:09:28.050640+0000","last_undegraded":"2024-03-21T13:09:28.050640+0000","last_fullsized":"2024-03-21T13:09:28.050640+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:08:05.448403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:08:05.448403+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":642,"num_read_kb":637,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":508,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.599906+0000","last_change":"2024-03-21T13:07:50.821737+0000","last_active":"2024-03-21T13:09:26.599906+0000","last_peered":"2024-03-21T13:09:26.599906+0000","last_clean":"2024-03-21T13:09:26.599906+0000","last_became_active":"2024-03-21T13:06:29.526702+0000","last_became_peered":"2024-03-21T13:06:29.526702+0000","last_unstale":"2024-03-21T13:09:26.599906+0000","last_undegraded":"2024-03-21T13:09:26.599906+0000","last_fullsized":"2024-03-21T13:09:26.599906+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:07:50.821119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:07:50.821119+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":65,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.604573+0000","last_change":"2024-03-21T13:08:56.630611+0000","last_active":"2024-03-21T13:09:26.604573+0000","last_peered":"2024-03-21T13:09:26.604573+0000","last_clean":"2024-03-21T13:09:26.604573+0000","last_became_active":"2024-03-21T13:08:20.892193+0000","last_became_peered":"2024-03-21T13:08:20.892193+0000","last_unstale":"2024-03-21T13:09:26.604573+0000","last_undegraded":"2024-03-21T13:09:26.604573+0000","last_fullsized":"2024-03-21T13:09:26.604573+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:56.630492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:56.630492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:58.893366+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"251'4109","reported_seq":4156,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:29.841576+0000","last_change":"2024-03-21T13:08:45.885732+0000","last_active":"2024-03-21T13:09:29.841576+0000","last_peered":"2024-03-21T13:09:29.841576+0000","last_clean":"2024-03-21T13:09:29.841576+0000","last_became_active":"2024-03-21T13:08:45.885382+0000","last_became_peered":"2024-03-21T13:08:45.885382+0000","last_unstale":"2024-03-21T13:09:29.841576+0000","last_undegraded":"2024-03-21T13:09:29.841576+0000","last_fullsized":"2024-03-21T13:09:29.841576+0000","mapping_epoch":234,"log_start":"240'1139","ondisk_log_start":"240'1139","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":2970,"log_dups_size":725,"ondisk_log_size":2970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:01.318388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269287424,"num_objects":4109,"num_object_clones":0,"num_object_copies":12327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4109,"num_write_kb":262976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":303,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.603053+0000","last_change":"2024-03-21T13:08:57.641050+0000","last_active":"2024-03-21T13:09:26.603053+0000","last_peered":"2024-03-21T13:09:26.603053+0000","last_clean":"2024-03-21T13:09:26.603053+0000","last_became_active":"2024-03-21T13:06:29.489464+0000","last_became_peered":"2024-03-21T13:06:29.489464+0000","last_unstale":"2024-03-21T13:09:26.603053+0000","last_undegraded":"2024-03-21T13:09:26.603053+0000","last_fullsized":"2024-03-21T13:09:26.603053+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:57.640892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.640892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:20.853004+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":152,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.602791+0000","last_change":"2024-03-21T13:08:22.012978+0000","last_active":"2024-03-21T13:09:26.602791+0000","last_peered":"2024-03-21T13:09:26.602791+0000","last_clean":"2024-03-21T13:09:26.602791+0000","last_became_active":"2024-03-21T13:06:29.484174+0000","last_became_peered":"2024-03-21T13:06:29.484174+0000","last_unstale":"2024-03-21T13:09:26.602791+0000","last_undegraded":"2024-03-21T13:09:26.602791+0000","last_fullsized":"2024-03-21T13:09:26.602791+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:22.012892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:22.012892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:37.561153+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":397,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.612303+0000","last_change":"2024-03-21T13:09:15.777415+0000","last_active":"2024-03-21T13:09:26.612303+0000","last_peered":"2024-03-21T13:09:26.612303+0000","last_clean":"2024-03-21T13:09:26.612303+0000","last_became_active":"2024-03-21T13:06:29.492612+0000","last_became_peered":"2024-03-21T13:06:29.492612+0000","last_unstale":"2024-03-21T13:09:26.612303+0000","last_undegraded":"2024-03-21T13:09:26.612303+0000","last_fullsized":"2024-03-21T13:09:26.612303+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-21T13:09:15.777140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:15.777140+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:30.629715+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":425,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.602375+0000","last_change":"2024-03-21T13:09:16.577277+0000","last_active":"2024-03-21T13:09:26.602375+0000","last_peered":"2024-03-21T13:09:26.602375+0000","last_clean":"2024-03-21T13:09:26.602375+0000","last_became_active":"2024-03-21T13:06:29.496679+0000","last_became_peered":"2024-03-21T13:06:29.496679+0000","last_unstale":"2024-03-21T13:09:26.602375+0000","last_undegraded":"2024-03-21T13:09:26.602375+0000","last_fullsized":"2024-03-21T13:09:26.602375+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:16.577170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:16.577170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:44.049843+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"250'3794","reported_seq":3837,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:25.802736+0000","last_change":"2024-03-21T13:08:45.619068+0000","last_active":"2024-03-21T13:09:25.802736+0000","last_peered":"2024-03-21T13:09:25.802736+0000","last_clean":"2024-03-21T13:09:25.802736+0000","last_became_active":"2024-03-21T13:08:45.618800+0000","last_became_peered":"2024-03-21T13:08:45.618800+0000","last_unstale":"2024-03-21T13:09:25.802736+0000","last_undegraded":"2024-03-21T13:09:25.802736+0000","last_fullsized":"2024-03-21T13:09:25.802736+0000","mapping_epoch":234,"log_start":"239'955","ondisk_log_start":"239'955","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":2839,"log_dups_size":622,"ondisk_log_size":2839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:47.996948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248643584,"num_objects":3794,"num_object_clones":0,"num_object_copies":11382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3794,"num_write_kb":242816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":307,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.329434+0000","last_change":"2024-03-21T13:08:55.488120+0000","last_active":"2024-03-21T13:09:21.329434+0000","last_peered":"2024-03-21T13:09:21.329434+0000","last_clean":"2024-03-21T13:09:21.329434+0000","last_became_active":"2024-03-21T13:06:29.501983+0000","last_became_peered":"2024-03-21T13:06:29.501983+0000","last_unstale":"2024-03-21T13:09:21.329434+0000","last_undegraded":"2024-03-21T13:09:21.329434+0000","last_fullsized":"2024-03-21T13:09:21.329434+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:55.487977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:55.487977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:57.490278+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":281,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.330340+0000","last_change":"2024-03-21T13:06:29.511420+0000","last_active":"2024-03-21T13:09:21.330340+0000","last_peered":"2024-03-21T13:09:21.330340+0000","last_clean":"2024-03-21T13:09:21.330340+0000","last_became_active":"2024-03-21T13:06:29.510833+0000","last_became_peered":"2024-03-21T13:06:29.510833+0000","last_unstale":"2024-03-21T13:09:21.330340+0000","last_undegraded":"2024-03-21T13:09:21.330340+0000","last_fullsized":"2024-03-21T13:09:21.330340+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.324701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.324701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678597+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"234'25","reported_seq":384,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.594951+0000","last_change":"2024-03-21T13:08:57.835622+0000","last_active":"2024-03-21T13:09:26.594951+0000","last_peered":"2024-03-21T13:09:26.594951+0000","last_clean":"2024-03-21T13:09:26.594951+0000","last_became_active":"2024-03-21T13:06:29.492863+0000","last_became_peered":"2024-03-21T13:06:29.492863+0000","last_unstale":"2024-03-21T13:09:26.594951+0000","last_undegraded":"2024-03-21T13:09:26.594951+0000","last_fullsized":"2024-03-21T13:09:26.594951+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-21T13:08:57.835430+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.835430+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:22.275431+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"125'12","reported_seq":441,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.605236+0000","last_change":"2024-03-21T13:08:18.149427+0000","last_active":"2024-03-21T13:09:26.605236+0000","last_peered":"2024-03-21T13:09:26.605236+0000","last_clean":"2024-03-21T13:09:26.605236+0000","last_became_active":"2024-03-21T13:02:26.253720+0000","last_became_peered":"2024-03-21T13:02:26.253720+0000","last_unstale":"2024-03-21T13:09:26.605236+0000","last_undegraded":"2024-03-21T13:09:26.605236+0000","last_fullsized":"2024-03-21T13:09:26.605236+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'12","last_scrub_stamp":"2024-03-21T13:08:18.149295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.149295+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"250'3749","reported_seq":3794,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:25.831977+0000","last_change":"2024-03-21T13:08:45.618681+0000","last_active":"2024-03-21T13:09:25.831977+0000","last_peered":"2024-03-21T13:09:25.831977+0000","last_clean":"2024-03-21T13:09:25.831977+0000","last_became_active":"2024-03-21T13:08:45.618487+0000","last_became_peered":"2024-03-21T13:08:45.618487+0000","last_unstale":"2024-03-21T13:09:25.831977+0000","last_undegraded":"2024-03-21T13:09:25.831977+0000","last_fullsized":"2024-03-21T13:09:25.831977+0000","mapping_epoch":234,"log_start":"239'914","ondisk_log_start":"239'914","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":622,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:13.133492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245694464,"num_objects":3749,"num_object_clones":0,"num_object_copies":11247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3749,"num_write_kb":239936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":405,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.612412+0000","last_change":"2024-03-21T13:06:29.489617+0000","last_active":"2024-03-21T13:09:26.612412+0000","last_peered":"2024-03-21T13:09:26.612412+0000","last_clean":"2024-03-21T13:09:26.612412+0000","last_became_active":"2024-03-21T13:06:29.489289+0000","last_became_peered":"2024-03-21T13:06:29.489289+0000","last_unstale":"2024-03-21T13:09:26.612412+0000","last_undegraded":"2024-03-21T13:09:26.612412+0000","last_fullsized":"2024-03-21T13:09:26.612412+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.951655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.951655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":455,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.599986+0000","last_change":"2024-03-21T13:09:09.025341+0000","last_active":"2024-03-21T13:09:26.599986+0000","last_peered":"2024-03-21T13:09:26.599986+0000","last_clean":"2024-03-21T13:09:26.599986+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:09:26.599986+0000","last_undegraded":"2024-03-21T13:09:26.599986+0000","last_fullsized":"2024-03-21T13:09:26.599986+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:09:09.025225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:09.025225+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:20.173155+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'35","reported_seq":398,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.602011+0000","last_change":"2024-03-21T13:08:54.645581+0000","last_active":"2024-03-21T13:09:26.602011+0000","last_peered":"2024-03-21T13:09:26.602011+0000","last_clean":"2024-03-21T13:09:26.602011+0000","last_became_active":"2024-03-21T13:02:26.227165+0000","last_became_peered":"2024-03-21T13:02:26.227165+0000","last_unstale":"2024-03-21T13:09:26.602011+0000","last_undegraded":"2024-03-21T13:09:26.602011+0000","last_fullsized":"2024-03-21T13:09:26.602011+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:08:54.645360+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:08:54.645360+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:11.752625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"226'10","reported_seq":545,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.330192+0000","last_change":"2024-03-21T13:08:16.739276+0000","last_active":"2024-03-21T13:09:21.330192+0000","last_peered":"2024-03-21T13:09:21.330192+0000","last_clean":"2024-03-21T13:09:21.330192+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:09:21.330192+0000","last_undegraded":"2024-03-21T13:09:21.330192+0000","last_fullsized":"2024-03-21T13:09:21.330192+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:08:16.739190+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:29.735435+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"251'4033","reported_seq":4080,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:29.858003+0000","last_change":"2024-03-21T13:08:45.621307+0000","last_active":"2024-03-21T13:09:29.858003+0000","last_peered":"2024-03-21T13:09:29.858003+0000","last_clean":"2024-03-21T13:09:29.858003+0000","last_became_active":"2024-03-21T13:08:45.620972+0000","last_became_peered":"2024-03-21T13:08:45.620972+0000","last_unstale":"2024-03-21T13:09:29.858003+0000","last_undegraded":"2024-03-21T13:09:29.858003+0000","last_fullsized":"2024-03-21T13:09:29.858003+0000","mapping_epoch":234,"log_start":"240'1112","ondisk_log_start":"240'1112","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":2921,"log_dups_size":725,"ondisk_log_size":2921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:07.398438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264306688,"num_objects":4033,"num_object_clones":0,"num_object_copies":12099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4033,"num_write_kb":258112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":321,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.332720+0000","last_change":"2024-03-21T13:06:29.491923+0000","last_active":"2024-03-21T13:09:21.332720+0000","last_peered":"2024-03-21T13:09:21.332720+0000","last_clean":"2024-03-21T13:09:21.332720+0000","last_became_active":"2024-03-21T13:06:29.491598+0000","last_became_peered":"2024-03-21T13:06:29.491598+0000","last_unstale":"2024-03-21T13:09:21.332720+0000","last_undegraded":"2024-03-21T13:09:21.332720+0000","last_fullsized":"2024-03-21T13:09:21.332720+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.416262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.416262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678767+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":402,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.332022+0000","last_change":"2024-03-21T13:04:20.538204+0000","last_active":"2024-03-21T13:09:21.332022+0000","last_peered":"2024-03-21T13:09:21.332022+0000","last_clean":"2024-03-21T13:09:21.332022+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:09:21.332022+0000","last_undegraded":"2024-03-21T13:09:21.332022+0000","last_fullsized":"2024-03-21T13:09:21.332022+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:04:20.538115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:04:20.538115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"234'30","reported_seq":449,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.329978+0000","last_change":"2024-03-21T13:05:12.322522+0000","last_active":"2024-03-21T13:09:21.329978+0000","last_peered":"2024-03-21T13:09:21.329978+0000","last_clean":"2024-03-21T13:09:21.329978+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:09:21.329978+0000","last_undegraded":"2024-03-21T13:09:21.329978+0000","last_fullsized":"2024-03-21T13:09:21.329978+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T13:05:12.322284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:12.322284+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":405,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.598356+0000","last_change":"2024-03-21T13:08:04.095062+0000","last_active":"2024-03-21T13:09:26.598356+0000","last_peered":"2024-03-21T13:09:26.598356+0000","last_clean":"2024-03-21T13:09:26.598356+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:09:26.598356+0000","last_undegraded":"2024-03-21T13:09:26.598356+0000","last_fullsized":"2024-03-21T13:09:26.598356+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:04.094946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:04.094946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"250'3547","reported_seq":3598,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:25.743552+0000","last_change":"2024-03-21T13:09:18.462019+0000","last_active":"2024-03-21T13:09:25.743552+0000","last_peered":"2024-03-21T13:09:25.743552+0000","last_clean":"2024-03-21T13:09:25.743552+0000","last_became_active":"2024-03-21T13:09:18.458845+0000","last_became_peered":"2024-03-21T13:09:18.458845+0000","last_unstale":"2024-03-21T13:09:25.743552+0000","last_undegraded":"2024-03-21T13:09:25.743552+0000","last_fullsized":"2024-03-21T13:09:25.743552+0000","mapping_epoch":234,"log_start":"239'300","ondisk_log_start":"239'300","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3247,"log_dups_size":0,"ondisk_log_size":3247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:29.811979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124223488,"num_objects":1895,"num_object_clones":0,"num_object_copies":5685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1895,"num_write_kb":121312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":245,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.603019+0000","last_change":"2024-03-21T13:06:29.489514+0000","last_active":"2024-03-21T13:09:26.603019+0000","last_peered":"2024-03-21T13:09:26.603019+0000","last_clean":"2024-03-21T13:09:26.603019+0000","last_became_active":"2024-03-21T13:06:29.488826+0000","last_became_peered":"2024-03-21T13:06:29.488826+0000","last_unstale":"2024-03-21T13:09:26.603019+0000","last_undegraded":"2024-03-21T13:09:26.603019+0000","last_fullsized":"2024-03-21T13:09:26.603019+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.328925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.328925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.e","version":"196'11","reported_seq":426,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.600177+0000","last_change":"2024-03-21T13:08:04.934320+0000","last_active":"2024-03-21T13:09:26.600177+0000","last_peered":"2024-03-21T13:09:26.600177+0000","last_clean":"2024-03-21T13:09:26.600177+0000","last_became_active":"2024-03-21T13:06:29.505444+0000","last_became_peered":"2024-03-21T13:06:29.505444+0000","last_unstale":"2024-03-21T13:09:26.600177+0000","last_undegraded":"2024-03-21T13:09:26.600177+0000","last_fullsized":"2024-03-21T13:09:26.600177+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'11","last_scrub_stamp":"2024-03-21T13:08:04.934196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:04.934196+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":416,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.329861+0000","last_change":"2024-03-21T13:09:15.640885+0000","last_active":"2024-03-21T13:09:21.329861+0000","last_peered":"2024-03-21T13:09:21.329861+0000","last_clean":"2024-03-21T13:09:21.329861+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:09:21.329861+0000","last_undegraded":"2024-03-21T13:09:21.329861+0000","last_fullsized":"2024-03-21T13:09:21.329861+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:15.640781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:15.640781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:35.737356+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"234'34","reported_seq":372,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.332131+0000","last_change":"2024-03-21T13:08:22.736499+0000","last_active":"2024-03-21T13:09:21.332131+0000","last_peered":"2024-03-21T13:09:21.332131+0000","last_clean":"2024-03-21T13:09:21.332131+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:09:21.332131+0000","last_undegraded":"2024-03-21T13:09:21.332131+0000","last_fullsized":"2024-03-21T13:09:21.332131+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:08:22.736371+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:52.513483+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"251'3694","reported_seq":3744,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:29.806912+0000","last_change":"2024-03-21T13:09:18.452545+0000","last_active":"2024-03-21T13:09:29.806912+0000","last_peered":"2024-03-21T13:09:29.806912+0000","last_clean":"2024-03-21T13:09:29.806912+0000","last_became_active":"2024-03-21T13:09:18.448090+0000","last_became_peered":"2024-03-21T13:09:18.448090+0000","last_unstale":"2024-03-21T13:09:29.806912+0000","last_undegraded":"2024-03-21T13:09:29.806912+0000","last_fullsized":"2024-03-21T13:09:29.806912+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:30.621303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134479872,"num_objects":2052,"num_object_clones":0,"num_object_copies":6156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2052,"num_write_kb":131328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":401,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.337786+0000","last_change":"2024-03-21T13:05:39.518051+0000","last_active":"2024-03-21T13:09:21.337786+0000","last_peered":"2024-03-21T13:09:21.337786+0000","last_clean":"2024-03-21T13:09:21.337786+0000","last_became_active":"2024-03-21T13:02:26.243985+0000","last_became_peered":"2024-03-21T13:02:26.243985+0000","last_unstale":"2024-03-21T13:09:21.337786+0000","last_undegraded":"2024-03-21T13:09:21.337786+0000","last_fullsized":"2024-03-21T13:09:21.337786+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:39.517591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.517591+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678785+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":299,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.599298+0000","last_change":"2024-03-21T13:08:08.054195+0000","last_active":"2024-03-21T13:09:26.599298+0000","last_peered":"2024-03-21T13:09:26.599298+0000","last_clean":"2024-03-21T13:09:26.599298+0000","last_became_active":"2024-03-21T13:06:29.501230+0000","last_became_peered":"2024-03-21T13:06:29.501230+0000","last_unstale":"2024-03-21T13:09:26.599298+0000","last_undegraded":"2024-03-21T13:09:26.599298+0000","last_fullsized":"2024-03-21T13:09:26.599298+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:08.054077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:08.054077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:28.130473+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":427,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.598484+0000","last_change":"2024-03-21T13:07:55.115719+0000","last_active":"2024-03-21T13:09:26.598484+0000","last_peered":"2024-03-21T13:09:26.598484+0000","last_clean":"2024-03-21T13:09:26.598484+0000","last_became_active":"2024-03-21T13:06:29.499291+0000","last_became_peered":"2024-03-21T13:06:29.499291+0000","last_unstale":"2024-03-21T13:09:26.598484+0000","last_undegraded":"2024-03-21T13:09:26.598484+0000","last_fullsized":"2024-03-21T13:09:26.598484+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:55.115613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:55.115613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":165,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.602856+0000","last_change":"2024-03-21T13:06:45.412306+0000","last_active":"2024-03-21T13:09:26.602856+0000","last_peered":"2024-03-21T13:09:26.602856+0000","last_clean":"2024-03-21T13:09:26.602856+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:09:26.602856+0000","last_undegraded":"2024-03-21T13:09:26.602856+0000","last_fullsized":"2024-03-21T13:09:26.602856+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:26.195507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:26.195507+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"251'3629","reported_seq":3685,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:29.889188+0000","last_change":"2024-03-21T13:09:18.463718+0000","last_active":"2024-03-21T13:09:29.889188+0000","last_peered":"2024-03-21T13:09:29.889188+0000","last_clean":"2024-03-21T13:09:29.889188+0000","last_became_active":"2024-03-21T13:09:18.458686+0000","last_became_peered":"2024-03-21T13:09:18.458686+0000","last_unstale":"2024-03-21T13:09:29.889188+0000","last_undegraded":"2024-03-21T13:09:29.889188+0000","last_fullsized":"2024-03-21T13:09:29.889188+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3629,"log_dups_size":0,"ondisk_log_size":3629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:30.495782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131203072,"num_objects":2002,"num_object_clones":0,"num_object_copies":6006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2002,"num_write_kb":128128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":121,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.331610+0000","last_change":"2024-03-21T13:06:29.508869+0000","last_active":"2024-03-21T13:09:21.331610+0000","last_peered":"2024-03-21T13:09:21.331610+0000","last_clean":"2024-03-21T13:09:21.331610+0000","last_became_active":"2024-03-21T13:06:29.508508+0000","last_became_peered":"2024-03-21T13:06:29.508508+0000","last_unstale":"2024-03-21T13:09:21.331610+0000","last_undegraded":"2024-03-21T13:09:21.331610+0000","last_fullsized":"2024-03-21T13:09:21.331610+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:36.362197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:05:36.362197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":305,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.597047+0000","last_change":"2024-03-21T13:08:05.143175+0000","last_active":"2024-03-21T13:09:26.597047+0000","last_peered":"2024-03-21T13:09:26.597047+0000","last_clean":"2024-03-21T13:09:26.597047+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:09:26.597047+0000","last_undegraded":"2024-03-21T13:09:26.597047+0000","last_fullsized":"2024-03-21T13:09:26.597047+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:05.142807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:05.142807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":427,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.602217+0000","last_change":"2024-03-21T13:06:30.274121+0000","last_active":"2024-03-21T13:09:26.602217+0000","last_peered":"2024-03-21T13:09:26.602217+0000","last_clean":"2024-03-21T13:09:26.602217+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:09:26.602217+0000","last_undegraded":"2024-03-21T13:09:26.602217+0000","last_fullsized":"2024-03-21T13:09:26.602217+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:30.274033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:30.274033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"234'31","reported_seq":254,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.604753+0000","last_change":"2024-03-21T13:09:03.608492+0000","last_active":"2024-03-21T13:09:26.604753+0000","last_peered":"2024-03-21T13:09:26.604753+0000","last_clean":"2024-03-21T13:09:26.604753+0000","last_became_active":"2024-03-21T13:06:29.489192+0000","last_became_peered":"2024-03-21T13:06:29.489192+0000","last_unstale":"2024-03-21T13:09:26.604753+0000","last_undegraded":"2024-03-21T13:09:26.604753+0000","last_fullsized":"2024-03-21T13:09:26.604753+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'31","last_scrub_stamp":"2024-03-21T13:09:03.608302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:03.608302+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:10:29.850099+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"250'3582","reported_seq":3633,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:25.650278+0000","last_change":"2024-03-21T13:09:18.460636+0000","last_active":"2024-03-21T13:09:25.650278+0000","last_peered":"2024-03-21T13:09:25.650278+0000","last_clean":"2024-03-21T13:09:25.650278+0000","last_became_active":"2024-03-21T13:09:18.458493+0000","last_became_peered":"2024-03-21T13:09:18.458493+0000","last_unstale":"2024-03-21T13:09:25.650278+0000","last_undegraded":"2024-03-21T13:09:25.650278+0000","last_fullsized":"2024-03-21T13:09:25.650278+0000","mapping_epoch":234,"log_start":"239'300","ondisk_log_start":"239'300","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3282,"log_dups_size":0,"ondisk_log_size":3282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:29.811956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124682240,"num_objects":1902,"num_object_clones":0,"num_object_copies":5706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1902,"num_write_kb":121760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":146,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.600960+0000","last_change":"2024-03-21T13:06:29.497122+0000","last_active":"2024-03-21T13:09:26.600960+0000","last_peered":"2024-03-21T13:09:26.600960+0000","last_clean":"2024-03-21T13:09:26.600960+0000","last_became_active":"2024-03-21T13:06:29.496779+0000","last_became_peered":"2024-03-21T13:06:29.496779+0000","last_unstale":"2024-03-21T13:09:26.600960+0000","last_undegraded":"2024-03-21T13:09:26.600960+0000","last_fullsized":"2024-03-21T13:09:26.600960+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.291369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.291369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"196'11","reported_seq":370,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-21T13:09:26.601205+0000","last_change":"2024-03-21T13:06:29.522757+0000","last_active":"2024-03-21T13:09:26.601205+0000","last_peered":"2024-03-21T13:09:26.601205+0000","last_clean":"2024-03-21T13:09:26.601205+0000","last_became_active":"2024-03-21T13:06:29.501395+0000","last_became_peered":"2024-03-21T13:06:29.501395+0000","last_unstale":"2024-03-21T13:09:26.601205+0000","last_undegraded":"2024-03-21T13:09:26.601205+0000","last_fullsized":"2024-03-21T13:09:26.601205+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:05:30.300605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:30.300605+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":402,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.332077+0000","last_change":"2024-03-21T13:05:38.492137+0000","last_active":"2024-03-21T13:09:21.332077+0000","last_peered":"2024-03-21T13:09:21.332077+0000","last_clean":"2024-03-21T13:09:21.332077+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:09:21.332077+0000","last_undegraded":"2024-03-21T13:09:21.332077+0000","last_fullsized":"2024-03-21T13:09:21.332077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:38.492015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.492015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678885+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"234'20","reported_seq":364,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-21T13:09:21.350744+0000","last_change":"2024-03-21T13:05:22.406498+0000","last_active":"2024-03-21T13:09:21.350744+0000","last_peered":"2024-03-21T13:09:21.350744+0000","last_clean":"2024-03-21T13:09:21.350744+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:09:21.350744+0000","last_undegraded":"2024-03-21T13:09:21.350744+0000","last_fullsized":"2024-03-21T13:09:21.350744+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:22.406367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.406367+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:09:22.678650+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":4154458112,"num_objects":63392,"num_object_clones":0,"num_object_copies":190176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3352,"num_objects_misplaced":13138,"num_objects_unfound":0,"num_objects_dirty":63392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63392,"num_write_kb":4057088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2904,"num_bytes_recovered":190316544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12930580480,"data_stored":12930580480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87758,"ondisk_log_size":87758,"up":78,"acting":74,"num_store_stats":5},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":642,"num_read_kb":637,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":5213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4428,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":15782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":729,"ondisk_log_size":729,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168711,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3902832,"kb_used_data":3420456,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":89842320,"statfs":{"total":95995035648,"available":91998535680,"internally_reserved":0,"allocated":3502546944,"data_stored":3502006850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":493943242},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463053,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2831004,"kb_used_data":2334740,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":90914148,"statfs":{"total":95995035648,"available":93096087552,"internally_reserved":0,"allocated":2390773760,"data_stored":2390253418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738539,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2668244,"kb_used_data":2276108,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91076908,"statfs":{"total":95995035648,"available":93262753792,"internally_reserved":0,"allocated":2330734592,"data_stored":2330221641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411489,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182555,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3514788,"kb_used_data":2861660,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":90230364,"statfs":{"total":95995035648,"available":92395892736,"internally_reserved":0,"allocated":2930339840,"data_stored":2929803855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":668793287},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019030,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2283024,"kb_used_data":1696520,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":91462128,"statfs":{"total":95995035648,"available":93657219072,"internally_reserved":0,"allocated":1737236480,"data_stored":1736727738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724689,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2329083904,"data_stored":2329083904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2930900992,"data_stored":2930900992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1750269952,"data_stored":1750269952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2406809600,"data_stored":2406809600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3513516032,"data_stored":3513516032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:09:39.561 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-21T13:09:39.580 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:39.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:39.683 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:39.720 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:39.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:39.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:39 smithi204 ceph-mon[33153]: osdmap e256: 8 total, 8 up, 5 in 2024-03-21T13:09:39.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:39 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:09:39.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:39 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:09:39.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:39 smithi163 ceph-mon[29621]: osdmap e256: 8 total, 8 up, 5 in 2024-03-21T13:09:39.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:39 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:09:39.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:39 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:09:39.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:39 smithi163 ceph-mon[40675]: osdmap e256: 8 total, 8 up, 5 in 2024-03-21T13:09:39.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:39 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:09:39.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:39 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:09:39.822 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:39.860 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:39.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:39.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:39.962 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:39.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:39.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:39.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:40.100 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:40.139 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:40.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:40.241 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:40.281 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:40.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:40.383 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:40.407 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:40.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:40.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:40.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:40.528 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 47 16 76261 76245 101.373 79.875 0.0158719 0.00985947 2024-03-21T13:09:40.545 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:40.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:40.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:40.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:40.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:40.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:40.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:40.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:40 smithi163 ceph-mon[29621]: pgmap v490: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 150 active+clean, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 4.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 853 B/s rd, 55 MiB/s wr, 888 op/s; 3352/212733 objects degraded (1.576%); 9834/212733 objects misplaced (4.623%); 32 MiB/s, 512 objects/s recovering 2024-03-21T13:09:40.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:40 smithi163 ceph-mon[29621]: osdmap e257: 8 total, 8 up, 5 in 2024-03-21T13:09:40.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:40 smithi163 ceph-mon[40675]: pgmap v490: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 150 active+clean, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 4.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 853 B/s rd, 55 MiB/s wr, 888 op/s; 3352/212733 objects degraded (1.576%); 9834/212733 objects misplaced (4.623%); 32 MiB/s, 512 objects/s recovering 2024-03-21T13:09:40.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:40 smithi163 ceph-mon[40675]: osdmap e257: 8 total, 8 up, 5 in 2024-03-21T13:09:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:40 smithi204 ceph-mon[33153]: pgmap v490: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 150 active+clean, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 4.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 853 B/s rd, 55 MiB/s wr, 888 op/s; 3352/212733 objects degraded (1.576%); 9834/212733 objects misplaced (4.623%); 32 MiB/s, 512 objects/s recovering 2024-03-21T13:09:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:40 smithi204 ceph-mon[33153]: osdmap e257: 8 total, 8 up, 5 in 2024-03-21T13:09:40.787 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:40.830 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:40.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:40.931 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:40.961 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:41.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:41.002 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:41.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:41.103 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:41.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-21T13:09:41.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:41.159 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:41.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:41.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:41.294 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:41.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:41.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:41.422 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:41 smithi204 ceph-mon[33153]: Health check update: Degraded data redundancy: 3352/212733 objects degraded (1.576%), 1 pg degraded (PG_DEGRADED) 2024-03-21T13:09:41.422 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:41 smithi204 ceph-mon[33153]: osdmap e258: 8 total, 8 up, 5 in 2024-03-21T13:09:41.430 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:41.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:41.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:41.528 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 48 16 77567 77551 100.961 81.625 0.0113543 0.00989835 2024-03-21T13:09:41.532 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:41.565 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:41.568 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:41.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:41.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:41.670 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:41.705 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:41.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:41.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:41.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:41 smithi163 ceph-mon[29621]: Health check update: Degraded data redundancy: 3352/212733 objects degraded (1.576%), 1 pg degraded (PG_DEGRADED) 2024-03-21T13:09:41.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:41 smithi163 ceph-mon[29621]: osdmap e258: 8 total, 8 up, 5 in 2024-03-21T13:09:41.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:41 smithi163 ceph-mon[40675]: Health check update: Degraded data redundancy: 3352/212733 objects degraded (1.576%), 1 pg degraded (PG_DEGRADED) 2024-03-21T13:09:41.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:41 smithi163 ceph-mon[40675]: osdmap e258: 8 total, 8 up, 5 in 2024-03-21T13:09:41.808 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:41.842 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:41.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:41.944 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:41.975 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:41.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:41.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:42.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:09:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:09:42.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:42.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:42.104 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:42.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:42.189 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:09:42.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:42.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:42.240 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:42.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:42.341 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:42.381 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:42.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:42.482 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:42.510 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:42.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:42.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:42.529 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 49 16 78178 78162 99.6801 38.1875 0.0136228 0.0100181 2024-03-21T13:09:42.612 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:42.644 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:42.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:42.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:42 smithi204 ceph-mon[33153]: pgmap v492: 158 pgs: 1 active+remapped+backfill_toofull, 151 active+clean, 4 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait; 4.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.1 KiB/s rd, 50 MiB/s wr, 803 op/s; 11904/223530 objects misplaced (5.325%); 20 MiB/s, 312 objects/s recovering 2024-03-21T13:09:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:42 smithi204 ceph-mon[33153]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3352/212733 objects degraded (1.576%), 1 pg degraded) 2024-03-21T13:09:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:42 smithi204 ceph-mon[33153]: osdmap e259: 8 total, 8 up, 5 in 2024-03-21T13:09:42.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:42 smithi163 ceph-mon[40675]: pgmap v492: 158 pgs: 1 active+remapped+backfill_toofull, 151 active+clean, 4 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait; 4.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.1 KiB/s rd, 50 MiB/s wr, 803 op/s; 11904/223530 objects misplaced (5.325%); 20 MiB/s, 312 objects/s recovering 2024-03-21T13:09:42.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:42 smithi163 ceph-mon[40675]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3352/212733 objects degraded (1.576%), 1 pg degraded) 2024-03-21T13:09:42.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:42 smithi163 ceph-mon[40675]: osdmap e259: 8 total, 8 up, 5 in 2024-03-21T13:09:42.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:42 smithi163 ceph-mon[29621]: pgmap v492: 158 pgs: 1 active+remapped+backfill_toofull, 151 active+clean, 4 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait; 4.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.1 KiB/s rd, 50 MiB/s wr, 803 op/s; 11904/223530 objects misplaced (5.325%); 20 MiB/s, 312 objects/s recovering 2024-03-21T13:09:42.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:42 smithi163 ceph-mon[29621]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3352/212733 objects degraded (1.576%), 1 pg degraded) 2024-03-21T13:09:42.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:42 smithi163 ceph-mon[29621]: osdmap e259: 8 total, 8 up, 5 in 2024-03-21T13:09:42.781 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:42.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:42.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:42.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:42.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:42.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:43.018 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:43.047 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:43.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:43.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:43.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:43.182 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:43.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:43.283 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:43.308 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:43.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:43.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:43.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:43.453 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:43.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:43.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:43.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:43.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:43 smithi163 ceph-mon[29621]: osdmap e260: 8 total, 8 up, 5 in 2024-03-21T13:09:43.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:43.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:43 smithi163 ceph-mon[40675]: osdmap e260: 8 total, 8 up, 5 in 2024-03-21T13:09:43.528 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 50 15 78649 78634 98.2765 29.5 0.00299743 0.0101598 2024-03-21T13:09:43.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:43.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:43.586 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:43.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:43.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:43.718 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:43.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:43.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:43.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:43 smithi204 ceph-mon[33153]: osdmap e260: 8 total, 8 up, 5 in 2024-03-21T13:09:43.820 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:43.848 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:43.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:43.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:43.980 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:43.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:43.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:44.082 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:44.107 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:44.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:44.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:44.209 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:44.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:44.238 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:44.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:44.339 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:44.373 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:44.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:44.474 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:44.507 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:44.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:44.528 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 51 16 79088 79072 96.8864 27.375 0.00366705 0.010316 2024-03-21T13:09:44.609 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:44.644 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:44.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:44.695 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:44 smithi204 ceph-mon[33153]: pgmap v495: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean, 4 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait; 4.6 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 10444/228396 objects misplaced (4.573%); 37 MiB/s, 587 objects/s recovering 2024-03-21T13:09:44.695 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:44 smithi204 ceph-mon[33153]: osdmap e261: 8 total, 8 up, 5 in 2024-03-21T13:09:44.747 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:44.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:44 smithi163 ceph-mon[29621]: pgmap v495: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean, 4 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait; 4.6 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 10444/228396 objects misplaced (4.573%); 37 MiB/s, 587 objects/s recovering 2024-03-21T13:09:44.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:44 smithi163 ceph-mon[29621]: osdmap e261: 8 total, 8 up, 5 in 2024-03-21T13:09:44.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:44 smithi163 ceph-mon[40675]: pgmap v495: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean, 4 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait; 4.6 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 10444/228396 objects misplaced (4.573%); 37 MiB/s, 587 objects/s recovering 2024-03-21T13:09:44.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:44 smithi163 ceph-mon[40675]: osdmap e261: 8 total, 8 up, 5 in 2024-03-21T13:09:44.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:44.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:44.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:44.919 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:44.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:44.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:45.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:45.071 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:45.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:45.173 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:45.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:45.206 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:45.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:45.266 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:09:45.266 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:09:45.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.266 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25226.0:80090 9.1 e261/246 9:87c9bd03:::benchmark_data_smithi163_131650_object80089:head v 261'4200, mlcod=261'4198)", 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:45.261186+0000", 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0027286469999999998, 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027589759999999998, 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261186+0000", 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261186+0000", 2024-03-21T13:09:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261186+0000", 2024-03-21T13:09:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1099999999999996e-07 2024-03-21T13:09:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261231+0000", 2024-03-21T13:09:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4441999999999999e-05 2024-03-21T13:09:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261243+0000", 2024-03-21T13:09:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.221e-05 2024-03-21T13:09:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261250+0000", 2024-03-21T13:09:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0299999999999996e-06 2024-03-21T13:09:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261310+0000", 2024-03-21T13:09:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9837999999999999e-05 2024-03-21T13:09:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261330+0000", 2024-03-21T13:09:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0772e-05 2024-03-21T13:09:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:80091 9.b 9:d46ec3e3:::benchmark_data_smithi163_131650_object80090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-03-21T13:09:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:45.261290+0000", 2024-03-21T13:09:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0026238059999999998, 2024-03-21T13:09:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027345889999999999, 2024-03-21T13:09:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:09:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:09:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:09:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:09:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 80091 2024-03-21T13:09:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261290+0000", 2024-03-21T13:09:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261290+0000", 2024-03-21T13:09:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261296+0000", 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1499999999999998e-06 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261339+0000", 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3126e-05 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261348+0000", 2024-03-21T13:09:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.003e-06 2024-03-21T13:09:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261355+0000", 2024-03-21T13:09:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8940000000000004e-06 2024-03-21T13:09:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261530+0000", 2024-03-21T13:09:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000175756 2024-03-21T13:09:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261662+0000", 2024-03-21T13:09:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013168500000000001 2024-03-21T13:09:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:09:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261801+0000", 2024-03-21T13:09:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013853800000000001 2024-03-21T13:09:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25226.0:80089 9.11 e261/248 9:8b7d1c4c:::benchmark_data_smithi163_131650_object80088:head v 261'4235, mlcod=261'4233)", 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:45.261432+0000", 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0024825369999999999, 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026748340000000001, 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261432+0000", 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261432+0000", 2024-03-21T13:09:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261432+0000", 2024-03-21T13:09:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0600000000000001e-07 2024-03-21T13:09:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261453+0000", 2024-03-21T13:09:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0733999999999998e-05 2024-03-21T13:09:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261460+0000", 2024-03-21T13:09:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8759999999999999e-06 2024-03-21T13:09:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.261466+0000", 2024-03-21T13:09:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7649999999999999e-06 2024-03-21T13:09:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.262609+0000", 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011437730000000001 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.262626+0000", 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6728e-05 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25226.0:80095 9.a e261/234 9:54e02cbd:::benchmark_data_smithi163_131650_object80094:head v 261'5007, mlcod=261'5004)", 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:45.263394+0000", 2024-03-21T13:09:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.00052044400000000001, 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00077032900000000004, 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263394+0000", 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263394+0000", 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263395+0000", 2024-03-21T13:09:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8100000000000002e-07 2024-03-21T13:09:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263420+0000", 2024-03-21T13:09:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5570000000000001e-05 2024-03-21T13:09:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263433+0000", 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2578000000000001e-05 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263440+0000", 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2300000000000002e-06 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263493+0000", 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3508000000000003e-05 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263512+0000", 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8091999999999999e-05 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25226.0:80096 9.7 e261/246 9:e52cff89:::benchmark_data_smithi163_131650_object80095:head v 261'4115, mlcod=261'4113)", 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:45.263888+0000", 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 2.6628999999999999e-05, 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033696399999999998, 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:09:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263888+0000", 2024-03-21T13:09:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263888+0000", 2024-03-21T13:09:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263888+0000", 2024-03-21T13:09:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1999999999999999e-07 2024-03-21T13:09:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263911+0000", 2024-03-21T13:09:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2402999999999999e-05 2024-03-21T13:09:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263924+0000", 2024-03-21T13:09:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3006999999999999e-05 2024-03-21T13:09:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263930+0000", 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5610000000000004e-06 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.263980+0000", 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9317000000000002e-05 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.264000+0000", 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0894e-05 2024-03-21T13:09:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25226.0:80088 9.16 e261/255 9:6f834503:::benchmark_data_smithi163_131650_object80087:head v 261'4060, mlcod=261'4058)", 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:45.259161+0000", 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0047530619999999997, 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051222760000000003, 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:09:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.259161+0000", 2024-03-21T13:09:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:09:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.259161+0000", 2024-03-21T13:09:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:09:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:09:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.259162+0000", 2024-03-21T13:09:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1289999999999999e-06 2024-03-21T13:09:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:09:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.259188+0000", 2024-03-21T13:09:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5445000000000001e-05 2024-03-21T13:09:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:09:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.259203+0000", 2024-03-21T13:09:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5594999999999999e-05 2024-03-21T13:09:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.259211+0000", 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6310000000000003e-06 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.259294+0000", 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2538000000000002e-05 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:45.259315+0000", 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1950000000000002e-05 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:09:45.292 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:45.292 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:09:45.292 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-03-21T13:09:45.292 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 6 2024-03-21T13:09:45.292 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:09:45.306 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:45.341 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:45.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:45.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:45.443 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:45.480 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:45.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:45.528 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 52 16 80482 80466 96.6985 87.125 0.0137384 0.0103367 2024-03-21T13:09:45.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:45.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:45.615 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:45.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:45.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:45.754 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:45.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:45.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:45.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:45.893 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:45.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:45.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:45.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:46.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:09:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:09:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:09:46.037 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:46.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:46.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:46.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-21T13:09:46.139 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:46.171 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:46.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:46.272 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:46.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:46.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:46.341 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:09:46.410 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:46.444 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:46.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:46.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[29621]: pgmap v498: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean, 4 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait; 4.6 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 25 MiB/s wr, 405 op/s; 10444/228396 objects misplaced (4.573%); 19 MiB/s, 299 objects/s recovering 2024-03-21T13:09:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:09:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:09:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1102614965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:09:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:09:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1102614965' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:09:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[29621]: osdmap e262: 8 total, 8 up, 5 in 2024-03-21T13:09:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[40675]: pgmap v498: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean, 4 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait; 4.6 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 25 MiB/s wr, 405 op/s; 10444/228396 objects misplaced (4.573%); 19 MiB/s, 299 objects/s recovering 2024-03-21T13:09:46.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:09:46.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:09:46.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1102614965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:09:46.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:09:46.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1102614965' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:09:46.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:46 smithi163 ceph-mon[40675]: osdmap e262: 8 total, 8 up, 5 in 2024-03-21T13:09:46.528 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 53 16 81785 81769 96.4105 81.4375 0.00920433 0.010368 2024-03-21T13:09:46.530 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:46.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:46.609 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:46.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:46.710 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:46.715 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:46 smithi204 ceph-mon[33153]: pgmap v498: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean, 4 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait; 4.6 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 25 MiB/s wr, 405 op/s; 10444/228396 objects misplaced (4.573%); 19 MiB/s, 299 objects/s recovering 2024-03-21T13:09:46.715 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:46 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:09:46.715 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:46 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:09:46.715 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:46 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1102614965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:09:46.715 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:46 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:09:46.715 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:46 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1102614965' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:09:46.715 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:46 smithi204 ceph-mon[33153]: osdmap e262: 8 total, 8 up, 5 in 2024-03-21T13:09:46.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:46.743 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:46.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:46.844 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:46.880 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:46.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:46.982 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:47.016 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:47.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:47.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:47.118 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:47.160 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:47.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:47.262 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:47.297 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:47.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:47.399 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:47.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:47.439 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:47.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:47.529 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 54 16 83189 83173 96.2499 87.75 0.0129081 0.0103847 2024-03-21T13:09:47.540 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:47.571 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:47.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:47.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:47.636 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:47.674 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:47.709 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:47.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:47.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:47 smithi204 ceph-mon[33153]: 9.14 scrub starts 2024-03-21T13:09:47.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:47 smithi163 ceph-mon[29621]: 9.14 scrub starts 2024-03-21T13:09:47.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:47 smithi163 ceph-mon[40675]: 9.14 scrub starts 2024-03-21T13:09:47.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:47.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:47.836 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:47.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:47.938 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:47.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:47.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:47.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:48.073 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:48.100 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:48.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:48.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:48.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:48.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:48.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:48.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:48.357 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:48.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:48.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:48.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:48.460 INFO:teuthology.orchestra.run.smithi163.stderr:set pool 1 pgp_num to 1 2024-03-21T13:09:48.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:48.494 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:48.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:48.529 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 55 16 83753 83737 95.1394 35.25 0.00886866 0.0105062 2024-03-21T13:09:48.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:48.634 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:48.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:48.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:48.736 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:48.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:48 smithi204 ceph-mon[33153]: pgmap v500: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean, 1 active+recovering+undersized+remapped; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 918 op/s; 6414/240738 objects misplaced (2.664%); 50 MiB/s, 796 objects/s recovering 2024-03-21T13:09:48.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:48 smithi204 ceph-mon[33153]: 9.14 scrub ok 2024-03-21T13:09:48.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:48 smithi204 ceph-mon[33153]: osdmap e263: 8 total, 8 up, 5 in 2024-03-21T13:09:48.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:48 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1613566157' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-21T13:09:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:48 smithi163 ceph-mon[29621]: pgmap v500: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean, 1 active+recovering+undersized+remapped; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 918 op/s; 6414/240738 objects misplaced (2.664%); 50 MiB/s, 796 objects/s recovering 2024-03-21T13:09:48.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:48 smithi163 ceph-mon[29621]: 9.14 scrub ok 2024-03-21T13:09:48.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:48 smithi163 ceph-mon[29621]: osdmap e263: 8 total, 8 up, 5 in 2024-03-21T13:09:48.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:48 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1613566157' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-21T13:09:48.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:48 smithi163 ceph-mon[40675]: pgmap v500: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean, 1 active+recovering+undersized+remapped; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 918 op/s; 6414/240738 objects misplaced (2.664%); 50 MiB/s, 796 objects/s recovering 2024-03-21T13:09:48.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:48 smithi163 ceph-mon[40675]: 9.14 scrub ok 2024-03-21T13:09:48.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:48 smithi163 ceph-mon[40675]: osdmap e263: 8 total, 8 up, 5 in 2024-03-21T13:09:48.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:48 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1613566157' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-21T13:09:48.773 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:48.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:48.875 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:48.910 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:48.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:48.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:49.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:49.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:49.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:49.159 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:49.196 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:49.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:49.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:49.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:49.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:49.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:49.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:49.475 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:49.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:49.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:49.529 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 56 16 85172 85156 95.0241 88.6875 0.0153979 0.0105168 2024-03-21T13:09:49.549 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:09:49.549 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-03-21T13:09:49.549 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-03-21T13:09:49.549 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-03-21T13:09:49.550 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:09:49.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:49.618 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:49.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:49.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:49 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1613566157' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-21T13:09:49.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:49 smithi163 ceph-mon[40675]: osdmap e264: 8 total, 8 up, 5 in 2024-03-21T13:09:49.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:49 smithi163 ceph-mon[40675]: 9.8 scrub starts 2024-03-21T13:09:49.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:49 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1613566157' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-21T13:09:49.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:49 smithi163 ceph-mon[29621]: osdmap e264: 8 total, 8 up, 5 in 2024-03-21T13:09:49.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:49 smithi163 ceph-mon[29621]: 9.8 scrub starts 2024-03-21T13:09:49.630 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:09:49.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:49.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:49 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1613566157' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-21T13:09:49.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:49 smithi204 ceph-mon[33153]: osdmap e264: 8 total, 8 up, 5 in 2024-03-21T13:09:49.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:49 smithi204 ceph-mon[33153]: 9.8 scrub starts 2024-03-21T13:09:49.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:49.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:49.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:49.838 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:09:49.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:49.881 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:09:49.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:49.901 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:49.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:49.921 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:09:49.957 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:09:49.998 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:09:50.003 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:50.035 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:09:50.038 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:50.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:50.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:50.075 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:09:50.110 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:09:50.114 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:09:50.114 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:09:50.114 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:09:50.114 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:09:50.114 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:09:50.114 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:09:50.114 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:09:50.114 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:09:50.141 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:50.188 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:50.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:50.290 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:50.322 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-21T13:09:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:50.323 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:50.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:50.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:50.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:50.465 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:50.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:50.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:50 smithi163 ceph-mon[40675]: pgmap v503: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean, 1 active+recovering+undersized+remapped; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 918 op/s; 6414/240738 objects misplaced (2.664%); 50 MiB/s, 796 objects/s recovering 2024-03-21T13:09:50.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:50 smithi163 ceph-mon[40675]: 9.8 scrub ok 2024-03-21T13:09:50.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:50 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1690428783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:50.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:50.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:09:50.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:50 smithi163 ceph-mon[40675]: 9.1 scrub starts 2024-03-21T13:09:50.529 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 57 16 86544 86528 94.8613 85.75 0.00622345 0.0105368 2024-03-21T13:09:50.566 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:50.596 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:50.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:50.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:50.730 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:50.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:50.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:50 smithi204 ceph-mon[33153]: pgmap v503: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean, 1 active+recovering+undersized+remapped; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 918 op/s; 6414/240738 objects misplaced (2.664%); 50 MiB/s, 796 objects/s recovering 2024-03-21T13:09:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:50 smithi204 ceph-mon[33153]: 9.8 scrub ok 2024-03-21T13:09:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:50 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1690428783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:09:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:50 smithi204 ceph-mon[33153]: 9.1 scrub starts 2024-03-21T13:09:50.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:50 smithi163 ceph-mon[29621]: pgmap v503: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean, 1 active+recovering+undersized+remapped; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 918 op/s; 6414/240738 objects misplaced (2.664%); 50 MiB/s, 796 objects/s recovering 2024-03-21T13:09:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:50 smithi163 ceph-mon[29621]: 9.8 scrub ok 2024-03-21T13:09:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:50 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1690428783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:09:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:09:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:50 smithi163 ceph-mon[29621]: 9.1 scrub starts 2024-03-21T13:09:50.767 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:50.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:50.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:50.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:50.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:50.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:51.022 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:51.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:51.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:51.088 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:51.124 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:51.164 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:51.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:51.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:51.266 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:51.342 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:51.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:51.443 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:51.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:51.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:51.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:51.529 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 58 16 87951 87935 94.7418 87.9375 0.0146868 0.0105499 2024-03-21T13:09:51.587 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:51.616 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:51.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:51.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:51 smithi204 ceph-mon[33153]: 9.1 scrub ok 2024-03-21T13:09:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:51 smithi204 ceph-mon[33153]: 9.2 scrub starts 2024-03-21T13:09:51.757 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:51.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:51.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:51 smithi163 ceph-mon[29621]: 9.1 scrub ok 2024-03-21T13:09:51.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:51 smithi163 ceph-mon[29621]: 9.2 scrub starts 2024-03-21T13:09:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:51 smithi163 ceph-mon[40675]: 9.1 scrub ok 2024-03-21T13:09:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:51 smithi163 ceph-mon[40675]: 9.2 scrub starts 2024-03-21T13:09:51.859 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:51.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:51.899 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:51.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:52.000 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:52.034 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:52.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:52.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:52.169 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:52.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:52.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:52.253 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:09:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:09:52.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:52.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:52.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:52.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:52.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:52.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:52.530 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 59 16 89310 89294 94.5742 84.9375 0.0183305 0.0105696 2024-03-21T13:09:52.566 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:52.600 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:52.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:52.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:52.616 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:52 smithi204 ceph-mon[33153]: pgmap v504: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 154 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.7 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 5139/253233 objects misplaced (2.029%); 50 MiB/s, 796 objects/s recovering 2024-03-21T13:09:52.617 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:52 smithi204 ceph-mon[33153]: 9.2 scrub ok 2024-03-21T13:09:52.617 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:52 smithi204 ceph-mon[33153]: 9.0 scrub starts 2024-03-21T13:09:52.617 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:52 smithi204 ceph-mon[33153]: 9.4 scrub starts 2024-03-21T13:09:52.617 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:52 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:52.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:52.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:52.741 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:52.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:52 smithi163 ceph-mon[29621]: pgmap v504: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 154 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.7 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 5139/253233 objects misplaced (2.029%); 50 MiB/s, 796 objects/s recovering 2024-03-21T13:09:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:52 smithi163 ceph-mon[29621]: 9.2 scrub ok 2024-03-21T13:09:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:52 smithi163 ceph-mon[29621]: 9.0 scrub starts 2024-03-21T13:09:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:52 smithi163 ceph-mon[29621]: 9.4 scrub starts 2024-03-21T13:09:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:52 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:52.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:52 smithi163 ceph-mon[40675]: pgmap v504: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 154 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.7 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 5139/253233 objects misplaced (2.029%); 50 MiB/s, 796 objects/s recovering 2024-03-21T13:09:52.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:52 smithi163 ceph-mon[40675]: 9.2 scrub ok 2024-03-21T13:09:52.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:52 smithi163 ceph-mon[40675]: 9.0 scrub starts 2024-03-21T13:09:52.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:52 smithi163 ceph-mon[40675]: 9.4 scrub starts 2024-03-21T13:09:52.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:52 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:09:52.834 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:09:52.834 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:09:52.843 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:52.875 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:52.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:52.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:52.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:53.017 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:53.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:53.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:09:53.119 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:53.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:53.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:53.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:53.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:53.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:53.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:53.400 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:53.456 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:53.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:53.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:53 smithi163 ceph-mon[40675]: 9.0 scrub ok 2024-03-21T13:09:53.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:53 smithi163 ceph-mon[40675]: 9.4 scrub ok 2024-03-21T13:09:53.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:53 smithi163 ceph-mon[40675]: 9.6 scrub starts 2024-03-21T13:09:53.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:53 smithi163 ceph-mon[40675]: pgmap v505: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 154 active+clean; 5.3 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 789 B/s rd, 0 op/s; 4949/258849 objects misplaced (1.912%) 2024-03-21T13:09:53.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:53 smithi163 ceph-mon[40675]: 9.19 scrub starts 2024-03-21T13:09:53.504 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:53.530 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:09:53.531139+0000 min lat: 0.00122048 max lat: 2.05061 avg lat: 0.0105989 2024-03-21T13:09:53.531 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:09:53.531 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 60 16 90569 90553 94.3094 78.6875 0.00650718 0.0105989 2024-03-21T13:09:53.558 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:53.591 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:53.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:53.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:53.693 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:09:53.732 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:53.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:09:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:53 smithi204 ceph-mon[33153]: 9.0 scrub ok 2024-03-21T13:09:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:53 smithi204 ceph-mon[33153]: 9.4 scrub ok 2024-03-21T13:09:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:53 smithi204 ceph-mon[33153]: 9.6 scrub starts 2024-03-21T13:09:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:53 smithi204 ceph-mon[33153]: pgmap v505: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 154 active+clean; 5.3 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 789 B/s rd, 0 op/s; 4949/258849 objects misplaced (1.912%) 2024-03-21T13:09:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:53 smithi204 ceph-mon[33153]: 9.19 scrub starts 2024-03-21T13:09:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:53 smithi163 ceph-mon[29621]: 9.0 scrub ok 2024-03-21T13:09:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:53 smithi163 ceph-mon[29621]: 9.4 scrub ok 2024-03-21T13:09:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:53 smithi163 ceph-mon[29621]: 9.6 scrub starts 2024-03-21T13:09:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:53 smithi163 ceph-mon[29621]: pgmap v505: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 154 active+clean; 5.3 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 789 B/s rd, 0 op/s; 4949/258849 objects misplaced (1.912%) 2024-03-21T13:09:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:53 smithi163 ceph-mon[29621]: 9.19 scrub starts 2024-03-21T13:09:53.834 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:53.867 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:53.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:53.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:53.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-21T13:09:53.968 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:54.004 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:54.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:54.106 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:54.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:54.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:54.250 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:54.286 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:54.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:54.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:54.389 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:54.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:54.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:54.444 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:54.502 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:09:54.530 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 61 15 91982 91967 94.212 88.375 0.0114798 0.0106103 2024-03-21T13:09:54.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:54.591 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:54.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:54.692 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:54.726 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:54.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:54.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:54 smithi204 ceph-mon[33153]: 9.6 scrub ok 2024-03-21T13:09:54.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:54 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1228690973' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:09:54.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:54 smithi204 ceph-mon[33153]: 9.19 scrub ok 2024-03-21T13:09:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:54 smithi163 ceph-mon[29621]: 9.6 scrub ok 2024-03-21T13:09:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:54 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1228690973' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:09:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:54 smithi163 ceph-mon[29621]: 9.19 scrub ok 2024-03-21T13:09:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:54 smithi163 ceph-mon[40675]: 9.6 scrub ok 2024-03-21T13:09:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:54 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1228690973' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:09:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:54 smithi163 ceph-mon[40675]: 9.19 scrub ok 2024-03-21T13:09:54.828 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:54.861 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:54.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:54.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:55.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:55.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:55.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:55.155 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:55.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:55.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:55.207 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:09:55.207 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:09:55.207 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-03-21T13:09:55.207 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd in 2 2024-03-21T13:09:55.257 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:55.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:55.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:55.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:55.431 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:55.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:55.491 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:55 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:09:55.491 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:55 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1228690973' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:09:55.491 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:55 smithi163 ceph-mon[29621]: osdmap e265: 8 total, 8 up, 5 in 2024-03-21T13:09:55.491 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:55 smithi163 ceph-mon[29621]: pgmap v507: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 154 active+clean; 5.3 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 694 B/s rd, 0 op/s; 4949/258849 objects misplaced (1.912%) 2024-03-21T13:09:55.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:55 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:09:55.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:55 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1228690973' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:09:55.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:55 smithi163 ceph-mon[40675]: osdmap e265: 8 total, 8 up, 5 in 2024-03-21T13:09:55.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:55 smithi163 ceph-mon[40675]: pgmap v507: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 154 active+clean; 5.3 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 694 B/s rd, 0 op/s; 4949/258849 objects misplaced (1.912%) 2024-03-21T13:09:55.530 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 62 16 93441 93425 94.1621 91.125 0.00800348 0.0106159 2024-03-21T13:09:55.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:55.571 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:55.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:55.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:55.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:55.713 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:55.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:55.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:55.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:55 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:09:55.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:55 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1228690973' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:09:55.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:55 smithi204 ceph-mon[33153]: osdmap e265: 8 total, 8 up, 5 in 2024-03-21T13:09:55.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:55 smithi204 ceph-mon[33153]: pgmap v507: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 154 active+clean; 5.3 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 694 B/s rd, 0 op/s; 4949/258849 objects misplaced (1.912%) 2024-03-21T13:09:55.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:09:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:09:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:09:55.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:55.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:55.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:55.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:55.900 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:55.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:56.006 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:56.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:56.107 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:56.141 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:56.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:56.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:56.272 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:56.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:56.374 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:56.413 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:56.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:56.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:56.515 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:56.531 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 63 16 94854 94838 94.069 88.3125 0.00613597 0.0106127 2024-03-21T13:09:56.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:56.550 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:56.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:56.554 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:09:56.584 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:09:56.606 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:09:56.606 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:09:56.609 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:09:56.637 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:09:56.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:56 smithi163 ceph-mon[29621]: 3.7 scrub starts 2024-03-21T13:09:56.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:56 smithi163 ceph-mon[29621]: 3.7 scrub ok 2024-03-21T13:09:56.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:56 smithi163 ceph-mon[29621]: osdmap e266: 8 total, 8 up, 5 in 2024-03-21T13:09:56.638 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:56 smithi163 ceph-mon[40675]: 3.7 scrub starts 2024-03-21T13:09:56.638 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:56 smithi163 ceph-mon[40675]: 3.7 scrub ok 2024-03-21T13:09:56.638 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:56 smithi163 ceph-mon[40675]: osdmap e266: 8 total, 8 up, 5 in 2024-03-21T13:09:56.651 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:56.662 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:56 smithi204 ceph-mon[33153]: 3.7 scrub starts 2024-03-21T13:09:56.662 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:56 smithi204 ceph-mon[33153]: 3.7 scrub ok 2024-03-21T13:09:56.662 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:56 smithi204 ceph-mon[33153]: osdmap e266: 8 total, 8 up, 5 in 2024-03-21T13:09:56.665 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:09:56.680 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:56.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:56.694 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:09:56.721 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:09:56.747 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:09:56.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:09:56.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:09:56.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:09:56.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:09:56.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:09:56.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:09:56.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:09:56.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:09:56.782 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:56.811 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:56.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:56.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:56.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:56.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:56.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:56.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:57.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:57.064 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:57.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:57.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:57.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:57.191 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:57.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:57.292 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:57.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:57.317 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:57.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:09:57.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-21T13:09:57.373 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:09:57.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:57.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:57.467 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:57.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:57.531 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 64 16 95348 95332 93.0816 30.875 0.00545667 0.0107387 2024-03-21T13:09:57.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:57.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:57.610 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:57.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:57.711 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:09:57.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:57 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2841638068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:57.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:57 smithi204 ceph-mon[33153]: pgmap v509: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 155 active+clean; 5.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 3954/279819 objects misplaced (1.413%) 2024-03-21T13:09:57.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:57 smithi204 ceph-mon[33153]: 5.1c scrub starts 2024-03-21T13:09:57.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:57 smithi204 ceph-mon[33153]: 5.1c scrub ok 2024-03-21T13:09:57.755 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:57 smithi204 ceph-mon[33153]: osdmap e267: 8 total, 8 up, 5 in 2024-03-21T13:09:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:57.760 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:57.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:09:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:57 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2841638068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:57 smithi163 ceph-mon[29621]: pgmap v509: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 155 active+clean; 5.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 3954/279819 objects misplaced (1.413%) 2024-03-21T13:09:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:57 smithi163 ceph-mon[29621]: 5.1c scrub starts 2024-03-21T13:09:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:57 smithi163 ceph-mon[29621]: 5.1c scrub ok 2024-03-21T13:09:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:57 smithi163 ceph-mon[29621]: osdmap e267: 8 total, 8 up, 5 in 2024-03-21T13:09:57.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:57 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2841638068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:09:57.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:57 smithi163 ceph-mon[40675]: pgmap v509: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 155 active+clean; 5.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 3954/279819 objects misplaced (1.413%) 2024-03-21T13:09:57.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:57 smithi163 ceph-mon[40675]: 5.1c scrub starts 2024-03-21T13:09:57.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:57 smithi163 ceph-mon[40675]: 5.1c scrub ok 2024-03-21T13:09:57.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:57 smithi163 ceph-mon[40675]: osdmap e267: 8 total, 8 up, 5 in 2024-03-21T13:09:57.776 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:57.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:57.905 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:57.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:57.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:58.007 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:58.042 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:58.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:58.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:58.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:58.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:58.286 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:58.338 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:58.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:58.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:58.476 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:58.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:58.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:58.531 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 65 16 96798 96782 93.0437 90.625 0.0122952 0.0107427 2024-03-21T13:09:58.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:58.614 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:58.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:58.716 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:58.746 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:58.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:58 smithi163 ceph-mon[29621]: 4.12 scrub starts 2024-03-21T13:09:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:58 smithi163 ceph-mon[29621]: 4.12 scrub ok 2024-03-21T13:09:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:58 smithi163 ceph-mon[29621]: 6.7 scrub starts 2024-03-21T13:09:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:58 smithi163 ceph-mon[29621]: 6.7 scrub ok 2024-03-21T13:09:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:58 smithi163 ceph-mon[40675]: 4.12 scrub starts 2024-03-21T13:09:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:58 smithi163 ceph-mon[40675]: 4.12 scrub ok 2024-03-21T13:09:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:58 smithi163 ceph-mon[40675]: 6.7 scrub starts 2024-03-21T13:09:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:58 smithi163 ceph-mon[40675]: 6.7 scrub ok 2024-03-21T13:09:58.849 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:09:58.876 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:58.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:09:58.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:09:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:59.002 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:59.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:09:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:58 smithi204 ceph-mon[33153]: 4.12 scrub starts 2024-03-21T13:09:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:58 smithi204 ceph-mon[33153]: 4.12 scrub ok 2024-03-21T13:09:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:58 smithi204 ceph-mon[33153]: 6.7 scrub starts 2024-03-21T13:09:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:58 smithi204 ceph-mon[33153]: 6.7 scrub ok 2024-03-21T13:09:59.103 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:59.127 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:59.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:59.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:59.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:59.254 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:59.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:59.355 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:09:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:59.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:59.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:09:59.480 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:09:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:59.504 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:09:59.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:09:59.531 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 66 16 97202 97186 92.0166 25.25 0.00263651 0.0107424 2024-03-21T13:09:59.604 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:59.640 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:59.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:59.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:59.741 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:09:59.774 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:59.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:59.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:09:59.877 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:09:59.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:09:59.903 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:09:59.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:00.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:59 smithi204 ceph-mon[33153]: 9.3 scrub starts 2024-03-21T13:10:00.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:59 smithi204 ceph-mon[33153]: osdmap e268: 8 total, 8 up, 5 in 2024-03-21T13:10:00.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:59 smithi204 ceph-mon[33153]: 3.18 scrub starts 2024-03-21T13:10:00.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:59 smithi204 ceph-mon[33153]: 3.18 scrub ok 2024-03-21T13:10:00.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:59 smithi204 ceph-mon[33153]: pgmap v512: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 155 active+clean; 5.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 3954/279819 objects misplaced (1.413%) 2024-03-21T13:10:00.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:09:59 smithi204 ceph-mon[33153]: 9.3 scrub ok 2024-03-21T13:10:00.003 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:00.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:59 smithi163 ceph-mon[29621]: 9.3 scrub starts 2024-03-21T13:10:00.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:59 smithi163 ceph-mon[29621]: osdmap e268: 8 total, 8 up, 5 in 2024-03-21T13:10:00.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:59 smithi163 ceph-mon[29621]: 3.18 scrub starts 2024-03-21T13:10:00.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:59 smithi163 ceph-mon[29621]: 3.18 scrub ok 2024-03-21T13:10:00.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:59 smithi163 ceph-mon[29621]: pgmap v512: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 155 active+clean; 5.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 3954/279819 objects misplaced (1.413%) 2024-03-21T13:10:00.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:09:59 smithi163 ceph-mon[29621]: 9.3 scrub ok 2024-03-21T13:10:00.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:59 smithi163 ceph-mon[40675]: 9.3 scrub starts 2024-03-21T13:10:00.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:59 smithi163 ceph-mon[40675]: osdmap e268: 8 total, 8 up, 5 in 2024-03-21T13:10:00.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:59 smithi163 ceph-mon[40675]: 3.18 scrub starts 2024-03-21T13:10:00.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:59 smithi163 ceph-mon[40675]: 3.18 scrub ok 2024-03-21T13:10:00.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:59 smithi163 ceph-mon[40675]: pgmap v512: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 155 active+clean; 5.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 3954/279819 objects misplaced (1.413%) 2024-03-21T13:10:00.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:09:59 smithi163 ceph-mon[40675]: 9.3 scrub ok 2024-03-21T13:10:00.037 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:00.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:00.139 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:00.494 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:00.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:00.531 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 67 16 98348 98332 91.7122 71.625 0.0066604 0.0108971 2024-03-21T13:10:00.596 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:00.628 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:00.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:00.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:00 smithi204 ceph-mon[33153]: 9.12 scrub starts 2024-03-21T13:10:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:00 smithi204 ceph-mon[33153]: osdmap e269: 8 total, 8 up, 5 in 2024-03-21T13:10:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:00 smithi204 ceph-mon[33153]: 9.9 scrub starts 2024-03-21T13:10:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:00 smithi204 ceph-mon[33153]: 9.12 scrub ok 2024-03-21T13:10:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:00 smithi204 ceph-mon[33153]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-03-21T13:10:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:00 smithi204 ceph-mon[33153]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-03-21T13:10:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:00 smithi204 ceph-mon[33153]: pg 9.18 is active+remapped+backfill_toofull, acting [0,7,3] 2024-03-21T13:10:00.754 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:00.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:00.856 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:00.885 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:00.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:00.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:00.987 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:01.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[40675]: 9.12 scrub starts 2024-03-21T13:10:01.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[40675]: osdmap e269: 8 total, 8 up, 5 in 2024-03-21T13:10:01.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[40675]: 9.9 scrub starts 2024-03-21T13:10:01.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[40675]: 9.12 scrub ok 2024-03-21T13:10:01.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[40675]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-03-21T13:10:01.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[40675]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-03-21T13:10:01.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[40675]: pg 9.18 is active+remapped+backfill_toofull, acting [0,7,3] 2024-03-21T13:10:01.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[29621]: 9.12 scrub starts 2024-03-21T13:10:01.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[29621]: osdmap e269: 8 total, 8 up, 5 in 2024-03-21T13:10:01.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[29621]: 9.9 scrub starts 2024-03-21T13:10:01.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[29621]: 9.12 scrub ok 2024-03-21T13:10:01.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[29621]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-03-21T13:10:01.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[29621]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-03-21T13:10:01.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:00 smithi163 ceph-mon[29621]: pg 9.18 is active+remapped+backfill_toofull, acting [0,7,3] 2024-03-21T13:10:01.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:01.020 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:01.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:01.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:01.147 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:01.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:01.248 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:01.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:01.285 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:01.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:01.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:01.415 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:01.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:01.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:01.531 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 68 16 99298 99282 91.2366 59.375 0.00405454 0.0109559 2024-03-21T13:10:01.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:01.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:01.656 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:01.680 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:01.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:01.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:01.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[29621]: 3.3 scrub starts 2024-03-21T13:10:01.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[29621]: 3.3 scrub ok 2024-03-21T13:10:01.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[29621]: 9.7 scrub starts 2024-03-21T13:10:01.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[29621]: 9.9 scrub ok 2024-03-21T13:10:01.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[29621]: pgmap v514: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 5.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 3039/290079 objects misplaced (1.048%) 2024-03-21T13:10:01.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[29621]: 9.16 scrub starts 2024-03-21T13:10:01.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[29621]: 4.1b scrub starts 2024-03-21T13:10:01.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[29621]: 4.1b scrub ok 2024-03-21T13:10:01.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[29621]: osdmap e270: 8 total, 8 up, 5 in 2024-03-21T13:10:01.708 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[40675]: 3.3 scrub starts 2024-03-21T13:10:01.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[40675]: 3.3 scrub ok 2024-03-21T13:10:01.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[40675]: 9.7 scrub starts 2024-03-21T13:10:01.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[40675]: 9.9 scrub ok 2024-03-21T13:10:01.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[40675]: pgmap v514: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 5.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 3039/290079 objects misplaced (1.048%) 2024-03-21T13:10:01.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[40675]: 9.16 scrub starts 2024-03-21T13:10:01.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[40675]: 4.1b scrub starts 2024-03-21T13:10:01.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[40675]: 4.1b scrub ok 2024-03-21T13:10:01.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-mon[40675]: osdmap e270: 8 total, 8 up, 5 in 2024-03-21T13:10:01.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:01.806 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:01.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:01.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:01.857 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:01 smithi204 ceph-mon[33153]: 3.3 scrub starts 2024-03-21T13:10:01.857 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:01 smithi204 ceph-mon[33153]: 3.3 scrub ok 2024-03-21T13:10:01.857 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:01 smithi204 ceph-mon[33153]: 9.7 scrub starts 2024-03-21T13:10:01.857 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:01 smithi204 ceph-mon[33153]: 9.9 scrub ok 2024-03-21T13:10:01.857 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:01 smithi204 ceph-mon[33153]: pgmap v514: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 5.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 3039/290079 objects misplaced (1.048%) 2024-03-21T13:10:01.858 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:01 smithi204 ceph-mon[33153]: 9.16 scrub starts 2024-03-21T13:10:01.858 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:01 smithi204 ceph-mon[33153]: 4.1b scrub starts 2024-03-21T13:10:01.858 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:01 smithi204 ceph-mon[33153]: 4.1b scrub ok 2024-03-21T13:10:01.858 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:01 smithi204 ceph-mon[33153]: osdmap e270: 8 total, 8 up, 5 in 2024-03-21T13:10:01.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:01.947 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:01.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:01.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:01.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:01 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a[29617]: 2024-03-21T13:10:01.793+0000 7f8e1b000700 -1 mon.a@0(leader).osd e270 definitely_dead 0 2024-03-21T13:10:02.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:02.085 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:02.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:02.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:02.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:02.221 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:02.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:02.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:02.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:10:01 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:10:02.323 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:02.359 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:02.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:02.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:02.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:02.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:02.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:02.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:02.534 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 69 16 100335 100319 90.8536 64.8125 0.012799 0.0110023 2024-03-21T13:10:02.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:02.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:02.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:02.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:02.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:02.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:02.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:02.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:02 smithi163 ceph-mon[40675]: 9.5 scrub starts 2024-03-21T13:10:02.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:02 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1685568656' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-21T13:10:02.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:02 smithi163 ceph-mon[40675]: 9.16 scrub ok 2024-03-21T13:10:02.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:02 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:02.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:02 smithi163 ceph-mon[40675]: 9.7 scrub ok 2024-03-21T13:10:02.806 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:02 smithi163 ceph-mon[29621]: 9.5 scrub starts 2024-03-21T13:10:02.806 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:02 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1685568656' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-21T13:10:02.806 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:02 smithi163 ceph-mon[29621]: 9.16 scrub ok 2024-03-21T13:10:02.806 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:02 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:02.807 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:02 smithi163 ceph-mon[29621]: 9.7 scrub ok 2024-03-21T13:10:02.807 INFO:teuthology.orchestra.run.smithi163.stderr:marked in osd.2. 2024-03-21T13:10:02.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:02.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:02.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:02.981 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:03.007 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:02 smithi204 ceph-mon[33153]: 9.5 scrub starts 2024-03-21T13:10:03.007 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:02 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1685568656' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-21T13:10:03.007 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:02 smithi204 ceph-mon[33153]: 9.16 scrub ok 2024-03-21T13:10:03.008 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:02 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:03.008 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:02 smithi204 ceph-mon[33153]: 9.7 scrub ok 2024-03-21T13:10:03.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:03.009 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:03.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:03.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:03.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:03.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:03.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:03.236 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:03.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:03.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:03.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:03.387 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:03.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:03.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:03.514 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:03.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:03.531 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 70 16 100420 100404 89.6316 5.3125 0.00412016 0.0110032 2024-03-21T13:10:03.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:03.646 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:03.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:03.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:03.747 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:03.772 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:03.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:03.873 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:03.897 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:03.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:03.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:03.932 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:03 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1685568656' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-21T13:10:03.932 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:03 smithi204 ceph-mon[33153]: osdmap e271: 8 total, 8 up, 6 in 2024-03-21T13:10:03.932 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:03 smithi204 ceph-mon[33153]: 4.a scrub starts 2024-03-21T13:10:03.932 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:03 smithi204 ceph-mon[33153]: 4.a scrub ok 2024-03-21T13:10:03.932 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:03 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:10:03.932 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:03 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:10:03.932 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:03 smithi204 ceph-mon[33153]: 9.11 scrub starts 2024-03-21T13:10:03.932 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:03 smithi204 ceph-mon[33153]: 9.5 scrub ok 2024-03-21T13:10:03.933 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:03 smithi204 ceph-mon[33153]: pgmap v517: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 3039/294258 objects misplaced (1.033%) 2024-03-21T13:10:03.933 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:03 smithi204 ceph-mon[33153]: 9.11 scrub ok 2024-03-21T13:10:03.998 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1685568656' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-21T13:10:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[29621]: osdmap e271: 8 total, 8 up, 6 in 2024-03-21T13:10:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[29621]: 4.a scrub starts 2024-03-21T13:10:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[29621]: 4.a scrub ok 2024-03-21T13:10:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:10:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:10:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[29621]: 9.11 scrub starts 2024-03-21T13:10:04.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[29621]: 9.5 scrub ok 2024-03-21T13:10:04.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[29621]: pgmap v517: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 3039/294258 objects misplaced (1.033%) 2024-03-21T13:10:04.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[29621]: 9.11 scrub ok 2024-03-21T13:10:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1685568656' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-21T13:10:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[40675]: osdmap e271: 8 total, 8 up, 6 in 2024-03-21T13:10:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[40675]: 4.a scrub starts 2024-03-21T13:10:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[40675]: 4.a scrub ok 2024-03-21T13:10:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:10:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:10:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[40675]: 9.11 scrub starts 2024-03-21T13:10:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[40675]: 9.5 scrub ok 2024-03-21T13:10:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[40675]: pgmap v517: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 3039/294258 objects misplaced (1.033%) 2024-03-21T13:10:04.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:03 smithi163 ceph-mon[40675]: 9.11 scrub ok 2024-03-21T13:10:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:04.024 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:04.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:04.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:04.150 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:04.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:04.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:04.277 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:04.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:04.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:04.341 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:10:04.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:04.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:04.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:04.507 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:04.531 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 71 16 100420 100404 88.3693 0 - 0.0110032 2024-03-21T13:10:04.532 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:04.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:04.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:04.634 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:04.703 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:04.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:04 smithi163 ceph-mon[29621]: osdmap e272: 8 total, 8 up, 6 in 2024-03-21T13:10:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:04 smithi163 ceph-mon[29621]: 3.9 scrub starts 2024-03-21T13:10:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:04 smithi163 ceph-mon[29621]: 3.9 scrub ok 2024-03-21T13:10:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:04 smithi163 ceph-mon[29621]: 4.4 scrub starts 2024-03-21T13:10:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:04 smithi163 ceph-mon[29621]: 4.4 scrub ok 2024-03-21T13:10:04.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:04 smithi163 ceph-mon[40675]: osdmap e272: 8 total, 8 up, 6 in 2024-03-21T13:10:04.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:04 smithi163 ceph-mon[40675]: 3.9 scrub starts 2024-03-21T13:10:04.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:04 smithi163 ceph-mon[40675]: 3.9 scrub ok 2024-03-21T13:10:04.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:04 smithi163 ceph-mon[40675]: 4.4 scrub starts 2024-03-21T13:10:04.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:04 smithi163 ceph-mon[40675]: 4.4 scrub ok 2024-03-21T13:10:04.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:04.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:04.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:04.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:04.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:04.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:04.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:04.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:04 smithi204 ceph-mon[33153]: osdmap e272: 8 total, 8 up, 6 in 2024-03-21T13:10:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:04 smithi204 ceph-mon[33153]: 3.9 scrub starts 2024-03-21T13:10:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:04 smithi204 ceph-mon[33153]: 3.9 scrub ok 2024-03-21T13:10:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:04 smithi204 ceph-mon[33153]: 4.4 scrub starts 2024-03-21T13:10:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:04 smithi204 ceph-mon[33153]: 4.4 scrub ok 2024-03-21T13:10:05.085 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:05.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:05.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:05.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:05.221 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:05.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:05.258 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:05.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:05.360 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:05.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:05.490 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:05.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:05.531 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 72 16 101340 101324 87.9405 28.75 0.0045121 0.0113662 2024-03-21T13:10:05.591 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:05.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:05.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:05.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:10:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:10:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[29621]: 9.2 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4862 2024-03-21T13:10:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[29621]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 270'6390 2024-03-21T13:10:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[29621]: osdmap e273: 8 total, 8 up, 6 in 2024-03-21T13:10:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[29621]: 9.14 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4740 2024-03-21T13:10:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[29621]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4764 2024-03-21T13:10:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[29621]: 9.12 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4769 2024-03-21T13:10:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[29621]: 9.17 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4773 2024-03-21T13:10:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[29621]: 9.b starting backfill to osd.2 from (0'0,0'0] MAX to 270'6328 2024-03-21T13:10:05.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[29621]: pgmap v520: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 22 MiB/s wr, 348 op/s; 3039/294258 objects misplaced (1.033%) 2024-03-21T13:10:05.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:10:05.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:10:05.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[40675]: 9.2 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4862 2024-03-21T13:10:05.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[40675]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 270'6390 2024-03-21T13:10:05.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[40675]: osdmap e273: 8 total, 8 up, 6 in 2024-03-21T13:10:05.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[40675]: 9.14 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4740 2024-03-21T13:10:05.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[40675]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4764 2024-03-21T13:10:05.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[40675]: 9.12 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4769 2024-03-21T13:10:05.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[40675]: 9.17 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4773 2024-03-21T13:10:05.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[40675]: 9.b starting backfill to osd.2 from (0'0,0'0] MAX to 270'6328 2024-03-21T13:10:05.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[40675]: pgmap v520: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 22 MiB/s wr, 348 op/s; 3039/294258 objects misplaced (1.033%) 2024-03-21T13:10:05.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:10:05.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:10:05.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:05.774 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:05.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:05.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:05.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:05.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:05.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:05 smithi204 ceph-mon[33153]: 9.2 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4862 2024-03-21T13:10:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:05 smithi204 ceph-mon[33153]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 270'6390 2024-03-21T13:10:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:05 smithi204 ceph-mon[33153]: osdmap e273: 8 total, 8 up, 6 in 2024-03-21T13:10:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:05 smithi204 ceph-mon[33153]: 9.14 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4740 2024-03-21T13:10:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:05 smithi204 ceph-mon[33153]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4764 2024-03-21T13:10:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:05 smithi204 ceph-mon[33153]: 9.12 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4769 2024-03-21T13:10:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:05 smithi204 ceph-mon[33153]: 9.17 starting backfill to osd.2 from (0'0,0'0] MAX to 270'4773 2024-03-21T13:10:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:05 smithi204 ceph-mon[33153]: 9.b starting backfill to osd.2 from (0'0,0'0] MAX to 270'6328 2024-03-21T13:10:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:05 smithi204 ceph-mon[33153]: pgmap v520: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 22 MiB/s wr, 348 op/s; 3039/294258 objects misplaced (1.033%) 2024-03-21T13:10:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:10:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:10:06.020 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:06.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:06.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:06.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:06.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:06.197 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:06.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:06.299 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:06.324 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:06.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:06.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:06.426 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:06.466 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:06.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:06.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:06.532 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 73 16 102499 102483 87.7281 72.4375 0.0205129 0.0113942 2024-03-21T13:10:06.568 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:06.595 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-03-21T13:10:06.616 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:06.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:06.663 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:10:06.713 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:10:06.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:06.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:06.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:06.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:06.849 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:06.881 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:06.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:06.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:06.982 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:07.026 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:07.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:07.128 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:07.169 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:07.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:07.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:07.271 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:07.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:07.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:07.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:07.411 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:07.457 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:07.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:07.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:07.532 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 74 16 103519 103503 87.404 63.75 0.077794 0.0114286 2024-03-21T13:10:07.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:07.597 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:07.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:07.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:07.650 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:07 smithi204 ceph-mon[33153]: osdmap e274: 8 total, 8 up, 6 in 2024-03-21T13:10:07.699 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:07.734 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:07.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:07.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:07 smithi163 ceph-mon[29621]: osdmap e274: 8 total, 8 up, 6 in 2024-03-21T13:10:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:07 smithi163 ceph-mon[40675]: osdmap e274: 8 total, 8 up, 6 in 2024-03-21T13:10:07.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:07.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:07.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:07.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:07.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:08.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:08.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:08.104 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:08.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:08.141 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:08.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:08.242 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:08.294 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:08.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:08.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:08.439 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:08.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:08.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:08.532 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 75 16 104491 104475 87.0486 60.75 0.00892435 0.0114835 2024-03-21T13:10:08.540 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:08.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:08.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:08.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:08.681 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:08.716 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:08.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:08.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:08 smithi204 ceph-mon[33153]: pgmap v522: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 721 B/s rd, 0 op/s; 40963/303459 objects misplaced (13.499%) 2024-03-21T13:10:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:08 smithi204 ceph-mon[33153]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:10:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:08 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:10:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:08 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:10:08.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:08 smithi163 ceph-mon[29621]: pgmap v522: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 721 B/s rd, 0 op/s; 40963/303459 objects misplaced (13.499%) 2024-03-21T13:10:08.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:08 smithi163 ceph-mon[29621]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:10:08.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:08 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:10:08.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:08 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:10:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:08 smithi163 ceph-mon[40675]: pgmap v522: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 721 B/s rd, 0 op/s; 40963/303459 objects misplaced (13.499%) 2024-03-21T13:10:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:08 smithi163 ceph-mon[40675]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:10:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:08 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:10:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:08 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:10:08.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:08.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:08.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:08.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:08.934 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:10:08.934 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-03-21T13:10:08.934 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-03-21T13:10:08.934 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-03-21T13:10:08.934 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:10:08.953 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:08.990 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:08.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:08.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:09.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:09.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:09.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:09.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:09.220 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:09.259 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:09.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:09.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:09.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:09.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:09.407 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:09.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:09.508 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:09.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:09 smithi163 ceph-mon[29621]: osdmap e275: 8 total, 8 up, 6 in 2024-03-21T13:10:09.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:09 smithi163 ceph-mon[40675]: osdmap e275: 8 total, 8 up, 6 in 2024-03-21T13:10:09.532 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 76 15 105477 105462 86.7149 61.6875 0.0128143 0.0115277 2024-03-21T13:10:09.544 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:09.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:09.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:09.646 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:09.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:09.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:09.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:09 smithi204 ceph-mon[33153]: osdmap e275: 8 total, 8 up, 6 in 2024-03-21T13:10:09.777 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:09.816 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:09.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:09.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:09.918 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:10.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:10.039 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:10.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:10.140 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:10.182 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:10.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:10.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:10.241 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-21T13:10:10.255 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:10:10.283 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:10.293 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:10:10.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:10.324 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:10.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:10.334 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:10:10.367 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:10:10.404 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:10:10.425 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:10.452 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:10:10.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:10 smithi163 ceph-mon[29621]: pgmap v524: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 584 B/s rd, 0 op/s; 40963/303459 objects misplaced (13.499%) 2024-03-21T13:10:10.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:10 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1182213164' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:10:10.453 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:10 smithi163 ceph-mon[40675]: pgmap v524: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 584 B/s rd, 0 op/s; 40963/303459 objects misplaced (13.499%) 2024-03-21T13:10:10.453 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:10 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1182213164' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:10:10.466 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:10.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:10.492 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:10:10.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:10 smithi204 ceph-mon[33153]: pgmap v524: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 584 B/s rd, 0 op/s; 40963/303459 objects misplaced (13.499%) 2024-03-21T13:10:10.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:10 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1182213164' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:10:10.532 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 77 16 106616 106600 86.5123 71.125 0.0111195 0.0115546 2024-03-21T13:10:10.537 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:10:10.549 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:10:10.549 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:10:10.549 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:10:10.549 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:10:10.549 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:10:10.549 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:10:10.549 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:10:10.550 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:10:10.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:10.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:10.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:10.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:10.665 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:10:10.719 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:10.779 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:10.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:10.880 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:10.921 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:10.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:11.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:11.060 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:11.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:11.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:11.162 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:11.200 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:11.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:11.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:11.302 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:11.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:11.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:11.353 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:10:11.438 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:11.478 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:11.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:11.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:11 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/937635961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:10:11.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:11 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:10:11.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:11 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/937635961' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:10:11.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:11 smithi163 ceph-mon[29621]: osdmap e276: 8 total, 8 up, 6 in 2024-03-21T13:10:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:11 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/937635961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:10:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:11 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:10:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:11 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/937635961' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:10:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:11 smithi163 ceph-mon[40675]: osdmap e276: 8 total, 8 up, 6 in 2024-03-21T13:10:11.532 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 78 16 107703 107687 86.2741 67.9375 0.00748262 0.0115863 2024-03-21T13:10:11.580 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:11.596 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 2] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:10:11.596 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-21T13:10:11.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:11.621 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:11.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:11.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:11.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:11 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/937635961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:10:11.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:11 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:10:11.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:11 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/937635961' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:10:11.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:11 smithi204 ceph-mon[33153]: osdmap e276: 8 total, 8 up, 6 in 2024-03-21T13:10:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:11.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:11.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:11.856 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:11.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:11.899 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:11.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:12.000 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:12.047 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:12.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:12.149 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:12.201 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:12.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:12.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:10:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:10:12.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:12.341 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:12.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:12.443 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:12.532 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 79 16 108837 108821 86.0791 70.875 0.0147504 0.0116127 2024-03-21T13:10:12.536 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:12 smithi163 ceph-mon[29621]: pgmap v525: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.4 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 41095/314421 objects misplaced (13.070%) 2024-03-21T13:10:12.536 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:12 smithi163 ceph-mon[40675]: pgmap v525: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.4 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 41095/314421 objects misplaced (13.070%) 2024-03-21T13:10:12.734 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:12 smithi204 ceph-mon[33153]: pgmap v525: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.4 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 41095/314421 objects misplaced (13.070%) 2024-03-21T13:10:12.777 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:12.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:12.788 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:10:12.880 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:12.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:12.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:12.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:13.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:13.067 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:13.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:13.169 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:13.207 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:13.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:13.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:13.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:13.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:13.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:13.451 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:13.488 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:13.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:13.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:13 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:13.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:13 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:13.532 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:10:13.532933+0000 min lat: 0.00122048 max lat: 2.2562 avg lat: 0.0116416 2024-03-21T13:10:13.532 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:10:13.532 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 80 16 109934 109918 85.8601 68.5625 0.00948609 0.0116416 2024-03-21T13:10:13.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:13.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:13.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:13.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:13.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:13 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:13.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:13.769 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:13.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:13.808 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:10:13.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:13.900 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:13.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:13.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:14.002 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:14.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:14.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:14.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:14.146 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:14.183 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:14.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:14.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:14.285 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:14.323 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:14.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:14.425 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:14.473 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:14.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:14.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:14 smithi204 ceph-mon[33153]: pgmap v527: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 789 B/s rd, 51 MiB/s wr, 823 op/s; 41740/319470 objects misplaced (13.065%); 8.1 MiB/s, 129 objects/s recovering 2024-03-21T13:10:14.532 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 81 16 111018 111002 85.6365 67.75 0.0311735 0.0116724 2024-03-21T13:10:14.576 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:14.614 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:14.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:14.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:14.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:14 smithi163 ceph-mon[29621]: pgmap v527: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 789 B/s rd, 51 MiB/s wr, 823 op/s; 41740/319470 objects misplaced (13.065%); 8.1 MiB/s, 129 objects/s recovering 2024-03-21T13:10:14.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:14 smithi163 ceph-mon[40675]: pgmap v527: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 789 B/s rd, 51 MiB/s wr, 823 op/s; 41740/319470 objects misplaced (13.065%); 8.1 MiB/s, 129 objects/s recovering 2024-03-21T13:10:14.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:14.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:14.754 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:14.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:14.767 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:10:14.781 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:10:14.781 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:10:14.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:14.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:14.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:14.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:15.028 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:15.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:15.129 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:15.179 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:15.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:15.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:15.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:15.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:15.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:15.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:15.464 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:15.475 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:10:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=cleanup t=2024-03-21T13:10:15.034144278Z level=info msg="Completed cleanup jobs" duration=5.82443ms 2024-03-21T13:10:15.475 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:10:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=sqlstore.transactions t=2024-03-21T13:10:15.039571571Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-21T13:10:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:15.516 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:15.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:15.522 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-21T13:10:15.535 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 82 16 112128 112112 85.4351 69.375 0.0136117 0.0117008 2024-03-21T13:10:15.618 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:15.657 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:15.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:15.759 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:15.805 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:15.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:15.869 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:10:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:10:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:10:15.888 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:10:15.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:15.942 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:15.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:16.045 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:16.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:16.089 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:16.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:16.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:16.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:16.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:16.325 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:16.357 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:16.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:16.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:16.459 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:16.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:16.508 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:16.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:16.535 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 83 16 113208 113192 85.2189 67.5 0.0140659 0.0117298 2024-03-21T13:10:16.609 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:16.648 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:16.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:16.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:16 smithi163 ceph-mon[29621]: pgmap v528: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 42 MiB/s wr, 667 op/s; 41740/319470 objects misplaced (13.065%); 6.5 MiB/s, 104 objects/s recovering 2024-03-21T13:10:16.710 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:16 smithi163 ceph-mon[40675]: pgmap v528: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 42 MiB/s wr, 667 op/s; 41740/319470 objects misplaced (13.065%); 6.5 MiB/s, 104 objects/s recovering 2024-03-21T13:10:16.749 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:16 smithi204 ceph-mon[33153]: pgmap v528: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 42 MiB/s wr, 667 op/s; 41740/319470 objects misplaced (13.065%); 6.5 MiB/s, 104 objects/s recovering 2024-03-21T13:10:16.788 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:16.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:16.891 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:16.929 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:16.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:17.030 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:17.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:17.072 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:17.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:17.174 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:17.215 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:17.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:17.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:17.471 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.0 to 1 (10000) 2024-03-21T13:10:17.472 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:17 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/66526490' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:10:17.472 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:17 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:10:17.472 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:17 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/66526490' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:10:17.472 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:17 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:10:17.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:17.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:17.535 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 84 16 114455 114439 85.1322 77.9375 0.010999 0.0117421 2024-03-21T13:10:17.590 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:17.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:17.637 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:17.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:17.738 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:17 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/66526490' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:10:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:17 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:10:17.786 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:17.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:17.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:17.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:17.927 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:17.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:18.028 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:18.047 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:10:18.047 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:10:18.096 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:18.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:18.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:18.198 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:18.237 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:18.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:18.339 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:18.378 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:18.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:18.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:18.480 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:18.521 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-21T13:10:18.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:18.536 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:18.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:18.537 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 85 16 115682 115666 85.032 76.6875 0.00947227 0.0117555 2024-03-21T13:10:18.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:18.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:18.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:18.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:18.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:18 smithi163 ceph-mon[29621]: pgmap v529: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s; 42752/335979 objects misplaced (12.725%); 12 MiB/s, 189 objects/s recovering 2024-03-21T13:10:18.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:18 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:10:18.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:18 smithi163 ceph-mon[29621]: osdmap e277: 8 total, 8 up, 6 in 2024-03-21T13:10:18.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:18 smithi163 ceph-mon[40675]: pgmap v529: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s; 42752/335979 objects misplaced (12.725%); 12 MiB/s, 189 objects/s recovering 2024-03-21T13:10:18.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:18 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:10:18.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:18 smithi163 ceph-mon[40675]: osdmap e277: 8 total, 8 up, 6 in 2024-03-21T13:10:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:18 smithi204 ceph-mon[33153]: pgmap v529: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s; 42752/335979 objects misplaced (12.725%); 12 MiB/s, 189 objects/s recovering 2024-03-21T13:10:18.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:18 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:10:18.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:18 smithi204 ceph-mon[33153]: osdmap e277: 8 total, 8 up, 6 in 2024-03-21T13:10:18.796 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:18.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:18.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:18.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:18.937 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:19.000 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:19.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:19.101 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:19.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:19.146 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:19.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:19.247 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:19.290 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:19.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:19.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:19.336 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:10:19.374 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:10:19.393 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:19.416 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:10:19.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:19.448 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:19.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:19.462 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:10:19.508 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:10:19.536 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 86 16 116882 116866 84.9153 75 0.0103603 0.0117722 2024-03-21T13:10:19.549 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:19.550 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:10:19.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:19.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:19.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:19.598 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:10:19.639 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:10:19.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:19 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:10:19.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:19 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:10:19.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:19 smithi163 ceph-mon[29621]: osdmap e278: 8 total, 8 up, 6 in 2024-03-21T13:10:19.640 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:19 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3064952138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:10:19.640 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:19 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:10:19.640 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:19 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:10:19.640 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:19 smithi163 ceph-mon[40675]: osdmap e278: 8 total, 8 up, 6 in 2024-03-21T13:10:19.640 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:19 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3064952138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:10:19.644 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:10:19.644 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:10:19.645 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:10:19.645 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:10:19.645 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:10:19.645 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:10:19.645 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:10:19.645 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:10:19.685 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:19.720 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:19.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:19 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:10:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:19 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:10:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:19 smithi204 ceph-mon[33153]: osdmap e278: 8 total, 8 up, 6 in 2024-03-21T13:10:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:19 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3064952138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:10:19.820 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:19.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:19.859 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:19.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:19.960 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:19.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:20.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:20.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:20.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:20.140 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:20.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:20.241 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:20.278 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:20.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:20.380 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:20.421 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:20.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:20.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:20.523 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:20.537 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 87 16 118031 118015 84.7645 71.8125 0.0105193 0.011793 2024-03-21T13:10:20.568 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:20.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:20.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:20.707 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:20.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:20.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:20 smithi204 ceph-mon[33153]: pgmap v532: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 683 B/s rd, 46 MiB/s wr, 735 op/s; 42752/335979 objects misplaced (12.725%); 6.4 MiB/s, 101 objects/s recovering 2024-03-21T13:10:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:10:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:10:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:20 smithi163 ceph-mon[40675]: pgmap v532: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 683 B/s rd, 46 MiB/s wr, 735 op/s; 42752/335979 objects misplaced (12.725%); 6.4 MiB/s, 101 objects/s recovering 2024-03-21T13:10:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:10:20.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:10:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:20 smithi163 ceph-mon[29621]: pgmap v532: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 6.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 683 B/s rd, 46 MiB/s wr, 735 op/s; 42752/335979 objects misplaced (12.725%); 6.4 MiB/s, 101 objects/s recovering 2024-03-21T13:10:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:10:20.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:10:20.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:20.850 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:20.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:20.952 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:21.067 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:21.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:21.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:21.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:21.205 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:21.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:21.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:21.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:21.345 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:21.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:21.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:21.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:21.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:21.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:21.537 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 88 16 119189 119173 84.6236 72.375 0.0144008 0.0118126 2024-03-21T13:10:21.593 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:21.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:21.639 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:21.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:21.740 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:21.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:21 smithi204 ceph-mon[33153]: pgmap v533: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 7.1 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 43099/348027 objects misplaced (12.384%); 12 MiB/s, 186 objects/s recovering 2024-03-21T13:10:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:21 smithi163 ceph-mon[29621]: pgmap v533: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 7.1 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 43099/348027 objects misplaced (12.384%); 12 MiB/s, 186 objects/s recovering 2024-03-21T13:10:21.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:21 smithi163 ceph-mon[40675]: pgmap v533: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 7.1 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 43099/348027 objects misplaced (12.384%); 12 MiB/s, 186 objects/s recovering 2024-03-21T13:10:21.781 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:21.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:21.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:21.883 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:21.920 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:21.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:21.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:22.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:22.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:22.063 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:22.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:22.164 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:22.199 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:22.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:22.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:10:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:10:22.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:22.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:22.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:22.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:22.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:22.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:22.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:22.537 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 89 16 120309 120293 84.4593 70 0.0244937 0.0118352 2024-03-21T13:10:22.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:22.607 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:22.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:22.709 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:22.748 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:22.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:22.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:22 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:10:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:22 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:10:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:22 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:22 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:10:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:22 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:10:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:22 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:22.850 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:22.895 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:22.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:22.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:22.997 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:23.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:22 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:10:23.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:22 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:10:23.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:22 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:23.039 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:23.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:23.140 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:23.176 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:23.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:23.277 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:23.319 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:23.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:23.420 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:23.537 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Total time run: 90.0148 2024-03-21T13:10:23.537 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Total writes made: 121527 2024-03-21T13:10:23.537 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Write size: 65536 2024-03-21T13:10:23.537 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Object size: 65536 2024-03-21T13:10:23.537 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Bandwidth (MB/sec): 84.3799 2024-03-21T13:10:23.537 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev Bandwidth: 42.9723 2024-03-21T13:10:23.538 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max bandwidth (MB/sec): 220.25 2024-03-21T13:10:23.538 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min bandwidth (MB/sec): 0 2024-03-21T13:10:23.538 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Average IOPS: 1350 2024-03-21T13:10:23.538 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev IOPS: 687.556 2024-03-21T13:10:23.538 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max IOPS: 3524 2024-03-21T13:10:23.538 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min IOPS: 0 2024-03-21T13:10:23.538 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Average Latency(s): 0.0118468 2024-03-21T13:10:23.538 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev Latency(s): 0.0438558 2024-03-21T13:10:23.538 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max latency(s): 2.2562 2024-03-21T13:10:23.538 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min latency(s): 0.00122048 2024-03-21T13:10:23.544 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Cleaning up (deleting benchmark objects) 2024-03-21T13:10:23.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:23 smithi204 ceph-mon[33153]: osdmap e279: 8 total, 8 up, 6 in 2024-03-21T13:10:23.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:23 smithi204 ceph-mon[33153]: pgmap v535: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 7.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 852 B/s rd, 62 MiB/s wr, 990 op/s; 43871/353808 objects misplaced (12.400%); 7.6 MiB/s, 121 objects/s recovering 2024-03-21T13:10:23.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:23 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:10:23.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:23 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:10:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:23 smithi163 ceph-mon[29621]: osdmap e279: 8 total, 8 up, 6 in 2024-03-21T13:10:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:23 smithi163 ceph-mon[29621]: pgmap v535: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 7.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 852 B/s rd, 62 MiB/s wr, 990 op/s; 43871/353808 objects misplaced (12.400%); 7.6 MiB/s, 121 objects/s recovering 2024-03-21T13:10:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:23 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:10:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:23 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:10:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:23 smithi163 ceph-mon[40675]: osdmap e279: 8 total, 8 up, 6 in 2024-03-21T13:10:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:23 smithi163 ceph-mon[40675]: pgmap v535: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 7.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 852 B/s rd, 62 MiB/s wr, 990 op/s; 43871/353808 objects misplaced (12.400%); 7.6 MiB/s, 121 objects/s recovering 2024-03-21T13:10:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:23 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:10:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:23 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:10:23.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:23.806 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:23.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:23.907 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:23.934 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:23.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:23.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:24.036 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:24.156 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:24.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:24.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:24.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:24.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:24.393 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:24.421 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:24.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:24.522 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:24.560 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:24.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:24.662 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:24.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:24.699 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:24.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:24 smithi204 ceph-mon[33153]: osdmap e280: 8 total, 8 up, 6 in 2024-03-21T13:10:24.800 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:24.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:24.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:24.936 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:24.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:24.979 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:24.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:24 smithi163 ceph-mon[29621]: osdmap e280: 8 total, 8 up, 6 in 2024-03-21T13:10:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:24 smithi163 ceph-mon[40675]: osdmap e280: 8 total, 8 up, 6 in 2024-03-21T13:10:25.081 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:25.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:25.230 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:25.270 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:25.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:25.372 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:25.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:25.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:25.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:25.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:25.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:25.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:25.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:25 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:10:25.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:25 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:10:25.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:25 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:25.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:25 smithi163 ceph-mon[29621]: pgmap v537: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 7.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 805 B/s rd, 58 MiB/s wr, 935 op/s; 43871/353808 objects misplaced (12.400%); 7.2 MiB/s, 114 objects/s recovering 2024-03-21T13:10:25.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:25 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:10:25.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:25 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:10:25.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:25 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:25.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:25 smithi163 ceph-mon[40675]: pgmap v537: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 7.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 805 B/s rd, 58 MiB/s wr, 935 op/s; 43871/353808 objects misplaced (12.400%); 7.2 MiB/s, 114 objects/s recovering 2024-03-21T13:10:25.641 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:25.673 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:25.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:25.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:25.775 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:25.824 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:25.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:25.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:25.926 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:25.968 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:25.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:25.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:26.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:25 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:10:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:25 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:10:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:25 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:25 smithi204 ceph-mon[33153]: pgmap v537: 158 pgs: 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 7.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 805 B/s rd, 58 MiB/s wr, 935 op/s; 43871/353808 objects misplaced (12.400%); 7.2 MiB/s, 114 objects/s recovering 2024-03-21T13:10:26.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:10:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:10:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:10:26.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:26.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:26.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:26.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:26.225 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:26.263 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:26.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:26.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:26.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:26.401 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:26.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:26.502 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:26.545 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:26.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:26.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:26.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:26.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:26.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:26.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:26.785 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:26.789 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[29621]: osdmap e281: 8 total, 8 up, 6 in 2024-03-21T13:10:26.789 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[29621]: osdmap e282: 8 total, 8 up, 6 in 2024-03-21T13:10:26.789 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:26.789 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:26.790 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:26.790 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:26.790 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:10:26.790 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[40675]: osdmap e281: 8 total, 8 up, 6 in 2024-03-21T13:10:26.790 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[40675]: osdmap e282: 8 total, 8 up, 6 in 2024-03-21T13:10:26.790 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:26.790 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:26.790 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:26.790 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:26.790 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:26 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:10:26.823 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:26.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:26.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:26.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:26.963 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:26.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:26 smithi204 ceph-mon[33153]: osdmap e281: 8 total, 8 up, 6 in 2024-03-21T13:10:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:26 smithi204 ceph-mon[33153]: osdmap e282: 8 total, 8 up, 6 in 2024-03-21T13:10:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:26 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:26 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:26 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:26 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:26 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:10:27.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:27.108 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:27.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:27.135 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:10:27.209 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:27.255 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:27.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:27.356 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:27.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:27.398 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:27.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:27.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:27.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:27.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:27.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:27.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:27.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:27.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:27 smithi163 ceph-mon[29621]: Adjusting osd_memory_target on smithi163 to 3248M 2024-03-21T13:10:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:27 smithi163 ceph-mon[29621]: pgmap v540: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 7.3 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.4 KiB/s rd, 54 MiB/s wr, 1.35k op/s; 43164/358770 objects misplaced (12.031%); 17 MiB/s, 276 objects/s recovering 2024-03-21T13:10:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:27 smithi163 ceph-mon[40675]: Adjusting osd_memory_target on smithi163 to 3248M 2024-03-21T13:10:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:27 smithi163 ceph-mon[40675]: pgmap v540: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 7.3 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.4 KiB/s rd, 54 MiB/s wr, 1.35k op/s; 43164/358770 objects misplaced (12.031%); 17 MiB/s, 276 objects/s recovering 2024-03-21T13:10:27.773 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:27.812 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:27.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:27.913 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:27.948 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:27.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:28.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:27 smithi204 ceph-mon[33153]: Adjusting osd_memory_target on smithi163 to 3248M 2024-03-21T13:10:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:27 smithi204 ceph-mon[33153]: pgmap v540: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 7.3 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.4 KiB/s rd, 54 MiB/s wr, 1.35k op/s; 43164/358770 objects misplaced (12.031%); 17 MiB/s, 276 objects/s recovering 2024-03-21T13:10:28.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:28.083 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:28.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:28.111 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:10:28.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:28.218 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:28.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:28.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:28.320 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:28.369 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:10:28.369 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:10:28.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:10:28.369 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:126820 9.6 9.78bc4a46 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-03-21T13:10:28.370 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:27.798385+0000", 2024-03-21T13:10:28.370 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.030722156, 2024-03-21T13:10:28.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.030750161000000002, 2024-03-21T13:10:28.370 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:10:28.370 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "queued for pg", 2024-03-21T13:10:28.370 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:10:28.370 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:10:28.370 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:10:28.370 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 126820 2024-03-21T13:10:28.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:10:28.370 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:10:28.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:10:28.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:27.798385+0000", 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:27.798385+0000", 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:27.798387+0000", 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.35e-06 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:10:28.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:10:28.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:10:28.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:27.798391+0000", 2024-03-21T13:10:28.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.634e-06 2024-03-21T13:10:28.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:10:28.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:10:28.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:10:28.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:27.798400+0000", 2024-03-21T13:10:28.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1800000000000002e-06 2024-03-21T13:10:28.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:10:28.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:10:28.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:10:28.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:27.798408+0000", 2024-03-21T13:10:28.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9359999999999999e-06 2024-03-21T13:10:28.373 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:10:28.373 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:10:28.373 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:10:28.373 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:10:28.373 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:10:28.373 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 1 2024-03-21T13:10:28.373 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:10:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:28.374 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:28.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:28.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:28.513 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:28.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:28.614 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:28.710 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:28.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:28.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:28.851 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:28.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:28.952 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:28.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:28.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:28.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:29.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:28 smithi204 ceph-mon[33153]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:10:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:28 smithi163 ceph-mon[29621]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:10:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:28 smithi163 ceph-mon[40675]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:10:29.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:29.116 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:29.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:29.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:29.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:29.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:29.255 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:29.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:29.356 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:29.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:29.400 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:29.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:29.501 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:29.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:29.548 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:29.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:29.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:29.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:29.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:29.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:29.825 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:29.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:29.928 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:29.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:29.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:30.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:29 smithi204 ceph-mon[33153]: pgmap v541: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 7.3 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 39 MiB/s wr, 974 op/s; 43164/358770 objects misplaced (12.031%); 12 MiB/s, 199 objects/s recovering 2024-03-21T13:10:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:29 smithi163 ceph-mon[29621]: pgmap v541: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 7.3 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 39 MiB/s wr, 974 op/s; 43164/358770 objects misplaced (12.031%); 12 MiB/s, 199 objects/s recovering 2024-03-21T13:10:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:29 smithi163 ceph-mon[40675]: pgmap v541: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 7.3 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 39 MiB/s wr, 974 op/s; 43164/358770 objects misplaced (12.031%); 12 MiB/s, 199 objects/s recovering 2024-03-21T13:10:30.085 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:30.117 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:30.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:30.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:30.219 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:30.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:30.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:30.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:30.396 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:30.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:30.498 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:30.527 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:30.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:30.629 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:30.668 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:30.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:30.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:30.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:30 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:30.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:30 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:30.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:30 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:10:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:30 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:30 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:30 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:10:30.770 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:30.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:30.807 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:30.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:30.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:30.946 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:30.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:30 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:31.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:30 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:31.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:30 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:10:31.047 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:31.087 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:31.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:31.189 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:31.214 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:31.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:31.315 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:31.353 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:31.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:31.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:31.495 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:31.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:31.596 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:31.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:31.641 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:31.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:31.742 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:31.781 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:31.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:31.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:31.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:31.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:31.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:32.022 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:32.060 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:32.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:32.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:32.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:32.197 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:32.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:32.253 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:10:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:10:32.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:32.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:32.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:32.438 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:32.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:32.477 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:32.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:32.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:32 smithi163 ceph-mon[40675]: pgmap v542: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 7.0 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.5 KiB/s rd, 32 MiB/s wr, 1.38k op/s; 40597/346119 objects misplaced (11.729%); 22 MiB/s, 353 objects/s recovering 2024-03-21T13:10:32.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:32 smithi163 ceph-mon[29621]: pgmap v542: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 7.0 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.5 KiB/s rd, 32 MiB/s wr, 1.38k op/s; 40597/346119 objects misplaced (11.729%); 22 MiB/s, 353 objects/s recovering 2024-03-21T13:10:32.579 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:32.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:32.619 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:32.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:32.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:32 smithi204 ceph-mon[33153]: pgmap v542: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 7.0 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.5 KiB/s rd, 32 MiB/s wr, 1.38k op/s; 40597/346119 objects misplaced (11.729%); 22 MiB/s, 353 objects/s recovering 2024-03-21T13:10:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:32.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:32.863 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:32.899 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:32.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:33.000 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-21T13:10:33.001 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:33.086 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:33.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:33.188 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:33.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:33.238 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:33.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:33.339 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:33.377 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:33.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:33.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:33.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:33 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:33.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:33 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:33.527 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:33.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:33.629 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:33.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:33.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:33.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:33 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:33.766 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:33.802 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:33.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:33.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:33.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:33.946 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:33.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:34.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:34.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:34.083 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:34.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:34.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:34.221 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:34.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:34.323 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:34.357 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:34.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:34.441 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:10:34.458 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:34.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:34.484 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:34.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:34.573 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:34 smithi163 ceph-mon[29621]: pgmap v543: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.4 KiB/s rd, 29 MiB/s wr, 1.48k op/s; 39883/340689 objects misplaced (11.707%); 20 MiB/s, 323 objects/s recovering 2024-03-21T13:10:34.574 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:34 smithi163 ceph-mon[40675]: pgmap v543: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.4 KiB/s rd, 29 MiB/s wr, 1.48k op/s; 39883/340689 objects misplaced (11.707%); 20 MiB/s, 323 objects/s recovering 2024-03-21T13:10:34.586 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:34.614 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:34.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:34.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:34.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:34.749 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:34.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:34.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:34 smithi204 ceph-mon[33153]: pgmap v543: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.4 KiB/s rd, 29 MiB/s wr, 1.48k op/s; 39883/340689 objects misplaced (11.707%); 20 MiB/s, 323 objects/s recovering 2024-03-21T13:10:34.852 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:34.878 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:34.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:34.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:35.016 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:35.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:35.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:35.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:35.162 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:35.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:35.264 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:35.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:35.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:35.401 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:35.416 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:10:35.439 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:35.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:35.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:35.541 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:35.591 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:35.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:35.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:35.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:35.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:35.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:35.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:35 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4005534934' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:10:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:35 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:10:35.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:10:35.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:35 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4005534934' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:10:35.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:35 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:10:35.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:10:35.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:10:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:10:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:10:35.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:35 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4005534934' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:10:35.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:35 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:10:35.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:10:35.830 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:35.873 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:35.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:35.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:36.238 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:36.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:36.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:36.340 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:36.377 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:36.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:36.478 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:36.521 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:36.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:36.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:36.623 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:36.659 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:36.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:36.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:36 smithi204 ceph-mon[33153]: pgmap v544: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 550 B/s rd, 6.7 MiB/s wr, 850 op/s; 39883/340689 objects misplaced (11.707%); 9.4 MiB/s, 149 objects/s recovering 2024-03-21T13:10:36.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:36 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:10:36.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:36 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:10:36.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:36 smithi204 ceph-mon[33153]: osdmap e283: 8 total, 8 up, 6 in 2024-03-21T13:10:36.757 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:36 smithi163 ceph-mon[29621]: pgmap v544: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 550 B/s rd, 6.7 MiB/s wr, 850 op/s; 39883/340689 objects misplaced (11.707%); 9.4 MiB/s, 149 objects/s recovering 2024-03-21T13:10:36.757 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:36 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:10:36.757 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:36 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:10:36.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:36 smithi163 ceph-mon[29621]: osdmap e283: 8 total, 8 up, 6 in 2024-03-21T13:10:36.758 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:36 smithi163 ceph-mon[40675]: pgmap v544: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 550 B/s rd, 6.7 MiB/s wr, 850 op/s; 39883/340689 objects misplaced (11.707%); 9.4 MiB/s, 149 objects/s recovering 2024-03-21T13:10:36.758 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:36 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:10:36.758 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:36 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:10:36.758 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:36 smithi163 ceph-mon[40675]: osdmap e283: 8 total, 8 up, 6 in 2024-03-21T13:10:36.760 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:36.808 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:36.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:36.909 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:36.948 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:36.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:37.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:37.089 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:37.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:37.191 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:37.233 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:37.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:37.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:37.375 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:37.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:37.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:37.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:37.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:37.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:37.604 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:37.639 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:37.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:37.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:37.741 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:37.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:37 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1715506920' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:10:37.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:37 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:10:37.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:37 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1715506920' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:10:37.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:37 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:10:37.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:37 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1715506920' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:10:37.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:37 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:10:37.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:37.777 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:37.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:37.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:37.916 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:37.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:38.018 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:38.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:38.051 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:38.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:38.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:38.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:38.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:38.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:38.288 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:38.329 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:38.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:38.430 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:38.467 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:38.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:38.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:38.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:38.603 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:38.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:38.704 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:38.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:38.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:38.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:38.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:38 smithi204 ceph-mon[33153]: pgmap v546: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 147 active+clean; 6.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s; 36644/322893 objects misplaced (11.349%); 18 MiB/s, 284 objects/s recovering 2024-03-21T13:10:38.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:38 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:10:38.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:38 smithi204 ceph-mon[33153]: osdmap e284: 8 total, 8 up, 6 in 2024-03-21T13:10:38.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:38 smithi163 ceph-mon[29621]: pgmap v546: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 147 active+clean; 6.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s; 36644/322893 objects misplaced (11.349%); 18 MiB/s, 284 objects/s recovering 2024-03-21T13:10:38.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:38 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:10:38.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:38 smithi163 ceph-mon[29621]: osdmap e284: 8 total, 8 up, 6 in 2024-03-21T13:10:38.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:38 smithi163 ceph-mon[40675]: pgmap v546: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 147 active+clean; 6.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s; 36644/322893 objects misplaced (11.349%); 18 MiB/s, 284 objects/s recovering 2024-03-21T13:10:38.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:38 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:10:38.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:38 smithi163 ceph-mon[40675]: osdmap e284: 8 total, 8 up, 6 in 2024-03-21T13:10:38.839 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:38.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:38.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:39.016 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:39.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:39.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:39.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:39.155 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:39.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:39.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:39.257 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:39.283 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:39.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:39.384 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:39.423 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:39.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:39.525 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:39.549 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:39.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:39.650 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:39.687 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:39.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:39.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:39.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:39.815 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:39.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:39.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:39.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:39.958 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:39.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:40.059 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:40.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:40.083 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:40.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:40.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:40.221 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:40.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:40.323 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:40.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:40.361 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:40.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:40.462 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:40.500 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:40.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:40.520 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.3 to 1 (10000) 2024-03-21T13:10:40.601 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:40.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:40.628 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:40.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:40.729 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:40 smithi204 ceph-mon[33153]: pgmap v548: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 147 active+clean; 6.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 968 op/s; 36644/322893 objects misplaced (11.349%); 11 MiB/s, 181 objects/s recovering 2024-03-21T13:10:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:40.760 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:40.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:40.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:40 smithi163 ceph-mon[29621]: pgmap v548: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 147 active+clean; 6.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 968 op/s; 36644/322893 objects misplaced (11.349%); 11 MiB/s, 181 objects/s recovering 2024-03-21T13:10:40.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:40 smithi163 ceph-mon[40675]: pgmap v548: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 147 active+clean; 6.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 968 op/s; 36644/322893 objects misplaced (11.349%); 11 MiB/s, 181 objects/s recovering 2024-03-21T13:10:40.862 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:40.899 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:40.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:40.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:41.001 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:41.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:41.042 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:41.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:41.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:41.181 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:41.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:41.283 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:41.319 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:41.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:41.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:41.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:41.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:41.560 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:41.600 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:41.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:41.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:41.702 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:41.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:41.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:41.840 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:41.878 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:41.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:41.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:41.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:42.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:42.017 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:42.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:42.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:42.158 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:42.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:42.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:10:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:10:42.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:42.302 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:42.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:42.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:42.404 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:42.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:42.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:42.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:42.550 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:42.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:42.587 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:42.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:42.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:42 smithi163 ceph-mon[29621]: pgmap v549: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 6.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 33980/311034 objects misplaced (10.925%); 24 MiB/s, 388 objects/s recovering 2024-03-21T13:10:42.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:42 smithi163 ceph-mon[40675]: pgmap v549: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 6.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 33980/311034 objects misplaced (10.925%); 24 MiB/s, 388 objects/s recovering 2024-03-21T13:10:42.688 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:42.729 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:42.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:42.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:42 smithi204 ceph-mon[33153]: pgmap v549: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 6.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 33980/311034 objects misplaced (10.925%); 24 MiB/s, 388 objects/s recovering 2024-03-21T13:10:42.830 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:42.874 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:42.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:42.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:42.976 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:43.010 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:43.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:43.093 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-21T13:10:43.112 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:43.184 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:43.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:43.286 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:43.327 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:43.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:43.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:43.429 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:43.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:43.476 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:43.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:43.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:43.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:43.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:43.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:43.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:43.759 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:43.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:43.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:43.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:43.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:43.861 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:43.898 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:43.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:43.999 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:44.002 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:10:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:44.037 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:44.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:44.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:44.155 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:10:44.155 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:10:44.156 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:10:44.156 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:10:44.156 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:10:44.156 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:10:44.156 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:10:44.156 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:10:44.156 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:10:44.156 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:10:44.156 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:10:44.156 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:10:44.156 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:10:44.157 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:10:44.157 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:10:44.157 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:10:44.172 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:44.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:44.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:44.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:44.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:44.328 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:44.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:44.429 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:44.479 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:44.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:44.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:44.496 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:10:44.581 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:44.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:44.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:44.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:44 smithi204 ceph-mon[33153]: pgmap v550: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 33282/305580 objects misplaced (10.891%); 24 MiB/s, 388 objects/s recovering 2024-03-21T13:10:44.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1469485388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:10:44.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:44 smithi163 ceph-mon[29621]: pgmap v550: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 33282/305580 objects misplaced (10.891%); 24 MiB/s, 388 objects/s recovering 2024-03-21T13:10:44.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1469485388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:10:44.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:44 smithi163 ceph-mon[40675]: pgmap v550: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 33282/305580 objects misplaced (10.891%); 24 MiB/s, 388 objects/s recovering 2024-03-21T13:10:44.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1469485388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:10:44.766 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:44.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:44.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:44.868 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:44.907 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:44.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:44.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:45.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:45.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:45.044 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:45.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:45.125 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:10:45.145 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:45.193 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:45.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:45.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:45.333 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:45.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:45.434 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:45.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:45.476 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:45.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:45.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:45.613 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:45.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:45.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:45.715 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:45.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:45.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:45.763 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:10:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:10:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:10:45.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:45.893 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:45.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:45.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:46.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:46.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:46.134 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:46.170 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:46.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:46.271 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:46.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:46.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:46.406 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:46.442 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:46.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:46.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:46.569 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:46.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:46.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:46.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:46.699 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:46.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:46.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:46.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:46 smithi204 ceph-mon[33153]: pgmap v551: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 541 B/s rd, 0 B/s wr, 805 op/s; 33282/305580 objects misplaced (10.891%); 11 MiB/s, 175 objects/s recovering 2024-03-21T13:10:46.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:46 smithi163 ceph-mon[29621]: pgmap v551: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 541 B/s rd, 0 B/s wr, 805 op/s; 33282/305580 objects misplaced (10.891%); 11 MiB/s, 175 objects/s recovering 2024-03-21T13:10:46.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:46 smithi163 ceph-mon[40675]: pgmap v551: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 541 B/s rd, 0 B/s wr, 805 op/s; 33282/305580 objects misplaced (10.891%); 11 MiB/s, 175 objects/s recovering 2024-03-21T13:10:46.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:46.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:46.826 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:46.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:46.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:46.961 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:46.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:46.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:47.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:47.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:47.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:47.111 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:10:47.199 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:47.241 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:47.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:47.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:47.343 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:47.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:47.378 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:47.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:47.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:47.531 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:47.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:47.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:47.635 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:47.683 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:47.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:47.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:47.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:47 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:10:47.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:47 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:10:47.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:47 smithi204 ceph-mon[33153]: osdmap e285: 8 total, 8 up, 6 in 2024-03-21T13:10:47.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:47 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:10:47.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:47 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:10:47.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:47 smithi163 ceph-mon[29621]: osdmap e285: 8 total, 8 up, 6 in 2024-03-21T13:10:47.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:47 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:10:47.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:47 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:10:47.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:47 smithi163 ceph-mon[40675]: osdmap e285: 8 total, 8 up, 6 in 2024-03-21T13:10:47.786 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:47.826 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:47.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:47.927 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:47.964 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:47.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:47.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:48.066 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:48.103 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:48.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:48.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:48.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:48.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:48.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:48.367 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:48.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:48.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:48.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:48.504 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:48.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:48.606 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:48.619 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:10:48.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:48.662 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:48.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:48.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:48 smithi204 ceph-mon[33153]: pgmap v553: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 5.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.30k op/s; 29559/286026 objects misplaced (10.334%); 23 MiB/s, 366 objects/s recovering 2024-03-21T13:10:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:48 smithi163 ceph-mon[29621]: pgmap v553: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 5.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.30k op/s; 29559/286026 objects misplaced (10.334%); 23 MiB/s, 366 objects/s recovering 2024-03-21T13:10:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:48 smithi163 ceph-mon[40675]: pgmap v553: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 5.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.30k op/s; 29559/286026 objects misplaced (10.334%); 23 MiB/s, 366 objects/s recovering 2024-03-21T13:10:48.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:48.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:48.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:48.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:48.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:48.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:48.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:49.040 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:49.078 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:49.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:49.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:49.216 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:49.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:49.317 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:49.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:49.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:49.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:49.490 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:49.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:49.592 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:49.628 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:49.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:49.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:49.730 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:49.762 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:49.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:49.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:49.864 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:49.892 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:49.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:49.993 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:50.033 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:50.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:50.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:50.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:50.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:50.264 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:50.295 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:50.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:50.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:50.397 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:50.427 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:50.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:50.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:50.529 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:50.568 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:50.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:50.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:50.670 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:50.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:50 smithi163 ceph-mon[29621]: pgmap v554: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 5.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 29559/286026 objects misplaced (10.334%); 22 MiB/s, 346 objects/s recovering 2024-03-21T13:10:50.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:50 smithi163 ceph-mon[29621]: osdmap e286: 8 total, 8 up, 6 in 2024-03-21T13:10:50.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:10:50.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:50 smithi163 ceph-mon[40675]: pgmap v554: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 5.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 29559/286026 objects misplaced (10.334%); 22 MiB/s, 346 objects/s recovering 2024-03-21T13:10:50.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:50 smithi163 ceph-mon[40675]: osdmap e286: 8 total, 8 up, 6 in 2024-03-21T13:10:50.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:10:50.707 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:50.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:50.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:50 smithi204 ceph-mon[33153]: pgmap v554: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 147 active+clean; 5.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 29559/286026 objects misplaced (10.334%); 22 MiB/s, 346 objects/s recovering 2024-03-21T13:10:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:50 smithi204 ceph-mon[33153]: osdmap e286: 8 total, 8 up, 6 in 2024-03-21T13:10:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:10:50.809 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:50.853 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:50.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:50.954 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:50.999 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:50.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:51.009 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:10:51.009 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:10:51.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:51.139 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:51.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:51.240 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:51.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:51.280 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:51.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:51.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:51.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:51.421 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:51.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:51.522 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:51.560 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:51.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:51.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:51.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:51 smithi163 ceph-mon[29621]: osdmap e287: 8 total, 8 up, 6 in 2024-03-21T13:10:51.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:51 smithi163 ceph-mon[29621]: pgmap v557: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 creating+peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 147 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.71k op/s; 26228/274848 objects misplaced (9.543%); 44 MiB/s, 706 objects/s recovering 2024-03-21T13:10:51.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:51 smithi163 ceph-mon[29621]: osdmap e288: 8 total, 8 up, 6 in 2024-03-21T13:10:51.651 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:51 smithi163 ceph-mon[40675]: osdmap e287: 8 total, 8 up, 6 in 2024-03-21T13:10:51.651 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:51 smithi163 ceph-mon[40675]: pgmap v557: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 creating+peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 147 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.71k op/s; 26228/274848 objects misplaced (9.543%); 44 MiB/s, 706 objects/s recovering 2024-03-21T13:10:51.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:51 smithi163 ceph-mon[40675]: osdmap e288: 8 total, 8 up, 6 in 2024-03-21T13:10:51.662 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:51.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:51.698 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:51.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:51.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:51.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:51.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:51.913 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:51 smithi204 ceph-mon[33153]: osdmap e287: 8 total, 8 up, 6 in 2024-03-21T13:10:51.913 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:51 smithi204 ceph-mon[33153]: pgmap v557: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 creating+peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 147 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.71k op/s; 26228/274848 objects misplaced (9.543%); 44 MiB/s, 706 objects/s recovering 2024-03-21T13:10:51.914 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:51 smithi204 ceph-mon[33153]: osdmap e288: 8 total, 8 up, 6 in 2024-03-21T13:10:51.936 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:51.964 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:51.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:51.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:52.066 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:52.093 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:52.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:52.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:52.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:52.223 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:52.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:52.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:10:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:10:52.324 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:52.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:52.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:52.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:52.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:52.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:52.504 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-21T13:10:52.604 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:52.637 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:52.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:52.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:52.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:52.772 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:52.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:52.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:52.874 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:52.903 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:52.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:52.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:53.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:53.031 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:53.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:53.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:53.160 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:53.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:53.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:53.262 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:53.292 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:53.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:53.393 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:53.424 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:53.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:53.526 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:53.561 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:53.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:53.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:53.663 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:53.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:53.701 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:53.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:53 smithi204 ceph-mon[33153]: osdmap e289: 8 total, 8 up, 6 in 2024-03-21T13:10:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:53 smithi163 ceph-mon[40675]: osdmap e289: 8 total, 8 up, 6 in 2024-03-21T13:10:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:10:53.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:53 smithi163 ceph-mon[29621]: osdmap e289: 8 total, 8 up, 6 in 2024-03-21T13:10:53.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:53.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:53.830 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:53.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:53.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:53.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:53.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:53.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:54.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:54.103 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:54.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:54.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:54.242 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:54.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:54.343 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:54.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:54.380 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:54.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:54.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:54.520 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:54.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:54.622 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:54.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:54.662 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:54.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:54.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:54 smithi163 ceph-mon[29621]: pgmap v560: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 creating+peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 147 active+clean; 5.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 25034/270234 objects misplaced (9.264%); 47 MiB/s, 751 objects/s recovering 2024-03-21T13:10:54.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:54 smithi163 ceph-mon[29621]: osdmap e290: 8 total, 8 up, 6 in 2024-03-21T13:10:54.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:54 smithi163 ceph-mon[40675]: pgmap v560: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 creating+peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 147 active+clean; 5.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 25034/270234 objects misplaced (9.264%); 47 MiB/s, 751 objects/s recovering 2024-03-21T13:10:54.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:54 smithi163 ceph-mon[40675]: osdmap e290: 8 total, 8 up, 6 in 2024-03-21T13:10:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:54 smithi204 ceph-mon[33153]: pgmap v560: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 creating+peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 147 active+clean; 5.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 25034/270234 objects misplaced (9.264%); 47 MiB/s, 751 objects/s recovering 2024-03-21T13:10:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:54 smithi204 ceph-mon[33153]: osdmap e290: 8 total, 8 up, 6 in 2024-03-21T13:10:54.764 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:54.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:54.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:54.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:54.905 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:54.945 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:54.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:55.047 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:55.073 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:55.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:55.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:55.176 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:55.213 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:55.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:55.314 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:55.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:55.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:55.442 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:55.479 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:55.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:55.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:55.581 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:55.617 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:55.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:55.718 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:55.753 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:55.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:55.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:55.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:55.890 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:55.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:55.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:56.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:10:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:10:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:10:56.025 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:56.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:56.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:56.127 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:56.179 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:56.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:56.280 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:56.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:56.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:56.396 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:56 smithi163 ceph-mon[29621]: pgmap v562: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 creating+peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 147 active+clean; 5.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 353 op/s; 25034/270234 objects misplaced (9.264%); 8.2 MiB/s, 131 objects/s recovering 2024-03-21T13:10:56.397 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:56 smithi163 ceph-mon[40675]: pgmap v562: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 creating+peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 147 active+clean; 5.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 353 op/s; 25034/270234 objects misplaced (9.264%); 8.2 MiB/s, 131 objects/s recovering 2024-03-21T13:10:56.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:56.453 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:56.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:56.554 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:56.591 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:56.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:56.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:56.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:56.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:56.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:56 smithi204 ceph-mon[33153]: pgmap v562: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 creating+peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 147 active+clean; 5.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 353 op/s; 25034/270234 objects misplaced (9.264%); 8.2 MiB/s, 131 objects/s recovering 2024-03-21T13:10:56.830 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:56.872 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:56.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:56.913 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:10:56.973 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:10:57.020 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:57.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:10:57.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:10:57.160 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:57.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:57.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:10:57.193 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:10:57.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:57.296 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:57.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:57.398 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:57.442 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:57.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:57.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:57.544 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:57.586 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:57.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:57.688 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:57.728 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:57.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:57.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:57.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:57.867 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:57.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:57.968 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:58.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:58.004 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:58.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:58.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:58.141 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:58.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:58.243 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:58.282 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:58.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:58.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:58.384 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:58.415 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:10:58.421 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.4 to 1 (10000) 2024-03-21T13:10:58.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:58.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:58.528 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:10:58.564 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:58.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:10:58.666 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:58.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:58 smithi163 ceph-mon[29621]: pgmap v563: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s; 19882/255612 objects misplaced (7.778%); 48 MiB/s, 760 objects/s recovering 2024-03-21T13:10:58.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:58 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3025146529' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:10:58.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:58 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2245192551' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:10:58.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:58 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:10:58.669 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:58 smithi163 ceph-mon[40675]: pgmap v563: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s; 19882/255612 objects misplaced (7.778%); 48 MiB/s, 760 objects/s recovering 2024-03-21T13:10:58.669 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:58 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3025146529' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:10:58.669 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:58 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2245192551' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:10:58.669 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:58 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:10:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:58.703 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:58.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:58 smithi204 ceph-mon[33153]: pgmap v563: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s; 19882/255612 objects misplaced (7.778%); 48 MiB/s, 760 objects/s recovering 2024-03-21T13:10:58.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:58 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3025146529' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:10:58.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:58 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2245192551' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:10:58.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:58 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:10:58.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:10:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:58.843 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:58.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:10:58.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:58.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:58.989 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:58.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:59.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:59.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:59.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:59.229 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:59.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:59.269 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:59.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:59.370 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:10:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:59.407 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:59.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:10:59.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:59.545 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:59.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:10:59.646 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:10:59.682 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:10:59.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:59.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:10:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:59 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:10:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:59 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3025146529' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:10:59.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:59 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:10:59.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:10:59 smithi204 ceph-mon[33153]: osdmap e291: 8 total, 8 up, 6 in 2024-03-21T13:10:59.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:59 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:10:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:59 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3025146529' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:10:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:59 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:10:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:10:59 smithi163 ceph-mon[29621]: osdmap e291: 8 total, 8 up, 6 in 2024-03-21T13:10:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:59 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:10:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:59 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3025146529' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:10:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:59 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:10:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:10:59 smithi163 ceph-mon[40675]: osdmap e291: 8 total, 8 up, 6 in 2024-03-21T13:10:59.784 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:10:59.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:59.821 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:59.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:10:59.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:10:59.963 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:10:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:59.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:00.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:00.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:00.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:00.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:00.237 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:00.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:00.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:00.339 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:00.374 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:00.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:00.475 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:00.514 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:00.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:00.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:00.616 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:00.641 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:00.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:00.741 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:00.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:00 smithi204 ceph-mon[33153]: pgmap v565: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 790 B/s rd, 0 B/s wr, 753 op/s; 19882/255612 objects misplaced (7.778%); 38 MiB/s, 616 objects/s recovering 2024-03-21T13:11:00.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:00 smithi163 ceph-mon[29621]: pgmap v565: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 790 B/s rd, 0 B/s wr, 753 op/s; 19882/255612 objects misplaced (7.778%); 38 MiB/s, 616 objects/s recovering 2024-03-21T13:11:00.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:00 smithi163 ceph-mon[40675]: pgmap v565: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 790 B/s rd, 0 B/s wr, 753 op/s; 19882/255612 objects misplaced (7.778%); 38 MiB/s, 616 objects/s recovering 2024-03-21T13:11:00.777 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:00.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:00.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:00.879 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:00.910 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:00.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:01.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:01.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:01.037 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:01.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:01.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:01.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:01.176 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:01.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:01.277 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:01.314 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:01.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:01.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:01.453 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:01.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:01.461 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:11:01.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:01 smithi163 ceph-mon[29621]: 4.e scrub starts 2024-03-21T13:11:01.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:01 smithi163 ceph-mon[29621]: 4.e scrub ok 2024-03-21T13:11:01.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:01 smithi163 ceph-mon[40675]: 4.e scrub starts 2024-03-21T13:11:01.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:01 smithi163 ceph-mon[40675]: 4.e scrub ok 2024-03-21T13:11:01.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:01.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:01.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:01.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:01.695 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:01.741 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:01.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:01.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:01 smithi204 ceph-mon[33153]: 4.e scrub starts 2024-03-21T13:11:01.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:01 smithi204 ceph-mon[33153]: 4.e scrub ok 2024-03-21T13:11:01.842 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:01.895 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:01.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:01.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:01.997 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:02.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:02.032 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:02.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:02.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:02.171 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:02.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:02.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:11:01 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:11:02.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:02.305 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:02.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:02.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:02.311 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-21T13:11:02.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:02.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:02.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:02.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:02 smithi163 ceph-mon[29621]: pgmap v566: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 4.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 17822/241005 objects misplaced (7.395%); 41 MiB/s, 651 objects/s recovering 2024-03-21T13:11:02.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:02 smithi163 ceph-mon[29621]: 6.1e scrub starts 2024-03-21T13:11:02.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:02 smithi163 ceph-mon[29621]: 6.1e scrub ok 2024-03-21T13:11:02.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:02 smithi163 ceph-mon[29621]: 5.18 scrub starts 2024-03-21T13:11:02.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:02 smithi163 ceph-mon[29621]: 5.18 scrub ok 2024-03-21T13:11:02.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:02 smithi163 ceph-mon[40675]: pgmap v566: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 4.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 17822/241005 objects misplaced (7.395%); 41 MiB/s, 651 objects/s recovering 2024-03-21T13:11:02.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:02 smithi163 ceph-mon[40675]: 6.1e scrub starts 2024-03-21T13:11:02.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:02 smithi163 ceph-mon[40675]: 6.1e scrub ok 2024-03-21T13:11:02.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:02 smithi163 ceph-mon[40675]: 5.18 scrub starts 2024-03-21T13:11:02.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:02 smithi163 ceph-mon[40675]: 5.18 scrub ok 2024-03-21T13:11:02.545 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:02.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:02.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:02.686 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:02.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:02.730 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:02.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:02 smithi204 ceph-mon[33153]: pgmap v566: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 4.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 17822/241005 objects misplaced (7.395%); 41 MiB/s, 651 objects/s recovering 2024-03-21T13:11:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:02 smithi204 ceph-mon[33153]: 6.1e scrub starts 2024-03-21T13:11:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:02 smithi204 ceph-mon[33153]: 6.1e scrub ok 2024-03-21T13:11:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:02 smithi204 ceph-mon[33153]: 5.18 scrub starts 2024-03-21T13:11:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:02 smithi204 ceph-mon[33153]: 5.18 scrub ok 2024-03-21T13:11:02.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:02.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:02.900 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:02.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:03.001 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:03.040 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:03.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:03.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:03.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:03.182 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:03.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:03.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:03.284 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:03.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:03.467 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:03.507 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:03.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:03.608 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:03.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:03.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:03 smithi163 ceph-mon[29621]: 5.c scrub starts 2024-03-21T13:11:03.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:03 smithi163 ceph-mon[29621]: 5.c scrub ok 2024-03-21T13:11:03.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:03 smithi163 ceph-mon[29621]: 6.a scrub starts 2024-03-21T13:11:03.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:03 smithi163 ceph-mon[29621]: 6.a scrub ok 2024-03-21T13:11:03.612 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:03.612 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:03 smithi163 ceph-mon[40675]: 5.c scrub starts 2024-03-21T13:11:03.612 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:03 smithi163 ceph-mon[40675]: 5.c scrub ok 2024-03-21T13:11:03.612 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:03 smithi163 ceph-mon[40675]: 6.a scrub starts 2024-03-21T13:11:03.613 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:03 smithi163 ceph-mon[40675]: 6.a scrub ok 2024-03-21T13:11:03.639 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:03.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:03.703 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:03.703 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:03.718 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:11:03.741 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:03.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:03.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:03 smithi204 ceph-mon[33153]: 5.c scrub starts 2024-03-21T13:11:03.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:03 smithi204 ceph-mon[33153]: 5.c scrub ok 2024-03-21T13:11:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:03 smithi204 ceph-mon[33153]: 6.a scrub starts 2024-03-21T13:11:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:03 smithi204 ceph-mon[33153]: 6.a scrub ok 2024-03-21T13:11:03.756 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:11:03.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:03.776 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:03.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:03.790 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:11:03.820 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:11:03.851 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:11:03.877 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:03.882 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:11:03.908 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:03.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:03.912 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:11:03.941 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:11:03.943 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:11:03.943 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:11:03.943 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:11:03.943 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:11:03.943 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:11:03.943 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:11:03.943 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:11:03.943 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:11:04.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:04.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:04.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:04.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:04.160 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:04.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:04.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:04.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:04.287 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:04.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:04.388 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:04.419 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:04.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:04.521 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:04.561 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:04.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:04.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:04.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:04.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:04.697 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:04.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:04.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:04 smithi163 ceph-mon[29621]: pgmap v567: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 4.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s; 16340/234174 objects misplaced (6.978%); 39 MiB/s, 630 objects/s recovering 2024-03-21T13:11:04.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:04 smithi163 ceph-mon[29621]: osdmap e292: 8 total, 8 up, 6 in 2024-03-21T13:11:04.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:04 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1008205288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:11:04.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:04 smithi163 ceph-mon[29621]: 9.18 scrub starts 2024-03-21T13:11:04.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:04 smithi163 ceph-mon[29621]: 9.18 scrub ok 2024-03-21T13:11:04.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:04 smithi163 ceph-mon[40675]: pgmap v567: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 4.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s; 16340/234174 objects misplaced (6.978%); 39 MiB/s, 630 objects/s recovering 2024-03-21T13:11:04.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:04 smithi163 ceph-mon[40675]: osdmap e292: 8 total, 8 up, 6 in 2024-03-21T13:11:04.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:04 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1008205288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:11:04.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:04 smithi163 ceph-mon[40675]: 9.18 scrub starts 2024-03-21T13:11:04.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:04 smithi163 ceph-mon[40675]: 9.18 scrub ok 2024-03-21T13:11:04.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:04 smithi204 ceph-mon[33153]: pgmap v567: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 4.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s; 16340/234174 objects misplaced (6.978%); 39 MiB/s, 630 objects/s recovering 2024-03-21T13:11:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:04 smithi204 ceph-mon[33153]: osdmap e292: 8 total, 8 up, 6 in 2024-03-21T13:11:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:04 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1008205288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:11:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:04 smithi204 ceph-mon[33153]: 9.18 scrub starts 2024-03-21T13:11:04.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:04 smithi204 ceph-mon[33153]: 9.18 scrub ok 2024-03-21T13:11:04.798 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:04.832 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:04.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:04.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:04.968 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:04.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:05.070 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:05.109 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:05.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:05.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:05.211 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:05.212 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:11:05.212 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-03-21T13:11:05.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-03-21T13:11:05.213 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:11:05.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.213 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:197997 8.e 8:72fc6c89:::benchmark_data_smithi163_107079_object62124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.213 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.339543+0000", 2024-03-21T13:11:05.213 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.86877694099999, 2024-03-21T13:11:05.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98943099899999998, 2024-03-21T13:11:05.213 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.213 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.213 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.213 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.213 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.213 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 197997 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339543+0000", 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339543+0000", 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339544+0000", 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7400000000000002e-07 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339548+0000", 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0400000000000003e-06 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339559+0000", 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0631e-05 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339566+0000", 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.661e-06 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339624+0000", 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7519999999999998e-05 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339632+0000", 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5690000000000005e-06 2024-03-21T13:11:05.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.325535+0000", 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98590222599999999 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.325675+0000", 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014081 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.325799+0000", 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012380099999999999 2024-03-21T13:11:05.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.218 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.218 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.328858+0000", 2024-03-21T13:11:05.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003058659 2024-03-21T13:11:05.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.218 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.218 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.328892+0000", 2024-03-21T13:11:05.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4462999999999999e-05 2024-03-21T13:11:05.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.218 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.218 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.328903+0000", 2024-03-21T13:11:05.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0261000000000001e-05 2024-03-21T13:11:05.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.328933+0000", 2024-03-21T13:11:05.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9887e-05 2024-03-21T13:11:05.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.328974+0000", 2024-03-21T13:11:05.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1696999999999997e-05 2024-03-21T13:11:05.220 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.220 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.220 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.220 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198019 8.e 8:79bb99bc:::benchmark_data_smithi163_107079_object62146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.220 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.353797+0000", 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.85452277900001, 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.97779139900000001, 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198019 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353797+0000", 2024-03-21T13:11:05.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353797+0000", 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353798+0000", 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9100000000000004e-07 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353801+0000", 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4060000000000001e-06 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353809+0000", 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8849999999999999e-06 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353816+0000", 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3969999999999999e-06 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.223 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353870+0000", 2024-03-21T13:11:05.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4196000000000001e-05 2024-03-21T13:11:05.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353876+0000", 2024-03-21T13:11:05.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.241e-06 2024-03-21T13:11:05.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.327830+0000", 2024-03-21T13:11:05.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.97395389600000004 2024-03-21T13:11:05.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.327900+0000", 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9678000000000006e-05 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.328004+0000", 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104032 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331359+0000", 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033555260000000002 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331410+0000", 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0231000000000001e-05 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331476+0000", 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6135000000000003e-05 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331534+0000", 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8661e-05 2024-03-21T13:11:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331589+0000", 2024-03-21T13:11:05.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4524000000000002e-05 2024-03-21T13:11:05.228 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.228 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.228 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198149 8.e 8:7688df86:::benchmark_data_smithi163_107079_object62276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.432999+0000", 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.775321029, 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.90199298900000002, 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198149 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.229 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.432999+0000", 2024-03-21T13:11:05.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.432999+0000", 2024-03-21T13:11:05.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.433001+0000", 2024-03-21T13:11:05.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.395e-06 2024-03-21T13:11:05.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.433014+0000", 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3903000000000001e-05 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.433035+0000", 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0395e-05 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.433043+0000", 2024-03-21T13:11:05.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5750000000000001e-06 2024-03-21T13:11:05.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.433126+0000", 2024-03-21T13:11:05.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.297e-05 2024-03-21T13:11:05.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.433138+0000", 2024-03-21T13:11:05.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1861e-05 2024-03-21T13:11:05.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.329093+0000", 2024-03-21T13:11:05.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.89595484400000003 2024-03-21T13:11:05.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.329200+0000", 2024-03-21T13:11:05.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107267 2024-03-21T13:11:05.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.329265+0000", 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4258999999999996e-05 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331600+0000", 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023357 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.334786+0000", 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031856190000000002 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.334840+0000", 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4367999999999999e-05 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.334942+0000", 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000101543 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.334992+0000", 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0290000000000001e-05 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.236 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.237 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198166 8.e 8:70e11697:::benchmark_data_smithi163_107079_object62293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.237 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.448097+0000", 2024-03-21T13:11:05.237 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.76022304200001, 2024-03-21T13:11:05.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.89185160900000005, 2024-03-21T13:11:05.237 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.237 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.237 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.237 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.237 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.237 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198166 2024-03-21T13:11:05.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448097+0000", 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448097+0000", 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448099+0000", 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.685e-06 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448105+0000", 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2999999999999998e-06 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448123+0000", 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8187000000000001e-05 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448133+0000", 2024-03-21T13:11:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7790000000000007e-06 2024-03-21T13:11:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448218+0000", 2024-03-21T13:11:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5099000000000006e-05 2024-03-21T13:11:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448229+0000", 2024-03-21T13:11:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0497e-05 2024-03-21T13:11:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.330366+0000", 2024-03-21T13:11:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.88213745099999996 2024-03-21T13:11:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.330435+0000", 2024-03-21T13:11:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8779e-05 2024-03-21T13:11:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.330499+0000", 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4129999999999998e-05 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.334736+0000", 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0042372149999999999 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.335060+0000", 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032360700000000001 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.339846+0000", 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047860309999999996 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.339902+0000", 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.588e-05 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.339949+0000", 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6968999999999997e-05 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198169 8.e 8:7f3d2014:::benchmark_data_smithi163_107079_object62296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.452482+0000", 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.75583777099999, 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.88774095099999994, 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198169 2024-03-21T13:11:05.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452482+0000", 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452482+0000", 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452484+0000", 2024-03-21T13:11:05.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.486e-06 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452491+0000", 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8390000000000001e-06 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452503+0000", 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.253e-05 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452512+0000", 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9330000000000007e-06 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452623+0000", 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110587 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452632+0000", 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7290000000000006e-06 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331656+0000", 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.879023997 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331767+0000", 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000111239 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331890+0000", 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000123306 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.334800+0000", 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0029102360000000001 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.335161+0000", 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00036037599999999998 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340133+0000", 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049728039999999999 2024-03-21T13:11:05.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340184+0000", 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0417999999999997e-05 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340223+0000", 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9471000000000002e-05 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198177 8.e 8:704f885f:::benchmark_data_smithi163_107079_object62304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.456512+0000", 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.751808453, 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.883838499, 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198177 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456512+0000", 2024-03-21T13:11:05.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456512+0000", 2024-03-21T13:11:05.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456513+0000", 2024-03-21T13:11:05.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4750000000000001e-06 2024-03-21T13:11:05.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456520+0000", 2024-03-21T13:11:05.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7900000000000002e-06 2024-03-21T13:11:05.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456538+0000", 2024-03-21T13:11:05.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7883000000000001e-05 2024-03-21T13:11:05.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456548+0000", 2024-03-21T13:11:05.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8560000000000002e-06 2024-03-21T13:11:05.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456610+0000", 2024-03-21T13:11:05.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2644000000000005e-05 2024-03-21T13:11:05.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456617+0000", 2024-03-21T13:11:05.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9349999999999997e-06 2024-03-21T13:11:05.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.333229+0000", 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.87661202800000004 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.333343+0000", 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114021 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.333468+0000", 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124461 2024-03-21T13:11:05.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.335178+0000", 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0017099280000000001 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.339788+0000", 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0046099510000000002 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340273+0000", 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048556599999999999 2024-03-21T13:11:05.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340316+0000", 2024-03-21T13:11:05.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2725000000000002e-05 2024-03-21T13:11:05.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340350+0000", 2024-03-21T13:11:05.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4236000000000002e-05 2024-03-21T13:11:05.258 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.258 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.258 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.258 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198178 8.e 8:721bb05b:::benchmark_data_smithi163_107079_object62305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.456814+0000", 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.751505799, 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.88372909700000002, 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198178 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456814+0000", 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456814+0000", 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456815+0000", 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456820+0000", 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7890000000000002e-06 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456832+0000", 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1558e-05 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456839+0000", 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8380000000000002e-06 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456893+0000", 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3254999999999997e-05 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456903+0000", 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9019999999999994e-06 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.335208+0000", 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.87830587000000004 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.335859+0000", 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00065099199999999998 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.335970+0000", 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110944 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.339961+0000", 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039907049999999998 2024-03-21T13:11:05.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340080+0000", 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118522 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340388+0000", 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030865299999999999 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340503+0000", 2024-03-21T13:11:05.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011429099999999999 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340543+0000", 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0973999999999998e-05 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198182 8.e 8:7bcb4e00:::benchmark_data_smithi163_107079_object62309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.457774+0000", 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.750545711, 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.88472258800000003, 2024-03-21T13:11:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198182 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457774+0000", 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457774+0000", 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457775+0000", 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4300000000000001e-07 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457781+0000", 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2780000000000001e-06 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457795+0000", 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4754e-05 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457804+0000", 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3270000000000001e-06 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457874+0000", 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9811e-05 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457881+0000", 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2799999999999998e-06 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.337893+0000", 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.88001165699999995 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.338013+0000", 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120188 2024-03-21T13:11:05.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.338128+0000", 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115423 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340236+0000", 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021079570000000001 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342388+0000", 2024-03-21T13:11:05.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021521140000000001 2024-03-21T13:11:05.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342434+0000", 2024-03-21T13:11:05.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5337999999999999e-05 2024-03-21T13:11:05.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342469+0000", 2024-03-21T13:11:05.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5061999999999997e-05 2024-03-21T13:11:05.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342497+0000", 2024-03-21T13:11:05.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8456e-05 2024-03-21T13:11:05.272 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.272 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.272 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198184 8.e 8:7a1e0247:::benchmark_data_smithi163_107079_object62311:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.457973+0000", 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.750346933, 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.89018627299999997, 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198184 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457973+0000", 2024-03-21T13:11:05.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457973+0000", 2024-03-21T13:11:05.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457974+0000", 2024-03-21T13:11:05.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8899999999999998e-07 2024-03-21T13:11:05.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457979+0000", 2024-03-21T13:11:05.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2100000000000001e-06 2024-03-21T13:11:05.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457990+0000", 2024-03-21T13:11:05.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0969e-05 2024-03-21T13:11:05.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457997+0000", 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8329999999999996e-06 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.458072+0000", 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4911000000000002e-05 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.458079+0000", 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0140000000000001e-06 2024-03-21T13:11:05.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340576+0000", 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.88249676899999996 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340684+0000", 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000108677 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340796+0000", 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000111854 2024-03-21T13:11:05.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.347782+0000", 2024-03-21T13:11:05.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0069861200000000002 2024-03-21T13:11:05.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.347830+0000", 2024-03-21T13:11:05.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7663000000000001e-05 2024-03-21T13:11:05.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.347939+0000", 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010846899999999999 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348109+0000", 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017050400000000001 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348160+0000", 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0491000000000002e-05 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.279 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198188 8.e 8:71a99677:::benchmark_data_smithi163_107079_object62315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.462609+0000", 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.745711234, 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.89143663399999995, 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198188 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.280 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462609+0000", 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462609+0000", 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462611+0000", 2024-03-21T13:11:05.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6360000000000001e-06 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462617+0000", 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4749999999999998e-06 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462630+0000", 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2551999999999999e-05 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462638+0000", 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6030000000000005e-06 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462696+0000", 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7834999999999998e-05 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462706+0000", 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6360000000000006e-06 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342529+0000", 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.87982343500000004 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342627+0000", 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7701999999999994e-05 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342753+0000", 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012617499999999999 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.347843+0000", 2024-03-21T13:11:05.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050896400000000003 2024-03-21T13:11:05.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348204+0000", 2024-03-21T13:11:05.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00036116300000000001 2024-03-21T13:11:05.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.353943+0000", 2024-03-21T13:11:05.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057395900000000001 2024-03-21T13:11:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:05.286 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:05.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:05.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.353996+0000", 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2973000000000003e-05 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354046+0000", 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9218999999999998e-05 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198243 8.e 8:700a35e6:::benchmark_data_smithi163_107079_object62370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.506276+0000", 2024-03-21T13:11:05.288 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.702044046, 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.84810814400000001, 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198243 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506276+0000", 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506276+0000", 2024-03-21T13:11:05.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506278+0000", 2024-03-21T13:11:05.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3659999999999999e-06 2024-03-21T13:11:05.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506284+0000", 2024-03-21T13:11:05.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0560000000000001e-06 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506295+0000", 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1175000000000001e-05 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506312+0000", 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.753e-05 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506379+0000", 2024-03-21T13:11:05.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6618000000000005e-05 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506390+0000", 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0638e-05 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.344552+0000", 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.83816236499999996 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.344660+0000", 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010765 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.344726+0000", 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6173000000000005e-05 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.347849+0000", 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031237109999999999 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348241+0000", 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039150099999999999 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354126+0000", 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0058847830000000002 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354338+0000", 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00021213800000000001 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354384+0000", 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6440000000000003e-05 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198261 8.e 8:72991b00:::benchmark_data_smithi163_107079_object62388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.518817+0000", 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.68950369999999, 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.83579388399999999, 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.295 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198261 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518817+0000", 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518817+0000", 2024-03-21T13:11:05.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518818+0000", 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.776e-06 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518824+0000", 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8599999999999998e-06 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518840+0000", 2024-03-21T13:11:05.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5987000000000001e-05 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518849+0000", 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4409999999999993e-06 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518913+0000", 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4414999999999999e-05 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518922+0000", 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6500000000000002e-06 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.346413+0000", 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.82749177500000004 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.346487+0000", 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3609000000000006e-05 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.346554+0000", 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6489999999999995e-05 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.353886+0000", 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0073328710000000004 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.353898+0000", 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354507+0000", 2024-03-21T13:11:05.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00060872200000000002 2024-03-21T13:11:05.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354572+0000", 2024-03-21T13:11:05.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5137999999999999e-05 2024-03-21T13:11:05.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354610+0000", 2024-03-21T13:11:05.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8352000000000001e-05 2024-03-21T13:11:05.301 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198285 8.e 8:7c6267f4:::benchmark_data_smithi163_107079_object62412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.540290+0000", 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.66802976599999, 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.81888756500000004, 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.302 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198285 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540290+0000", 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540290+0000", 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540292+0000", 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3570000000000001e-06 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540297+0000", 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2429999999999996e-06 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540307+0000", 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.026e-05 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540315+0000", 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5889999999999997e-06 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540375+0000", 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9960000000000002e-05 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540384+0000", 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6509999999999996e-06 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348268+0000", 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.80788329000000003 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348361+0000", 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3122999999999996e-05 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348463+0000", 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000102512 2024-03-21T13:11:05.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.353903+0000", 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054392429999999999 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354086+0000", 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018381300000000001 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359076+0000", 2024-03-21T13:11:05.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049894859999999996 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359131+0000", 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5441e-05 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359178+0000", 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6597000000000001e-05 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198325 8.e 8:7a044c9a:::benchmark_data_smithi163_107079_object62452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.588828+0000", 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.61949211800001, 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.770531403, 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198325 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588828+0000", 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588828+0000", 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588829+0000", 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.26e-06 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588834+0000", 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4100000000000001e-06 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588843+0000", 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2439999999999999e-06 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588849+0000", 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8680000000000004e-06 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588891+0000", 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2385999999999997e-05 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588897+0000", 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1940000000000003e-06 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.350134+0000", 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.76123635999999995 2024-03-21T13:11:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.350236+0000", 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000102503 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.350345+0000", 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010907499999999999 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.353907+0000", 2024-03-21T13:11:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035621030000000001 2024-03-21T13:11:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359012+0000", 2024-03-21T13:11:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0051040729999999998 2024-03-21T13:11:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359275+0000", 2024-03-21T13:11:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000263341 2024-03-21T13:11:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359323+0000", 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7743e-05 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359359+0000", 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6842999999999999e-05 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198333 8.e 8:7573d736:::benchmark_data_smithi163_107079_object62460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:11:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.597489+0000", 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 218.61083090100001, 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.76345962700000003, 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198333 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.597489+0000", 2024-03-21T13:11:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.597489+0000", 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.597491+0000", 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.325e-06 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.597497+0000", 2024-03-21T13:11:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9900000000000002e-06 2024-03-21T13:11:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.597509+0000", 2024-03-21T13:11:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2274e-05 2024-03-21T13:11:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.597516+0000", 2024-03-21T13:11:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2660000000000004e-06 2024-03-21T13:11:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.597580+0000", 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3578999999999996e-05 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.597589+0000", 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2329999999999995e-06 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.352006+0000", 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.75441725100000001 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.352093+0000", 2024-03-21T13:11:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6304999999999994e-05 2024-03-21T13:11:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:11:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.352163+0000", 2024-03-21T13:11:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0098000000000005e-05 2024-03-21T13:11:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.356832+0000", 2024-03-21T13:11:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0046689490000000004 2024-03-21T13:11:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359224+0000", 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023921110000000002 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.360805+0000", 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001580906 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.360899+0000", 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4790999999999994e-05 2024-03-21T13:11:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.360949+0000", 2024-03-21T13:11:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9549000000000003e-05 2024-03-21T13:11:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:54244 9.12 9.7c67c952 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-03-21T13:11:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:09:19.445638+0000", 2024-03-21T13:11:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.762682688, 2024-03-21T13:11:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.99024263899999998, 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 54244 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445638+0000", 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445638+0000", 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445639+0000", 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0890000000000001e-06 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445668+0000", 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9726e-05 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445676+0000", 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0660000000000004e-06 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445683+0000", 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3990000000000001e-06 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445741+0000", 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7972e-05 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445751+0000", 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0227999999999999e-05 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:20.435868+0000", 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.99011690600000002 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:20.435880+0000", 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2252999999999999e-05 2024-03-21T13:11:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:54255 9.12 9.40534e52 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-03-21T13:11:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:09:19.449680+0000", 2024-03-21T13:11:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.758640189, 2024-03-21T13:11:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98623570800000004, 2024-03-21T13:11:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:11:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:11:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:11:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 54255 2024-03-21T13:11:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449680+0000", 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449680+0000", 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449681+0000", 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2559999999999999e-06 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449704+0000", 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2748e-05 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449719+0000", 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4584e-05 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449726+0000", 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6990000000000003e-06 2024-03-21T13:11:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449788+0000", 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1667000000000004e-05 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449797+0000", 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.799e-06 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:20.435907+0000", 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98611042400000004 2024-03-21T13:11:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:20.435916+0000", 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5310000000000001e-06 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:100377 9.12 9:4e5aa4e2:::benchmark_data_smithi163_131650_object100376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:02.557198+0000", 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 62.651122289, 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1069594089999999, 2024-03-21T13:11:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100377 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557198+0000", 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557198+0000", 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557199+0000", 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.344e-06 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557224+0000", 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888e-05 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557238+0000", 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3759e-05 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557245+0000", 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3529999999999997e-06 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557311+0000", 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5513999999999996e-05 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557434+0000", 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012332599999999999 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557755+0000", 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000320718 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.560432+0000", 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026767940000000001 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.561289+0000", 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00085723799999999997 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.573020+0000", 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011730908999999999 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.573043+0000", 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3289999999999999e-05 2024-03-21T13:11:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584602+0000", 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0115587029999999 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584621+0000", 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9125000000000001e-05 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.664079+0000", 2024-03-21T13:11:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0794580840000001 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.664157+0000", 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8363999999999997e-05 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:100402 9.12 9:4d69482a:::benchmark_data_smithi163_131650_object100401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:02.595353+0000", 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 62.612966978000003, 2024-03-21T13:11:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0785798529999999, 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100402 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595353+0000", 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595353+0000", 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595354+0000", 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6400000000000002e-07 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595374+0000", 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0288e-05 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595381+0000", 2024-03-21T13:11:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3690000000000001e-06 2024-03-21T13:11:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595388+0000", 2024-03-21T13:11:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-03-21T13:11:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595445+0000", 2024-03-21T13:11:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7123000000000002e-05 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595461+0000", 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5758000000000001e-05 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584764+0000", 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98930377599999997 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584774+0000", 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7000000000000003e-06 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.664696+0000", 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.079921769 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.664847+0000", 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000151249 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,3,7", 2024-03-21T13:11:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.665217+0000", 2024-03-21T13:11:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00037034999999999998 2024-03-21T13:11:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673509+0000", 2024-03-21T13:11:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082912909999999992 2024-03-21T13:11:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673550+0000", 2024-03-21T13:11:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1844000000000001e-05 2024-03-21T13:11:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673597+0000", 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.651e-05 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673749+0000", 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000152082 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673821+0000", 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2371000000000002e-05 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673933+0000", 2024-03-21T13:11:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000111629 2024-03-21T13:11:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:100408 9.12 9:4f7d3001:::benchmark_data_smithi163_131650_object100407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:02.605697+0000", 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 62.602623442999999, 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0683809150000001, 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100408 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605697+0000", 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605697+0000", 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605701+0000", 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.985e-06 2024-03-21T13:11:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605730+0000", 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8864999999999999e-05 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605741+0000", 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.148e-05 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605748+0000", 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3839999999999999e-06 2024-03-21T13:11:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605818+0000", 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9108000000000003e-05 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605829+0000", 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1874e-05 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584799+0000", 2024-03-21T13:11:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.97896959400000005 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584808+0000", 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2320000000000007e-06 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.667194+0000", 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.082386031 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.667343+0000", 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000148672 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,3,7", 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.667701+0000", 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035776999999999998 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673640+0000", 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.005939405 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673684+0000", 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4180000000000001e-05 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673730+0000", 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5911000000000001e-05 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673945+0000", 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000214606 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673989+0000", 2024-03-21T13:11:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3665000000000003e-05 2024-03-21T13:11:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.674078+0000", 2024-03-21T13:11:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9153000000000005e-05 2024-03-21T13:11:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:05.357 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:11:05.388 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:05.428 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:05.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:05.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:05 smithi163 ceph-mon[29621]: osdmap e293: 8 total, 8 up, 6 in 2024-03-21T13:11:05.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:05.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:05 smithi163 ceph-mon[40675]: osdmap e293: 8 total, 8 up, 6 in 2024-03-21T13:11:05.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:05.530 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:05.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:05.571 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:05.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:05.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:05.703 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:05.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:05.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:05 smithi204 ceph-mon[33153]: osdmap e293: 8 total, 8 up, 6 in 2024-03-21T13:11:05.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:05.805 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:05.838 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:05.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:05.940 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:05.970 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:05.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:06.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:11:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:11:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:11:06.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:06.094 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:06.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:06.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:06.220 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:06.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:06.322 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:06.352 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:06.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:06.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:06.454 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:06.481 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:06.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:06.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:06.584 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:06.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:06.626 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:06.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:06.727 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:06.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:06 smithi204 ceph-mon[33153]: pgmap v570: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 4.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 795 B/s rd, 0 B/s wr, 1.11k op/s; 16340/234174 objects misplaced (6.978%); 19 MiB/s, 308 objects/s recovering 2024-03-21T13:11:06.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:06 smithi204 ceph-mon[33153]: osdmap e294: 8 total, 8 up, 6 in 2024-03-21T13:11:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:06 smithi163 ceph-mon[29621]: pgmap v570: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 4.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 795 B/s rd, 0 B/s wr, 1.11k op/s; 16340/234174 objects misplaced (6.978%); 19 MiB/s, 308 objects/s recovering 2024-03-21T13:11:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:06 smithi163 ceph-mon[29621]: osdmap e294: 8 total, 8 up, 6 in 2024-03-21T13:11:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:06 smithi163 ceph-mon[40675]: pgmap v570: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 151 active+clean; 4.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 795 B/s rd, 0 B/s wr, 1.11k op/s; 16340/234174 objects misplaced (6.978%); 19 MiB/s, 308 objects/s recovering 2024-03-21T13:11:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:06 smithi163 ceph-mon[40675]: osdmap e294: 8 total, 8 up, 6 in 2024-03-21T13:11:06.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:06.768 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:06.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:06.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:06.903 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:06.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:06.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:07.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:07.039 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:07.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:07.140 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:07.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:07.177 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:07.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:07.278 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:07.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:07.302 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:07.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:07.344 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:11:07.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:07.438 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:07.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:07.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:07.540 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:07.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:07.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:07.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:07.712 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:07.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:07.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:07 smithi204 ceph-mon[33153]: osdmap e295: 8 total, 8 up, 6 in 2024-03-21T13:11:07.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:07 smithi204 ceph-mon[33153]: 3.1 scrub starts 2024-03-21T13:11:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:07 smithi204 ceph-mon[33153]: 3.1 scrub ok 2024-03-21T13:11:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:07 smithi204 ceph-mon[33153]: 9.10 scrub starts 2024-03-21T13:11:07.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:07 smithi163 ceph-mon[29621]: osdmap e295: 8 total, 8 up, 6 in 2024-03-21T13:11:07.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:07 smithi163 ceph-mon[29621]: 3.1 scrub starts 2024-03-21T13:11:07.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:07 smithi163 ceph-mon[29621]: 3.1 scrub ok 2024-03-21T13:11:07.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:07 smithi163 ceph-mon[29621]: 9.10 scrub starts 2024-03-21T13:11:07.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:07 smithi163 ceph-mon[40675]: osdmap e295: 8 total, 8 up, 6 in 2024-03-21T13:11:07.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:07 smithi163 ceph-mon[40675]: 3.1 scrub starts 2024-03-21T13:11:07.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:07 smithi163 ceph-mon[40675]: 3.1 scrub ok 2024-03-21T13:11:07.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:07 smithi163 ceph-mon[40675]: 9.10 scrub starts 2024-03-21T13:11:07.814 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:07.852 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:07.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:07.954 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:07.993 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:07.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:08.094 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:08.132 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:08.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:08.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:08.270 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:08.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:08.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:08.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:08.406 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:08.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:08.507 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:08.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:08.546 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:08.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:08.648 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:08.684 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:08.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:08.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:08 smithi204 ceph-mon[33153]: pgmap v573: 158 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 182 op/s; 12200/216285 objects misplaced (5.641%) 2024-03-21T13:11:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:08 smithi204 ceph-mon[33153]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:11:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:08 smithi204 ceph-mon[33153]: 9.10 scrub ok 2024-03-21T13:11:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:08 smithi204 ceph-mon[33153]: 9.13 scrub starts 2024-03-21T13:11:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:08 smithi163 ceph-mon[29621]: pgmap v573: 158 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 182 op/s; 12200/216285 objects misplaced (5.641%) 2024-03-21T13:11:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:08 smithi163 ceph-mon[29621]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:11:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:08 smithi163 ceph-mon[29621]: 9.10 scrub ok 2024-03-21T13:11:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:08 smithi163 ceph-mon[29621]: 9.13 scrub starts 2024-03-21T13:11:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:08 smithi163 ceph-mon[40675]: pgmap v573: 158 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 182 op/s; 12200/216285 objects misplaced (5.641%) 2024-03-21T13:11:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:08 smithi163 ceph-mon[40675]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:11:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:08 smithi163 ceph-mon[40675]: 9.10 scrub ok 2024-03-21T13:11:08.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:08 smithi163 ceph-mon[40675]: 9.13 scrub starts 2024-03-21T13:11:08.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:08.820 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:08.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:08.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:08.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-21T13:11:08.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:08.955 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:08.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:09.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:09.087 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:09.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:09.188 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:09.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:09.233 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:09.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:09.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:09.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:09.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:09.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:09.476 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:09.527 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:09.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:09.628 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:09.663 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:09.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:09.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:09.765 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:09.781 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:09 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:11:09.781 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:09 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:11:09.781 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:09 smithi163 ceph-mon[29621]: 9.13 scrub ok 2024-03-21T13:11:09.781 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:09 smithi163 ceph-mon[29621]: pgmap v574: 158 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 946 B/s rd, 0 B/s wr, 134 op/s; 12200/216285 objects misplaced (5.641%) 2024-03-21T13:11:09.781 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:09 smithi163 ceph-mon[29621]: 9.15 scrub starts 2024-03-21T13:11:09.781 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:09 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:11:09.781 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:09 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:11:09.782 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:09 smithi163 ceph-mon[40675]: 9.13 scrub ok 2024-03-21T13:11:09.782 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:09 smithi163 ceph-mon[40675]: pgmap v574: 158 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 946 B/s rd, 0 B/s wr, 134 op/s; 12200/216285 objects misplaced (5.641%) 2024-03-21T13:11:09.782 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:09 smithi163 ceph-mon[40675]: 9.15 scrub starts 2024-03-21T13:11:09.797 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:09.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:09.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:09.899 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:09.938 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:09.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:10.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:09 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:11:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:09 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:11:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:09 smithi204 ceph-mon[33153]: 9.13 scrub ok 2024-03-21T13:11:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:09 smithi204 ceph-mon[33153]: pgmap v574: 158 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 946 B/s rd, 0 B/s wr, 134 op/s; 12200/216285 objects misplaced (5.641%) 2024-03-21T13:11:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:09 smithi204 ceph-mon[33153]: 9.15 scrub starts 2024-03-21T13:11:10.039 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:10.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:10.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:10.178 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:10.215 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:10.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:10.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:10.374 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:10.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:10.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:10.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:10.514 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:10.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:10.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:10.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:10.651 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:10.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:10.752 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:10 smithi163 ceph-mon[29621]: osdmap e296: 8 total, 8 up, 6 in 2024-03-21T13:11:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:10 smithi163 ceph-mon[29621]: 4.b scrub starts 2024-03-21T13:11:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:10 smithi163 ceph-mon[29621]: 4.b scrub ok 2024-03-21T13:11:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:10 smithi163 ceph-mon[29621]: 9.15 scrub ok 2024-03-21T13:11:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:10 smithi163 ceph-mon[29621]: 9.17 scrub starts 2024-03-21T13:11:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:10 smithi163 ceph-mon[40675]: osdmap e296: 8 total, 8 up, 6 in 2024-03-21T13:11:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:10 smithi163 ceph-mon[40675]: 4.b scrub starts 2024-03-21T13:11:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:10 smithi163 ceph-mon[40675]: 4.b scrub ok 2024-03-21T13:11:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:10 smithi163 ceph-mon[40675]: 9.15 scrub ok 2024-03-21T13:11:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:10 smithi163 ceph-mon[40675]: 9.17 scrub starts 2024-03-21T13:11:10.790 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:10.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:10.892 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:10.927 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:10.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:10.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:10 smithi204 ceph-mon[33153]: osdmap e296: 8 total, 8 up, 6 in 2024-03-21T13:11:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:10 smithi204 ceph-mon[33153]: 4.b scrub starts 2024-03-21T13:11:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:10 smithi204 ceph-mon[33153]: 4.b scrub ok 2024-03-21T13:11:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:10 smithi204 ceph-mon[33153]: 9.15 scrub ok 2024-03-21T13:11:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:10 smithi204 ceph-mon[33153]: 9.17 scrub starts 2024-03-21T13:11:11.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:11.065 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:11.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:11.166 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:11.207 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:11.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:11.308 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:11.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:11.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:11.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:11.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:11.472 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:11.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:11.573 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:11.616 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:11.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:11.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:11.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:11.742 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:11.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:11.844 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:11.885 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:11.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:11.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:11.933 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:11 smithi204 ceph-mon[33153]: 9.17 scrub ok 2024-03-21T13:11:11.933 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:11 smithi204 ceph-mon[33153]: pgmap v576: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s; 10717/203493 objects misplaced (5.267%) 2024-03-21T13:11:11.933 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:11 smithi204 ceph-mon[33153]: 5.4 scrub starts 2024-03-21T13:11:11.933 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:11 smithi204 ceph-mon[33153]: 5.4 scrub ok 2024-03-21T13:11:11.987 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:11 smithi163 ceph-mon[29621]: 9.17 scrub ok 2024-03-21T13:11:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:11 smithi163 ceph-mon[29621]: pgmap v576: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s; 10717/203493 objects misplaced (5.267%) 2024-03-21T13:11:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:11 smithi163 ceph-mon[29621]: 5.4 scrub starts 2024-03-21T13:11:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:11 smithi163 ceph-mon[29621]: 5.4 scrub ok 2024-03-21T13:11:12.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:11 smithi163 ceph-mon[40675]: 9.17 scrub ok 2024-03-21T13:11:12.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:11 smithi163 ceph-mon[40675]: pgmap v576: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s; 10717/203493 objects misplaced (5.267%) 2024-03-21T13:11:12.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:11 smithi163 ceph-mon[40675]: 5.4 scrub starts 2024-03-21T13:11:12.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:11 smithi163 ceph-mon[40675]: 5.4 scrub ok 2024-03-21T13:11:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:12.017 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:12.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:12.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:12.159 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:12.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:12.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:11:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:11:12.261 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:12.299 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:12.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:12.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:12.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:12.436 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:12.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:12.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:12.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:12.569 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:12.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:12.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:12.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:12.703 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:12.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:12.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:12.834 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:12 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:11:12.834 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:12 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:11:12.834 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:12 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:12 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:11:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:12 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:11:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:12 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:12.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:12.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:12.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:12.937 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:12.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:12.979 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:12.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:13.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:12 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:11:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:12 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:11:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:12 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:13.080 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:13.122 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:13.124 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:13.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:13.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:13.226 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:13.280 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:13.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:13.382 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:13.416 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:13.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:13.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:13.517 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:13.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:13.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:13.663 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:13.701 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:13.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:13 smithi163 ceph-mon[29621]: osdmap e297: 8 total, 8 up, 6 in 2024-03-21T13:11:13.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:13 smithi163 ceph-mon[29621]: pgmap v578: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 802 B/s rd, 0 op/s; 8859/197337 objects misplaced (4.489%) 2024-03-21T13:11:13.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:13 smithi163 ceph-mon[40675]: osdmap e297: 8 total, 8 up, 6 in 2024-03-21T13:11:13.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:13 smithi163 ceph-mon[40675]: pgmap v578: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 802 B/s rd, 0 op/s; 8859/197337 objects misplaced (4.489%) 2024-03-21T13:11:13.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:13.842 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:13.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:13.902 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:13.943 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:13.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:13.995 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:13.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:14.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:13 smithi204 ceph-mon[33153]: osdmap e297: 8 total, 8 up, 6 in 2024-03-21T13:11:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:13 smithi204 ceph-mon[33153]: pgmap v578: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 802 B/s rd, 0 op/s; 8859/197337 objects misplaced (4.489%) 2024-03-21T13:11:14.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:14.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:14.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:14.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:14.235 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:14.271 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:14.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:14.374 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:14.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:14.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:14.510 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:14.547 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:14.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:14.565 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.6 to 1 (10000) 2024-03-21T13:11:14.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:14.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:14.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:14.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:14 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1362570685' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:11:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:14 smithi163 ceph-mon[40675]: 5.1e scrub starts 2024-03-21T13:11:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:14 smithi163 ceph-mon[40675]: 5.1e scrub ok 2024-03-21T13:11:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:14 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1362570685' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:11:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:14 smithi163 ceph-mon[29621]: 5.1e scrub starts 2024-03-21T13:11:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:14 smithi163 ceph-mon[29621]: 5.1e scrub ok 2024-03-21T13:11:14.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:14.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:14.822 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:14.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:14.923 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:14.968 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:14.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:14 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1362570685' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:11:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:14 smithi204 ceph-mon[33153]: 5.1e scrub starts 2024-03-21T13:11:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:14 smithi204 ceph-mon[33153]: 5.1e scrub ok 2024-03-21T13:11:15.070 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:15.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:15.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:15.214 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:15.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:15.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:15.350 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:15.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:15.389 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:15.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:15.490 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:15.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:15.531 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:15.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:15.632 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:15.670 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-21T13:11:15.678 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:15.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:15.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:15.818 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:15.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:15.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:15 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1362570685' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:11:15.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:15 smithi163 ceph-mon[29621]: osdmap e298: 8 total, 8 up, 6 in 2024-03-21T13:11:15.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:15 smithi163 ceph-mon[29621]: 4.18 scrub starts 2024-03-21T13:11:15.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:15 smithi163 ceph-mon[29621]: 4.18 scrub ok 2024-03-21T13:11:15.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:15 smithi163 ceph-mon[29621]: pgmap v580: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 0 op/s; 8859/197337 objects misplaced (4.489%) 2024-03-21T13:11:15.915 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:11:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:11:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:11:15.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:15 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1362570685' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:11:15.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:15 smithi163 ceph-mon[40675]: osdmap e298: 8 total, 8 up, 6 in 2024-03-21T13:11:15.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:15 smithi163 ceph-mon[40675]: 4.18 scrub starts 2024-03-21T13:11:15.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:15 smithi163 ceph-mon[40675]: 4.18 scrub ok 2024-03-21T13:11:15.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:15 smithi163 ceph-mon[40675]: pgmap v580: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 0 op/s; 8859/197337 objects misplaced (4.489%) 2024-03-21T13:11:15.919 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:15.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:15.949 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:15.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:15 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1362570685' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:11:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:15 smithi204 ceph-mon[33153]: osdmap e298: 8 total, 8 up, 6 in 2024-03-21T13:11:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:15 smithi204 ceph-mon[33153]: 4.18 scrub starts 2024-03-21T13:11:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:15 smithi204 ceph-mon[33153]: 4.18 scrub ok 2024-03-21T13:11:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:15 smithi204 ceph-mon[33153]: pgmap v580: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 152 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 0 op/s; 8859/197337 objects misplaced (4.489%) 2024-03-21T13:11:16.051 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:16.083 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:16.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:16.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:16.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:16.219 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:16.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:16.320 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:16.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:16.460 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:16.502 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:16.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:16.604 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:16.643 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:16.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:16.744 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:16.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:16.787 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:16.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:16.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:16.919 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:16 smithi163 ceph-mon[29621]: 4.1e scrub starts 2024-03-21T13:11:16.919 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:16 smithi163 ceph-mon[29621]: 4.1e scrub ok 2024-03-21T13:11:16.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:16 smithi163 ceph-mon[40675]: 4.1e scrub starts 2024-03-21T13:11:16.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:16 smithi163 ceph-mon[40675]: 4.1e scrub ok 2024-03-21T13:11:16.922 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:16.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:16.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:17.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:16 smithi204 ceph-mon[33153]: 4.1e scrub starts 2024-03-21T13:11:17.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:16 smithi204 ceph-mon[33153]: 4.1e scrub ok 2024-03-21T13:11:17.024 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:17.072 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:17.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:17.173 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:17.218 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:17.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:17.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:17.356 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:17.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:17.458 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:17.501 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:17.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:17.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:17.638 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:17.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:17.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:17.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:17.764 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:17.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:17.866 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:17.892 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:17.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:17.942 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:17 smithi163 ceph-mon[29621]: pgmap v581: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 153 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s; 6244/176412 objects misplaced (3.539%) 2024-03-21T13:11:17.942 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:17 smithi163 ceph-mon[29621]: 3.d scrub starts 2024-03-21T13:11:17.942 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:17 smithi163 ceph-mon[29621]: 3.d scrub ok 2024-03-21T13:11:17.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:17 smithi163 ceph-mon[40675]: pgmap v581: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 153 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s; 6244/176412 objects misplaced (3.539%) 2024-03-21T13:11:17.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:17 smithi163 ceph-mon[40675]: 3.d scrub starts 2024-03-21T13:11:17.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:17 smithi163 ceph-mon[40675]: 3.d scrub ok 2024-03-21T13:11:17.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:18.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:17 smithi204 ceph-mon[33153]: pgmap v581: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 153 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s; 6244/176412 objects misplaced (3.539%) 2024-03-21T13:11:18.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:17 smithi204 ceph-mon[33153]: 3.d scrub starts 2024-03-21T13:11:18.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:17 smithi204 ceph-mon[33153]: 3.d scrub ok 2024-03-21T13:11:18.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:18.019 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:18.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:18.120 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:18.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:18.145 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:18.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:18.246 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:18.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:18.275 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:18.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:18.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:18.400 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:18.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:18.502 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:18.529 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:18.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:18.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:18.631 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:18.658 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:18.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:18.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:18.664 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:11:18.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:18 smithi163 ceph-mon[40675]: 4.6 scrub starts 2024-03-21T13:11:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:18 smithi163 ceph-mon[40675]: 4.6 scrub ok 2024-03-21T13:11:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:18 smithi163 ceph-mon[40675]: osdmap e299: 8 total, 8 up, 6 in 2024-03-21T13:11:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:18 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1973948496' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:11:18.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:18 smithi163 ceph-mon[29621]: 4.6 scrub starts 2024-03-21T13:11:18.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:18 smithi163 ceph-mon[29621]: 4.6 scrub ok 2024-03-21T13:11:18.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:18 smithi163 ceph-mon[29621]: osdmap e299: 8 total, 8 up, 6 in 2024-03-21T13:11:18.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:18 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1973948496' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:11:18.791 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:18.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:18.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:18.927 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:18.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:18.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:19.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:18 smithi204 ceph-mon[33153]: 4.6 scrub starts 2024-03-21T13:11:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:18 smithi204 ceph-mon[33153]: 4.6 scrub ok 2024-03-21T13:11:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:18 smithi204 ceph-mon[33153]: osdmap e299: 8 total, 8 up, 6 in 2024-03-21T13:11:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:18 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1973948496' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:11:19.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:19.067 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:19.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:19.168 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:19.206 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:19.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:19.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:19.345 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:19.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:19.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:19.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:19.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:19.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:19.586 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:19.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:19.624 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:19.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:19.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:19.760 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:19.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:19.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:19.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:19.897 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:19.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:19.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:20.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:19 smithi204 ceph-mon[33153]: 5.e scrub starts 2024-03-21T13:11:20.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:19 smithi204 ceph-mon[33153]: 5.e scrub ok 2024-03-21T13:11:20.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:19 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:11:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:19 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1973948496' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:11:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:19 smithi204 ceph-mon[33153]: osdmap e300: 8 total, 8 up, 6 in 2024-03-21T13:11:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:19 smithi204 ceph-mon[33153]: pgmap v584: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 153 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 232 op/s; 6244/176412 objects misplaced (3.539%) 2024-03-21T13:11:20.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:19 smithi163 ceph-mon[29621]: 5.e scrub starts 2024-03-21T13:11:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:19 smithi163 ceph-mon[29621]: 5.e scrub ok 2024-03-21T13:11:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:19 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:11:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:19 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1973948496' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:11:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:19 smithi163 ceph-mon[29621]: osdmap e300: 8 total, 8 up, 6 in 2024-03-21T13:11:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:19 smithi163 ceph-mon[29621]: pgmap v584: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 153 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 232 op/s; 6244/176412 objects misplaced (3.539%) 2024-03-21T13:11:20.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:19 smithi163 ceph-mon[40675]: 5.e scrub starts 2024-03-21T13:11:20.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:19 smithi163 ceph-mon[40675]: 5.e scrub ok 2024-03-21T13:11:20.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:19 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:11:20.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:19 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1973948496' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:11:20.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:19 smithi163 ceph-mon[40675]: osdmap e300: 8 total, 8 up, 6 in 2024-03-21T13:11:20.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:19 smithi163 ceph-mon[40675]: pgmap v584: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 153 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 232 op/s; 6244/176412 objects misplaced (3.539%) 2024-03-21T13:11:20.031 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:20.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:20.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:20.133 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:20.158 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:20.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:20.259 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:20.299 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:20.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:20.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:20.436 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:20.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:20.537 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:20.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:20.574 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:20.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:20.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:20.716 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:20.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:20.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:20.851 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:20.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:20.953 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:20.989 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:20.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:21.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-03-21T13:11:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 7]}]: dispatch 2024-03-21T13:11:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:11:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-03-21T13:11:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 7]}]: dispatch 2024-03-21T13:11:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:11:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:21.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-03-21T13:11:21.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 7]}]: dispatch 2024-03-21T13:11:21.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:11:21.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:21.091 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:21.128 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:21.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:21.230 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:21.267 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:21.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:21.368 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:21.425 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:21.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:21.527 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:21.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:21.569 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:21.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:21.595 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:11:21.595 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:11:21.595 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-03-21T13:11:21.595 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:11:21.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:21.710 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:21.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:21.795 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:21 smithi163 ceph-mon[29621]: 5.1b scrub starts 2024-03-21T13:11:21.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:21 smithi163 ceph-mon[29621]: 5.1b scrub ok 2024-03-21T13:11:21.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]': finished 2024-03-21T13:11:21.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 7]}]': finished 2024-03-21T13:11:21.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:21 smithi163 ceph-mon[29621]: osdmap e301: 8 total, 8 up, 6 in 2024-03-21T13:11:21.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:21 smithi163 ceph-mon[29621]: pgmap v586: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 154 active+clean; 3.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 871 op/s; 4852/164928 objects misplaced (2.942%) 2024-03-21T13:11:21.796 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:21 smithi163 ceph-mon[40675]: 5.1b scrub starts 2024-03-21T13:11:21.796 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:21 smithi163 ceph-mon[40675]: 5.1b scrub ok 2024-03-21T13:11:21.796 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]': finished 2024-03-21T13:11:21.796 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 7]}]': finished 2024-03-21T13:11:21.796 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:21 smithi163 ceph-mon[40675]: osdmap e301: 8 total, 8 up, 6 in 2024-03-21T13:11:21.796 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:21 smithi163 ceph-mon[40675]: pgmap v586: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 154 active+clean; 3.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 871 op/s; 4852/164928 objects misplaced (2.942%) 2024-03-21T13:11:21.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:21.851 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:21.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:21.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:21.943 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:21 smithi204 ceph-mon[33153]: 5.1b scrub starts 2024-03-21T13:11:21.943 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:21 smithi204 ceph-mon[33153]: 5.1b scrub ok 2024-03-21T13:11:21.943 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]': finished 2024-03-21T13:11:21.943 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 7]}]': finished 2024-03-21T13:11:21.944 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:21 smithi204 ceph-mon[33153]: osdmap e301: 8 total, 8 up, 6 in 2024-03-21T13:11:21.944 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:21 smithi204 ceph-mon[33153]: pgmap v586: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 154 active+clean; 3.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 871 op/s; 4852/164928 objects misplaced (2.942%) 2024-03-21T13:11:21.953 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:21.995 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:21.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:21.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:22.097 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:22.137 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:22.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:22.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:22.239 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:22.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:11:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:11:22.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:22.283 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:22.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:22.384 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:22.414 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:11:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:22.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:22.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:22.453 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:11:22.495 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:11:22.520 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:22.540 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:11:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:22.550 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:22.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:22.579 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:11:22.619 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:11:22.652 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:22.662 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:11:22.685 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:22.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:22.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:22.703 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:11:22.704 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:22 smithi163 ceph-mon[29621]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:11:22.704 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:22 smithi163 ceph-mon[29621]: osdmap e302: 8 total, 8 up, 6 in 2024-03-21T13:11:22.704 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:22 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:22.704 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:22 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1926326538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:11:22.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:22 smithi163 ceph-mon[40675]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:11:22.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:22 smithi163 ceph-mon[40675]: osdmap e302: 8 total, 8 up, 6 in 2024-03-21T13:11:22.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:22 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:22.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:22 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1926326538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:11:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:11:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:11:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:11:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:11:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:11:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:11:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:11:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:11:22.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:22.821 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:22.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:22.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:22.923 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:22.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:22.963 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:22.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:23.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:22 smithi204 ceph-mon[33153]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:11:23.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:22 smithi204 ceph-mon[33153]: osdmap e302: 8 total, 8 up, 6 in 2024-03-21T13:11:23.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:22 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:23.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:22 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1926326538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:11:23.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:23.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:23.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:23.200 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:23.237 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:23.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:23.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:23.339 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:23.376 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:23.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:23.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:23.478 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:23.502 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:23.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:23.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:23.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:23.639 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:23.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:23.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:23.775 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:23.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:23.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:23.912 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:23.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:23 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:11:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:23 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:11:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:23 smithi204 ceph-mon[33153]: 5.1 scrub starts 2024-03-21T13:11:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:23 smithi204 ceph-mon[33153]: 5.1 scrub ok 2024-03-21T13:11:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:23 smithi204 ceph-mon[33153]: pgmap v588: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 973 B/s rd, 0 B/s wr, 1.07k op/s; 3548/159573 objects misplaced (2.223%); 26 MiB/s, 423 objects/s recovering 2024-03-21T13:11:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:23 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:11:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:23 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:11:24.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:11:24.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:11:24.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[29621]: 5.1 scrub starts 2024-03-21T13:11:24.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[29621]: 5.1 scrub ok 2024-03-21T13:11:24.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[29621]: pgmap v588: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 973 B/s rd, 0 B/s wr, 1.07k op/s; 3548/159573 objects misplaced (2.223%); 26 MiB/s, 423 objects/s recovering 2024-03-21T13:11:24.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:11:24.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:11:24.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:11:24.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:11:24.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[40675]: 5.1 scrub starts 2024-03-21T13:11:24.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[40675]: 5.1 scrub ok 2024-03-21T13:11:24.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[40675]: pgmap v588: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 973 B/s rd, 0 B/s wr, 1.07k op/s; 3548/159573 objects misplaced (2.223%); 26 MiB/s, 423 objects/s recovering 2024-03-21T13:11:24.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:11:24.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:23 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:11:24.014 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:24.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:24.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:24.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:24.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:24.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:24.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:24.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:24.324 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:24.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:24.425 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:24.467 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:24.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:24.569 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:24.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:24.605 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:24.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:24.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:24.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:24.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:24.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:24.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:24 smithi163 ceph-mon[29621]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:11:24.759 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:24 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:11:24.759 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:24 smithi163 ceph-mon[29621]: osdmap e303: 8 total, 8 up, 6 in 2024-03-21T13:11:24.759 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:24 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:11:24.759 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:24 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:11:24.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:24 smithi163 ceph-mon[40675]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:11:24.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:24 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:11:24.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:24 smithi163 ceph-mon[40675]: osdmap e303: 8 total, 8 up, 6 in 2024-03-21T13:11:24.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:24 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:11:24.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:24 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:11:24.764 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-21T13:11:24.837 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:24.882 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:24.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:24.983 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:24 smithi204 ceph-mon[33153]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:11:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:24 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:11:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:24 smithi204 ceph-mon[33153]: osdmap e303: 8 total, 8 up, 6 in 2024-03-21T13:11:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:24 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:11:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:24 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:11:25.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:25.028 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:25.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:25.129 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:25.167 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:25.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:25.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:25.269 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:25.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:25.306 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:25.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:25.407 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:25.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:25.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:25.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:25.581 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:25.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:25.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:25.683 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:25.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:25.719 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:25.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:25.821 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:25.866 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:25.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:25.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:25 smithi204 ceph-mon[33153]: osdmap e304: 8 total, 8 up, 6 in 2024-03-21T13:11:26.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:25 smithi204 ceph-mon[33153]: pgmap v591: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 427 op/s; 3548/159573 objects misplaced (2.223%); 17 MiB/s, 278 objects/s recovering 2024-03-21T13:11:26.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:26.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:26.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:25 smithi163 ceph-mon[29621]: osdmap e304: 8 total, 8 up, 6 in 2024-03-21T13:11:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:25 smithi163 ceph-mon[29621]: pgmap v591: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 427 op/s; 3548/159573 objects misplaced (2.223%); 17 MiB/s, 278 objects/s recovering 2024-03-21T13:11:26.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:25 smithi163 ceph-mon[40675]: osdmap e304: 8 total, 8 up, 6 in 2024-03-21T13:11:26.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:25 smithi163 ceph-mon[40675]: pgmap v591: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 427 op/s; 3548/159573 objects misplaced (2.223%); 17 MiB/s, 278 objects/s recovering 2024-03-21T13:11:26.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:11:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:11:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:11:26.059 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:11:26.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:26.143 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:26.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:26.244 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:26.298 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:26.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:26.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:26.401 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:26.438 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:26.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:26.539 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:26.574 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:26.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:26.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:26.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:26.710 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:26.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:26.812 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:26.845 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:26.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:26.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:26.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:26.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:26 smithi204 ceph-mon[33153]: 3.1c scrub starts 2024-03-21T13:11:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:26 smithi204 ceph-mon[33153]: 3.1c scrub ok 2024-03-21T13:11:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:26 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:11:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:26 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:11:27.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:26 smithi163 ceph-mon[29621]: 3.1c scrub starts 2024-03-21T13:11:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:26 smithi163 ceph-mon[29621]: 3.1c scrub ok 2024-03-21T13:11:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:26 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:11:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:26 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:11:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:26 smithi163 ceph-mon[40675]: 3.1c scrub starts 2024-03-21T13:11:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:26 smithi163 ceph-mon[40675]: 3.1c scrub ok 2024-03-21T13:11:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:26 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:11:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:26 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:11:27.086 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:27.116 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:27.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:27.218 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:27.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:27.243 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:27.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:27.344 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:27.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:27.369 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:27.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:27.470 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:27.495 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:27.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:27.596 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:27.621 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:27.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:27.723 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:27.750 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:27.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:27.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:27.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:27.886 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:27.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:27.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:27.988 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:28.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:27 smithi204 ceph-mon[33153]: 9.12 scrub starts 2024-03-21T13:11:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:27 smithi204 ceph-mon[33153]: 5.17 scrub starts 2024-03-21T13:11:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:27 smithi204 ceph-mon[33153]: 5.17 scrub ok 2024-03-21T13:11:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:27 smithi204 ceph-mon[33153]: osdmap e305: 8 total, 8 up, 6 in 2024-03-21T13:11:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:27 smithi204 ceph-mon[33153]: pgmap v593: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 792 B/s rd, 0 B/s wr, 1.45k op/s; 1731/137133 objects misplaced (1.262%); 17 MiB/s, 274 objects/s recovering 2024-03-21T13:11:28.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:27 smithi204 ceph-mon[33153]: 9.12 scrub ok 2024-03-21T13:11:28.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:27 smithi163 ceph-mon[29621]: 9.12 scrub starts 2024-03-21T13:11:28.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:27 smithi163 ceph-mon[29621]: 5.17 scrub starts 2024-03-21T13:11:28.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:27 smithi163 ceph-mon[29621]: 5.17 scrub ok 2024-03-21T13:11:28.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:27 smithi163 ceph-mon[29621]: osdmap e305: 8 total, 8 up, 6 in 2024-03-21T13:11:28.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:27 smithi163 ceph-mon[29621]: pgmap v593: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 792 B/s rd, 0 B/s wr, 1.45k op/s; 1731/137133 objects misplaced (1.262%); 17 MiB/s, 274 objects/s recovering 2024-03-21T13:11:28.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:27 smithi163 ceph-mon[29621]: 9.12 scrub ok 2024-03-21T13:11:28.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:27 smithi163 ceph-mon[40675]: 9.12 scrub starts 2024-03-21T13:11:28.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:27 smithi163 ceph-mon[40675]: 5.17 scrub starts 2024-03-21T13:11:28.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:27 smithi163 ceph-mon[40675]: 5.17 scrub ok 2024-03-21T13:11:28.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:27 smithi163 ceph-mon[40675]: osdmap e305: 8 total, 8 up, 6 in 2024-03-21T13:11:28.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:27 smithi163 ceph-mon[40675]: pgmap v593: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 792 B/s rd, 0 B/s wr, 1.45k op/s; 1731/137133 objects misplaced (1.262%); 17 MiB/s, 274 objects/s recovering 2024-03-21T13:11:28.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:27 smithi163 ceph-mon[40675]: 9.12 scrub ok 2024-03-21T13:11:28.017 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:28.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:28.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:28.119 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:28.155 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:28.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:28.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:28.257 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:28.289 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:28.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:28.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:28.429 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:28.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:28.530 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:28.569 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:28.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:28.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:28.666 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:28.671 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:28.721 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:28.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:28.822 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:28.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:28.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:28.909 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:28.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:28.989 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:28.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:29.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:28 smithi204 ceph-mon[33153]: osdmap e306: 8 total, 8 up, 6 in 2024-03-21T13:11:29.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:28 smithi204 ceph-mon[33153]: 5.f scrub starts 2024-03-21T13:11:29.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:28 smithi204 ceph-mon[33153]: 5.f scrub ok 2024-03-21T13:11:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:28 smithi163 ceph-mon[40675]: osdmap e306: 8 total, 8 up, 6 in 2024-03-21T13:11:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:28 smithi163 ceph-mon[40675]: 5.f scrub starts 2024-03-21T13:11:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:28 smithi163 ceph-mon[40675]: 5.f scrub ok 2024-03-21T13:11:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:28 smithi163 ceph-mon[29621]: osdmap e306: 8 total, 8 up, 6 in 2024-03-21T13:11:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:28 smithi163 ceph-mon[29621]: 5.f scrub starts 2024-03-21T13:11:29.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:28 smithi163 ceph-mon[29621]: 5.f scrub ok 2024-03-21T13:11:29.091 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:29.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:29.130 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:29.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:29.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:29.495 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:29.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:29.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:29.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:29.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:29.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:29.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:29.761 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:29.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:29.862 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:29.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:29.900 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:29.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:30.001 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:29 smithi204 ceph-mon[33153]: 5.1d scrub starts 2024-03-21T13:11:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:29 smithi204 ceph-mon[33153]: 5.1d scrub ok 2024-03-21T13:11:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:29 smithi204 ceph-mon[33153]: pgmap v595: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 792 B/s rd, 0 B/s wr, 1.45k op/s; 1731/137133 objects misplaced (1.262%); 17 MiB/s, 274 objects/s recovering 2024-03-21T13:11:30.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:29 smithi204 ceph-mon[33153]: 4.19 scrub starts 2024-03-21T13:11:30.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:29 smithi204 ceph-mon[33153]: 4.19 scrub ok 2024-03-21T13:11:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:29 smithi163 ceph-mon[29621]: 5.1d scrub starts 2024-03-21T13:11:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:29 smithi163 ceph-mon[29621]: 5.1d scrub ok 2024-03-21T13:11:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:29 smithi163 ceph-mon[29621]: pgmap v595: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 792 B/s rd, 0 B/s wr, 1.45k op/s; 1731/137133 objects misplaced (1.262%); 17 MiB/s, 274 objects/s recovering 2024-03-21T13:11:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:29 smithi163 ceph-mon[29621]: 4.19 scrub starts 2024-03-21T13:11:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:29 smithi163 ceph-mon[29621]: 4.19 scrub ok 2024-03-21T13:11:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:29 smithi163 ceph-mon[40675]: 5.1d scrub starts 2024-03-21T13:11:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:29 smithi163 ceph-mon[40675]: 5.1d scrub ok 2024-03-21T13:11:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:29 smithi163 ceph-mon[40675]: pgmap v595: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 792 B/s rd, 0 B/s wr, 1.45k op/s; 1731/137133 objects misplaced (1.262%); 17 MiB/s, 274 objects/s recovering 2024-03-21T13:11:30.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:29 smithi163 ceph-mon[40675]: 4.19 scrub starts 2024-03-21T13:11:30.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:29 smithi163 ceph-mon[40675]: 4.19 scrub ok 2024-03-21T13:11:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:30.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:30.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:30.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:30.177 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:30.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:30.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:30.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:30.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:30.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:30.418 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:30.460 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:30.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:30.562 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:30.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:30.610 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:30.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:30.659 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:11:30.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:30.741 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:30.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:30.842 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:30.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:30.892 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:30.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:30.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:30 smithi204 ceph-mon[33153]: 4.13 scrub starts 2024-03-21T13:11:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:30 smithi204 ceph-mon[33153]: 4.13 scrub ok 2024-03-21T13:11:31.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:30 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:31.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:30 smithi163 ceph-mon[29621]: 4.13 scrub starts 2024-03-21T13:11:31.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:30 smithi163 ceph-mon[29621]: 4.13 scrub ok 2024-03-21T13:11:31.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:30 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:31.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:30 smithi163 ceph-mon[40675]: 4.13 scrub starts 2024-03-21T13:11:31.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:30 smithi163 ceph-mon[40675]: 4.13 scrub ok 2024-03-21T13:11:31.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:30 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:31.036 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:31.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:31.137 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:31.179 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:31.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:31.281 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:31.319 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:31.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:31.420 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:31.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:31.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:31.566 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:31.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:31.613 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:31.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:31.714 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:31.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:31.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:31.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:31.901 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:31.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:32.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:31 smithi204 ceph-mon[33153]: pgmap v596: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.92k op/s; 573/124128 objects misplaced (0.462%); 25 MiB/s, 393 objects/s recovering 2024-03-21T13:11:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:31 smithi204 ceph-mon[33153]: 3.f scrub starts 2024-03-21T13:11:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:31 smithi204 ceph-mon[33153]: 3.f scrub ok 2024-03-21T13:11:32.003 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:11:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:11:32.003 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:32.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:31 smithi163 ceph-mon[29621]: pgmap v596: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.92k op/s; 573/124128 objects misplaced (0.462%); 25 MiB/s, 393 objects/s recovering 2024-03-21T13:11:32.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:31 smithi163 ceph-mon[29621]: 3.f scrub starts 2024-03-21T13:11:32.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:31 smithi163 ceph-mon[29621]: 3.f scrub ok 2024-03-21T13:11:32.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:31 smithi163 ceph-mon[40675]: pgmap v596: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.92k op/s; 573/124128 objects misplaced (0.462%); 25 MiB/s, 393 objects/s recovering 2024-03-21T13:11:32.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:31 smithi163 ceph-mon[40675]: 3.f scrub starts 2024-03-21T13:11:32.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:31 smithi163 ceph-mon[40675]: 3.f scrub ok 2024-03-21T13:11:32.047 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:32.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:32.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:32.149 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:32.187 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:32.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:32.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:32.289 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:32.323 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:32.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:32.339 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:32.425 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:32.463 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:32.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:32.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:32.601 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:32.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:32.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:32.703 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:32.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:32.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:32.840 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:32.886 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:32.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:32.988 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:33.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:32 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:33.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:32 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:33.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:32 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:33.027 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:33.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:33.129 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:33.168 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:33.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:33.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:33.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:33.310 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:33.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:33.411 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:33.443 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:33.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:33.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:33.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:33.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:33.586 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:33.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:33.687 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:33.729 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:33.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:33.835 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:33.867 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:33.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:33.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:33.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:34.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:34.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:34.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:34.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:33 smithi163 ceph-mon[29621]: 6.1b scrub starts 2024-03-21T13:11:34.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:33 smithi163 ceph-mon[29621]: pgmap v597: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.72k op/s; 509/118191 objects misplaced (0.431%); 19 MiB/s, 310 objects/s recovering 2024-03-21T13:11:34.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:33 smithi163 ceph-mon[29621]: 6.1b scrub ok 2024-03-21T13:11:34.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:33 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:34.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:33 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:34.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:33 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:11:34.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:33 smithi163 ceph-mon[40675]: 6.1b scrub starts 2024-03-21T13:11:34.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:33 smithi163 ceph-mon[40675]: pgmap v597: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.72k op/s; 509/118191 objects misplaced (0.431%); 19 MiB/s, 310 objects/s recovering 2024-03-21T13:11:34.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:33 smithi163 ceph-mon[40675]: 6.1b scrub ok 2024-03-21T13:11:34.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:33 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:34.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:33 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:34.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:33 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:11:34.107 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:34.143 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:34.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:34.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:34.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:33 smithi204 ceph-mon[33153]: 6.1b scrub starts 2024-03-21T13:11:34.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:33 smithi204 ceph-mon[33153]: pgmap v597: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.72k op/s; 509/118191 objects misplaced (0.431%); 19 MiB/s, 310 objects/s recovering 2024-03-21T13:11:34.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:33 smithi204 ceph-mon[33153]: 6.1b scrub ok 2024-03-21T13:11:34.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:33 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:34.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:33 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:34.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:33 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:11:34.282 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:34.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:34.384 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:34.389 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:11:34.389 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:11:34.419 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:34.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:34.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:34.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:34.545 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:34.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:34.646 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:34.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:34.670 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:34.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:34.771 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:34.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:34.797 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:34.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:34.898 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:34.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:34.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:34.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:35.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[40675]: 3.1d scrub starts 2024-03-21T13:11:35.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[40675]: 3.1d scrub ok 2024-03-21T13:11:35.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[40675]: 5.b scrub starts 2024-03-21T13:11:35.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[40675]: 5.b scrub ok 2024-03-21T13:11:35.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[40675]: 6.13 scrub starts 2024-03-21T13:11:35.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[40675]: 6.13 scrub ok 2024-03-21T13:11:35.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[40675]: 6.11 scrub starts 2024-03-21T13:11:35.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[40675]: 6.11 scrub ok 2024-03-21T13:11:35.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:11:35.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:11:35.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[40675]: osdmap e307: 8 total, 8 up, 6 in 2024-03-21T13:11:35.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[29621]: 3.1d scrub starts 2024-03-21T13:11:35.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[29621]: 3.1d scrub ok 2024-03-21T13:11:35.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[29621]: 5.b scrub starts 2024-03-21T13:11:35.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[29621]: 5.b scrub ok 2024-03-21T13:11:35.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[29621]: 6.13 scrub starts 2024-03-21T13:11:35.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[29621]: 6.13 scrub ok 2024-03-21T13:11:35.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[29621]: 6.11 scrub starts 2024-03-21T13:11:35.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[29621]: 6.11 scrub ok 2024-03-21T13:11:35.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:11:35.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:11:35.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:34 smithi163 ceph-mon[29621]: osdmap e307: 8 total, 8 up, 6 in 2024-03-21T13:11:35.024 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:35.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:35.049 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:35.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:35.150 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:35.175 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:35.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:35.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:34 smithi204 ceph-mon[33153]: 3.1d scrub starts 2024-03-21T13:11:35.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:34 smithi204 ceph-mon[33153]: 3.1d scrub ok 2024-03-21T13:11:35.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:34 smithi204 ceph-mon[33153]: 5.b scrub starts 2024-03-21T13:11:35.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:34 smithi204 ceph-mon[33153]: 5.b scrub ok 2024-03-21T13:11:35.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:34 smithi204 ceph-mon[33153]: 6.13 scrub starts 2024-03-21T13:11:35.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:34 smithi204 ceph-mon[33153]: 6.13 scrub ok 2024-03-21T13:11:35.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:34 smithi204 ceph-mon[33153]: 6.11 scrub starts 2024-03-21T13:11:35.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:34 smithi204 ceph-mon[33153]: 6.11 scrub ok 2024-03-21T13:11:35.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:34 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:11:35.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:34 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:11:35.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:34 smithi204 ceph-mon[33153]: osdmap e307: 8 total, 8 up, 6 in 2024-03-21T13:11:35.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:35.302 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:35.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:35.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:35.424 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-21T13:11:35.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:35.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:35.537 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:35.586 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:35.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:35.674 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:11:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:11:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:11:35.688 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:35.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:35.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:35.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:35.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:35.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:35.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:35.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:35.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:35.995 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:35.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:36.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:36.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:36.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:36.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:36.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:35 smithi204 ceph-mon[33153]: 3.10 scrub starts 2024-03-21T13:11:36.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:35 smithi204 ceph-mon[33153]: 3.10 scrub ok 2024-03-21T13:11:36.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:35 smithi204 ceph-mon[33153]: pgmap v599: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 789 op/s; 509/118191 objects misplaced (0.431%); 8.3 MiB/s, 133 objects/s recovering 2024-03-21T13:11:36.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:11:36.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:36.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:35 smithi204 ceph-mon[33153]: osdmap e308: 8 total, 8 up, 6 in 2024-03-21T13:11:36.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:35 smithi163 ceph-mon[29621]: 3.10 scrub starts 2024-03-21T13:11:36.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:35 smithi163 ceph-mon[29621]: 3.10 scrub ok 2024-03-21T13:11:36.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:35 smithi163 ceph-mon[29621]: pgmap v599: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 789 op/s; 509/118191 objects misplaced (0.431%); 8.3 MiB/s, 133 objects/s recovering 2024-03-21T13:11:36.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:11:36.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:36.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:35 smithi163 ceph-mon[29621]: osdmap e308: 8 total, 8 up, 6 in 2024-03-21T13:11:36.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:35 smithi163 ceph-mon[40675]: 3.10 scrub starts 2024-03-21T13:11:36.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:35 smithi163 ceph-mon[40675]: 3.10 scrub ok 2024-03-21T13:11:36.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:35 smithi163 ceph-mon[40675]: pgmap v599: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 789 op/s; 509/118191 objects misplaced (0.431%); 8.3 MiB/s, 133 objects/s recovering 2024-03-21T13:11:36.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:11:36.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:36.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:35 smithi163 ceph-mon[40675]: osdmap e308: 8 total, 8 up, 6 in 2024-03-21T13:11:36.272 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:36.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:36.374 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:36.411 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:36.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:36.474 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:36.512 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:36.575 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:36.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:36.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:36.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:36.712 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:36.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:36.813 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:36.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:36.852 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:36.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:36.952 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:36.982 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.7 to 1 (10000) 2024-03-21T13:11:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:37.006 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:37.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:37.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:37.141 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:37.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:37.243 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:37.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:36 smithi204 ceph-mon[33153]: 6.f scrub starts 2024-03-21T13:11:37.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:36 smithi204 ceph-mon[33153]: 3.1a scrub starts 2024-03-21T13:11:37.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:36 smithi204 ceph-mon[33153]: 3.1a scrub ok 2024-03-21T13:11:37.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:36 smithi204 ceph-mon[33153]: 6.f scrub ok 2024-03-21T13:11:37.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:36 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2072289578' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:11:37.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:36 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:11:37.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:36 smithi163 ceph-mon[29621]: 6.f scrub starts 2024-03-21T13:11:37.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:36 smithi163 ceph-mon[29621]: 3.1a scrub starts 2024-03-21T13:11:37.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:36 smithi163 ceph-mon[29621]: 3.1a scrub ok 2024-03-21T13:11:37.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:36 smithi163 ceph-mon[29621]: 6.f scrub ok 2024-03-21T13:11:37.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:36 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2072289578' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:11:37.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:36 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:11:37.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:36 smithi163 ceph-mon[40675]: 6.f scrub starts 2024-03-21T13:11:37.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:36 smithi163 ceph-mon[40675]: 3.1a scrub starts 2024-03-21T13:11:37.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:36 smithi163 ceph-mon[40675]: 3.1a scrub ok 2024-03-21T13:11:37.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:36 smithi163 ceph-mon[40675]: 6.f scrub ok 2024-03-21T13:11:37.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:36 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2072289578' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:11:37.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:36 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:11:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:37.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:37.389 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:37.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:37.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:37.526 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:37.562 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:37.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:37.664 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:37.701 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:37.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:37.765 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-21T13:11:37.804 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:37.860 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:37.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:37.962 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:38.008 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:38.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:38.109 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:38.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:38.152 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:38.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:38.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:38.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:37 smithi204 ceph-mon[33153]: pgmap v601: 158 pgs: 1 peering, 157 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s; 12 MiB/s, 192 objects/s recovering 2024-03-21T13:11:38.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:37 smithi204 ceph-mon[33153]: 6.4 scrub starts 2024-03-21T13:11:38.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:37 smithi204 ceph-mon[33153]: 6.4 scrub ok 2024-03-21T13:11:38.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:37 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:11:38.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:37 smithi204 ceph-mon[33153]: osdmap e309: 8 total, 8 up, 6 in 2024-03-21T13:11:38.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:37 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2983221346' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:11:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:37 smithi163 ceph-mon[29621]: pgmap v601: 158 pgs: 1 peering, 157 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s; 12 MiB/s, 192 objects/s recovering 2024-03-21T13:11:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:37 smithi163 ceph-mon[29621]: 6.4 scrub starts 2024-03-21T13:11:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:37 smithi163 ceph-mon[29621]: 6.4 scrub ok 2024-03-21T13:11:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:37 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:11:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:37 smithi163 ceph-mon[29621]: osdmap e309: 8 total, 8 up, 6 in 2024-03-21T13:11:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:37 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2983221346' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:11:38.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:37 smithi163 ceph-mon[40675]: pgmap v601: 158 pgs: 1 peering, 157 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s; 12 MiB/s, 192 objects/s recovering 2024-03-21T13:11:38.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:37 smithi163 ceph-mon[40675]: 6.4 scrub starts 2024-03-21T13:11:38.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:37 smithi163 ceph-mon[40675]: 6.4 scrub ok 2024-03-21T13:11:38.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:37 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:11:38.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:37 smithi163 ceph-mon[40675]: osdmap e309: 8 total, 8 up, 6 in 2024-03-21T13:11:38.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:37 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2983221346' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:11:38.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:38.289 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:38.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:38.390 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:38.391 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:11:38.433 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:38.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:38.535 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:38.574 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:38.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:38.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:38.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:38.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:38.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:38.815 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:38.853 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:38.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:38.954 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:38.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:38.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:39.094 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:39.136 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:39.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:39.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:39.238 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:39.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:38 smithi204 ceph-mon[33153]: 4.2 scrub starts 2024-03-21T13:11:39.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:38 smithi204 ceph-mon[33153]: 4.2 scrub ok 2024-03-21T13:11:39.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:38 smithi204 ceph-mon[33153]: 6.9 scrub starts 2024-03-21T13:11:39.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:38 smithi204 ceph-mon[33153]: 6.9 scrub ok 2024-03-21T13:11:39.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:38 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:11:39.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:38 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2983221346' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:11:39.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:38 smithi204 ceph-mon[33153]: osdmap e310: 8 total, 8 up, 6 in 2024-03-21T13:11:39.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[29621]: 4.2 scrub starts 2024-03-21T13:11:39.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[29621]: 4.2 scrub ok 2024-03-21T13:11:39.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[29621]: 6.9 scrub starts 2024-03-21T13:11:39.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[29621]: 6.9 scrub ok 2024-03-21T13:11:39.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:11:39.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2983221346' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:11:39.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[29621]: osdmap e310: 8 total, 8 up, 6 in 2024-03-21T13:11:39.264 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[40675]: 4.2 scrub starts 2024-03-21T13:11:39.264 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[40675]: 4.2 scrub ok 2024-03-21T13:11:39.264 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[40675]: 6.9 scrub starts 2024-03-21T13:11:39.264 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[40675]: 6.9 scrub ok 2024-03-21T13:11:39.264 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:11:39.265 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2983221346' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:11:39.265 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:38 smithi163 ceph-mon[40675]: osdmap e310: 8 total, 8 up, 6 in 2024-03-21T13:11:39.276 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:39.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:39.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:39.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:39.415 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:39.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:39.516 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:39.560 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:39.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:39.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:39.662 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:39.701 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:39.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:39.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:39.803 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:39.842 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:39.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:39.943 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:39.965 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:39.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:39.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:39.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:40.041 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:40.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:40.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:40.120 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:40.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:40.222 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:40.262 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:40.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:40.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:40.400 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:40.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:40.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:40.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:40.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:40.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:40.548 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:40 smithi163 ceph-mon[29621]: pgmap v604: 158 pgs: 1 peering, 157 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.45k op/s; 6.6 MiB/s, 105 objects/s recovering 2024-03-21T13:11:40.548 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:40 smithi163 ceph-mon[40675]: pgmap v604: 158 pgs: 1 peering, 157 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.45k op/s; 6.6 MiB/s, 105 objects/s recovering 2024-03-21T13:11:40.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:40.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:40.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:40.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:40 smithi204 ceph-mon[33153]: pgmap v604: 158 pgs: 1 peering, 157 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.45k op/s; 6.6 MiB/s, 105 objects/s recovering 2024-03-21T13:11:40.776 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:40.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:40.814 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:40.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:40.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:40.954 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:40.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:40.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:41.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:41.089 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:41.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:41.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:41.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:41.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:41.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:41.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:41.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:41.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:41.365 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:41.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:41.466 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:41.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:41.505 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:41.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:41.576 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:11:41.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:41.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:41.635 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:41.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:41.737 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:41.781 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:41.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:41.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:41.916 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:41.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:42.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:42.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:42.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:42.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:42.156 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:42.191 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:42.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:42.253 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:11:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:11:42.292 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:42.326 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:42.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:42.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:42.428 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:42.460 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:42.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:42.561 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:42.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:42.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:42.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:42.734 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:42.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:42.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:42.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:42 smithi204 ceph-mon[33153]: pgmap v605: 158 pgs: 158 active+clean; 1.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.00k op/s; 4.9 MiB/s, 79 objects/s recovering 2024-03-21T13:11:42.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:42 smithi204 ceph-mon[33153]: osdmap e311: 8 total, 8 up, 6 in 2024-03-21T13:11:42.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:42 smithi163 ceph-mon[29621]: pgmap v605: 158 pgs: 158 active+clean; 1.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.00k op/s; 4.9 MiB/s, 79 objects/s recovering 2024-03-21T13:11:42.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:42 smithi163 ceph-mon[29621]: osdmap e311: 8 total, 8 up, 6 in 2024-03-21T13:11:42.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:42 smithi163 ceph-mon[40675]: pgmap v605: 158 pgs: 158 active+clean; 1.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.00k op/s; 4.9 MiB/s, 79 objects/s recovering 2024-03-21T13:11:42.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:42 smithi163 ceph-mon[40675]: osdmap e311: 8 total, 8 up, 6 in 2024-03-21T13:11:42.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:42.866 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:42.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:42.969 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:43.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:43.010 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:43.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:43.111 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:43.147 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:43.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:43.248 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:43.292 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:43.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:43.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:43.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:43.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:43.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:43.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:43.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:43.667 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:43.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:43.702 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:43.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:43.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:43.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:43.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:43.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:43.836 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:43.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:43.937 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:43.973 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:43.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:44.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:44.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:44.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:44.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:44.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:44.239 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:44.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:44.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:44.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:44.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:44.477 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:44.516 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:44.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:44.618 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:44.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:44.643 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:44.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:44.744 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:44.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:44 smithi204 ceph-mon[33153]: pgmap v607: 158 pgs: 158 active+clean; 1.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.34k op/s 2024-03-21T13:11:44.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:44 smithi163 ceph-mon[29621]: pgmap v607: 158 pgs: 158 active+clean; 1.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.34k op/s 2024-03-21T13:11:44.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:44 smithi163 ceph-mon[40675]: pgmap v607: 158 pgs: 158 active+clean; 1.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.34k op/s 2024-03-21T13:11:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:44.782 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:44.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:44.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:44.919 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:44.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:45.020 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:45.056 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:45.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:45.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:45.157 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:45.195 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:45.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:45.296 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:45.335 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:45.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:45.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:45.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:45.471 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:45.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:45.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:45.607 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:45.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:45.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:45.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:45.742 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:45.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:45.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:45.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:45.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:45.979 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:46.013 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:11:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:11:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:11:46.013 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:46.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:46.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:46.116 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:46.151 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:46.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:46.253 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:46.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:46.297 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:46.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:46.398 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:46.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:46.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:46.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:46.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:46 smithi163 ceph-mon[29621]: pgmap v608: 158 pgs: 158 active+clean; 1.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 648 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-21T13:11:46.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:46 smithi163 ceph-mon[40675]: pgmap v608: 158 pgs: 158 active+clean; 1.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 648 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-21T13:11:46.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:46.569 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:46.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:46.670 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:46.708 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:46.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:46.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:46 smithi204 ceph-mon[33153]: pgmap v608: 158 pgs: 158 active+clean; 1.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 648 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-21T13:11:46.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:46.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:46.845 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:46.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:46.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:46.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:46.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:46.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:47.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:47.106 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:47.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:47.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:47.208 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:47.242 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:47.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:47.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:47.343 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:47.388 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:47.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:47.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:47.490 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:47.523 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:47.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:47.625 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:47.661 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:47.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:47.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:47.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:47 smithi163 ceph-mon[29621]: osdmap e312: 8 total, 8 up, 6 in 2024-03-21T13:11:47.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:47 smithi163 ceph-mon[40675]: osdmap e312: 8 total, 8 up, 6 in 2024-03-21T13:11:47.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:47 smithi204 ceph-mon[33153]: osdmap e312: 8 total, 8 up, 6 in 2024-03-21T13:11:47.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:47.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:47.795 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:47.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:47.896 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:47.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:47.938 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:47.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:48.039 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:48.076 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:48.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:48.080 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:11:48.127 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:11:48.167 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:11:48.178 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:48.210 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:11:48.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:48.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:48.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:48.250 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:11:48.288 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:11:48.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:48.335 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:11:48.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:48.361 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:48.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:48.378 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:11:48.385 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:11:48.385 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:11:48.385 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:11:48.385 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:11:48.385 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:11:48.385 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:11:48.385 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:11:48.386 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:11:48.462 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:48.505 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:48.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:48.606 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:48.644 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:48.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:48.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:48.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:48.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:48 smithi163 ceph-mon[29621]: pgmap v610: 158 pgs: 158 active+clean; 1.0 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.97k op/s 2024-03-21T13:11:48.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:48 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3089074878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:11:48.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:48 smithi163 ceph-mon[40675]: pgmap v610: 158 pgs: 158 active+clean; 1.0 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.97k op/s 2024-03-21T13:11:48.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:48 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3089074878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:11:48.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:48 smithi204 ceph-mon[33153]: pgmap v610: 158 pgs: 158 active+clean; 1.0 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.97k op/s 2024-03-21T13:11:48.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:48 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3089074878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:11:48.779 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:48.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:48.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:48.881 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:48.916 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:48.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:48.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:49.018 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:49.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:49.057 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:49.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:49.159 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:49.197 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:49.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:49.298 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:49.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:49.338 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:49.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:49.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.2 to 1 (10000) 2024-03-21T13:11:49.471 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:49.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:49.573 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:49.606 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:11:49.606 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-03-21T13:11:49.606 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-03-21T13:11:49.606 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:11:49.606 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.606 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:89863 7.9 7:901f84a5:::benchmark_data_smithi163_80312_object89862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:11:49.606 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:54.714348+0000", 2024-03-21T13:11:49.606 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 594.31789598600005, 2024-03-21T13:11:49.606 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067425181000000001, 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 89863 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714348+0000", 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714348+0000", 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714349+0000", 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4219999999999999e-06 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714368+0000", 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8301e-05 2024-03-21T13:11:49.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714378+0000", 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0594000000000001e-05 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714386+0000", 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6639999999999998e-06 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714461+0000", 2024-03-21T13:11:49.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4826999999999994e-05 2024-03-21T13:11:49.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.610 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.610 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:49.610 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714602+0000", 2024-03-21T13:11:49.610 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141686 2024-03-21T13:11:49.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.610 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.610 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:11:49.610 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.714759+0000", 2024-03-21T13:11:49.610 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015627600000000001 2024-03-21T13:11:49.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.610 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.610 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:49.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.721998+0000", 2024-03-21T13:11:49.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0072397199999999998 2024-03-21T13:11:49.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723198+0000", 2024-03-21T13:11:49.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001200095 2024-03-21T13:11:49.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.781671+0000", 2024-03-21T13:11:49.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.058472203 2024-03-21T13:11:49.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.781725+0000", 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4168e-05 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.781773+0000", 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8225e-05 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:89877 7.9 7:9aa81cc8:::benchmark_data_smithi163_80312_object89876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:54.723093+0000", 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 594.30915074999996, 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.070265444999999996, 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 89877 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723093+0000", 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723093+0000", 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723095+0000", 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.564e-06 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723117+0000", 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2660999999999999e-05 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723132+0000", 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5092999999999999e-05 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723140+0000", 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8669999999999994e-06 2024-03-21T13:11:49.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723227+0000", 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6644999999999995e-05 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723339+0000", 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011218900000000001 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.723472+0000", 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013277499999999999 2024-03-21T13:11:49.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726766+0000", 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032936649999999999 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.730039+0000", 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032733950000000001 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.793215+0000", 2024-03-21T13:11:49.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063175855000000003 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.793312+0000", 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7078999999999997e-05 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.793359+0000", 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6656999999999997e-05 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:89880 7.9 7:9d968a4b:::benchmark_data_smithi163_80312_object89879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:54.726832+0000", 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 594.30541209199998, 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.066681890999999993, 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 89880 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:49.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726832+0000", 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726832+0000", 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726833+0000", 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5300000000000002e-07 2024-03-21T13:11:49.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726851+0000", 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8651000000000002e-05 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726863+0000", 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1615e-05 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726871+0000", 2024-03-21T13:11:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3629999999999994e-06 2024-03-21T13:11:49.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.622 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:49.622 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.726926+0000", 2024-03-21T13:11:49.622 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4891e-05 2024-03-21T13:11:49.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.622 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:49.622 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.727174+0000", 2024-03-21T13:11:49.622 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024773599999999997 2024-03-21T13:11:49.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.622 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:11:49.623 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.727352+0000", 2024-03-21T13:11:49.623 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017773600000000001 2024-03-21T13:11:49.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.623 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:49.623 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.730133+0000", 2024-03-21T13:11:49.623 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027817240000000002 2024-03-21T13:11:49.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.623 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.623 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.730525+0000", 2024-03-21T13:11:49.623 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039137999999999998 2024-03-21T13:11:49.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.793397+0000", 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062871705999999999 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.793476+0000", 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9856000000000003e-05 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:54.793514+0000", 2024-03-21T13:11:49.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7280000000000002e-05 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:91006 7.b 7:d6eb55a7:::benchmark_data_smithi163_80312_object91005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:55.792977+0000", 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 593.23926688899996, 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062755527000000005, 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:11:49.625 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:11:49.626 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91006 2024-03-21T13:11:49.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.626 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:49.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:49.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.792977+0000", 2024-03-21T13:11:49.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:49.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.792977+0000", 2024-03-21T13:11:49.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.627 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:49.627 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.792978+0000", 2024-03-21T13:11:49.627 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7700000000000004e-07 2024-03-21T13:11:49.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.627 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:49.627 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.792997+0000", 2024-03-21T13:11:49.627 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9125000000000001e-05 2024-03-21T13:11:49.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.627 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:49.627 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.793007+0000", 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6590000000000002e-06 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.793013+0000", 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9489999999999999e-06 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.798124+0000", 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0051106149999999998 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:49.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.798240+0000", 2024-03-21T13:11:49.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115652 2024-03-21T13:11:49.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-03-21T13:11:49.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.798382+0000", 2024-03-21T13:11:49.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141748 2024-03-21T13:11:49.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:49.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.801043+0000", 2024-03-21T13:11:49.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026610359999999999 2024-03-21T13:11:49.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.809641+0000", 2024-03-21T13:11:49.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085982579999999992 2024-03-21T13:11:49.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.855609+0000", 2024-03-21T13:11:49.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045968588999999997 2024-03-21T13:11:49.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:49.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.855678+0000", 2024-03-21T13:11:49.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8659999999999997e-05 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.855733+0000", 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4459000000000002e-05 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:91014 7.b 7:da4ce39b:::benchmark_data_smithi163_80312_object91013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:55.799579+0000", 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 593.23266531499996, 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063691170000000005, 2024-03-21T13:11:49.631 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91014 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.799579+0000", 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.799579+0000", 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.799580+0000", 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.319e-06 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.799602+0000", 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2589e-05 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:49.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.799612+0000", 2024-03-21T13:11:49.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4919999999999999e-06 2024-03-21T13:11:49.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:49.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.799619+0000", 2024-03-21T13:11:49.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6420000000000001e-06 2024-03-21T13:11:49.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:49.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.801356+0000", 2024-03-21T13:11:49.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001737904 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.801469+0000", 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011238099999999999 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.801609+0000", 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014058300000000001 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.809681+0000", 2024-03-21T13:11:49.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0080712089999999993 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.812820+0000", 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031392690000000001 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.863124+0000", 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.050303632000000001 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:49.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.863219+0000", 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5036000000000003e-05 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.863270+0000", 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1113999999999999e-05 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:91025 7.b 7:d35260cc:::benchmark_data_smithi163_80312_object91024:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:55.808912+0000", 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 593.22333189400001, 2024-03-21T13:11:49.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055675924000000002, 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91025 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.808912+0000", 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.808912+0000", 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.808914+0000", 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.606e-06 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.808933+0000", 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9772999999999999e-05 2024-03-21T13:11:49.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.808945+0000", 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2098e-05 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.808954+0000", 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1079999999999993e-06 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.810074+0000", 2024-03-21T13:11:49.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001120832 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.810235+0000", 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000160338 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.810340+0000", 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010535300000000001 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.812963+0000", 2024-03-21T13:11:49.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026229090000000001 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.812976+0000", 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3481e-05 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.864500+0000", 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051523878000000002 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:49.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.864551+0000", 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0519000000000002e-05 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.864588+0000", 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7029000000000001e-05 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:91194 7.9 7:940b1401:::benchmark_data_smithi163_80312_object91193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:55.990360+0000", 2024-03-21T13:11:49.643 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 593.04188398999997, 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054572715000000001, 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91194 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990360+0000", 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990360+0000", 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990362+0000", 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8479999999999999e-06 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990386+0000", 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4181000000000001e-05 2024-03-21T13:11:49.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990402+0000", 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5821e-05 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990410+0000", 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4970000000000001e-06 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990486+0000", 2024-03-21T13:11:49.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5362000000000001e-05 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990616+0000", 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013085700000000001 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.990912+0000", 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029580900000000001 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:49.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.994547+0000", 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0036350950000000001 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:55.999053+0000", 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045052060000000003 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.044850+0000", 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045797787 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.044889+0000", 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8739999999999998e-05 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.044933+0000", 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3511999999999999e-05 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:92024 7.b 7:def0ff7d:::benchmark_data_smithi163_80312_object92023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-21T13:11:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:01:56.751202+0000", 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 592.28104161600004, 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.058877783000000003, 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92024 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.751202+0000", 2024-03-21T13:11:49.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.751202+0000", 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.751204+0000", 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5340000000000001e-06 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:11:49.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.751226+0000", 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2104e-05 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.751238+0000", 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2072999999999999e-05 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.751246+0000", 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3100000000000001e-06 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.752009+0000", 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00076259600000000004 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.752143+0000", 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013382499999999999 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.752284+0000", 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141025 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.755341+0000", 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0030574339999999999 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.755507+0000", 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000165556 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.809940+0000", 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054433606000000002 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.810017+0000", 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6401000000000003e-05 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:01:56.810080+0000", 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3318999999999995e-05 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:11:49.655 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:11:49.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:49.657 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:49.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:49.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:49 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2358693190' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:11:49.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:49 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:11:49.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:49.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:49 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2358693190' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:11:49.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:49 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:11:49.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:49 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2358693190' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:11:49.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:49 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:11:49.793 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:49.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:49.895 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:49.930 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:49.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:49.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:50.032 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:50.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:50.066 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:50.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:50.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:50.203 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:50.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:50.304 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:50.312 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-21T13:11:50.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:50.350 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:50.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:50.451 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:50.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:50.501 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:50.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:50.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[29621]: pgmap v611: 158 pgs: 158 active+clean; 1.0 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-21T13:11:50.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:11:50.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[29621]: osdmap e313: 8 total, 8 up, 6 in 2024-03-21T13:11:50.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:11:50.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:11:50.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:11:50.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:50.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[40675]: pgmap v611: 158 pgs: 158 active+clean; 1.0 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-21T13:11:50.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:11:50.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[40675]: osdmap e313: 8 total, 8 up, 6 in 2024-03-21T13:11:50.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:11:50.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:11:50.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:11:50.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:50.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:50.637 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:50.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:50.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:50.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:50 smithi204 ceph-mon[33153]: pgmap v611: 158 pgs: 158 active+clean; 1.0 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-21T13:11:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:50 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:11:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:50 smithi204 ceph-mon[33153]: osdmap e313: 8 total, 8 up, 6 in 2024-03-21T13:11:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:50 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:11:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:50 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:11:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:11:50.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:50.775 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:50.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:50.877 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:50.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:50.914 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:50.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:51.015 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:51.055 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:51.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:51.156 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:51.189 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:51.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:51.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:51.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:51.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:51.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:51.417 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:51.434 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:11:51.453 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:51.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:51.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:51.556 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:51.594 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:51.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:51.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:51.696 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:51.721 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:51.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:51.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:51 smithi204 ceph-mon[33153]: osdmap e314: 8 total, 8 up, 6 in 2024-03-21T13:11:51.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:51 smithi204 ceph-mon[33153]: osdmap e315: 8 total, 8 up, 6 in 2024-03-21T13:11:51.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:51 smithi163 ceph-mon[29621]: osdmap e314: 8 total, 8 up, 6 in 2024-03-21T13:11:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:51 smithi163 ceph-mon[29621]: osdmap e315: 8 total, 8 up, 6 in 2024-03-21T13:11:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:51 smithi163 ceph-mon[40675]: osdmap e314: 8 total, 8 up, 6 in 2024-03-21T13:11:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:51 smithi163 ceph-mon[40675]: osdmap e315: 8 total, 8 up, 6 in 2024-03-21T13:11:51.822 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:51.859 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:51.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:51.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:51.961 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:51.998 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:51.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:52.097 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-21T13:11:52.097 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:11:52.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:52.115 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:52.128 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:52.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:52.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:52.230 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:52.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:11:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:11:52.277 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:52.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:52.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:52.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:52.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:52.514 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:52.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:52.552 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:52.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:52.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:52.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:52 smithi163 ceph-mon[29621]: pgmap v614: 158 pgs: 158 active+clean; 716 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.19k op/s 2024-03-21T13:11:52.671 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:52 smithi163 ceph-mon[40675]: pgmap v614: 158 pgs: 158 active+clean; 716 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.19k op/s 2024-03-21T13:11:52.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:52.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:52.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:52 smithi204 ceph-mon[33153]: pgmap v614: 158 pgs: 158 active+clean; 716 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.19k op/s 2024-03-21T13:11:52.788 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:52.824 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:52.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:52.926 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:52.962 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:52.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:53.062 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:53.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:53.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:53.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:53.200 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:53.240 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:53.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:53.341 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:53.378 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:53.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:53.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:53.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:53.514 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:53.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:53.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:53.616 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:53.622 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:53.663 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:53.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:53.694 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:53.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:53.734 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:53.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:53.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:53.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:53.844 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:53.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:53.945 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:53.981 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:53.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:54.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:54.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:54.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:54.221 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:54.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:54.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:54.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:54.394 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:54.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:54.496 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:54.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:54 smithi163 ceph-mon[29621]: pgmap v616: 158 pgs: 158 active+clean; 555 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s 2024-03-21T13:11:54.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:54 smithi163 ceph-mon[40675]: pgmap v616: 158 pgs: 158 active+clean; 555 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s 2024-03-21T13:11:54.532 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:54.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:54.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:54.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:54.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:54 smithi204 ceph-mon[33153]: pgmap v616: 158 pgs: 158 active+clean; 555 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s 2024-03-21T13:11:54.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:54.808 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:54.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:54.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:54.910 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:54.948 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:54.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:55.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:55.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:55.090 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:55.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:55.191 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:55.236 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:55.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:55.337 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:55.376 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:55.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:55.477 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:55.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:55.515 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:55.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:55.616 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:55.652 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:55.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:55.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:55.785 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Removed 121527 objects 2024-03-21T13:11:55.785 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Clean up completed and total clean up time :92.2411 2024-03-21T13:11:55.788 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:55.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:55.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:55.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:55.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:55.915 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:55.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:55.953 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:11:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:11:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:11:56.016 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:56.041 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:56.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:56.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:56.167 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:56.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:56.268 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:56.292 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:56.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:56.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:56.394 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:56.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:56.458 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:56.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:56.509 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-21T13:11:56.509 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-21T13:11:56.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:56 smithi163 ceph-mon[29621]: pgmap v617: 158 pgs: 158 active+clean; 555 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s 2024-03-21T13:11:56.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:56 smithi163 ceph-mon[40675]: pgmap v617: 158 pgs: 158 active+clean; 555 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s 2024-03-21T13:11:56.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:56.583 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:56.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:56.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:56.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:56.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:56.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:56.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:56 smithi204 ceph-mon[33153]: pgmap v617: 158 pgs: 158 active+clean; 555 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s 2024-03-21T13:11:56.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:56.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:56.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:56.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:56.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:56.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:56.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:56.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:57.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:57.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:57.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:57.204 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:57.230 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:57.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:57.331 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:11:57.355 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:57.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:57.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:11:57.457 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:57.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:57.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:57.587 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:57.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:57.612 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:57.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:57.713 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:57.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:57.737 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:57.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:57.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:57.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:57.867 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:57.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:57.968 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:57.993 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:57.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:58.066 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:11:58.087 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:11:58.094 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:58.105 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:11:58.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:58.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:58.124 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:11:58.144 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:11:58.164 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:11:58.185 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:11:58.206 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:11:58.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:11:58.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:11:58.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:11:58.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:11:58.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:11:58.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:11:58.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:11:58.207 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:11:58.224 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:58.248 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:58.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:58.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:58.350 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:58.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:58.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:58.496 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:11:58.504 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:58.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:58.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:58.606 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:11:58.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:58.631 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:58.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:11:58.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:11:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:58 smithi204 ceph-mon[33153]: pgmap v618: 158 pgs: 158 active+clean; 49 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 2.17k op/s 2024-03-21T13:11:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:58 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3600622834' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:11:58.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:58 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/313598681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:11:58.760 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:58 smithi163 ceph-mon[29621]: pgmap v618: 158 pgs: 158 active+clean; 49 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 2.17k op/s 2024-03-21T13:11:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:58 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3600622834' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:11:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:58 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/313598681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:11:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:58 smithi163 ceph-mon[40675]: pgmap v618: 158 pgs: 158 active+clean; 49 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 2.17k op/s 2024-03-21T13:11:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:58 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3600622834' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:11:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:58 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/313598681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:11:58.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:58.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:11:58.863 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:58.887 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:58.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:58.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:11:58.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:59.019 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:59.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:59.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:59.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:59.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:59.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:59.251 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:59.279 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:59.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:59.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:11:59.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:59.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:11:59.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:11:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:59.535 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:59.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:11:59.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:11:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:59.662 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:11:59.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:11:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:59 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:11:59.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:59 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3600622834' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:11:59.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:11:59 smithi204 ceph-mon[33153]: osdmap e316: 8 total, 8 up, 6 in 2024-03-21T13:11:59.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:59 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:11:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:59 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3600622834' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:11:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:11:59 smithi163 ceph-mon[29621]: osdmap e316: 8 total, 8 up, 6 in 2024-03-21T13:11:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:59 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:11:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:59 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3600622834' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:11:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:11:59 smithi163 ceph-mon[40675]: osdmap e316: 8 total, 8 up, 6 in 2024-03-21T13:11:59.763 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:11:59.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:59.789 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:59.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:11:59.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:11:59.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:11:59.915 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:11:59.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:00.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:00.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:00.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:00.149 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:00.173 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:00.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:00.261 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:00.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:00.302 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:00.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:00.404 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:00.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:00.429 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:00.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:00.472 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:12:00.475 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:12:00.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:00.560 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:00.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:00.662 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:00.687 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:00.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:00.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:00 smithi163 ceph-mon[29621]: pgmap v620: 158 pgs: 158 active+clean; 49 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.33k op/s 2024-03-21T13:12:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:00 smithi163 ceph-mon[40675]: pgmap v620: 158 pgs: 158 active+clean; 49 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.33k op/s 2024-03-21T13:12:00.789 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:00.813 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:00.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:00.915 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:00.939 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:00.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:00 smithi204 ceph-mon[33153]: pgmap v620: 158 pgs: 158 active+clean; 49 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.33k op/s 2024-03-21T13:12:01.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:01.064 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:01.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:01.166 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:01.192 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:01.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:01.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:01.202 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":620,"stamp":"2024-03-21T13:11:58.871980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51059136,"num_objects":969,"num_object_clones":0,"num_object_copies":2907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":969,"num_whiteouts":0,"num_read":2132,"num_read_kb":1978,"num_write":172722,"num_write_kb":3799454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25765,"num_bytes_recovered":1669735601,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93579,"ondisk_log_size":93579,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":4934328,"kb_used_data":716808,"kb_used_omap":9,"kb_used_meta":4217206,"kb_avail":557536584,"statfs":{"total":575970213888,"available":570917462016,"internally_reserved":0,"allocated":734011392,"data_stored":731182122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":4318419622},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,4,5,4,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-699203584,"num_objects":-10669,"num_object_clones":0,"num_object_copies":-32007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10669,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":10669,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002072"},"pg_stats":[{"pgid":"9.15","version":"315'9291","reported_seq":14775,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.188510+0000","last_change":"2024-03-21T13:11:09.626943+0000","last_active":"2024-03-21T13:11:55.188510+0000","last_peered":"2024-03-21T13:11:55.188510+0000","last_clean":"2024-03-21T13:11:55.188510+0000","last_became_active":"2024-03-21T13:09:57.348575+0000","last_became_peered":"2024-03-21T13:09:57.348575+0000","last_unstale":"2024-03-21T13:11:55.188510+0000","last_undegraded":"2024-03-21T13:11:55.188510+0000","last_fullsized":"2024-03-21T13:11:55.188510+0000","mapping_epoch":266,"log_start":"282'5831","ondisk_log_start":"282'5831","created":246,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"296'7242","last_scrub_stamp":"2024-03-21T13:11:09.626710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:11:09.626710+0000","objects_scrubbed":2091,"log_size":3460,"log_dups_size":0,"ondisk_log_size":3460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:12.927209+0000","scrub_duration":579,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703936,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2065,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":390,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390459+0000","last_change":"2024-03-21T13:09:11.718192+0000","last_active":"2024-03-21T13:11:51.390459+0000","last_peered":"2024-03-21T13:11:51.390459+0000","last_clean":"2024-03-21T13:11:51.390459+0000","last_became_active":"2024-03-21T13:06:45.404217+0000","last_became_peered":"2024-03-21T13:06:45.404217+0000","last_unstale":"2024-03-21T13:11:51.390459+0000","last_undegraded":"2024-03-21T13:11:51.390459+0000","last_fullsized":"2024-03-21T13:11:51.390459+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:11.717992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:09:11.717992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":433,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390293+0000","last_change":"2024-03-21T13:10:03.600751+0000","last_active":"2024-03-21T13:11:51.390293+0000","last_peered":"2024-03-21T13:11:51.390293+0000","last_clean":"2024-03-21T13:11:51.390293+0000","last_became_active":"2024-03-21T13:10:03.600240+0000","last_became_peered":"2024-03-21T13:10:03.600240+0000","last_unstale":"2024-03-21T13:11:51.390293+0000","last_undegraded":"2024-03-21T13:11:51.390293+0000","last_fullsized":"2024-03-21T13:11:51.390293+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:54.360939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:07:54.360939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":97,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.389130+0000","last_change":"2024-03-21T13:10:03.605517+0000","last_active":"2024-03-21T13:11:51.389130+0000","last_peered":"2024-03-21T13:11:51.389130+0000","last_clean":"2024-03-21T13:11:51.389130+0000","last_became_active":"2024-03-21T13:10:03.605149+0000","last_became_peered":"2024-03-21T13:10:03.605149+0000","last_unstale":"2024-03-21T13:11:51.389130+0000","last_undegraded":"2024-03-21T13:11:51.389130+0000","last_fullsized":"2024-03-21T13:11:51.389130+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:49.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.260693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"234'19","reported_seq":614,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.380604+0000","last_change":"2024-03-21T13:11:14.697504+0000","last_active":"2024-03-21T13:11:51.380604+0000","last_peered":"2024-03-21T13:11:51.380604+0000","last_clean":"2024-03-21T13:11:51.380604+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:11:51.380604+0000","last_undegraded":"2024-03-21T13:11:51.380604+0000","last_fullsized":"2024-03-21T13:11:51.380604+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:11:14.697379+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:11:14.697379+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:31.632762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"315'9203","reported_seq":17474,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.740769+0000","last_change":"2024-03-21T13:11:27.751265+0000","last_active":"2024-03-21T13:11:55.740769+0000","last_peered":"2024-03-21T13:11:55.740769+0000","last_clean":"2024-03-21T13:11:55.740769+0000","last_became_active":"2024-03-21T13:11:27.736009+0000","last_became_peered":"2024-03-21T13:11:27.736009+0000","last_unstale":"2024-03-21T13:11:55.740769+0000","last_undegraded":"2024-03-21T13:11:55.740769+0000","last_fullsized":"2024-03-21T13:11:55.740769+0000","mapping_epoch":305,"log_start":"282'5696","ondisk_log_start":"282'5696","created":246,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":5,"last_scrub":"262'4197","last_scrub_stamp":"2024-03-21T13:09:47.353983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:09:47.353983+0000","objects_scrubbed":2533,"log_size":3507,"log_dups_size":0,"ondisk_log_size":3507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":575,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5017,"num_write_kb":79488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1638,"num_bytes_recovered":107347968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":529,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390009+0000","last_change":"2024-03-21T13:11:32.995208+0000","last_active":"2024-03-21T13:11:51.390009+0000","last_peered":"2024-03-21T13:11:51.390009+0000","last_clean":"2024-03-21T13:11:51.390009+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:11:51.390009+0000","last_undegraded":"2024-03-21T13:11:51.390009+0000","last_fullsized":"2024-03-21T13:11:51.390009+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:32.995084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:11:32.995084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:56.298187+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":480,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386428+0000","last_change":"2024-03-21T13:11:01.803199+0000","last_active":"2024-03-21T13:11:51.386428+0000","last_peered":"2024-03-21T13:11:51.386428+0000","last_clean":"2024-03-21T13:11:51.386428+0000","last_became_active":"2024-03-21T13:10:03.591543+0000","last_became_peered":"2024-03-21T13:10:03.591543+0000","last_unstale":"2024-03-21T13:11:51.386428+0000","last_undegraded":"2024-03-21T13:11:51.386428+0000","last_fullsized":"2024-03-21T13:11:51.386428+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:01.803105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:11:01.803105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.266826+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":561,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390143+0000","last_change":"2024-03-21T13:10:03.642319+0000","last_active":"2024-03-21T13:11:51.390143+0000","last_peered":"2024-03-21T13:11:51.390143+0000","last_clean":"2024-03-21T13:11:51.390143+0000","last_became_active":"2024-03-21T13:10:03.610318+0000","last_became_peered":"2024-03-21T13:10:03.610318+0000","last_unstale":"2024-03-21T13:11:51.390143+0000","last_undegraded":"2024-03-21T13:11:51.390143+0000","last_fullsized":"2024-03-21T13:11:51.390143+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:09:05.740634+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_clean_scrub_stamp":"2024-03-21T13:09:05.740634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998754+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":61,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390374+0000","last_change":"2024-03-21T13:11:28.976696+0000","last_active":"2024-03-21T13:11:51.390374+0000","last_peered":"2024-03-21T13:11:51.390374+0000","last_clean":"2024-03-21T13:11:51.390374+0000","last_became_active":"2024-03-21T13:11:21.702209+0000","last_became_peered":"2024-03-21T13:11:21.702209+0000","last_unstale":"2024-03-21T13:11:51.390374+0000","last_undegraded":"2024-03-21T13:11:51.390374+0000","last_fullsized":"2024-03-21T13:11:51.390374+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:11:28.976545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:28.976545+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:35.180894+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"315'9318","reported_seq":19767,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.201951+0000","last_change":"2024-03-21T13:11:10.626046+0000","last_active":"2024-03-21T13:11:55.201951+0000","last_peered":"2024-03-21T13:11:55.201951+0000","last_clean":"2024-03-21T13:11:55.201951+0000","last_became_active":"2024-03-21T13:11:06.479959+0000","last_became_peered":"2024-03-21T13:11:06.479959+0000","last_unstale":"2024-03-21T13:11:55.201951+0000","last_undegraded":"2024-03-21T13:11:55.201951+0000","last_fullsized":"2024-03-21T13:11:55.201951+0000","mapping_epoch":294,"log_start":"283'5936","ondisk_log_start":"283'5936","created":246,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":5,"last_scrub":"296'7217","last_scrub_stamp":"2024-03-21T13:11:10.623185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:11:10.623185+0000","objects_scrubbed":2145,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:18.634450+0000","scrub_duration":602,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1966080,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2115,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":440,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.388659+0000","last_change":"2024-03-21T13:10:03.598720+0000","last_active":"2024-03-21T13:11:51.388659+0000","last_peered":"2024-03-21T13:11:51.388659+0000","last_clean":"2024-03-21T13:11:51.388659+0000","last_became_active":"2024-03-21T13:10:03.597963+0000","last_became_peered":"2024-03-21T13:10:03.597963+0000","last_unstale":"2024-03-21T13:11:51.388659+0000","last_undegraded":"2024-03-21T13:11:51.388659+0000","last_fullsized":"2024-03-21T13:11:51.388659+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:57.627967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.627967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:57.134714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":437,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.387987+0000","last_change":"2024-03-21T13:11:20.342434+0000","last_active":"2024-03-21T13:11:51.387987+0000","last_peered":"2024-03-21T13:11:51.387987+0000","last_clean":"2024-03-21T13:11:51.387987+0000","last_became_active":"2024-03-21T13:02:26.236707+0000","last_became_peered":"2024-03-21T13:02:26.236707+0000","last_unstale":"2024-03-21T13:11:51.387987+0000","last_undegraded":"2024-03-21T13:11:51.387987+0000","last_fullsized":"2024-03-21T13:11:51.387987+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:20.342189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:11:20.342189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:35.037790+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":413,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386401+0000","last_change":"2024-03-21T13:11:33.515344+0000","last_active":"2024-03-21T13:11:51.386401+0000","last_peered":"2024-03-21T13:11:51.386401+0000","last_clean":"2024-03-21T13:11:51.386401+0000","last_became_active":"2024-03-21T13:02:26.245402+0000","last_became_peered":"2024-03-21T13:02:26.245402+0000","last_unstale":"2024-03-21T13:11:51.386401+0000","last_undegraded":"2024-03-21T13:11:51.386401+0000","last_fullsized":"2024-03-21T13:11:51.386401+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:33.515220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:33.515220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:53.156657+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":532,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.393395+0000","last_change":"2024-03-21T13:05:51.580203+0000","last_active":"2024-03-21T13:11:51.393395+0000","last_peered":"2024-03-21T13:11:51.393395+0000","last_clean":"2024-03-21T13:11:51.393395+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:11:51.393395+0000","last_undegraded":"2024-03-21T13:11:51.393395+0000","last_fullsized":"2024-03-21T13:11:51.393395+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-21T13:05:51.580074+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:05:51.580074+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998192+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"315'9042","reported_seq":18856,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.023743+0000","last_change":"2024-03-21T13:10:50.513893+0000","last_active":"2024-03-21T13:11:55.023743+0000","last_peered":"2024-03-21T13:11:55.023743+0000","last_clean":"2024-03-21T13:11:55.023743+0000","last_became_active":"2024-03-21T13:10:50.503988+0000","last_became_peered":"2024-03-21T13:10:50.503988+0000","last_unstale":"2024-03-21T13:11:55.023743+0000","last_undegraded":"2024-03-21T13:11:55.023743+0000","last_fullsized":"2024-03-21T13:11:55.023743+0000","mapping_epoch":286,"log_start":"278'5200","ondisk_log_start":"278'5200","created":246,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":5,"last_scrub":"270'4656","last_scrub_stamp":"2024-03-21T13:10:01.828233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:10:01.828233+0000","objects_scrubbed":3071,"log_size":3842,"log_dups_size":0,"ondisk_log_size":3842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:57.805022+0000","scrub_duration":1077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3604480,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4400,"num_write_kb":44288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3106,"num_bytes_recovered":203554816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":537,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.396489+0000","last_change":"2024-03-21T13:10:03.611898+0000","last_active":"2024-03-21T13:11:51.396489+0000","last_peered":"2024-03-21T13:11:51.396489+0000","last_clean":"2024-03-21T13:11:51.396489+0000","last_became_active":"2024-03-21T13:10:03.611240+0000","last_became_peered":"2024-03-21T13:10:03.611240+0000","last_unstale":"2024-03-21T13:11:51.396489+0000","last_undegraded":"2024-03-21T13:11:51.396489+0000","last_fullsized":"2024-03-21T13:11:51.396489+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:17.747360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:17.747360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998490+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":545,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390823+0000","last_change":"2024-03-21T13:08:09.443525+0000","last_active":"2024-03-21T13:11:51.390823+0000","last_peered":"2024-03-21T13:11:51.390823+0000","last_clean":"2024-03-21T13:11:51.390823+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:11:51.390823+0000","last_undegraded":"2024-03-21T13:11:51.390823+0000","last_fullsized":"2024-03-21T13:11:51.390823+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:09.443411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:09.443411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":561,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.384316+0000","last_change":"2024-03-21T13:11:25.792475+0000","last_active":"2024-03-21T13:11:51.384316+0000","last_peered":"2024-03-21T13:11:51.384316+0000","last_clean":"2024-03-21T13:11:51.384316+0000","last_became_active":"2024-03-21T13:02:26.230057+0000","last_became_peered":"2024-03-21T13:02:26.230057+0000","last_unstale":"2024-03-21T13:11:51.384316+0000","last_undegraded":"2024-03-21T13:11:51.384316+0000","last_fullsized":"2024-03-21T13:11:51.384316+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:25.792349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:25.792349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:48.325353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"234'9","reported_seq":560,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.389752+0000","last_change":"2024-03-21T13:10:03.627839+0000","last_active":"2024-03-21T13:11:51.389752+0000","last_peered":"2024-03-21T13:11:51.389752+0000","last_clean":"2024-03-21T13:11:51.389752+0000","last_became_active":"2024-03-21T13:10:03.595745+0000","last_became_peered":"2024-03-21T13:10:03.595745+0000","last_unstale":"2024-03-21T13:11:51.389752+0000","last_undegraded":"2024-03-21T13:11:51.389752+0000","last_fullsized":"2024-03-21T13:11:51.389752+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'9","last_scrub_stamp":"2024-03-21T13:10:00.872509+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:10:00.872509+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.999481+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"315'9263","reported_seq":9488,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.188715+0000","last_change":"2024-03-21T13:10:03.146863+0000","last_active":"2024-03-21T13:11:55.188715+0000","last_peered":"2024-03-21T13:11:55.188715+0000","last_clean":"2024-03-21T13:11:55.188715+0000","last_became_active":"2024-03-21T13:09:20.445112+0000","last_became_peered":"2024-03-21T13:09:20.445112+0000","last_unstale":"2024-03-21T13:11:55.188715+0000","last_undegraded":"2024-03-21T13:11:55.188715+0000","last_fullsized":"2024-03-21T13:11:55.188715+0000","mapping_epoch":248,"log_start":"282'5807","ondisk_log_start":"282'5807","created":246,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"271'4851","last_scrub_stamp":"2024-03-21T13:10:03.146744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:10:03.146744+0000","objects_scrubbed":3157,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:57.137429+0000","scrub_duration":441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4412,"num_write_kb":41088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":560,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.385102+0000","last_change":"2024-03-21T13:11:01.737787+0000","last_active":"2024-03-21T13:11:51.385102+0000","last_peered":"2024-03-21T13:11:51.385102+0000","last_clean":"2024-03-21T13:11:51.385102+0000","last_became_active":"2024-03-21T13:02:26.226675+0000","last_became_peered":"2024-03-21T13:02:26.226675+0000","last_unstale":"2024-03-21T13:11:51.385102+0000","last_undegraded":"2024-03-21T13:11:51.385102+0000","last_fullsized":"2024-03-21T13:11:51.385102+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:01.737083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:01.737083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:09.465335+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":447,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.384801+0000","last_change":"2024-03-21T13:11:28.742357+0000","last_active":"2024-03-21T13:11:51.384801+0000","last_peered":"2024-03-21T13:11:51.384801+0000","last_clean":"2024-03-21T13:11:51.384801+0000","last_became_active":"2024-03-21T13:02:26.231879+0000","last_became_peered":"2024-03-21T13:02:26.231879+0000","last_unstale":"2024-03-21T13:11:51.384801+0000","last_undegraded":"2024-03-21T13:11:51.384801+0000","last_fullsized":"2024-03-21T13:11:51.384801+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:28.742139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:11:28.742139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:45.280576+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":432,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.389565+0000","last_change":"2024-03-21T13:06:28.985227+0000","last_active":"2024-03-21T13:11:51.389565+0000","last_peered":"2024-03-21T13:11:51.389565+0000","last_clean":"2024-03-21T13:11:51.389565+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:11:51.389565+0000","last_undegraded":"2024-03-21T13:11:51.389565+0000","last_fullsized":"2024-03-21T13:11:51.389565+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:28.985145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:28.985145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":113,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386529+0000","last_change":"2024-03-21T13:10:03.707422+0000","last_active":"2024-03-21T13:11:51.386529+0000","last_peered":"2024-03-21T13:11:51.386529+0000","last_clean":"2024-03-21T13:11:51.386529+0000","last_became_active":"2024-03-21T13:10:03.604616+0000","last_became_peered":"2024-03-21T13:10:03.604616+0000","last_unstale":"2024-03-21T13:11:51.386529+0000","last_undegraded":"2024-03-21T13:11:51.386529+0000","last_fullsized":"2024-03-21T13:11:51.386529+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:20.158184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.158184+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"315'9151","reported_seq":12701,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.186543+0000","last_change":"2024-03-21T13:11:07.607008+0000","last_active":"2024-03-21T13:11:55.186543+0000","last_peered":"2024-03-21T13:11:55.186543+0000","last_clean":"2024-03-21T13:11:55.186543+0000","last_became_active":"2024-03-21T13:09:42.349288+0000","last_became_peered":"2024-03-21T13:09:42.349288+0000","last_unstale":"2024-03-21T13:11:55.186543+0000","last_undegraded":"2024-03-21T13:11:55.186543+0000","last_fullsized":"2024-03-21T13:11:55.186543+0000","mapping_epoch":258,"log_start":"282'5698","ondisk_log_start":"282'5698","created":246,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":5,"last_scrub":"295'7003","last_scrub_stamp":"2024-03-21T13:11:07.606658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:11:07.606658+0000","objects_scrubbed":2180,"log_size":3453,"log_dups_size":0,"ondisk_log_size":3453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:25.888053+0000","scrub_duration":561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703936,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":540,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.396449+0000","last_change":"2024-03-21T13:10:03.611239+0000","last_active":"2024-03-21T13:11:51.396449+0000","last_peered":"2024-03-21T13:11:51.396449+0000","last_clean":"2024-03-21T13:11:51.396449+0000","last_became_active":"2024-03-21T13:10:03.610319+0000","last_became_peered":"2024-03-21T13:10:03.610319+0000","last_unstale":"2024-03-21T13:11:51.396449+0000","last_undegraded":"2024-03-21T13:11:51.396449+0000","last_fullsized":"2024-03-21T13:11:51.396449+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:44.816722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:44.816722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998273+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":466,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392725+0000","last_change":"2024-03-21T13:10:03.613920+0000","last_active":"2024-03-21T13:11:51.392725+0000","last_peered":"2024-03-21T13:11:51.392725+0000","last_clean":"2024-03-21T13:11:51.392725+0000","last_became_active":"2024-03-21T13:10:03.612409+0000","last_became_peered":"2024-03-21T13:10:03.612409+0000","last_unstale":"2024-03-21T13:11:51.392725+0000","last_undegraded":"2024-03-21T13:11:51.392725+0000","last_fullsized":"2024-03-21T13:11:51.392725+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:56.868411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:56.868411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.999513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":583,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.380762+0000","last_change":"2024-03-21T13:11:35.734699+0000","last_active":"2024-03-21T13:11:51.380762+0000","last_peered":"2024-03-21T13:11:51.380762+0000","last_clean":"2024-03-21T13:11:51.380762+0000","last_became_active":"2024-03-21T13:10:03.586563+0000","last_became_peered":"2024-03-21T13:10:03.586563+0000","last_unstale":"2024-03-21T13:11:51.380762+0000","last_undegraded":"2024-03-21T13:11:51.380762+0000","last_fullsized":"2024-03-21T13:11:51.380762+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:35.734553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:20.956778+0000","last_clean_scrub_stamp":"2024-03-21T13:11:35.734553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:48.178822+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":597,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392666+0000","last_change":"2024-03-21T13:09:02.714077+0000","last_active":"2024-03-21T13:11:51.392666+0000","last_peered":"2024-03-21T13:11:51.392666+0000","last_clean":"2024-03-21T13:11:51.392666+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:11:51.392666+0000","last_undegraded":"2024-03-21T13:11:51.392666+0000","last_fullsized":"2024-03-21T13:11:51.392666+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-21T13:09:02.713898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:02.713898+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998664+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"315'9319","reported_seq":12965,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.178945+0000","last_change":"2024-03-21T13:11:08.696529+0000","last_active":"2024-03-21T13:11:55.178945+0000","last_peered":"2024-03-21T13:11:55.178945+0000","last_clean":"2024-03-21T13:11:55.178945+0000","last_became_active":"2024-03-21T13:09:43.347868+0000","last_became_peered":"2024-03-21T13:09:43.347868+0000","last_unstale":"2024-03-21T13:11:55.178945+0000","last_undegraded":"2024-03-21T13:11:55.178945+0000","last_fullsized":"2024-03-21T13:11:55.178945+0000","mapping_epoch":259,"log_start":"282'5916","ondisk_log_start":"282'5916","created":246,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":5,"last_scrub":"295'7186","last_scrub_stamp":"2024-03-21T13:11:08.696152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:11:08.696152+0000","objects_scrubbed":2177,"log_size":3403,"log_dups_size":0,"ondisk_log_size":3403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:19.436707+0000","scrub_duration":625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2162688,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2144,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":101,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386627+0000","last_change":"2024-03-21T13:10:03.664415+0000","last_active":"2024-03-21T13:11:51.386627+0000","last_peered":"2024-03-21T13:11:51.386627+0000","last_clean":"2024-03-21T13:11:51.386627+0000","last_became_active":"2024-03-21T13:10:03.605708+0000","last_became_peered":"2024-03-21T13:10:03.605708+0000","last_unstale":"2024-03-21T13:11:51.386627+0000","last_undegraded":"2024-03-21T13:11:51.386627+0000","last_fullsized":"2024-03-21T13:11:51.386627+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:50.295057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.295057+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":705,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386574+0000","last_change":"2024-03-21T13:05:38.022572+0000","last_active":"2024-03-21T13:11:51.386574+0000","last_peered":"2024-03-21T13:11:51.386574+0000","last_clean":"2024-03-21T13:11:51.386574+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:11:51.386574+0000","last_undegraded":"2024-03-21T13:11:51.386574+0000","last_fullsized":"2024-03-21T13:11:51.386574+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:05:38.022439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.022439+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":440,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.396805+0000","last_change":"2024-03-21T13:08:53.540108+0000","last_active":"2024-03-21T13:11:51.396805+0000","last_peered":"2024-03-21T13:11:51.396805+0000","last_clean":"2024-03-21T13:11:51.396805+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:11:51.396805+0000","last_undegraded":"2024-03-21T13:11:51.396805+0000","last_fullsized":"2024-03-21T13:11:51.396805+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:08:53.539909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:53.539909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":490,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392095+0000","last_change":"2024-03-21T13:11:16.033191+0000","last_active":"2024-03-21T13:11:51.392095+0000","last_peered":"2024-03-21T13:11:51.392095+0000","last_clean":"2024-03-21T13:11:51.392095+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:11:51.392095+0000","last_undegraded":"2024-03-21T13:11:51.392095+0000","last_fullsized":"2024-03-21T13:11:51.392095+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:11:16.033036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:16.033036+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:43.964982+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"315'9126","reported_seq":17111,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.246115+0000","last_change":"2024-03-21T13:11:26.946552+0000","last_active":"2024-03-21T13:11:55.246115+0000","last_peered":"2024-03-21T13:11:55.246115+0000","last_clean":"2024-03-21T13:11:55.246115+0000","last_became_active":"2024-03-21T13:11:18.670395+0000","last_became_peered":"2024-03-21T13:11:18.670395+0000","last_unstale":"2024-03-21T13:11:55.246115+0000","last_undegraded":"2024-03-21T13:11:55.246115+0000","last_fullsized":"2024-03-21T13:11:55.246115+0000","mapping_epoch":299,"log_start":"278'5292","ondisk_log_start":"278'5292","created":246,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":5,"last_scrub":"305'7838","last_scrub_stamp":"2024-03-21T13:11:26.945885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:11:26.945885+0000","objects_scrubbed":1320,"log_size":3834,"log_dups_size":0,"ondisk_log_size":3834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:29.677998+0000","scrub_duration":457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4448,"num_write_kb":46272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1958,"num_bytes_recovered":128319488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":435,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.384368+0000","last_change":"2024-03-21T13:10:03.587701+0000","last_active":"2024-03-21T13:11:51.384368+0000","last_peered":"2024-03-21T13:11:51.384368+0000","last_clean":"2024-03-21T13:11:51.384368+0000","last_became_active":"2024-03-21T13:10:03.586620+0000","last_became_peered":"2024-03-21T13:10:03.586620+0000","last_unstale":"2024-03-21T13:11:51.384368+0000","last_undegraded":"2024-03-21T13:11:51.384368+0000","last_fullsized":"2024-03-21T13:11:51.384368+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:01.881154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:09:01.881154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":453,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.391944+0000","last_change":"2024-03-21T13:11:13.977186+0000","last_active":"2024-03-21T13:11:51.391944+0000","last_peered":"2024-03-21T13:11:51.391944+0000","last_clean":"2024-03-21T13:11:51.391944+0000","last_became_active":"2024-03-21T13:10:03.599880+0000","last_became_peered":"2024-03-21T13:10:03.599880+0000","last_unstale":"2024-03-21T13:11:51.391944+0000","last_undegraded":"2024-03-21T13:11:51.391944+0000","last_fullsized":"2024-03-21T13:11:51.391944+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:13.977069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:13.977069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:41.212846+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":559,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392947+0000","last_change":"2024-03-21T13:10:03.610999+0000","last_active":"2024-03-21T13:11:51.392947+0000","last_peered":"2024-03-21T13:11:51.392947+0000","last_clean":"2024-03-21T13:11:51.392947+0000","last_became_active":"2024-03-21T13:10:03.610489+0000","last_became_peered":"2024-03-21T13:10:03.610489+0000","last_unstale":"2024-03-21T13:11:51.392947+0000","last_undegraded":"2024-03-21T13:11:51.392947+0000","last_fullsized":"2024-03-21T13:11:51.392947+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:58.820007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:58.820007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.999425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"234'19","reported_seq":605,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386631+0000","last_change":"2024-03-21T13:05:49.982963+0000","last_active":"2024-03-21T13:11:51.386631+0000","last_peered":"2024-03-21T13:11:51.386631+0000","last_clean":"2024-03-21T13:11:51.386631+0000","last_became_active":"2024-03-21T13:02:26.254436+0000","last_became_peered":"2024-03-21T13:02:26.254436+0000","last_unstale":"2024-03-21T13:11:51.386631+0000","last_undegraded":"2024-03-21T13:11:51.386631+0000","last_fullsized":"2024-03-21T13:11:51.386631+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:49.982812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.982812+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.407028+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":539,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.394345+0000","last_change":"2024-03-21T13:10:03.598837+0000","last_active":"2024-03-21T13:11:51.394345+0000","last_peered":"2024-03-21T13:11:51.394345+0000","last_clean":"2024-03-21T13:11:51.394345+0000","last_became_active":"2024-03-21T13:10:03.598068+0000","last_became_peered":"2024-03-21T13:10:03.598068+0000","last_unstale":"2024-03-21T13:11:51.394345+0000","last_undegraded":"2024-03-21T13:11:51.394345+0000","last_fullsized":"2024-03-21T13:11:51.394345+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:29.980750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:06:29.980750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":541,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.385861+0000","last_change":"2024-03-21T13:09:10.565138+0000","last_active":"2024-03-21T13:11:51.385861+0000","last_peered":"2024-03-21T13:11:51.385861+0000","last_clean":"2024-03-21T13:11:51.385861+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:11:51.385861+0000","last_undegraded":"2024-03-21T13:11:51.385861+0000","last_fullsized":"2024-03-21T13:11:51.385861+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:10.565030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.565030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.407643+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":407,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.396648+0000","last_change":"2024-03-21T13:05:28.499322+0000","last_active":"2024-03-21T13:11:51.396648+0000","last_peered":"2024-03-21T13:11:51.396648+0000","last_clean":"2024-03-21T13:11:51.396648+0000","last_became_active":"2024-03-21T13:02:26.242973+0000","last_became_peered":"2024-03-21T13:02:26.242973+0000","last_unstale":"2024-03-21T13:11:51.396648+0000","last_undegraded":"2024-03-21T13:11:51.396648+0000","last_fullsized":"2024-03-21T13:11:51.396648+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.499185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.499185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.997901+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":553,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.393042+0000","last_change":"2024-03-21T13:08:18.740907+0000","last_active":"2024-03-21T13:11:51.393042+0000","last_peered":"2024-03-21T13:11:51.393042+0000","last_clean":"2024-03-21T13:11:51.393042+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:11:51.393042+0000","last_undegraded":"2024-03-21T13:11:51.393042+0000","last_fullsized":"2024-03-21T13:11:51.393042+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-21T13:08:18.740815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.740815+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":529,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390181+0000","last_change":"2024-03-21T13:11:33.943308+0000","last_active":"2024-03-21T13:11:51.390181+0000","last_peered":"2024-03-21T13:11:51.390181+0000","last_clean":"2024-03-21T13:11:51.390181+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:11:51.390181+0000","last_undegraded":"2024-03-21T13:11:51.390181+0000","last_fullsized":"2024-03-21T13:11:51.390181+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:33.943015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:33.943015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:42.054652+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":187,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386204+0000","last_change":"2024-03-21T13:08:20.955637+0000","last_active":"2024-03-21T13:11:51.386204+0000","last_peered":"2024-03-21T13:11:51.386204+0000","last_clean":"2024-03-21T13:11:51.386204+0000","last_became_active":"2024-03-21T13:08:20.955293+0000","last_became_peered":"2024-03-21T13:08:20.955293+0000","last_unstale":"2024-03-21T13:11:51.386204+0000","last_undegraded":"2024-03-21T13:11:51.386204+0000","last_fullsized":"2024-03-21T13:11:51.386204+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:03.434682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:03.434682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.407061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":541,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.391769+0000","last_change":"2024-03-21T13:10:03.599838+0000","last_active":"2024-03-21T13:11:51.391769+0000","last_peered":"2024-03-21T13:11:51.391769+0000","last_clean":"2024-03-21T13:11:51.391769+0000","last_became_active":"2024-03-21T13:10:03.599520+0000","last_became_peered":"2024-03-21T13:10:03.599520+0000","last_unstale":"2024-03-21T13:11:51.391769+0000","last_undegraded":"2024-03-21T13:11:51.391769+0000","last_fullsized":"2024-03-21T13:11:51.391769+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:50.428251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:50.428251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":275,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.396998+0000","last_change":"2024-03-21T13:06:29.532046+0000","last_active":"2024-03-21T13:11:51.396998+0000","last_peered":"2024-03-21T13:11:51.396998+0000","last_clean":"2024-03-21T13:11:51.396998+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:11:51.396998+0000","last_undegraded":"2024-03-21T13:11:51.396998+0000","last_fullsized":"2024-03-21T13:11:51.396998+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:40.262277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:40.262277+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998137+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":437,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.385435+0000","last_change":"2024-03-21T13:08:20.069216+0000","last_active":"2024-03-21T13:11:51.385435+0000","last_peered":"2024-03-21T13:11:51.385435+0000","last_clean":"2024-03-21T13:11:51.385435+0000","last_became_active":"2024-03-21T13:02:26.252488+0000","last_became_peered":"2024-03-21T13:02:26.252488+0000","last_unstale":"2024-03-21T13:11:51.385435+0000","last_undegraded":"2024-03-21T13:11:51.385435+0000","last_fullsized":"2024-03-21T13:11:51.385435+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:20.069134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:20.069134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.407113+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":537,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.389930+0000","last_change":"2024-03-21T13:05:31.496412+0000","last_active":"2024-03-21T13:11:51.389930+0000","last_peered":"2024-03-21T13:11:51.389930+0000","last_clean":"2024-03-21T13:11:51.389930+0000","last_became_active":"2024-03-21T13:02:26.232682+0000","last_became_peered":"2024-03-21T13:02:26.232682+0000","last_unstale":"2024-03-21T13:11:51.389930+0000","last_undegraded":"2024-03-21T13:11:51.389930+0000","last_fullsized":"2024-03-21T13:11:51.389930+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.495639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.495639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998107+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":539,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.391894+0000","last_change":"2024-03-21T13:07:48.347291+0000","last_active":"2024-03-21T13:11:51.391894+0000","last_peered":"2024-03-21T13:11:51.391894+0000","last_clean":"2024-03-21T13:11:51.391894+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:11:51.391894+0000","last_undegraded":"2024-03-21T13:11:51.391894+0000","last_fullsized":"2024-03-21T13:11:51.391894+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:48.347175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:48.347175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":546,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.387649+0000","last_change":"2024-03-21T13:10:03.615159+0000","last_active":"2024-03-21T13:11:51.387649+0000","last_peered":"2024-03-21T13:11:51.387649+0000","last_clean":"2024-03-21T13:11:51.387649+0000","last_became_active":"2024-03-21T13:10:03.591857+0000","last_became_peered":"2024-03-21T13:10:03.591857+0000","last_unstale":"2024-03-21T13:11:51.387649+0000","last_undegraded":"2024-03-21T13:11:51.387649+0000","last_fullsized":"2024-03-21T13:11:51.387649+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:09:57.596217+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:09:57.596217+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.407771+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":374,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.388765+0000","last_change":"2024-03-21T13:11:34.090139+0000","last_active":"2024-03-21T13:11:51.388765+0000","last_peered":"2024-03-21T13:11:51.388765+0000","last_clean":"2024-03-21T13:11:51.388765+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:11:51.388765+0000","last_undegraded":"2024-03-21T13:11:51.388765+0000","last_fullsized":"2024-03-21T13:11:51.388765+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:34.087671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:34.087671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:44.240480+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":408,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.393496+0000","last_change":"2024-03-21T13:08:05.697231+0000","last_active":"2024-03-21T13:11:51.393496+0000","last_peered":"2024-03-21T13:11:51.393496+0000","last_clean":"2024-03-21T13:11:51.393496+0000","last_became_active":"2024-03-21T13:02:26.242491+0000","last_became_peered":"2024-03-21T13:02:26.242491+0000","last_unstale":"2024-03-21T13:11:51.393496+0000","last_undegraded":"2024-03-21T13:11:51.393496+0000","last_fullsized":"2024-03-21T13:11:51.393496+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:05.697141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:05.697141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":252,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390227+0000","last_change":"2024-03-21T13:08:18.389543+0000","last_active":"2024-03-21T13:11:51.390227+0000","last_peered":"2024-03-21T13:11:51.390227+0000","last_clean":"2024-03-21T13:11:51.390227+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:11:51.390227+0000","last_undegraded":"2024-03-21T13:11:51.390227+0000","last_fullsized":"2024-03-21T13:11:51.390227+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:18.389423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.389423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":656,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.381397+0000","last_change":"2024-03-21T13:11:29.734394+0000","last_active":"2024-03-21T13:11:51.381397+0000","last_peered":"2024-03-21T13:11:51.381397+0000","last_clean":"2024-03-21T13:11:51.381397+0000","last_became_active":"2024-03-21T13:10:03.594316+0000","last_became_peered":"2024-03-21T13:10:03.594316+0000","last_unstale":"2024-03-21T13:11:51.381397+0000","last_undegraded":"2024-03-21T13:11:51.381397+0000","last_fullsized":"2024-03-21T13:11:51.381397+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:11:29.734236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:29.734236+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:40.247988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.19","version":"315'9008","reported_seq":14318,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.761510+0000","last_change":"2024-03-21T13:09:53.527456+0000","last_active":"2024-03-21T13:11:55.761510+0000","last_peered":"2024-03-21T13:11:55.761510+0000","last_clean":"2024-03-21T13:11:55.761510+0000","last_became_active":"2024-03-21T13:09:44.342930+0000","last_became_peered":"2024-03-21T13:09:44.342930+0000","last_unstale":"2024-03-21T13:11:55.761510+0000","last_undegraded":"2024-03-21T13:11:55.761510+0000","last_fullsized":"2024-03-21T13:11:55.761510+0000","mapping_epoch":260,"log_start":"282'5582","ondisk_log_start":"282'5582","created":246,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":5,"last_scrub":"264'4378","last_scrub_stamp":"2024-03-21T13:09:53.527177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:09:53.527177+0000","objects_scrubbed":2766,"log_size":3426,"log_dups_size":0,"ondisk_log_size":3426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":690,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4642,"num_write_kb":60032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":440,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.391681+0000","last_change":"2024-03-21T13:08:56.597251+0000","last_active":"2024-03-21T13:11:51.391681+0000","last_peered":"2024-03-21T13:11:51.391681+0000","last_clean":"2024-03-21T13:11:51.391681+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:11:51.391681+0000","last_undegraded":"2024-03-21T13:11:51.391681+0000","last_fullsized":"2024-03-21T13:11:51.391681+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:56.597146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:56.597146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"125'10","reported_seq":702,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392519+0000","last_change":"2024-03-21T13:08:15.777606+0000","last_active":"2024-03-21T13:11:51.392519+0000","last_peered":"2024-03-21T13:11:51.392519+0000","last_clean":"2024-03-21T13:11:51.392519+0000","last_became_active":"2024-03-21T13:02:26.234494+0000","last_became_peered":"2024-03-21T13:02:26.234494+0000","last_unstale":"2024-03-21T13:11:51.392519+0000","last_undegraded":"2024-03-21T13:11:51.392519+0000","last_fullsized":"2024-03-21T13:11:51.392519+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:08:15.777522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:15.777522+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":551,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.389363+0000","last_change":"2024-03-21T13:10:03.599799+0000","last_active":"2024-03-21T13:11:51.389363+0000","last_peered":"2024-03-21T13:11:51.389363+0000","last_clean":"2024-03-21T13:11:51.389363+0000","last_became_active":"2024-03-21T13:10:03.599265+0000","last_became_peered":"2024-03-21T13:10:03.599265+0000","last_unstale":"2024-03-21T13:11:51.389363+0000","last_undegraded":"2024-03-21T13:11:51.389363+0000","last_fullsized":"2024-03-21T13:11:51.389363+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.042372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.042372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'20","reported_seq":613,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.393058+0000","last_change":"2024-03-21T13:10:03.707937+0000","last_active":"2024-03-21T13:11:51.393058+0000","last_peered":"2024-03-21T13:11:51.393058+0000","last_clean":"2024-03-21T13:11:51.393058+0000","last_became_active":"2024-03-21T13:10:03.613119+0000","last_became_peered":"2024-03-21T13:10:03.613119+0000","last_unstale":"2024-03-21T13:11:51.393058+0000","last_undegraded":"2024-03-21T13:11:51.393058+0000","last_fullsized":"2024-03-21T13:11:51.393058+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:23.379004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.379004+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998048+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"315'9069","reported_seq":15671,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.008093+0000","last_change":"2024-03-21T13:11:04.001810+0000","last_active":"2024-03-21T13:11:55.008093+0000","last_peered":"2024-03-21T13:11:55.008093+0000","last_clean":"2024-03-21T13:11:55.008093+0000","last_became_active":"2024-03-21T13:10:52.378423+0000","last_became_peered":"2024-03-21T13:10:52.378423+0000","last_unstale":"2024-03-21T13:11:55.008093+0000","last_undegraded":"2024-03-21T13:11:55.008093+0000","last_fullsized":"2024-03-21T13:11:55.008093+0000","mapping_epoch":288,"log_start":"277'5150","ondisk_log_start":"277'5150","created":246,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":5,"last_scrub":"292'6887","last_scrub_stamp":"2024-03-21T13:11:04.001678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:11:04.001678+0000","objects_scrubbed":2230,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:31.655294+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2184,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":97,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.388928+0000","last_change":"2024-03-21T13:10:03.604060+0000","last_active":"2024-03-21T13:11:51.388928+0000","last_peered":"2024-03-21T13:11:51.388928+0000","last_clean":"2024-03-21T13:11:51.388928+0000","last_became_active":"2024-03-21T13:10:03.603547+0000","last_became_peered":"2024-03-21T13:10:03.603547+0000","last_unstale":"2024-03-21T13:11:51.388928+0000","last_undegraded":"2024-03-21T13:11:51.388928+0000","last_fullsized":"2024-03-21T13:11:51.388928+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:21.157295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.157295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"272'10","reported_seq":714,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.393674+0000","last_change":"2024-03-21T13:10:03.624386+0000","last_active":"2024-03-21T13:11:51.393674+0000","last_peered":"2024-03-21T13:11:51.393674+0000","last_clean":"2024-03-21T13:11:51.393674+0000","last_became_active":"2024-03-21T13:10:03.594651+0000","last_became_peered":"2024-03-21T13:10:03.594651+0000","last_unstale":"2024-03-21T13:11:51.393674+0000","last_undegraded":"2024-03-21T13:11:51.393674+0000","last_fullsized":"2024-03-21T13:11:51.393674+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-03-21T13:07:52.673289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:07:52.673289+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":440,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390557+0000","last_change":"2024-03-21T13:08:21.365260+0000","last_active":"2024-03-21T13:11:51.390557+0000","last_peered":"2024-03-21T13:11:51.390557+0000","last_clean":"2024-03-21T13:11:51.390557+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:11:51.390557+0000","last_undegraded":"2024-03-21T13:11:51.390557+0000","last_fullsized":"2024-03-21T13:11:51.390557+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:08:21.365173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"234'15","reported_seq":467,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392175+0000","last_change":"2024-03-21T13:08:08.413361+0000","last_active":"2024-03-21T13:11:51.392175+0000","last_peered":"2024-03-21T13:11:51.392175+0000","last_clean":"2024-03-21T13:11:51.392175+0000","last_became_active":"2024-03-21T13:02:26.239161+0000","last_became_peered":"2024-03-21T13:02:26.239161+0000","last_unstale":"2024-03-21T13:11:51.392175+0000","last_undegraded":"2024-03-21T13:11:51.392175+0000","last_fullsized":"2024-03-21T13:11:51.392175+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:08:08.413234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:08.413234+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":97,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386338+0000","last_change":"2024-03-21T13:10:03.614199+0000","last_active":"2024-03-21T13:11:51.386338+0000","last_peered":"2024-03-21T13:11:51.386338+0000","last_clean":"2024-03-21T13:11:51.386338+0000","last_became_active":"2024-03-21T13:10:03.613841+0000","last_became_peered":"2024-03-21T13:10:03.613841+0000","last_unstale":"2024-03-21T13:11:51.386338+0000","last_undegraded":"2024-03-21T13:11:51.386338+0000","last_fullsized":"2024-03-21T13:11:51.386338+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:58.785444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:08:58.785444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":539,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.397092+0000","last_change":"2024-03-21T13:11:26.697833+0000","last_active":"2024-03-21T13:11:51.397092+0000","last_peered":"2024-03-21T13:11:51.397092+0000","last_clean":"2024-03-21T13:11:51.397092+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:11:51.397092+0000","last_undegraded":"2024-03-21T13:11:51.397092+0000","last_fullsized":"2024-03-21T13:11:51.397092+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:26.697728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:26.697728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:29.793406+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":553,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390099+0000","last_change":"2024-03-21T13:09:13.694824+0000","last_active":"2024-03-21T13:11:51.390099+0000","last_peered":"2024-03-21T13:11:51.390099+0000","last_clean":"2024-03-21T13:11:51.390099+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:11:51.390099+0000","last_undegraded":"2024-03-21T13:11:51.390099+0000","last_fullsized":"2024-03-21T13:11:51.390099+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:09:13.694702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:13.694702+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'15","reported_seq":476,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390049+0000","last_change":"2024-03-21T13:09:10.698886+0000","last_active":"2024-03-21T13:11:51.390049+0000","last_peered":"2024-03-21T13:11:51.390049+0000","last_clean":"2024-03-21T13:11:51.390049+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:11:51.390049+0000","last_undegraded":"2024-03-21T13:11:51.390049+0000","last_fullsized":"2024-03-21T13:11:51.390049+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:09:10.698706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.698706+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998814+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":527,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.391065+0000","last_change":"2024-03-21T13:07:52.425295+0000","last_active":"2024-03-21T13:11:51.391065+0000","last_peered":"2024-03-21T13:11:51.391065+0000","last_clean":"2024-03-21T13:11:51.391065+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:11:51.391065+0000","last_undegraded":"2024-03-21T13:11:51.391065+0000","last_fullsized":"2024-03-21T13:11:51.391065+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:52.424928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:52.424928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":521,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390290+0000","last_change":"2024-03-21T13:05:25.466812+0000","last_active":"2024-03-21T13:11:51.390290+0000","last_peered":"2024-03-21T13:11:51.390290+0000","last_clean":"2024-03-21T13:11:51.390290+0000","last_became_active":"2024-03-21T13:02:26.242502+0000","last_became_peered":"2024-03-21T13:02:26.242502+0000","last_unstale":"2024-03-21T13:11:51.390290+0000","last_undegraded":"2024-03-21T13:11:51.390290+0000","last_fullsized":"2024-03-21T13:11:51.390290+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:25.466695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:25.466695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.997986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":555,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386269+0000","last_change":"2024-03-21T13:11:34.524448+0000","last_active":"2024-03-21T13:11:51.386269+0000","last_peered":"2024-03-21T13:11:51.386269+0000","last_clean":"2024-03-21T13:11:51.386269+0000","last_became_active":"2024-03-21T13:02:26.252406+0000","last_became_peered":"2024-03-21T13:02:26.252406+0000","last_unstale":"2024-03-21T13:11:51.386269+0000","last_undegraded":"2024-03-21T13:11:51.386269+0000","last_fullsized":"2024-03-21T13:11:51.386269+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:34.524370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:34.524370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:13:00.539197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":597,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.396901+0000","last_change":"2024-03-21T13:10:03.647440+0000","last_active":"2024-03-21T13:11:51.396901+0000","last_peered":"2024-03-21T13:11:51.396901+0000","last_clean":"2024-03-21T13:11:51.396901+0000","last_became_active":"2024-03-21T13:10:03.610810+0000","last_became_peered":"2024-03-21T13:10:03.610810+0000","last_unstale":"2024-03-21T13:11:51.396901+0000","last_undegraded":"2024-03-21T13:11:51.396901+0000","last_fullsized":"2024-03-21T13:11:51.396901+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-21T13:09:12.712840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:12.712840+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998723+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"315'9327","reported_seq":9549,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.153955+0000","last_change":"2024-03-21T13:10:02.794810+0000","last_active":"2024-03-21T13:11:55.153955+0000","last_peered":"2024-03-21T13:11:55.153955+0000","last_clean":"2024-03-21T13:11:55.153955+0000","last_became_active":"2024-03-21T13:09:18.453217+0000","last_became_peered":"2024-03-21T13:09:18.453217+0000","last_unstale":"2024-03-21T13:11:55.153955+0000","last_undegraded":"2024-03-21T13:11:55.153955+0000","last_fullsized":"2024-03-21T13:11:55.153955+0000","mapping_epoch":234,"log_start":"282'5934","ondisk_log_start":"282'5934","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'4846","last_scrub_stamp":"2024-03-21T13:10:02.794680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:10:02.794680+0000","objects_scrubbed":3147,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:57.137469+0000","scrub_duration":1138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4494,"num_write_kb":44032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":426,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386918+0000","last_change":"2024-03-21T13:11:02.758245+0000","last_active":"2024-03-21T13:11:51.386918+0000","last_peered":"2024-03-21T13:11:51.386918+0000","last_clean":"2024-03-21T13:11:51.386918+0000","last_became_active":"2024-03-21T13:10:03.591230+0000","last_became_peered":"2024-03-21T13:10:03.591230+0000","last_unstale":"2024-03-21T13:11:51.386918+0000","last_undegraded":"2024-03-21T13:11:51.386918+0000","last_fullsized":"2024-03-21T13:11:51.386918+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:02.758017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:11:02.758017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:32.083279+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":583,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.391713+0000","last_change":"2024-03-21T13:11:31.040564+0000","last_active":"2024-03-21T13:11:51.391713+0000","last_peered":"2024-03-21T13:11:51.391713+0000","last_clean":"2024-03-21T13:11:51.391713+0000","last_became_active":"2024-03-21T13:06:29.487439+0000","last_became_peered":"2024-03-21T13:06:29.487439+0000","last_unstale":"2024-03-21T13:11:51.391713+0000","last_undegraded":"2024-03-21T13:11:51.391713+0000","last_fullsized":"2024-03-21T13:11:51.391713+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:11:31.040414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:31.040414+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:33.097785+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":492,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.389250+0000","last_change":"2024-03-21T13:08:10.436112+0000","last_active":"2024-03-21T13:11:51.389250+0000","last_peered":"2024-03-21T13:11:51.389250+0000","last_clean":"2024-03-21T13:11:51.389250+0000","last_became_active":"2024-03-21T13:02:26.237800+0000","last_became_peered":"2024-03-21T13:02:26.237800+0000","last_unstale":"2024-03-21T13:11:51.389250+0000","last_undegraded":"2024-03-21T13:11:51.389250+0000","last_fullsized":"2024-03-21T13:11:51.389250+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:08:10.435742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:10.435742+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":704,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390503+0000","last_change":"2024-03-21T13:08:53.619838+0000","last_active":"2024-03-21T13:11:51.390503+0000","last_peered":"2024-03-21T13:11:51.390503+0000","last_clean":"2024-03-21T13:11:51.390503+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:11:51.390503+0000","last_undegraded":"2024-03-21T13:11:51.390503+0000","last_fullsized":"2024-03-21T13:11:51.390503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:08:53.619709+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:08:53.619709+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:57.134610+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"315'9210","reported_seq":9427,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.766945+0000","last_change":"2024-03-21T13:09:52.378258+0000","last_active":"2024-03-21T13:11:55.766945+0000","last_peered":"2024-03-21T13:11:55.766945+0000","last_clean":"2024-03-21T13:11:55.766945+0000","last_became_active":"2024-03-21T13:09:18.434636+0000","last_became_peered":"2024-03-21T13:09:18.434636+0000","last_unstale":"2024-03-21T13:11:55.766945+0000","last_undegraded":"2024-03-21T13:11:55.766945+0000","last_fullsized":"2024-03-21T13:11:55.766945+0000","mapping_epoch":234,"log_start":"282'5712","ondisk_log_start":"282'5712","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'4350","last_scrub_stamp":"2024-03-21T13:09:52.377808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:09:52.377808+0000","objects_scrubbed":2780,"log_size":3498,"log_dups_size":0,"ondisk_log_size":3498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4874,"num_write_kb":67008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":527,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390098+0000","last_change":"2024-03-21T13:05:43.596914+0000","last_active":"2024-03-21T13:11:51.390098+0000","last_peered":"2024-03-21T13:11:51.390098+0000","last_clean":"2024-03-21T13:11:51.390098+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:11:51.390098+0000","last_undegraded":"2024-03-21T13:11:51.390098+0000","last_fullsized":"2024-03-21T13:11:51.390098+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.596691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998218+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":553,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.391853+0000","last_change":"2024-03-21T13:09:14.724844+0000","last_active":"2024-03-21T13:11:51.391853+0000","last_peered":"2024-03-21T13:11:51.391853+0000","last_clean":"2024-03-21T13:11:51.391853+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:11:51.391853+0000","last_undegraded":"2024-03-21T13:11:51.391853+0000","last_fullsized":"2024-03-21T13:11:51.391853+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:14.724681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:14.724681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"234'18","reported_seq":481,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.396904+0000","last_change":"2024-03-21T13:07:48.778302+0000","last_active":"2024-03-21T13:11:51.396904+0000","last_peered":"2024-03-21T13:11:51.396904+0000","last_clean":"2024-03-21T13:11:51.396904+0000","last_became_active":"2024-03-21T13:02:14.347102+0000","last_became_peered":"2024-03-21T13:02:14.347102+0000","last_unstale":"2024-03-21T13:11:51.396904+0000","last_undegraded":"2024-03-21T13:11:51.396904+0000","last_fullsized":"2024-03-21T13:11:51.396904+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:07:48.778145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:07:48.778145+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998247+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":97,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.384306+0000","last_change":"2024-03-21T13:10:03.605416+0000","last_active":"2024-03-21T13:11:51.384306+0000","last_peered":"2024-03-21T13:11:51.384306+0000","last_clean":"2024-03-21T13:11:51.384306+0000","last_became_active":"2024-03-21T13:10:03.604015+0000","last_became_peered":"2024-03-21T13:10:03.604015+0000","last_unstale":"2024-03-21T13:11:51.384306+0000","last_undegraded":"2024-03-21T13:11:51.384306+0000","last_fullsized":"2024-03-21T13:11:51.384306+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:10.712191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.712191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"315'9198","reported_seq":14815,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.159660+0000","last_change":"2024-03-21T13:11:04.468808+0000","last_active":"2024-03-21T13:11:55.159660+0000","last_peered":"2024-03-21T13:11:55.159660+0000","last_clean":"2024-03-21T13:11:55.159660+0000","last_became_active":"2024-03-21T13:11:04.459334+0000","last_became_peered":"2024-03-21T13:11:04.459334+0000","last_unstale":"2024-03-21T13:11:55.159660+0000","last_undegraded":"2024-03-21T13:11:55.159660+0000","last_fullsized":"2024-03-21T13:11:55.159660+0000","mapping_epoch":292,"log_start":"282'5788","ondisk_log_start":"282'5788","created":234,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'4749","last_scrub_stamp":"2024-03-21T13:10:02.069834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:10:02.069834+0000","objects_scrubbed":3119,"log_size":3410,"log_dups_size":0,"ondisk_log_size":3410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:57.137385+0000","scrub_duration":1447,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4460,"num_write_kb":43840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2691,"num_bytes_recovered":176357376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":548,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.388710+0000","last_change":"2024-03-21T13:10:03.598806+0000","last_active":"2024-03-21T13:11:51.388710+0000","last_peered":"2024-03-21T13:11:51.388710+0000","last_clean":"2024-03-21T13:11:51.388710+0000","last_became_active":"2024-03-21T13:10:03.598366+0000","last_became_peered":"2024-03-21T13:10:03.598366+0000","last_unstale":"2024-03-21T13:11:51.388710+0000","last_undegraded":"2024-03-21T13:11:51.388710+0000","last_fullsized":"2024-03-21T13:11:51.388710+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:08:14.371718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":105,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.388710+0000","last_change":"2024-03-21T13:11:33.743407+0000","last_active":"2024-03-21T13:11:51.388710+0000","last_peered":"2024-03-21T13:11:51.388710+0000","last_clean":"2024-03-21T13:11:51.388710+0000","last_became_active":"2024-03-21T13:10:03.615067+0000","last_became_peered":"2024-03-21T13:10:03.615067+0000","last_unstale":"2024-03-21T13:11:51.388710+0000","last_undegraded":"2024-03-21T13:11:51.388710+0000","last_fullsized":"2024-03-21T13:11:51.388710+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:33.743269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:33.743269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:45.180207+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":553,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392092+0000","last_change":"2024-03-21T13:11:17.019489+0000","last_active":"2024-03-21T13:11:51.392092+0000","last_peered":"2024-03-21T13:11:51.392092+0000","last_clean":"2024-03-21T13:11:51.392092+0000","last_became_active":"2024-03-21T13:02:26.236780+0000","last_became_peered":"2024-03-21T13:02:26.236780+0000","last_unstale":"2024-03-21T13:11:51.392092+0000","last_undegraded":"2024-03-21T13:11:51.392092+0000","last_fullsized":"2024-03-21T13:11:51.392092+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.019382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.019382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:18.671834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'35","reported_seq":653,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.387276+0000","last_change":"2024-03-21T13:10:02.601328+0000","last_active":"2024-03-21T13:11:51.387276+0000","last_peered":"2024-03-21T13:11:51.387276+0000","last_clean":"2024-03-21T13:11:51.387276+0000","last_became_active":"2024-03-21T13:02:26.240350+0000","last_became_peered":"2024-03-21T13:02:26.240350+0000","last_unstale":"2024-03-21T13:11:51.387276+0000","last_undegraded":"2024-03-21T13:11:51.387276+0000","last_fullsized":"2024-03-21T13:11:51.387276+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:10:02.601053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:10:02.601053+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.407805+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"315'8880","reported_seq":15127,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.120323+0000","last_change":"2024-03-21T13:10:50.516141+0000","last_active":"2024-03-21T13:11:55.120323+0000","last_peered":"2024-03-21T13:11:55.120323+0000","last_clean":"2024-03-21T13:11:55.120323+0000","last_became_active":"2024-03-21T13:10:50.503839+0000","last_became_peered":"2024-03-21T13:10:50.503839+0000","last_unstale":"2024-03-21T13:11:55.120323+0000","last_undegraded":"2024-03-21T13:11:55.120323+0000","last_fullsized":"2024-03-21T13:11:55.120323+0000","mapping_epoch":286,"log_start":"264'4298","ondisk_log_start":"264'4298","created":234,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'4300","last_scrub_stamp":"2024-03-21T13:09:53.341015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:09:53.341015+0000","objects_scrubbed":2679,"log_size":4582,"log_dups_size":0,"ondisk_log_size":4582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":764,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1966080,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":62208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3011,"num_bytes_recovered":197328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":452,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.389050+0000","last_change":"2024-03-21T13:11:38.166199+0000","last_active":"2024-03-21T13:11:51.389050+0000","last_peered":"2024-03-21T13:11:51.389050+0000","last_clean":"2024-03-21T13:11:51.389050+0000","last_became_active":"2024-03-21T13:10:03.606655+0000","last_became_peered":"2024-03-21T13:10:03.606655+0000","last_unstale":"2024-03-21T13:11:51.389050+0000","last_undegraded":"2024-03-21T13:11:51.389050+0000","last_fullsized":"2024-03-21T13:11:51.389050+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:38.166077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:38.166077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:47.674560+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":97,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.384612+0000","last_change":"2024-03-21T13:10:03.603979+0000","last_active":"2024-03-21T13:11:51.384612+0000","last_peered":"2024-03-21T13:11:51.384612+0000","last_clean":"2024-03-21T13:11:51.384612+0000","last_became_active":"2024-03-21T13:10:03.603646+0000","last_became_peered":"2024-03-21T13:10:03.603646+0000","last_unstale":"2024-03-21T13:11:51.384612+0000","last_undegraded":"2024-03-21T13:11:51.384612+0000","last_fullsized":"2024-03-21T13:11:51.384612+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:12.031099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:12.031099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":406,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392900+0000","last_change":"2024-03-21T13:08:57.570445+0000","last_active":"2024-03-21T13:11:51.392900+0000","last_peered":"2024-03-21T13:11:51.392900+0000","last_clean":"2024-03-21T13:11:51.392900+0000","last_became_active":"2024-03-21T13:02:26.243974+0000","last_became_peered":"2024-03-21T13:02:26.243974+0000","last_unstale":"2024-03-21T13:11:51.392900+0000","last_undegraded":"2024-03-21T13:11:51.392900+0000","last_fullsized":"2024-03-21T13:11:51.392900+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:57.570303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.570303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998635+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"234'19","reported_seq":521,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.381159+0000","last_change":"2024-03-21T13:11:09.615578+0000","last_active":"2024-03-21T13:11:51.381159+0000","last_peered":"2024-03-21T13:11:51.381159+0000","last_clean":"2024-03-21T13:11:51.381159+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:11:51.381159+0000","last_undegraded":"2024-03-21T13:11:51.381159+0000","last_fullsized":"2024-03-21T13:11:51.381159+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:11:09.615453+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:11:09.615453+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:14.465735+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"315'9054","reported_seq":9274,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.161718+0000","last_change":"2024-03-21T13:10:00.705745+0000","last_active":"2024-03-21T13:11:55.161718+0000","last_peered":"2024-03-21T13:11:55.161718+0000","last_clean":"2024-03-21T13:11:55.161718+0000","last_became_active":"2024-03-21T13:09:18.465669+0000","last_became_peered":"2024-03-21T13:09:18.465669+0000","last_unstale":"2024-03-21T13:11:55.161718+0000","last_undegraded":"2024-03-21T13:11:55.161718+0000","last_fullsized":"2024-03-21T13:11:55.161718+0000","mapping_epoch":234,"log_start":"282'5666","ondisk_log_start":"282'5666","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'4643","last_scrub_stamp":"2024-03-21T13:10:00.703828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:10:00.703828+0000","objects_scrubbed":2988,"log_size":3388,"log_dups_size":0,"ondisk_log_size":3388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:57.134902+0000","scrub_duration":1091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2293760,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4427,"num_write_kb":47168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":522,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390383+0000","last_change":"2024-03-21T13:09:11.661260+0000","last_active":"2024-03-21T13:11:51.390383+0000","last_peered":"2024-03-21T13:11:51.390383+0000","last_clean":"2024-03-21T13:11:51.390383+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:11:51.390383+0000","last_undegraded":"2024-03-21T13:11:51.390383+0000","last_fullsized":"2024-03-21T13:11:51.390383+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:11.661130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:09:11.661130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998785+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":437,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.387166+0000","last_change":"2024-03-21T13:05:43.961181+0000","last_active":"2024-03-21T13:11:51.387166+0000","last_peered":"2024-03-21T13:11:51.387166+0000","last_clean":"2024-03-21T13:11:51.387166+0000","last_became_active":"2024-03-21T13:02:26.243692+0000","last_became_peered":"2024-03-21T13:02:26.243692+0000","last_unstale":"2024-03-21T13:11:51.387166+0000","last_undegraded":"2024-03-21T13:11:51.387166+0000","last_fullsized":"2024-03-21T13:11:51.387166+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.961057+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.406971+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":252,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.387434+0000","last_change":"2024-03-21T13:10:00.606614+0000","last_active":"2024-03-21T13:11:51.387434+0000","last_peered":"2024-03-21T13:11:51.387434+0000","last_clean":"2024-03-21T13:11:51.387434+0000","last_became_active":"2024-03-21T13:06:45.410017+0000","last_became_peered":"2024-03-21T13:06:45.410017+0000","last_unstale":"2024-03-21T13:11:51.387434+0000","last_undegraded":"2024-03-21T13:11:51.387434+0000","last_fullsized":"2024-03-21T13:11:51.387434+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:10:00.606508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:10:00.606508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.407865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"272'43","reported_seq":150,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386237+0000","last_change":"2024-03-21T13:10:04.094557+0000","last_active":"2024-03-21T13:11:51.386237+0000","last_peered":"2024-03-21T13:11:51.386237+0000","last_clean":"2024-03-21T13:11:51.386237+0000","last_became_active":"2024-03-21T13:10:03.603889+0000","last_became_peered":"2024-03-21T13:10:03.603889+0000","last_unstale":"2024-03-21T13:11:51.386237+0000","last_undegraded":"2024-03-21T13:11:51.386237+0000","last_fullsized":"2024-03-21T13:11:51.386237+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'43","last_scrub_stamp":"2024-03-21T13:10:04.094455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:10:04.094455+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":256,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386142+0000","last_change":"2024-03-21T13:09:07.643588+0000","last_active":"2024-03-21T13:11:51.386142+0000","last_peered":"2024-03-21T13:11:51.386142+0000","last_clean":"2024-03-21T13:11:51.386142+0000","last_became_active":"2024-03-21T13:06:45.406734+0000","last_became_peered":"2024-03-21T13:06:45.406734+0000","last_unstale":"2024-03-21T13:11:51.386142+0000","last_undegraded":"2024-03-21T13:11:51.386142+0000","last_fullsized":"2024-03-21T13:11:51.386142+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:07.643482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:09:07.643482+0000","last_clean_scrub_stamp":"2024-03-21T13:09:07.643482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.407725+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.a","version":"315'15128","reported_seq":15304,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.199306+0000","last_change":"2024-03-21T13:08:45.620420+0000","last_active":"2024-03-21T13:11:55.199306+0000","last_peered":"2024-03-21T13:11:55.199306+0000","last_clean":"2024-03-21T13:11:55.199306+0000","last_became_active":"2024-03-21T13:08:45.620077+0000","last_became_peered":"2024-03-21T13:08:45.620077+0000","last_unstale":"2024-03-21T13:11:55.199306+0000","last_undegraded":"2024-03-21T13:11:55.199306+0000","last_fullsized":"2024-03-21T13:11:55.199306+0000","mapping_epoch":234,"log_start":"298'11755","ondisk_log_start":"298'11755","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3373,"log_dups_size":2639,"ondisk_log_size":3373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:57.134579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15128,"num_write_kb":486144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":536,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392264+0000","last_change":"2024-03-21T13:08:58.723773+0000","last_active":"2024-03-21T13:11:51.392264+0000","last_peered":"2024-03-21T13:11:51.392264+0000","last_clean":"2024-03-21T13:11:51.392264+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:11:51.392264+0000","last_undegraded":"2024-03-21T13:11:51.392264+0000","last_fullsized":"2024-03-21T13:11:51.392264+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:58.723653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:58.723653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":97,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.384369+0000","last_change":"2024-03-21T13:10:03.604686+0000","last_active":"2024-03-21T13:11:51.384369+0000","last_peered":"2024-03-21T13:11:51.384369+0000","last_clean":"2024-03-21T13:11:51.384369+0000","last_became_active":"2024-03-21T13:10:03.604303+0000","last_became_peered":"2024-03-21T13:10:03.604303+0000","last_unstale":"2024-03-21T13:11:51.384369+0000","last_undegraded":"2024-03-21T13:11:51.384369+0000","last_fullsized":"2024-03-21T13:11:51.384369+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:15.590828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:15.590828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.384045+0000","last_change":"2024-03-21T13:10:03.605166+0000","last_active":"2024-03-21T13:11:51.384045+0000","last_peered":"2024-03-21T13:11:51.384045+0000","last_clean":"2024-03-21T13:11:51.384045+0000","last_became_active":"2024-03-21T13:10:03.604427+0000","last_became_peered":"2024-03-21T13:10:03.604427+0000","last_unstale":"2024-03-21T13:11:51.384045+0000","last_undegraded":"2024-03-21T13:11:51.384045+0000","last_fullsized":"2024-03-21T13:11:51.384045+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:46.263644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:05:46.263644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"243'20","reported_seq":483,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.391459+0000","last_change":"2024-03-21T13:09:10.683920+0000","last_active":"2024-03-21T13:11:51.391459+0000","last_peered":"2024-03-21T13:11:51.391459+0000","last_clean":"2024-03-21T13:11:51.391459+0000","last_became_active":"2024-03-21T13:02:26.234230+0000","last_became_peered":"2024-03-21T13:02:26.234230+0000","last_unstale":"2024-03-21T13:11:51.391459+0000","last_undegraded":"2024-03-21T13:11:51.391459+0000","last_fullsized":"2024-03-21T13:11:51.391459+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'20","last_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_deep_scrub":"243'20","last_deep_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.683382+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":97,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.388319+0000","last_change":"2024-03-21T13:10:03.609881+0000","last_active":"2024-03-21T13:11:51.388319+0000","last_peered":"2024-03-21T13:11:51.388319+0000","last_clean":"2024-03-21T13:11:51.388319+0000","last_became_active":"2024-03-21T13:10:03.609581+0000","last_became_peered":"2024-03-21T13:10:03.609581+0000","last_unstale":"2024-03-21T13:11:51.388319+0000","last_undegraded":"2024-03-21T13:11:51.388319+0000","last_fullsized":"2024-03-21T13:11:51.388319+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:16.750077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:16.750077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"315'9269","reported_seq":9489,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.781042+0000","last_change":"2024-03-21T13:09:49.406385+0000","last_active":"2024-03-21T13:11:55.781042+0000","last_peered":"2024-03-21T13:11:55.781042+0000","last_clean":"2024-03-21T13:11:55.781042+0000","last_became_active":"2024-03-21T13:09:18.458922+0000","last_became_peered":"2024-03-21T13:09:18.458922+0000","last_unstale":"2024-03-21T13:11:55.781042+0000","last_undegraded":"2024-03-21T13:11:55.781042+0000","last_fullsized":"2024-03-21T13:11:55.781042+0000","mapping_epoch":234,"log_start":"282'5831","ondisk_log_start":"282'5831","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'4260","last_scrub_stamp":"2024-03-21T13:09:49.406249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:09:49.406249+0000","objects_scrubbed":2601,"log_size":3438,"log_dups_size":0,"ondisk_log_size":3438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":77440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":414,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.396990+0000","last_change":"2024-03-21T13:09:57.956198+0000","last_active":"2024-03-21T13:11:51.396990+0000","last_peered":"2024-03-21T13:11:51.396990+0000","last_clean":"2024-03-21T13:11:51.396990+0000","last_became_active":"2024-03-21T13:02:26.235640+0000","last_became_peered":"2024-03-21T13:02:26.235640+0000","last_unstale":"2024-03-21T13:11:51.396990+0000","last_undegraded":"2024-03-21T13:11:51.396990+0000","last_fullsized":"2024-03-21T13:11:51.396990+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:57.956080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:09:57.956080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.999447+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":798,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.388974+0000","last_change":"2024-03-21T13:08:07.454234+0000","last_active":"2024-03-21T13:11:51.388974+0000","last_peered":"2024-03-21T13:11:51.388974+0000","last_clean":"2024-03-21T13:11:51.388974+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:11:51.388974+0000","last_undegraded":"2024-03-21T13:11:51.388974+0000","last_fullsized":"2024-03-21T13:11:51.388974+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:08:07.454016+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":37,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.383725+0000","last_change":"2024-03-21T13:11:21.699418+0000","last_active":"2024-03-21T13:11:51.383725+0000","last_peered":"2024-03-21T13:11:51.383725+0000","last_clean":"2024-03-21T13:11:51.383725+0000","last_became_active":"2024-03-21T13:11:21.695629+0000","last_became_peered":"2024-03-21T13:11:21.695629+0000","last_unstale":"2024-03-21T13:11:51.383725+0000","last_undegraded":"2024-03-21T13:11:51.383725+0000","last_fullsized":"2024-03-21T13:11:51.383725+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:49.758485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:49.758485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"234'36","reported_seq":746,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386587+0000","last_change":"2024-03-21T13:05:42.033620+0000","last_active":"2024-03-21T13:11:51.386587+0000","last_peered":"2024-03-21T13:11:51.386587+0000","last_clean":"2024-03-21T13:11:51.386587+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:11:51.386587+0000","last_undegraded":"2024-03-21T13:11:51.386587+0000","last_fullsized":"2024-03-21T13:11:51.386587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'24","last_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.033266+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":558,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.390383+0000","last_change":"2024-03-21T13:11:11.001343+0000","last_active":"2024-03-21T13:11:51.390383+0000","last_peered":"2024-03-21T13:11:51.390383+0000","last_clean":"2024-03-21T13:11:51.390383+0000","last_became_active":"2024-03-21T13:10:03.599048+0000","last_became_peered":"2024-03-21T13:10:03.599048+0000","last_unstale":"2024-03-21T13:11:51.390383+0000","last_undegraded":"2024-03-21T13:11:51.390383+0000","last_fullsized":"2024-03-21T13:11:51.390383+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:11.001239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:11:11.001239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:36.367693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"315'15244","reported_seq":21357,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.035509+0000","last_change":"2024-03-21T13:11:35.370438+0000","last_active":"2024-03-21T13:11:55.035509+0000","last_peered":"2024-03-21T13:11:55.035509+0000","last_clean":"2024-03-21T13:11:55.035509+0000","last_became_active":"2024-03-21T13:11:35.340484+0000","last_became_peered":"2024-03-21T13:11:35.340484+0000","last_unstale":"2024-03-21T13:11:55.035509+0000","last_undegraded":"2024-03-21T13:11:55.035509+0000","last_fullsized":"2024-03-21T13:11:55.035509+0000","mapping_epoch":307,"log_start":"296'11306","ondisk_log_start":"296'11306","created":234,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3938,"log_dups_size":0,"ondisk_log_size":3938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:58.327760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":96,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15244,"num_write_kb":490880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2956,"num_bytes_recovered":193724416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":416,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392528+0000","last_change":"2024-03-21T13:11:36.890901+0000","last_active":"2024-03-21T13:11:51.392528+0000","last_peered":"2024-03-21T13:11:51.392528+0000","last_clean":"2024-03-21T13:11:51.392528+0000","last_became_active":"2024-03-21T13:02:26.242708+0000","last_became_peered":"2024-03-21T13:02:26.242708+0000","last_unstale":"2024-03-21T13:11:51.392528+0000","last_undegraded":"2024-03-21T13:11:51.392528+0000","last_fullsized":"2024-03-21T13:11:51.392528+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:36.890785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:36.890785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:13:03.917278+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":448,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.382018+0000","last_change":"2024-03-21T13:11:06.709643+0000","last_active":"2024-03-21T13:11:51.382018+0000","last_peered":"2024-03-21T13:11:51.382018+0000","last_clean":"2024-03-21T13:11:51.382018+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:11:51.382018+0000","last_undegraded":"2024-03-21T13:11:51.382018+0000","last_fullsized":"2024-03-21T13:11:51.382018+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:06.709498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:11:06.709498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:06.783266+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":1349,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:53.595210+0000","last_change":"2024-03-21T13:08:05.448565+0000","last_active":"2024-03-21T13:11:53.595210+0000","last_peered":"2024-03-21T13:11:53.595210+0000","last_clean":"2024-03-21T13:11:53.595210+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:11:53.595210+0000","last_undegraded":"2024-03-21T13:11:53.595210+0000","last_fullsized":"2024-03-21T13:11:53.595210+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:08:05.448403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:08:05.448403+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:57.133261+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":787,"num_read_kb":782,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":665,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.385011+0000","last_change":"2024-03-21T13:11:17.444422+0000","last_active":"2024-03-21T13:11:51.385011+0000","last_peered":"2024-03-21T13:11:51.385011+0000","last_clean":"2024-03-21T13:11:51.385011+0000","last_became_active":"2024-03-21T13:10:03.606672+0000","last_became_peered":"2024-03-21T13:10:03.606672+0000","last_unstale":"2024-03-21T13:11:51.385011+0000","last_undegraded":"2024-03-21T13:11:51.385011+0000","last_fullsized":"2024-03-21T13:11:51.385011+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-21T13:11:17.443750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.443750+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:28.261075+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":194,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386363+0000","last_change":"2024-03-21T13:08:56.630611+0000","last_active":"2024-03-21T13:11:51.386363+0000","last_peered":"2024-03-21T13:11:51.386363+0000","last_clean":"2024-03-21T13:11:51.386363+0000","last_became_active":"2024-03-21T13:08:20.892193+0000","last_became_peered":"2024-03-21T13:08:20.892193+0000","last_unstale":"2024-03-21T13:11:51.386363+0000","last_undegraded":"2024-03-21T13:11:51.386363+0000","last_fullsized":"2024-03-21T13:11:51.386363+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:56.630492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:56.630492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.407136+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"315'15025","reported_seq":15199,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.159520+0000","last_change":"2024-03-21T13:08:45.885732+0000","last_active":"2024-03-21T13:11:55.159520+0000","last_peered":"2024-03-21T13:11:55.159520+0000","last_clean":"2024-03-21T13:11:55.159520+0000","last_became_active":"2024-03-21T13:08:45.885382+0000","last_became_peered":"2024-03-21T13:08:45.885382+0000","last_unstale":"2024-03-21T13:11:55.159520+0000","last_undegraded":"2024-03-21T13:11:55.159520+0000","last_fullsized":"2024-03-21T13:11:55.159520+0000","mapping_epoch":234,"log_start":"298'11639","ondisk_log_start":"298'11639","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3386,"log_dups_size":2552,"ondisk_log_size":3386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:57.134627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3604480,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15025,"num_write_kb":482560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":438,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.385833+0000","last_change":"2024-03-21T13:10:03.591632+0000","last_active":"2024-03-21T13:11:51.385833+0000","last_peered":"2024-03-21T13:11:51.385833+0000","last_clean":"2024-03-21T13:11:51.385833+0000","last_became_active":"2024-03-21T13:10:03.591088+0000","last_became_peered":"2024-03-21T13:10:03.591088+0000","last_unstale":"2024-03-21T13:11:51.385833+0000","last_undegraded":"2024-03-21T13:11:51.385833+0000","last_fullsized":"2024-03-21T13:11:51.385833+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:57.640892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.640892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.407670+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":97,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386436+0000","last_change":"2024-03-21T13:10:03.606230+0000","last_active":"2024-03-21T13:11:51.386436+0000","last_peered":"2024-03-21T13:11:51.386436+0000","last_clean":"2024-03-21T13:11:51.386436+0000","last_became_active":"2024-03-21T13:10:03.605469+0000","last_became_peered":"2024-03-21T13:10:03.605469+0000","last_unstale":"2024-03-21T13:11:51.386436+0000","last_undegraded":"2024-03-21T13:11:51.386436+0000","last_fullsized":"2024-03-21T13:11:51.386436+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:22.012892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:22.012892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":550,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.389647+0000","last_change":"2024-03-21T13:10:03.649497+0000","last_active":"2024-03-21T13:11:51.389647+0000","last_peered":"2024-03-21T13:11:51.389647+0000","last_clean":"2024-03-21T13:11:51.389647+0000","last_became_active":"2024-03-21T13:10:03.598221+0000","last_became_peered":"2024-03-21T13:10:03.598221+0000","last_unstale":"2024-03-21T13:11:51.389647+0000","last_undegraded":"2024-03-21T13:11:51.389647+0000","last_fullsized":"2024-03-21T13:11:51.389647+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-21T13:09:15.777140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:15.777140+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":560,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.387415+0000","last_change":"2024-03-21T13:10:03.592045+0000","last_active":"2024-03-21T13:11:51.387415+0000","last_peered":"2024-03-21T13:11:51.387415+0000","last_clean":"2024-03-21T13:11:51.387415+0000","last_became_active":"2024-03-21T13:10:03.591328+0000","last_became_peered":"2024-03-21T13:10:03.591328+0000","last_unstale":"2024-03-21T13:11:51.387415+0000","last_undegraded":"2024-03-21T13:11:51.387415+0000","last_fullsized":"2024-03-21T13:11:51.387415+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:16.577170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:16.577170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.407698+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"315'15512","reported_seq":25406,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.784840+0000","last_change":"2024-03-21T13:11:18.697533+0000","last_active":"2024-03-21T13:11:55.784840+0000","last_peered":"2024-03-21T13:11:55.784840+0000","last_clean":"2024-03-21T13:11:55.784840+0000","last_became_active":"2024-03-21T13:11:18.671214+0000","last_became_peered":"2024-03-21T13:11:18.671214+0000","last_unstale":"2024-03-21T13:11:55.784840+0000","last_undegraded":"2024-03-21T13:11:55.784840+0000","last_fullsized":"2024-03-21T13:11:55.784840+0000","mapping_epoch":299,"log_start":"300'12055","ondisk_log_start":"300'12055","created":234,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3457,"log_dups_size":3000,"ondisk_log_size":3457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15512,"num_write_kb":496384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4850,"num_bytes_recovered":317849600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":445,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392855+0000","last_change":"2024-03-21T13:10:03.597778+0000","last_active":"2024-03-21T13:11:51.392855+0000","last_peered":"2024-03-21T13:11:51.392855+0000","last_clean":"2024-03-21T13:11:51.392855+0000","last_became_active":"2024-03-21T13:10:03.595454+0000","last_became_peered":"2024-03-21T13:10:03.595454+0000","last_unstale":"2024-03-21T13:11:51.392855+0000","last_undegraded":"2024-03-21T13:11:51.392855+0000","last_fullsized":"2024-03-21T13:11:51.392855+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:55.487977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:55.487977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998580+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":418,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392567+0000","last_change":"2024-03-21T13:10:03.611967+0000","last_active":"2024-03-21T13:11:51.392567+0000","last_peered":"2024-03-21T13:11:51.392567+0000","last_clean":"2024-03-21T13:11:51.392567+0000","last_became_active":"2024-03-21T13:10:03.611677+0000","last_became_peered":"2024-03-21T13:10:03.611677+0000","last_unstale":"2024-03-21T13:11:51.392567+0000","last_undegraded":"2024-03-21T13:11:51.392567+0000","last_fullsized":"2024-03-21T13:11:51.392567+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.324701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.324701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.997960+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"234'25","reported_seq":539,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.380855+0000","last_change":"2024-03-21T13:11:37.750033+0000","last_active":"2024-03-21T13:11:51.380855+0000","last_peered":"2024-03-21T13:11:51.380855+0000","last_clean":"2024-03-21T13:11:51.380855+0000","last_became_active":"2024-03-21T13:10:03.594571+0000","last_became_peered":"2024-03-21T13:10:03.594571+0000","last_unstale":"2024-03-21T13:11:51.380855+0000","last_undegraded":"2024-03-21T13:11:51.380855+0000","last_fullsized":"2024-03-21T13:11:51.380855+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-21T13:11:37.749901+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:11:37.749901+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:13:01.612839+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"125'12","reported_seq":599,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.387624+0000","last_change":"2024-03-21T13:08:18.149427+0000","last_active":"2024-03-21T13:11:51.387624+0000","last_peered":"2024-03-21T13:11:51.387624+0000","last_clean":"2024-03-21T13:11:51.387624+0000","last_became_active":"2024-03-21T13:02:26.253720+0000","last_became_peered":"2024-03-21T13:02:26.253720+0000","last_unstale":"2024-03-21T13:11:51.387624+0000","last_undegraded":"2024-03-21T13:11:51.387624+0000","last_fullsized":"2024-03-21T13:11:51.387624+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'12","last_scrub_stamp":"2024-03-21T13:08:18.149295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.149295+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"315'15282","reported_seq":15458,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.777702+0000","last_change":"2024-03-21T13:08:45.618681+0000","last_active":"2024-03-21T13:11:55.777702+0000","last_peered":"2024-03-21T13:11:55.777702+0000","last_clean":"2024-03-21T13:11:55.777702+0000","last_became_active":"2024-03-21T13:08:45.618487+0000","last_became_peered":"2024-03-21T13:08:45.618487+0000","last_unstale":"2024-03-21T13:11:55.777702+0000","last_undegraded":"2024-03-21T13:11:55.777702+0000","last_fullsized":"2024-03-21T13:11:55.777702+0000","mapping_epoch":234,"log_start":"300'11814","ondisk_log_start":"300'11814","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3468,"log_dups_size":3000,"ondisk_log_size":3468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15282,"num_write_kb":489024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":540,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.389732+0000","last_change":"2024-03-21T13:10:03.606897+0000","last_active":"2024-03-21T13:11:51.389732+0000","last_peered":"2024-03-21T13:11:51.389732+0000","last_clean":"2024-03-21T13:11:51.389732+0000","last_became_active":"2024-03-21T13:10:03.606557+0000","last_became_peered":"2024-03-21T13:10:03.606557+0000","last_unstale":"2024-03-21T13:11:51.389732+0000","last_undegraded":"2024-03-21T13:11:51.389732+0000","last_fullsized":"2024-03-21T13:11:51.389732+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.951655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.951655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":592,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.385117+0000","last_change":"2024-03-21T13:11:22.424995+0000","last_active":"2024-03-21T13:11:51.385117+0000","last_peered":"2024-03-21T13:11:51.385117+0000","last_clean":"2024-03-21T13:11:51.385117+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:11:51.385117+0000","last_undegraded":"2024-03-21T13:11:51.385117+0000","last_fullsized":"2024-03-21T13:11:51.385117+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:11:22.424870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:22.424870+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:50.879766+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'35","reported_seq":527,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.385660+0000","last_change":"2024-03-21T13:08:54.645581+0000","last_active":"2024-03-21T13:11:51.385660+0000","last_peered":"2024-03-21T13:11:51.385660+0000","last_clean":"2024-03-21T13:11:51.385660+0000","last_became_active":"2024-03-21T13:02:26.227165+0000","last_became_peered":"2024-03-21T13:02:26.227165+0000","last_unstale":"2024-03-21T13:11:51.385660+0000","last_undegraded":"2024-03-21T13:11:51.385660+0000","last_fullsized":"2024-03-21T13:11:51.385660+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:08:54.645360+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:08:54.645360+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.407611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"226'10","reported_seq":706,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.393893+0000","last_change":"2024-03-21T13:08:16.739276+0000","last_active":"2024-03-21T13:11:51.393893+0000","last_peered":"2024-03-21T13:11:51.393893+0000","last_clean":"2024-03-21T13:11:51.393893+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:11:51.393893+0000","last_undegraded":"2024-03-21T13:11:51.393893+0000","last_fullsized":"2024-03-21T13:11:51.393893+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:08:16.739190+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"315'15098","reported_seq":15275,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.190492+0000","last_change":"2024-03-21T13:08:45.621307+0000","last_active":"2024-03-21T13:11:55.190492+0000","last_peered":"2024-03-21T13:11:55.190492+0000","last_clean":"2024-03-21T13:11:55.190492+0000","last_became_active":"2024-03-21T13:08:45.620972+0000","last_became_peered":"2024-03-21T13:08:45.620972+0000","last_unstale":"2024-03-21T13:11:55.190492+0000","last_undegraded":"2024-03-21T13:11:55.190492+0000","last_fullsized":"2024-03-21T13:11:55.190492+0000","mapping_epoch":234,"log_start":"300'11712","ondisk_log_start":"300'11712","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:08:44.601980+0000","objects_scrubbed":0,"log_size":3386,"log_dups_size":2639,"ondisk_log_size":3386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:57.134661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570584,"num_objects":86,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15098,"num_write_kb":485825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":459,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.393651+0000","last_change":"2024-03-21T13:10:03.610862+0000","last_active":"2024-03-21T13:11:51.393651+0000","last_peered":"2024-03-21T13:11:51.393651+0000","last_clean":"2024-03-21T13:11:51.393651+0000","last_became_active":"2024-03-21T13:10:03.610471+0000","last_became_peered":"2024-03-21T13:10:03.610471+0000","last_unstale":"2024-03-21T13:11:51.393651+0000","last_undegraded":"2024-03-21T13:11:51.393651+0000","last_fullsized":"2024-03-21T13:11:51.393651+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.416262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.416262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":542,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.396729+0000","last_change":"2024-03-21T13:09:55.872269+0000","last_active":"2024-03-21T13:11:51.396729+0000","last_peered":"2024-03-21T13:11:51.396729+0000","last_clean":"2024-03-21T13:11:51.396729+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:11:51.396729+0000","last_undegraded":"2024-03-21T13:11:51.396729+0000","last_fullsized":"2024-03-21T13:11:51.396729+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:55.869919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:09:55.869919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998989+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"234'30","reported_seq":580,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.393700+0000","last_change":"2024-03-21T13:05:12.322522+0000","last_active":"2024-03-21T13:11:51.393700+0000","last_peered":"2024-03-21T13:11:51.393700+0000","last_clean":"2024-03-21T13:11:51.393700+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:11:51.393700+0000","last_undegraded":"2024-03-21T13:11:51.393700+0000","last_fullsized":"2024-03-21T13:11:51.393700+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T13:05:12.322284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:12.322284+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.997933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":542,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.381787+0000","last_change":"2024-03-21T13:11:22.759665+0000","last_active":"2024-03-21T13:11:51.381787+0000","last_peered":"2024-03-21T13:11:51.381787+0000","last_clean":"2024-03-21T13:11:51.381787+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:11:51.381787+0000","last_undegraded":"2024-03-21T13:11:51.381787+0000","last_fullsized":"2024-03-21T13:11:51.381787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:22.759551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:22.759551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:48.284181+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"315'9304","reported_seq":13765,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.760877+0000","last_change":"2024-03-21T13:11:18.692509+0000","last_active":"2024-03-21T13:11:55.760877+0000","last_peered":"2024-03-21T13:11:55.760877+0000","last_clean":"2024-03-21T13:11:55.760877+0000","last_became_active":"2024-03-21T13:11:18.670457+0000","last_became_peered":"2024-03-21T13:11:18.670457+0000","last_unstale":"2024-03-21T13:11:55.760877+0000","last_undegraded":"2024-03-21T13:11:55.760877+0000","last_fullsized":"2024-03-21T13:11:55.760877+0000","mapping_epoch":299,"log_start":"282'5868","ondisk_log_start":"282'5868","created":234,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'4446","last_scrub_stamp":"2024-03-21T13:09:51.547922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:09:51.547922+0000","objects_scrubbed":2777,"log_size":3436,"log_dups_size":0,"ondisk_log_size":3436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4875,"num_write_kb":67136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2110,"num_bytes_recovered":138280960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":97,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.387009+0000","last_change":"2024-03-21T13:10:03.604456+0000","last_active":"2024-03-21T13:11:51.387009+0000","last_peered":"2024-03-21T13:11:51.387009+0000","last_clean":"2024-03-21T13:11:51.387009+0000","last_became_active":"2024-03-21T13:10:03.604092+0000","last_became_peered":"2024-03-21T13:10:03.604092+0000","last_unstale":"2024-03-21T13:11:51.387009+0000","last_undegraded":"2024-03-21T13:11:51.387009+0000","last_fullsized":"2024-03-21T13:11:51.387009+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.328925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.328925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"272'12","reported_seq":602,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.387644+0000","last_change":"2024-03-21T13:11:18.400404+0000","last_active":"2024-03-21T13:11:51.387644+0000","last_peered":"2024-03-21T13:11:51.387644+0000","last_clean":"2024-03-21T13:11:51.387644+0000","last_became_active":"2024-03-21T13:10:03.606886+0000","last_became_peered":"2024-03-21T13:10:03.606886+0000","last_unstale":"2024-03-21T13:11:51.387644+0000","last_undegraded":"2024-03-21T13:11:51.387644+0000","last_fullsized":"2024-03-21T13:11:51.387644+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'12","last_scrub_stamp":"2024-03-21T13:11:18.400270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:18.400270+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:29.795823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":548,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.393516+0000","last_change":"2024-03-21T13:09:15.640885+0000","last_active":"2024-03-21T13:11:51.393516+0000","last_peered":"2024-03-21T13:11:51.393516+0000","last_clean":"2024-03-21T13:11:51.393516+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:11:51.393516+0000","last_undegraded":"2024-03-21T13:11:51.393516+0000","last_fullsized":"2024-03-21T13:11:51.393516+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:15.640781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:15.640781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998839+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"234'34","reported_seq":503,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.396809+0000","last_change":"2024-03-21T13:08:22.736499+0000","last_active":"2024-03-21T13:11:51.396809+0000","last_peered":"2024-03-21T13:11:51.396809+0000","last_clean":"2024-03-21T13:11:51.396809+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:11:51.396809+0000","last_undegraded":"2024-03-21T13:11:51.396809+0000","last_fullsized":"2024-03-21T13:11:51.396809+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:08:22.736371+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"315'9137","reported_seq":9353,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.137313+0000","last_change":"2024-03-21T13:09:59.088266+0000","last_active":"2024-03-21T13:11:55.137313+0000","last_peered":"2024-03-21T13:11:55.137313+0000","last_clean":"2024-03-21T13:11:55.137313+0000","last_became_active":"2024-03-21T13:09:18.448090+0000","last_became_peered":"2024-03-21T13:09:18.448090+0000","last_unstale":"2024-03-21T13:11:55.137313+0000","last_undegraded":"2024-03-21T13:11:55.137313+0000","last_fullsized":"2024-03-21T13:11:55.137313+0000","mapping_epoch":234,"log_start":"282'5752","ondisk_log_start":"282'5752","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'4654","last_scrub_stamp":"2024-03-21T13:09:59.088099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:09:59.088099+0000","objects_scrubbed":2991,"log_size":3385,"log_dups_size":0,"ondisk_log_size":3385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:57.134855+0000","scrub_duration":585,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2162688,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4484,"num_write_kb":48832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":532,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392629+0000","last_change":"2024-03-21T13:05:39.518051+0000","last_active":"2024-03-21T13:11:51.392629+0000","last_peered":"2024-03-21T13:11:51.392629+0000","last_clean":"2024-03-21T13:11:51.392629+0000","last_became_active":"2024-03-21T13:02:26.243985+0000","last_became_peered":"2024-03-21T13:02:26.243985+0000","last_unstale":"2024-03-21T13:11:51.392629+0000","last_undegraded":"2024-03-21T13:11:51.392629+0000","last_fullsized":"2024-03-21T13:11:51.392629+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:39.517591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.517591+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998079+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":442,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.384852+0000","last_change":"2024-03-21T13:11:27.742914+0000","last_active":"2024-03-21T13:11:51.384852+0000","last_peered":"2024-03-21T13:11:51.384852+0000","last_clean":"2024-03-21T13:11:51.384852+0000","last_became_active":"2024-03-21T13:10:03.594379+0000","last_became_peered":"2024-03-21T13:10:03.594379+0000","last_unstale":"2024-03-21T13:11:51.384852+0000","last_undegraded":"2024-03-21T13:11:51.384852+0000","last_fullsized":"2024-03-21T13:11:51.384852+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:27.742792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:27.742792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:52.061542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":570,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.381879+0000","last_change":"2024-03-21T13:10:03.725464+0000","last_active":"2024-03-21T13:11:51.381879+0000","last_peered":"2024-03-21T13:11:51.381879+0000","last_clean":"2024-03-21T13:11:51.381879+0000","last_became_active":"2024-03-21T13:10:03.586593+0000","last_became_peered":"2024-03-21T13:10:03.586593+0000","last_unstale":"2024-03-21T13:11:51.381879+0000","last_undegraded":"2024-03-21T13:11:51.381879+0000","last_fullsized":"2024-03-21T13:11:51.381879+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:10:03.725389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:10:03.725389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":302,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386210+0000","last_change":"2024-03-21T13:11:00.776320+0000","last_active":"2024-03-21T13:11:51.386210+0000","last_peered":"2024-03-21T13:11:51.386210+0000","last_clean":"2024-03-21T13:11:51.386210+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:11:51.386210+0000","last_undegraded":"2024-03-21T13:11:51.386210+0000","last_fullsized":"2024-03-21T13:11:51.386210+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:11:00.775958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:00.775958+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:19.616562+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"315'9099","reported_seq":15592,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.020047+0000","last_change":"2024-03-21T13:10:53.409947+0000","last_active":"2024-03-21T13:11:55.020047+0000","last_peered":"2024-03-21T13:11:55.020047+0000","last_clean":"2024-03-21T13:11:55.020047+0000","last_became_active":"2024-03-21T13:10:53.401083+0000","last_became_peered":"2024-03-21T13:10:53.401083+0000","last_unstale":"2024-03-21T13:11:55.020047+0000","last_undegraded":"2024-03-21T13:11:55.020047+0000","last_fullsized":"2024-03-21T13:11:55.020047+0000","mapping_epoch":289,"log_start":"277'5159","ondisk_log_start":"277'5159","created":234,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'4353","last_scrub_stamp":"2024-03-21T13:09:52.297192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:09:52.297192+0000","objects_scrubbed":2735,"log_size":3940,"log_dups_size":0,"ondisk_log_size":3940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:58.330058+0000","scrub_duration":734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4761,"num_write_kb":66240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3131,"num_bytes_recovered":205193216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":105,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.389034+0000","last_change":"2024-03-21T13:11:35.902330+0000","last_active":"2024-03-21T13:11:51.389034+0000","last_peered":"2024-03-21T13:11:51.389034+0000","last_clean":"2024-03-21T13:11:51.389034+0000","last_became_active":"2024-03-21T13:10:03.602752+0000","last_became_peered":"2024-03-21T13:10:03.602752+0000","last_unstale":"2024-03-21T13:11:51.389034+0000","last_undegraded":"2024-03-21T13:11:51.389034+0000","last_fullsized":"2024-03-21T13:11:51.389034+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:35.902195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:11:35.902195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:35.938985+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":442,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.384473+0000","last_change":"2024-03-21T13:11:02.693027+0000","last_active":"2024-03-21T13:11:51.384473+0000","last_peered":"2024-03-21T13:11:51.384473+0000","last_clean":"2024-03-21T13:11:51.384473+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:11:51.384473+0000","last_undegraded":"2024-03-21T13:11:51.384473+0000","last_fullsized":"2024-03-21T13:11:51.384473+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:02.692916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:02.692916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:08.411083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":556,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.387505+0000","last_change":"2024-03-21T13:06:30.274121+0000","last_active":"2024-03-21T13:11:51.387505+0000","last_peered":"2024-03-21T13:11:51.387505+0000","last_clean":"2024-03-21T13:11:51.387505+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:11:51.387505+0000","last_undegraded":"2024-03-21T13:11:51.387505+0000","last_fullsized":"2024-03-21T13:11:51.387505+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:30.274033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:30.274033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:52.407045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"234'31","reported_seq":115,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386920+0000","last_change":"2024-03-21T13:10:03.634167+0000","last_active":"2024-03-21T13:11:51.386920+0000","last_peered":"2024-03-21T13:11:51.386920+0000","last_clean":"2024-03-21T13:11:51.386920+0000","last_became_active":"2024-03-21T13:10:03.591832+0000","last_became_peered":"2024-03-21T13:10:03.591832+0000","last_unstale":"2024-03-21T13:11:51.386920+0000","last_undegraded":"2024-03-21T13:11:51.386920+0000","last_fullsized":"2024-03-21T13:11:51.386920+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'31","last_scrub_stamp":"2024-03-21T13:09:03.608302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:03.608302+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"315'9367","reported_seq":9585,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:55.760644+0000","last_change":"2024-03-21T13:09:50.510815+0000","last_active":"2024-03-21T13:11:55.760644+0000","last_peered":"2024-03-21T13:11:55.760644+0000","last_clean":"2024-03-21T13:11:55.760644+0000","last_became_active":"2024-03-21T13:09:18.458493+0000","last_became_peered":"2024-03-21T13:09:18.458493+0000","last_unstale":"2024-03-21T13:11:55.760644+0000","last_undegraded":"2024-03-21T13:11:55.760644+0000","last_fullsized":"2024-03-21T13:11:55.760644+0000","mapping_epoch":234,"log_start":"282'5908","ondisk_log_start":"282'5908","created":234,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'4418","last_scrub_stamp":"2024-03-21T13:09:50.510193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:44.601980+0000","last_clean_scrub_stamp":"2024-03-21T13:09:50.510193+0000","objects_scrubbed":2740,"log_size":3459,"log_dups_size":0,"ondisk_log_size":3459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4962,"num_write_kb":71104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":97,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.388247+0000","last_change":"2024-03-21T13:10:03.609956+0000","last_active":"2024-03-21T13:11:51.388247+0000","last_peered":"2024-03-21T13:11:51.388247+0000","last_clean":"2024-03-21T13:11:51.388247+0000","last_became_active":"2024-03-21T13:10:03.609620+0000","last_became_peered":"2024-03-21T13:10:03.609620+0000","last_unstale":"2024-03-21T13:11:51.388247+0000","last_undegraded":"2024-03-21T13:11:51.388247+0000","last_fullsized":"2024-03-21T13:11:51.388247+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.291369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.291369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"272'12","reported_seq":124,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.386872+0000","last_change":"2024-03-21T13:10:03.654783+0000","last_active":"2024-03-21T13:11:51.386872+0000","last_peered":"2024-03-21T13:11:51.386872+0000","last_clean":"2024-03-21T13:11:51.386872+0000","last_became_active":"2024-03-21T13:10:03.609698+0000","last_became_peered":"2024-03-21T13:10:03.609698+0000","last_unstale":"2024-03-21T13:11:51.386872+0000","last_undegraded":"2024-03-21T13:11:51.386872+0000","last_fullsized":"2024-03-21T13:11:51.386872+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:05:30.300605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:30.300605+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":533,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.389850+0000","last_change":"2024-03-21T13:05:38.492137+0000","last_active":"2024-03-21T13:11:51.389850+0000","last_peered":"2024-03-21T13:11:51.389850+0000","last_clean":"2024-03-21T13:11:51.389850+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:11:51.389850+0000","last_undegraded":"2024-03-21T13:11:51.389850+0000","last_fullsized":"2024-03-21T13:11:51.389850+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:38.492015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.492015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.998163+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"234'20","reported_seq":495,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-21T13:11:51.392808+0000","last_change":"2024-03-21T13:05:22.406498+0000","last_active":"2024-03-21T13:11:51.392808+0000","last_peered":"2024-03-21T13:11:51.392808+0000","last_clean":"2024-03-21T13:11:51.392808+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:11:51.392808+0000","last_undegraded":"2024-03-21T13:11:51.392808+0000","last_fullsized":"2024-03-21T13:11:51.392808+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:22.406367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.406367+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:11:51.997871+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":50462744,"num_objects":771,"num_object_clones":0,"num_object_copies":2313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":771,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171821,"num_write_kb":3796993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25451,"num_bytes_recovered":1667956736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":624439296,"data_stored":624427080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92619,"ondisk_log_size":92619,"up":78,"acting":78,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":787,"num_read_kb":782,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1108,"num_read_kb":865,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":285,"num_bytes_recovered":5776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":730,"ondisk_log_size":730,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168740,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1076380,"kb_used_data":350804,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92668772,"statfs":{"total":95995035648,"available":94892822528,"internally_reserved":0,"allocated":359223296,"data_stored":358744132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463082,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":784800,"kb_used_data":81688,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92960352,"statfs":{"total":95995035648,"available":95191400448,"internally_reserved":0,"allocated":83648512,"data_stored":83165428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":719976902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575042,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738568,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":772116,"kb_used_data":191180,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92973036,"statfs":{"total":95995035648,"available":95204388864,"internally_reserved":0,"allocated":195768320,"data_stored":195296699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411518,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":677044,"kb_used_data":41964,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93068108,"statfs":{"total":95995035648,"available":95301742592,"internally_reserved":0,"allocated":42971136,"data_stored":42531217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182585,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":830448,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92914704,"statfs":{"total":95995035648,"available":95144656896,"internally_reserved":0,"allocated":1024000,"data_stored":524318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":849344967},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019059,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":793540,"kb_used_data":50172,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92951612,"statfs":{"total":95995035648,"available":95182450688,"internally_reserved":0,"allocated":51376128,"data_stored":50920328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724718,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160301056,"data_stored":160301056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29949952,"data_stored":29949952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":39718912,"data_stored":39714840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":42930176,"data_stored":42926104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":351539200,"data_stored":351535128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:12:01.210 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:12:01.294 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:01.317 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:01.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:01.419 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:01.445 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:01.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:01.458 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:12:01.458 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:12:01.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:01.572 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:01.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:01.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:12:01.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:01.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:01.698 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:01.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:01 smithi163 ceph-mon[29621]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:12:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:01 smithi163 ceph-mon[29621]: osdmap e317: 8 total, 8 up, 6 in 2024-03-21T13:12:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:01 smithi163 ceph-mon[40675]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:12:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:01 smithi163 ceph-mon[40675]: osdmap e317: 8 total, 8 up, 6 in 2024-03-21T13:12:01.800 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:01.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:01.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:01.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:01.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:01.958 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:01.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:01.979 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:01 smithi204 ceph-mon[33153]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:12:01.979 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:01 smithi204 ceph-mon[33153]: osdmap e317: 8 total, 8 up, 6 in 2024-03-21T13:12:02.048 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:12:02.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:02.087 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:02.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:02.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:02.189 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:02.213 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:02.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:02.253 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:12:01 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:12:02.314 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:02.340 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:02.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:02.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:02.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:02.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:02.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:02.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:02.600 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:02.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:02.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:02.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:02.730 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:02.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:02 smithi163 ceph-mon[29621]: pgmap v621: 158 pgs: 158 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 936 op/s 2024-03-21T13:12:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:02 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:02.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:02 smithi163 ceph-mon[40675]: pgmap v621: 158 pgs: 158 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 936 op/s 2024-03-21T13:12:02.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:02 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:02.832 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:02.856 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:02.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:02.958 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:02.982 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:02.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:02.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:03.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:02 smithi204 ceph-mon[33153]: pgmap v621: 158 pgs: 158 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 936 op/s 2024-03-21T13:12:03.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:02 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:03.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:03.109 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:03.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:03.210 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:03.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:03.235 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:03.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:03.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:03.361 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:03.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:03.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-21T13:12:03.461 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:03.524 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:03.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:03.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:03.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:03.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:03.653 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:03.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:03.755 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:03 smithi163 ceph-mon[29621]: pgmap v623: 158 pgs: 158 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-21T13:12:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:03 smithi163 ceph-mon[40675]: pgmap v623: 158 pgs: 158 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-21T13:12:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:03.780 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:03.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:03.881 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:03.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:03.907 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:03.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:03.993 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:03 smithi204 ceph-mon[33153]: pgmap v623: 158 pgs: 158 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-21T13:12:04.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:04.019 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:04.036 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:04.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:04.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:04.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:04.166 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:04.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:04.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:04.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:04.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:04.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:04.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:04.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:04.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:04.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:04.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:04.552 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:04.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:04.557 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'unique_pool_2' removed 2024-03-21T13:12:04.652 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:04.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:04.678 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:04.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:04.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:04 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3075018663' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:12:04.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:04 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:12:04.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:04 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3075018663' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:12:04.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:04 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:12:04.779 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:04.805 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:04.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:04.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:04.931 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:04.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:04 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3075018663' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:12:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:04 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:12:05.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:05.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:05.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:05.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:05.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:05.184 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:05.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:05.285 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:05.311 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:05.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:05.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:05.445 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:05.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:05.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:05.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:05.573 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:05.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:05.674 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:05.700 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:05.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:05.770 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-21T13:12:05.771 INFO:tasks.radosbench:Beginning radosbench... 2024-03-21T13:12:05.772 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-21T13:12:05.772 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-21T13:12:05.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:05 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:12:05.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:05 smithi163 ceph-mon[29621]: osdmap e318: 8 total, 8 up, 6 in 2024-03-21T13:12:05.797 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:05 smithi163 ceph-mon[29621]: pgmap v625: 132 pgs: 132 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 641 B/s rd, 0 op/s 2024-03-21T13:12:05.797 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:05.797 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:05 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:12:05.798 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:05 smithi163 ceph-mon[40675]: osdmap e318: 8 total, 8 up, 6 in 2024-03-21T13:12:05.798 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:05 smithi163 ceph-mon[40675]: pgmap v625: 132 pgs: 132 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 641 B/s rd, 0 op/s 2024-03-21T13:12:05.798 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:05.798 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:12:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:12:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:12:05.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:05.825 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:05.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:05.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:05.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:05.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:05.951 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:05.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:05 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:12:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:05 smithi204 ceph-mon[33153]: osdmap e318: 8 total, 8 up, 6 in 2024-03-21T13:12:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:05 smithi204 ceph-mon[33153]: pgmap v625: 132 pgs: 132 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 641 B/s rd, 0 op/s 2024-03-21T13:12:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:06.053 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:06.079 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:06.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:06.180 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:06.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:06.204 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:06.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:06.305 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:06.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:06.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:06.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:06.431 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:06.454 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:06.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:06.556 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:06.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:06.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:06.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:06.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:06.706 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:06.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:06.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:06.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:06.831 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:06.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:06.932 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:06.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:06.957 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:06.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:07.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:07.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:07.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:07.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:07.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:07.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:07.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:07.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:07.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:07.334 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:07.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:07.435 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:07.463 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:07.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:07.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:07.590 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:07.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:07.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:07.715 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:07.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:07.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:07 smithi204 ceph-mon[33153]: osdmap e319: 8 total, 8 up, 6 in 2024-03-21T13:12:07.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:07 smithi163 ceph-mon[29621]: osdmap e319: 8 total, 8 up, 6 in 2024-03-21T13:12:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:07 smithi163 ceph-mon[40675]: osdmap e319: 8 total, 8 up, 6 in 2024-03-21T13:12:07.817 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:07.842 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:07.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:07.943 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:07.975 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:07.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:07.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:07.998 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:08.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:08.102 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:08.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:08.204 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:08.229 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:08.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:08.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:08.331 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:08.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:08.357 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:08.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:08.458 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:08.484 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:08.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:08.586 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:08.610 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:08.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:08.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:08.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:08 smithi163 ceph-mon[40675]: pgmap v627: 132 pgs: 132 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:12:08.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:08 smithi163 ceph-mon[29621]: pgmap v627: 132 pgs: 132 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:12:08.740 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:08.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:08.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:08.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:08 smithi204 ceph-mon[33153]: pgmap v627: 132 pgs: 132 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:12:08.842 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:08.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:08.867 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:08.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:08.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:08.997 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:08.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:09.098 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:09.123 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:09.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:09.225 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:09.249 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:09.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:09.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:09.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:09.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:09.381 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:09.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:09.482 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:09.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:09.510 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:09.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:09.612 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:09.640 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:09.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:09.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:09.742 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:09.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:09.772 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:09.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:09.873 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:09.901 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:09.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:09.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:10.003 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:10.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:10.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:10.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:10.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:10.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:10.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:10.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:10.263 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:10.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:10.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:10.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:10.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:10.420 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:10.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:10.521 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:10.547 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:10.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:10.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:10.672 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:10 smithi163 ceph-mon[40675]: pgmap v628: 132 pgs: 132 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 546 B/s rd, 0 op/s 2024-03-21T13:12:10.672 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:10 smithi163 ceph-mon[29621]: pgmap v628: 132 pgs: 132 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 546 B/s rd, 0 op/s 2024-03-21T13:12:10.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:10.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:10.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:10.677 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:10.688 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:12:10.688 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:12:10.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:10 smithi204 ceph-mon[33153]: pgmap v628: 132 pgs: 132 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 546 B/s rd, 0 op/s 2024-03-21T13:12:10.777 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:10.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:10.803 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:10.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:10.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:10.935 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:10.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:11.036 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:11.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:11.061 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:11.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:11.162 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:11.188 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:11.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:11.289 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:11.317 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:11.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:11.408 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:12:11.419 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:11.444 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:11.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:11.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:11.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:11.576 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:11.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:11.677 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:11.702 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:11.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:11.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:11 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/739972631' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:12:11.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:11 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/739972631' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:12:11.755 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:11 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/739972631' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:12:11.804 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:11.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:11.829 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:11.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:11.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:11.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:11.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:12.061 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:12.090 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:12.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:12.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:12.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:12.216 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:12.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:12.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:12:11 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:12:12.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:12.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:12.344 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:12.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:12.445 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:12.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:12.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:12.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:12.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:12.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:12.597 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:12.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:12.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:12.724 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:12.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:12.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:12 smithi204 ceph-mon[33153]: pgmap v629: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:12:12.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:12 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:12:12.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:12 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/739972631' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:12:12.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:12 smithi204 ceph-mon[33153]: osdmap e320: 8 total, 8 up, 6 in 2024-03-21T13:12:12.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:12 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:12:12.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:12 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:12:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:12 smithi163 ceph-mon[40675]: pgmap v629: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:12:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:12 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:12:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:12 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/739972631' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:12:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:12 smithi163 ceph-mon[40675]: osdmap e320: 8 total, 8 up, 6 in 2024-03-21T13:12:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:12 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:12:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:12 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:12:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:12 smithi163 ceph-mon[29621]: pgmap v629: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:12:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:12 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:12:12.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:12 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/739972631' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:12:12.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:12 smithi163 ceph-mon[29621]: osdmap e320: 8 total, 8 up, 6 in 2024-03-21T13:12:12.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:12 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:12:12.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:12 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:12:12.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:12.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:12.852 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:12.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:12.954 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:12.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:12.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:13.080 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:13.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:13.109 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:13.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:13.210 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:13.235 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:13.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:13.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:13.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:13.363 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:13.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:13.464 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:13.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:13.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:13.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:13.590 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:13.615 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:13.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:13.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:13 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:13.671 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:13 smithi163 ceph-mon[40675]: osdmap e321: 8 total, 8 up, 6 in 2024-03-21T13:12:13.671 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:13 smithi163 ceph-mon[40675]: 6.1d scrub starts 2024-03-21T13:12:13.671 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:13 smithi163 ceph-mon[40675]: 6.1d scrub ok 2024-03-21T13:12:13.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:13 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:13.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:13 smithi163 ceph-mon[29621]: osdmap e321: 8 total, 8 up, 6 in 2024-03-21T13:12:13.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:13 smithi163 ceph-mon[29621]: 6.1d scrub starts 2024-03-21T13:12:13.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:13 smithi163 ceph-mon[29621]: 6.1d scrub ok 2024-03-21T13:12:13.717 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:13.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:13.741 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:13.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:13 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:13 smithi204 ceph-mon[33153]: osdmap e321: 8 total, 8 up, 6 in 2024-03-21T13:12:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:13 smithi204 ceph-mon[33153]: 6.1d scrub starts 2024-03-21T13:12:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:13 smithi204 ceph-mon[33153]: 6.1d scrub ok 2024-03-21T13:12:13.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:13.868 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:13.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:13.969 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:13.995 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:13.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:14.096 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:14.120 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:14.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:14.222 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:14.247 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:14.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:14.348 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:14.373 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:14.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:14.474 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:14.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:14.498 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:14.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:14.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:14.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:14.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:14.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:14 smithi163 ceph-mon[40675]: pgmap v632: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 788 B/s rd, 0 op/s 2024-03-21T13:12:14.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:14 smithi163 ceph-mon[40675]: 3.9 scrub starts 2024-03-21T13:12:14.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:14 smithi163 ceph-mon[40675]: 3.9 scrub ok 2024-03-21T13:12:14.677 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:14 smithi163 ceph-mon[29621]: pgmap v632: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 788 B/s rd, 0 op/s 2024-03-21T13:12:14.677 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:14 smithi163 ceph-mon[29621]: 3.9 scrub starts 2024-03-21T13:12:14.677 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:14 smithi163 ceph-mon[29621]: 3.9 scrub ok 2024-03-21T13:12:14.703 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-21T13:12:14.728 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:14 smithi204 ceph-mon[33153]: pgmap v632: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 788 B/s rd, 0 op/s 2024-03-21T13:12:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:14 smithi204 ceph-mon[33153]: 3.9 scrub starts 2024-03-21T13:12:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:14 smithi204 ceph-mon[33153]: 3.9 scrub ok 2024-03-21T13:12:14.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:14.770 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:14.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:14.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:14.899 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:14.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:15.001 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:15.027 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:15.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:15.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:15.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:15.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:15.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:15.258 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:15.283 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:15.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:15.384 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:15.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:15.408 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:15.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:15.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:15.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:15.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:15.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:15.634 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:12:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:12:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:12:15.634 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:15 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:12:15.634 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:15 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:12:15.634 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:15 smithi163 ceph-mon[29621]: osdmap e322: 8 total, 8 up, 6 in 2024-03-21T13:12:15.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:15 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:12:15.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:15 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:12:15.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:15 smithi163 ceph-mon[40675]: osdmap e322: 8 total, 8 up, 6 in 2024-03-21T13:12:15.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:15.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:15.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:15.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:15 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:12:15.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:15 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:12:15.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:15 smithi204 ceph-mon[33153]: osdmap e322: 8 total, 8 up, 6 in 2024-03-21T13:12:15.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:15.793 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:15.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:15.880 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:12:15.896 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:15.900 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:12:15.920 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:12:15.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:15.921 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:15.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:15.957 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:12:15.981 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:12:15.984 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:12:15.984 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:12:15.999 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:12:16.022 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:16.023 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:12:16.042 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:12:16.043 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:12:16.043 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:12:16.044 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:12:16.044 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:12:16.044 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:12:16.044 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:12:16.044 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:12:16.044 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:12:16.045 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:16.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:16.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:16.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:16.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:16.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:16.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:16.277 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:16.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:16.404 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:16.428 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:16.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:16.531 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:16.554 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:16.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:16.656 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:16.680 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:16.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:16 smithi204 ceph-mon[33153]: pgmap v634: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:16 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3198339446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:12:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:16 smithi204 ceph-mon[33153]: 4.1c scrub starts 2024-03-21T13:12:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:16 smithi204 ceph-mon[33153]: 4.1c scrub ok 2024-03-21T13:12:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:16 smithi204 ceph-mon[33153]: osdmap e323: 8 total, 8 up, 6 in 2024-03-21T13:12:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:16 smithi163 ceph-mon[29621]: pgmap v634: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:16 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3198339446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:12:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:16 smithi163 ceph-mon[29621]: 4.1c scrub starts 2024-03-21T13:12:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:16 smithi163 ceph-mon[29621]: 4.1c scrub ok 2024-03-21T13:12:16.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:16 smithi163 ceph-mon[29621]: osdmap e323: 8 total, 8 up, 6 in 2024-03-21T13:12:16.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:16 smithi163 ceph-mon[40675]: pgmap v634: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:16.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:16 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3198339446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:12:16.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:16 smithi163 ceph-mon[40675]: 4.1c scrub starts 2024-03-21T13:12:16.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:16 smithi163 ceph-mon[40675]: 4.1c scrub ok 2024-03-21T13:12:16.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:16 smithi163 ceph-mon[40675]: osdmap e323: 8 total, 8 up, 6 in 2024-03-21T13:12:16.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:16.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:16.810 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:16.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:16.911 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:16.937 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:16.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:17.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:17.067 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:17.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:17.146 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:12:17.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:17.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:17.196 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:17.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:17.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:17.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:17.324 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:17.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:17.425 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:17.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:17.451 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:17.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:17.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:17.578 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:17.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:17.680 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:17.705 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:17.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:17.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:17 smithi204 ceph-mon[33153]: from='client.25436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:12:17.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:17 smithi204 ceph-mon[33153]: 2.2 scrub starts 2024-03-21T13:12:17.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:17 smithi204 ceph-mon[33153]: 2.2 scrub ok 2024-03-21T13:12:17.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:17 smithi204 ceph-mon[33153]: 6.17 scrub starts 2024-03-21T13:12:17.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:17 smithi204 ceph-mon[33153]: 6.17 scrub ok 2024-03-21T13:12:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:17 smithi163 ceph-mon[29621]: from='client.25436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:12:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:17 smithi163 ceph-mon[29621]: 2.2 scrub starts 2024-03-21T13:12:17.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:17 smithi163 ceph-mon[29621]: 2.2 scrub ok 2024-03-21T13:12:17.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:17 smithi163 ceph-mon[29621]: 6.17 scrub starts 2024-03-21T13:12:17.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:17 smithi163 ceph-mon[29621]: 6.17 scrub ok 2024-03-21T13:12:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:17 smithi163 ceph-mon[40675]: from='client.25436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:12:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:17 smithi163 ceph-mon[40675]: 2.2 scrub starts 2024-03-21T13:12:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:17 smithi163 ceph-mon[40675]: 2.2 scrub ok 2024-03-21T13:12:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:17 smithi163 ceph-mon[40675]: 6.17 scrub starts 2024-03-21T13:12:17.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:17 smithi163 ceph-mon[40675]: 6.17 scrub ok 2024-03-21T13:12:17.806 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:17.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:17.834 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:17.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:17.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:17.963 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:17.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:18.066 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:18.081 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:18.093 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:18.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:18.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:18.110 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":634,"stamp":"2024-03-21T13:12:14.875438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2146,"num_read_kb":1992,"num_write":901,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":1778865,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":960,"ondisk_log_size":960,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":401,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":4540212,"kb_used_data":306292,"kb_used_omap":9,"kb_used_meta":4233846,"kb_avail":557930700,"statfs":{"total":575970213888,"available":571321036800,"internally_reserved":0,"allocated":313643008,"data_stored":310672540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":4335458982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":66,"apply_latency_ms":66,"commit_latency_ns":66000000,"apply_latency_ns":66000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001337"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":398,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.431509+0000","last_change":"2024-03-21T13:09:11.718192+0000","last_active":"2024-03-21T13:12:06.431509+0000","last_peered":"2024-03-21T13:12:06.431509+0000","last_clean":"2024-03-21T13:12:06.431509+0000","last_became_active":"2024-03-21T13:06:45.404217+0000","last_became_peered":"2024-03-21T13:06:45.404217+0000","last_unstale":"2024-03-21T13:12:06.431509+0000","last_undegraded":"2024-03-21T13:12:06.431509+0000","last_fullsized":"2024-03-21T13:12:06.431509+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:11.717992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:09:11.717992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182678+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":442,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.432113+0000","last_change":"2024-03-21T13:10:03.600751+0000","last_active":"2024-03-21T13:12:06.432113+0000","last_peered":"2024-03-21T13:12:06.432113+0000","last_clean":"2024-03-21T13:12:06.432113+0000","last_became_active":"2024-03-21T13:10:03.600240+0000","last_became_peered":"2024-03-21T13:10:03.600240+0000","last_unstale":"2024-03-21T13:12:06.432113+0000","last_undegraded":"2024-03-21T13:12:06.432113+0000","last_fullsized":"2024-03-21T13:12:06.432113+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:54.360939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:07:54.360939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":106,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.385755+0000","last_change":"2024-03-21T13:10:03.605517+0000","last_active":"2024-03-21T13:12:06.385755+0000","last_peered":"2024-03-21T13:12:06.385755+0000","last_clean":"2024-03-21T13:12:06.385755+0000","last_became_active":"2024-03-21T13:10:03.605149+0000","last_became_peered":"2024-03-21T13:10:03.605149+0000","last_unstale":"2024-03-21T13:12:06.385755+0000","last_undegraded":"2024-03-21T13:12:06.385755+0000","last_fullsized":"2024-03-21T13:12:06.385755+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:49.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.260693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"234'19","reported_seq":622,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429267+0000","last_change":"2024-03-21T13:11:14.697504+0000","last_active":"2024-03-21T13:12:06.429267+0000","last_peered":"2024-03-21T13:12:06.429267+0000","last_clean":"2024-03-21T13:12:06.429267+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:12:06.429267+0000","last_undegraded":"2024-03-21T13:12:06.429267+0000","last_fullsized":"2024-03-21T13:12:06.429267+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:11:14.697379+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:11:14.697379+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:31.632762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":537,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428774+0000","last_change":"2024-03-21T13:11:32.995208+0000","last_active":"2024-03-21T13:12:06.428774+0000","last_peered":"2024-03-21T13:12:06.428774+0000","last_clean":"2024-03-21T13:12:06.428774+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:12:06.428774+0000","last_undegraded":"2024-03-21T13:12:06.428774+0000","last_fullsized":"2024-03-21T13:12:06.428774+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:32.995084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:11:32.995084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:56.298187+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":488,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429442+0000","last_change":"2024-03-21T13:11:01.803199+0000","last_active":"2024-03-21T13:12:06.429442+0000","last_peered":"2024-03-21T13:12:06.429442+0000","last_clean":"2024-03-21T13:12:06.429442+0000","last_became_active":"2024-03-21T13:10:03.591543+0000","last_became_peered":"2024-03-21T13:10:03.591543+0000","last_unstale":"2024-03-21T13:12:06.429442+0000","last_undegraded":"2024-03-21T13:12:06.429442+0000","last_fullsized":"2024-03-21T13:12:06.429442+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:01.803105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:11:01.803105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.266826+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":569,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429105+0000","last_change":"2024-03-21T13:10:03.642319+0000","last_active":"2024-03-21T13:12:06.429105+0000","last_peered":"2024-03-21T13:12:06.429105+0000","last_clean":"2024-03-21T13:12:06.429105+0000","last_became_active":"2024-03-21T13:10:03.610318+0000","last_became_peered":"2024-03-21T13:10:03.610318+0000","last_unstale":"2024-03-21T13:12:06.429105+0000","last_undegraded":"2024-03-21T13:12:06.429105+0000","last_fullsized":"2024-03-21T13:12:06.429105+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:09:05.740634+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:01:38.765896+0000","last_clean_scrub_stamp":"2024-03-21T13:09:05.740634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":69,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.432129+0000","last_change":"2024-03-21T13:11:28.976696+0000","last_active":"2024-03-21T13:12:06.432129+0000","last_peered":"2024-03-21T13:12:06.432129+0000","last_clean":"2024-03-21T13:12:06.432129+0000","last_became_active":"2024-03-21T13:11:21.702209+0000","last_became_peered":"2024-03-21T13:11:21.702209+0000","last_unstale":"2024-03-21T13:12:06.432129+0000","last_undegraded":"2024-03-21T13:12:06.432129+0000","last_fullsized":"2024-03-21T13:12:06.432129+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:11:28.976545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:28.976545+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:35.180894+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":448,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430134+0000","last_change":"2024-03-21T13:10:03.598720+0000","last_active":"2024-03-21T13:12:06.430134+0000","last_peered":"2024-03-21T13:12:06.430134+0000","last_clean":"2024-03-21T13:12:06.430134+0000","last_became_active":"2024-03-21T13:10:03.597963+0000","last_became_peered":"2024-03-21T13:10:03.597963+0000","last_unstale":"2024-03-21T13:12:06.430134+0000","last_undegraded":"2024-03-21T13:12:06.430134+0000","last_fullsized":"2024-03-21T13:12:06.430134+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:57.627967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.627967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":445,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.386907+0000","last_change":"2024-03-21T13:11:20.342434+0000","last_active":"2024-03-21T13:12:06.386907+0000","last_peered":"2024-03-21T13:12:06.386907+0000","last_clean":"2024-03-21T13:12:06.386907+0000","last_became_active":"2024-03-21T13:02:26.236707+0000","last_became_peered":"2024-03-21T13:02:26.236707+0000","last_unstale":"2024-03-21T13:12:06.386907+0000","last_undegraded":"2024-03-21T13:12:06.386907+0000","last_fullsized":"2024-03-21T13:12:06.386907+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:20.342189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:11:20.342189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:35.037790+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":421,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429100+0000","last_change":"2024-03-21T13:11:33.515344+0000","last_active":"2024-03-21T13:12:06.429100+0000","last_peered":"2024-03-21T13:12:06.429100+0000","last_clean":"2024-03-21T13:12:06.429100+0000","last_became_active":"2024-03-21T13:02:26.245402+0000","last_became_peered":"2024-03-21T13:02:26.245402+0000","last_unstale":"2024-03-21T13:12:06.429100+0000","last_undegraded":"2024-03-21T13:12:06.429100+0000","last_fullsized":"2024-03-21T13:12:06.429100+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:33.515220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:33.515220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:53.156657+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":540,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428517+0000","last_change":"2024-03-21T13:05:51.580203+0000","last_active":"2024-03-21T13:12:06.428517+0000","last_peered":"2024-03-21T13:12:06.428517+0000","last_clean":"2024-03-21T13:12:06.428517+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:12:06.428517+0000","last_undegraded":"2024-03-21T13:12:06.428517+0000","last_fullsized":"2024-03-21T13:12:06.428517+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-21T13:05:51.580074+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:05:51.580074+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":546,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428284+0000","last_change":"2024-03-21T13:10:03.611898+0000","last_active":"2024-03-21T13:12:06.428284+0000","last_peered":"2024-03-21T13:12:06.428284+0000","last_clean":"2024-03-21T13:12:06.428284+0000","last_became_active":"2024-03-21T13:10:03.611240+0000","last_became_peered":"2024-03-21T13:10:03.611240+0000","last_unstale":"2024-03-21T13:12:06.428284+0000","last_undegraded":"2024-03-21T13:12:06.428284+0000","last_fullsized":"2024-03-21T13:12:06.428284+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:17.747360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:17.747360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:12.990426+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":553,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.431762+0000","last_change":"2024-03-21T13:08:09.443525+0000","last_active":"2024-03-21T13:12:06.431762+0000","last_peered":"2024-03-21T13:12:06.431762+0000","last_clean":"2024-03-21T13:12:06.431762+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:12:06.431762+0000","last_undegraded":"2024-03-21T13:12:06.431762+0000","last_fullsized":"2024-03-21T13:12:06.431762+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:09.443411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:09.443411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182044+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":569,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430726+0000","last_change":"2024-03-21T13:11:25.792475+0000","last_active":"2024-03-21T13:12:06.430726+0000","last_peered":"2024-03-21T13:12:06.430726+0000","last_clean":"2024-03-21T13:12:06.430726+0000","last_became_active":"2024-03-21T13:02:26.230057+0000","last_became_peered":"2024-03-21T13:02:26.230057+0000","last_unstale":"2024-03-21T13:12:06.430726+0000","last_undegraded":"2024-03-21T13:12:06.430726+0000","last_fullsized":"2024-03-21T13:12:06.430726+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:25.792349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:25.792349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:48.325353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"234'9","reported_seq":568,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428538+0000","last_change":"2024-03-21T13:10:03.627839+0000","last_active":"2024-03-21T13:12:06.428538+0000","last_peered":"2024-03-21T13:12:06.428538+0000","last_clean":"2024-03-21T13:12:06.428538+0000","last_became_active":"2024-03-21T13:10:03.595745+0000","last_became_peered":"2024-03-21T13:10:03.595745+0000","last_unstale":"2024-03-21T13:12:06.428538+0000","last_undegraded":"2024-03-21T13:12:06.428538+0000","last_fullsized":"2024-03-21T13:12:06.428538+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'9","last_scrub_stamp":"2024-03-21T13:10:00.872509+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:10:00.872509+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":568,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.431559+0000","last_change":"2024-03-21T13:11:01.737787+0000","last_active":"2024-03-21T13:12:06.431559+0000","last_peered":"2024-03-21T13:12:06.431559+0000","last_clean":"2024-03-21T13:12:06.431559+0000","last_became_active":"2024-03-21T13:02:26.226675+0000","last_became_peered":"2024-03-21T13:02:26.226675+0000","last_unstale":"2024-03-21T13:12:06.431559+0000","last_undegraded":"2024-03-21T13:12:06.431559+0000","last_fullsized":"2024-03-21T13:12:06.431559+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:01.737083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:01.737083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:09.465335+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":455,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430464+0000","last_change":"2024-03-21T13:11:28.742357+0000","last_active":"2024-03-21T13:12:06.430464+0000","last_peered":"2024-03-21T13:12:06.430464+0000","last_clean":"2024-03-21T13:12:06.430464+0000","last_became_active":"2024-03-21T13:02:26.231879+0000","last_became_peered":"2024-03-21T13:02:26.231879+0000","last_unstale":"2024-03-21T13:12:06.430464+0000","last_undegraded":"2024-03-21T13:12:06.430464+0000","last_fullsized":"2024-03-21T13:12:06.430464+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:28.742139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:11:28.742139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:45.280576+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":440,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429203+0000","last_change":"2024-03-21T13:06:28.985227+0000","last_active":"2024-03-21T13:12:06.429203+0000","last_peered":"2024-03-21T13:12:06.429203+0000","last_clean":"2024-03-21T13:12:06.429203+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:12:06.429203+0000","last_undegraded":"2024-03-21T13:12:06.429203+0000","last_fullsized":"2024-03-21T13:12:06.429203+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:28.985145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:28.985145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":121,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.386040+0000","last_change":"2024-03-21T13:10:03.707422+0000","last_active":"2024-03-21T13:12:06.386040+0000","last_peered":"2024-03-21T13:12:06.386040+0000","last_clean":"2024-03-21T13:12:06.386040+0000","last_became_active":"2024-03-21T13:10:03.604616+0000","last_became_peered":"2024-03-21T13:10:03.604616+0000","last_unstale":"2024-03-21T13:12:06.386040+0000","last_undegraded":"2024-03-21T13:12:06.386040+0000","last_fullsized":"2024-03-21T13:12:06.386040+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:05:20.158184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:20.158184+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:07.821570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":548,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429166+0000","last_change":"2024-03-21T13:10:03.611239+0000","last_active":"2024-03-21T13:12:06.429166+0000","last_peered":"2024-03-21T13:12:06.429166+0000","last_clean":"2024-03-21T13:12:06.429166+0000","last_became_active":"2024-03-21T13:10:03.610319+0000","last_became_peered":"2024-03-21T13:10:03.610319+0000","last_unstale":"2024-03-21T13:12:06.429166+0000","last_undegraded":"2024-03-21T13:12:06.429166+0000","last_fullsized":"2024-03-21T13:12:06.429166+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:44.816722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:44.816722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":474,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428864+0000","last_change":"2024-03-21T13:10:03.613920+0000","last_active":"2024-03-21T13:12:06.428864+0000","last_peered":"2024-03-21T13:12:06.428864+0000","last_clean":"2024-03-21T13:12:06.428864+0000","last_became_active":"2024-03-21T13:10:03.612409+0000","last_became_peered":"2024-03-21T13:10:03.612409+0000","last_unstale":"2024-03-21T13:12:06.428864+0000","last_undegraded":"2024-03-21T13:12:06.428864+0000","last_fullsized":"2024-03-21T13:12:06.428864+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:56.868411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:56.868411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":591,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428851+0000","last_change":"2024-03-21T13:11:35.734699+0000","last_active":"2024-03-21T13:12:06.428851+0000","last_peered":"2024-03-21T13:12:06.428851+0000","last_clean":"2024-03-21T13:12:06.428851+0000","last_became_active":"2024-03-21T13:10:03.586563+0000","last_became_peered":"2024-03-21T13:10:03.586563+0000","last_unstale":"2024-03-21T13:12:06.428851+0000","last_undegraded":"2024-03-21T13:12:06.428851+0000","last_fullsized":"2024-03-21T13:12:06.428851+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:35.734553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:20.956778+0000","last_clean_scrub_stamp":"2024-03-21T13:11:35.734553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:48.178822+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":605,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429167+0000","last_change":"2024-03-21T13:09:02.714077+0000","last_active":"2024-03-21T13:12:06.429167+0000","last_peered":"2024-03-21T13:12:06.429167+0000","last_clean":"2024-03-21T13:12:06.429167+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:12:06.429167+0000","last_undegraded":"2024-03-21T13:12:06.429167+0000","last_fullsized":"2024-03-21T13:12:06.429167+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-21T13:09:02.713898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:02.713898+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":109,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.386142+0000","last_change":"2024-03-21T13:10:03.664415+0000","last_active":"2024-03-21T13:12:06.386142+0000","last_peered":"2024-03-21T13:12:06.386142+0000","last_clean":"2024-03-21T13:12:06.386142+0000","last_became_active":"2024-03-21T13:10:03.605708+0000","last_became_peered":"2024-03-21T13:10:03.605708+0000","last_unstale":"2024-03-21T13:12:06.386142+0000","last_undegraded":"2024-03-21T13:12:06.386142+0000","last_fullsized":"2024-03-21T13:12:06.386142+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:50.295057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:50.295057+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:07.821627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":716,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429193+0000","last_change":"2024-03-21T13:05:38.022572+0000","last_active":"2024-03-21T13:12:06.429193+0000","last_peered":"2024-03-21T13:12:06.429193+0000","last_clean":"2024-03-21T13:12:06.429193+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:12:06.429193+0000","last_undegraded":"2024-03-21T13:12:06.429193+0000","last_fullsized":"2024-03-21T13:12:06.429193+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:05:38.022439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.022439+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:12.350964+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":448,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428353+0000","last_change":"2024-03-21T13:08:53.540108+0000","last_active":"2024-03-21T13:12:06.428353+0000","last_peered":"2024-03-21T13:12:06.428353+0000","last_clean":"2024-03-21T13:12:06.428353+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:12:06.428353+0000","last_undegraded":"2024-03-21T13:12:06.428353+0000","last_fullsized":"2024-03-21T13:12:06.428353+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:08:53.539909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:53.539909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":498,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.431051+0000","last_change":"2024-03-21T13:11:16.033191+0000","last_active":"2024-03-21T13:12:06.431051+0000","last_peered":"2024-03-21T13:12:06.431051+0000","last_clean":"2024-03-21T13:12:06.431051+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:12:06.431051+0000","last_undegraded":"2024-03-21T13:12:06.431051+0000","last_fullsized":"2024-03-21T13:12:06.431051+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:11:16.033036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:16.033036+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:43.964982+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":443,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429954+0000","last_change":"2024-03-21T13:10:03.587701+0000","last_active":"2024-03-21T13:12:06.429954+0000","last_peered":"2024-03-21T13:12:06.429954+0000","last_clean":"2024-03-21T13:12:06.429954+0000","last_became_active":"2024-03-21T13:10:03.586620+0000","last_became_peered":"2024-03-21T13:10:03.586620+0000","last_unstale":"2024-03-21T13:12:06.429954+0000","last_undegraded":"2024-03-21T13:12:06.429954+0000","last_fullsized":"2024-03-21T13:12:06.429954+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:01.881154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:09:01.881154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.770606+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":461,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430898+0000","last_change":"2024-03-21T13:11:13.977186+0000","last_active":"2024-03-21T13:12:06.430898+0000","last_peered":"2024-03-21T13:12:06.430898+0000","last_clean":"2024-03-21T13:12:06.430898+0000","last_became_active":"2024-03-21T13:10:03.599880+0000","last_became_peered":"2024-03-21T13:10:03.599880+0000","last_unstale":"2024-03-21T13:12:06.430898+0000","last_undegraded":"2024-03-21T13:12:06.430898+0000","last_fullsized":"2024-03-21T13:12:06.430898+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:13.977069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:13.977069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:41.212846+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":567,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428292+0000","last_change":"2024-03-21T13:10:03.610999+0000","last_active":"2024-03-21T13:12:06.428292+0000","last_peered":"2024-03-21T13:12:06.428292+0000","last_clean":"2024-03-21T13:12:06.428292+0000","last_became_active":"2024-03-21T13:10:03.610489+0000","last_became_peered":"2024-03-21T13:10:03.610489+0000","last_unstale":"2024-03-21T13:12:06.428292+0000","last_undegraded":"2024-03-21T13:12:06.428292+0000","last_fullsized":"2024-03-21T13:12:06.428292+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:58.820007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:58.820007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"234'19","reported_seq":613,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428956+0000","last_change":"2024-03-21T13:05:49.982963+0000","last_active":"2024-03-21T13:12:06.428956+0000","last_peered":"2024-03-21T13:12:06.428956+0000","last_clean":"2024-03-21T13:12:06.428956+0000","last_became_active":"2024-03-21T13:02:26.254436+0000","last_became_peered":"2024-03-21T13:02:26.254436+0000","last_unstale":"2024-03-21T13:12:06.428956+0000","last_undegraded":"2024-03-21T13:12:06.428956+0000","last_fullsized":"2024-03-21T13:12:06.428956+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:49.982812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:49.982812+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":547,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.431765+0000","last_change":"2024-03-21T13:10:03.598837+0000","last_active":"2024-03-21T13:12:06.431765+0000","last_peered":"2024-03-21T13:12:06.431765+0000","last_clean":"2024-03-21T13:12:06.431765+0000","last_became_active":"2024-03-21T13:10:03.598068+0000","last_became_peered":"2024-03-21T13:10:03.598068+0000","last_unstale":"2024-03-21T13:12:06.431765+0000","last_undegraded":"2024-03-21T13:12:06.431765+0000","last_fullsized":"2024-03-21T13:12:06.431765+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:29.980750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:06:29.980750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.181988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":549,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.431001+0000","last_change":"2024-03-21T13:09:10.565138+0000","last_active":"2024-03-21T13:12:06.431001+0000","last_peered":"2024-03-21T13:12:06.431001+0000","last_clean":"2024-03-21T13:12:06.431001+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:12:06.431001+0000","last_undegraded":"2024-03-21T13:12:06.431001+0000","last_fullsized":"2024-03-21T13:12:06.431001+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:10.565030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.565030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":416,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429327+0000","last_change":"2024-03-21T13:05:28.499322+0000","last_active":"2024-03-21T13:12:06.429327+0000","last_peered":"2024-03-21T13:12:06.429327+0000","last_clean":"2024-03-21T13:12:06.429327+0000","last_became_active":"2024-03-21T13:02:26.242973+0000","last_became_peered":"2024-03-21T13:02:26.242973+0000","last_unstale":"2024-03-21T13:12:06.429327+0000","last_undegraded":"2024-03-21T13:12:06.429327+0000","last_fullsized":"2024-03-21T13:12:06.429327+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:28.499185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:28.499185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:12.989840+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":561,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428523+0000","last_change":"2024-03-21T13:08:18.740907+0000","last_active":"2024-03-21T13:12:06.428523+0000","last_peered":"2024-03-21T13:12:06.428523+0000","last_clean":"2024-03-21T13:12:06.428523+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:12:06.428523+0000","last_undegraded":"2024-03-21T13:12:06.428523+0000","last_fullsized":"2024-03-21T13:12:06.428523+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-21T13:08:18.740815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.740815+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":537,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428874+0000","last_change":"2024-03-21T13:11:33.943308+0000","last_active":"2024-03-21T13:12:06.428874+0000","last_peered":"2024-03-21T13:12:06.428874+0000","last_clean":"2024-03-21T13:12:06.428874+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:12:06.428874+0000","last_undegraded":"2024-03-21T13:12:06.428874+0000","last_fullsized":"2024-03-21T13:12:06.428874+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:33.943015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:33.943015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:42.054652+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":195,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428551+0000","last_change":"2024-03-21T13:08:20.955637+0000","last_active":"2024-03-21T13:12:06.428551+0000","last_peered":"2024-03-21T13:12:06.428551+0000","last_clean":"2024-03-21T13:12:06.428551+0000","last_became_active":"2024-03-21T13:08:20.955293+0000","last_became_peered":"2024-03-21T13:08:20.955293+0000","last_unstale":"2024-03-21T13:12:06.428551+0000","last_undegraded":"2024-03-21T13:12:06.428551+0000","last_fullsized":"2024-03-21T13:12:06.428551+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:03.434682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:03.434682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":550,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430750+0000","last_change":"2024-03-21T13:10:03.599838+0000","last_active":"2024-03-21T13:12:06.430750+0000","last_peered":"2024-03-21T13:12:06.430750+0000","last_clean":"2024-03-21T13:12:06.430750+0000","last_became_active":"2024-03-21T13:10:03.599520+0000","last_became_peered":"2024-03-21T13:10:03.599520+0000","last_unstale":"2024-03-21T13:12:06.430750+0000","last_undegraded":"2024-03-21T13:12:06.430750+0000","last_fullsized":"2024-03-21T13:12:06.430750+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:50.428251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:50.428251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":283,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428498+0000","last_change":"2024-03-21T13:06:29.532046+0000","last_active":"2024-03-21T13:12:06.428498+0000","last_peered":"2024-03-21T13:12:06.428498+0000","last_clean":"2024-03-21T13:12:06.428498+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:12:06.428498+0000","last_undegraded":"2024-03-21T13:12:06.428498+0000","last_fullsized":"2024-03-21T13:12:06.428498+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-21T13:05:40.262277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:40.262277+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":445,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428444+0000","last_change":"2024-03-21T13:08:20.069216+0000","last_active":"2024-03-21T13:12:06.428444+0000","last_peered":"2024-03-21T13:12:06.428444+0000","last_clean":"2024-03-21T13:12:06.428444+0000","last_became_active":"2024-03-21T13:02:26.252488+0000","last_became_peered":"2024-03-21T13:02:26.252488+0000","last_unstale":"2024-03-21T13:12:06.428444+0000","last_undegraded":"2024-03-21T13:12:06.428444+0000","last_fullsized":"2024-03-21T13:12:06.428444+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:20.069134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:20.069134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":545,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428750+0000","last_change":"2024-03-21T13:05:31.496412+0000","last_active":"2024-03-21T13:12:06.428750+0000","last_peered":"2024-03-21T13:12:06.428750+0000","last_clean":"2024-03-21T13:12:06.428750+0000","last_became_active":"2024-03-21T13:02:26.232682+0000","last_became_peered":"2024-03-21T13:02:26.232682+0000","last_unstale":"2024-03-21T13:12:06.428750+0000","last_undegraded":"2024-03-21T13:12:06.428750+0000","last_fullsized":"2024-03-21T13:12:06.428750+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.495639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.495639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":547,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430065+0000","last_change":"2024-03-21T13:07:48.347291+0000","last_active":"2024-03-21T13:12:06.430065+0000","last_peered":"2024-03-21T13:12:06.430065+0000","last_clean":"2024-03-21T13:12:06.430065+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:12:06.430065+0000","last_undegraded":"2024-03-21T13:12:06.430065+0000","last_fullsized":"2024-03-21T13:12:06.430065+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:48.347175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:48.347175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182125+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":554,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430037+0000","last_change":"2024-03-21T13:10:03.615159+0000","last_active":"2024-03-21T13:12:06.430037+0000","last_peered":"2024-03-21T13:12:06.430037+0000","last_clean":"2024-03-21T13:12:06.430037+0000","last_became_active":"2024-03-21T13:10:03.591857+0000","last_became_peered":"2024-03-21T13:10:03.591857+0000","last_unstale":"2024-03-21T13:12:06.430037+0000","last_undegraded":"2024-03-21T13:12:06.430037+0000","last_fullsized":"2024-03-21T13:12:06.430037+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:09:57.596217+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:09:57.596217+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":382,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.431782+0000","last_change":"2024-03-21T13:11:34.090139+0000","last_active":"2024-03-21T13:12:06.431782+0000","last_peered":"2024-03-21T13:12:06.431782+0000","last_clean":"2024-03-21T13:12:06.431782+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:12:06.431782+0000","last_undegraded":"2024-03-21T13:12:06.431782+0000","last_fullsized":"2024-03-21T13:12:06.431782+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:34.087671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:34.087671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:44.240480+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":416,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428627+0000","last_change":"2024-03-21T13:08:05.697231+0000","last_active":"2024-03-21T13:12:06.428627+0000","last_peered":"2024-03-21T13:12:06.428627+0000","last_clean":"2024-03-21T13:12:06.428627+0000","last_became_active":"2024-03-21T13:02:26.242491+0000","last_became_peered":"2024-03-21T13:02:26.242491+0000","last_unstale":"2024-03-21T13:12:06.428627+0000","last_undegraded":"2024-03-21T13:12:06.428627+0000","last_fullsized":"2024-03-21T13:12:06.428627+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:05.697141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:05.697141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":260,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429671+0000","last_change":"2024-03-21T13:08:18.389543+0000","last_active":"2024-03-21T13:12:06.429671+0000","last_peered":"2024-03-21T13:12:06.429671+0000","last_clean":"2024-03-21T13:12:06.429671+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:12:06.429671+0000","last_undegraded":"2024-03-21T13:12:06.429671+0000","last_fullsized":"2024-03-21T13:12:06.429671+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:18.389423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.389423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":664,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429646+0000","last_change":"2024-03-21T13:11:29.734394+0000","last_active":"2024-03-21T13:12:06.429646+0000","last_peered":"2024-03-21T13:12:06.429646+0000","last_clean":"2024-03-21T13:12:06.429646+0000","last_became_active":"2024-03-21T13:10:03.594316+0000","last_became_peered":"2024-03-21T13:10:03.594316+0000","last_unstale":"2024-03-21T13:12:06.429646+0000","last_undegraded":"2024-03-21T13:12:06.429646+0000","last_fullsized":"2024-03-21T13:12:06.429646+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:11:29.734236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:29.734236+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:40.247988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":448,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430604+0000","last_change":"2024-03-21T13:08:56.597251+0000","last_active":"2024-03-21T13:12:06.430604+0000","last_peered":"2024-03-21T13:12:06.430604+0000","last_clean":"2024-03-21T13:12:06.430604+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:12:06.430604+0000","last_undegraded":"2024-03-21T13:12:06.430604+0000","last_fullsized":"2024-03-21T13:12:06.430604+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:56.597146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:56.597146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182492+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"125'10","reported_seq":714,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:07.931096+0000","last_change":"2024-03-21T13:08:15.777606+0000","last_active":"2024-03-21T13:12:07.931096+0000","last_peered":"2024-03-21T13:12:07.931096+0000","last_clean":"2024-03-21T13:12:07.931096+0000","last_became_active":"2024-03-21T13:02:26.234494+0000","last_became_peered":"2024-03-21T13:02:26.234494+0000","last_unstale":"2024-03-21T13:12:07.931096+0000","last_undegraded":"2024-03-21T13:12:07.931096+0000","last_fullsized":"2024-03-21T13:12:07.931096+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:08:15.777522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:15.777522+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:12.990391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429005+0000","last_change":"2024-03-21T13:10:03.599799+0000","last_active":"2024-03-21T13:12:06.429005+0000","last_peered":"2024-03-21T13:12:06.429005+0000","last_clean":"2024-03-21T13:12:06.429005+0000","last_became_active":"2024-03-21T13:10:03.599265+0000","last_became_peered":"2024-03-21T13:10:03.599265+0000","last_unstale":"2024-03-21T13:12:06.429005+0000","last_undegraded":"2024-03-21T13:12:06.429005+0000","last_fullsized":"2024-03-21T13:12:06.429005+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:31.042372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:31.042372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.181930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'20","reported_seq":621,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429164+0000","last_change":"2024-03-21T13:10:03.707937+0000","last_active":"2024-03-21T13:12:06.429164+0000","last_peered":"2024-03-21T13:12:06.429164+0000","last_clean":"2024-03-21T13:12:06.429164+0000","last_became_active":"2024-03-21T13:10:03.613119+0000","last_became_peered":"2024-03-21T13:10:03.613119+0000","last_unstale":"2024-03-21T13:12:06.429164+0000","last_undegraded":"2024-03-21T13:12:06.429164+0000","last_fullsized":"2024-03-21T13:12:06.429164+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:23.379004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:23.379004+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":105,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.385525+0000","last_change":"2024-03-21T13:10:03.604060+0000","last_active":"2024-03-21T13:12:06.385525+0000","last_peered":"2024-03-21T13:12:06.385525+0000","last_clean":"2024-03-21T13:12:06.385525+0000","last_became_active":"2024-03-21T13:10:03.603547+0000","last_became_peered":"2024-03-21T13:10:03.603547+0000","last_unstale":"2024-03-21T13:12:06.385525+0000","last_undegraded":"2024-03-21T13:12:06.385525+0000","last_fullsized":"2024-03-21T13:12:06.385525+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:21.157295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:05:21.157295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:07.821589+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"272'10","reported_seq":725,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429189+0000","last_change":"2024-03-21T13:10:03.624386+0000","last_active":"2024-03-21T13:12:06.429189+0000","last_peered":"2024-03-21T13:12:06.429189+0000","last_clean":"2024-03-21T13:12:06.429189+0000","last_became_active":"2024-03-21T13:10:03.594651+0000","last_became_peered":"2024-03-21T13:10:03.594651+0000","last_unstale":"2024-03-21T13:12:06.429189+0000","last_undegraded":"2024-03-21T13:12:06.429189+0000","last_fullsized":"2024-03-21T13:12:06.429189+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-03-21T13:07:52.673289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:07:52.673289+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":448,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.431599+0000","last_change":"2024-03-21T13:08:21.365260+0000","last_active":"2024-03-21T13:12:06.431599+0000","last_peered":"2024-03-21T13:12:06.431599+0000","last_clean":"2024-03-21T13:12:06.431599+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:12:06.431599+0000","last_undegraded":"2024-03-21T13:12:06.431599+0000","last_fullsized":"2024-03-21T13:12:06.431599+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:08:21.365173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"234'15","reported_seq":476,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430429+0000","last_change":"2024-03-21T13:08:08.413361+0000","last_active":"2024-03-21T13:12:06.430429+0000","last_peered":"2024-03-21T13:12:06.430429+0000","last_clean":"2024-03-21T13:12:06.430429+0000","last_became_active":"2024-03-21T13:02:26.239161+0000","last_became_peered":"2024-03-21T13:02:26.239161+0000","last_unstale":"2024-03-21T13:12:06.430429+0000","last_undegraded":"2024-03-21T13:12:06.430429+0000","last_fullsized":"2024-03-21T13:12:06.430429+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-21T13:08:08.413234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:08.413234+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":105,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.385838+0000","last_change":"2024-03-21T13:10:03.614199+0000","last_active":"2024-03-21T13:12:06.385838+0000","last_peered":"2024-03-21T13:12:06.385838+0000","last_clean":"2024-03-21T13:12:06.385838+0000","last_became_active":"2024-03-21T13:10:03.613841+0000","last_became_peered":"2024-03-21T13:10:03.613841+0000","last_unstale":"2024-03-21T13:12:06.385838+0000","last_undegraded":"2024-03-21T13:12:06.385838+0000","last_fullsized":"2024-03-21T13:12:06.385838+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:58.785444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:08:58.785444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:07.821809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":547,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429460+0000","last_change":"2024-03-21T13:11:26.697833+0000","last_active":"2024-03-21T13:12:06.429460+0000","last_peered":"2024-03-21T13:12:06.429460+0000","last_clean":"2024-03-21T13:12:06.429460+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:12:06.429460+0000","last_undegraded":"2024-03-21T13:12:06.429460+0000","last_fullsized":"2024-03-21T13:12:06.429460+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:26.697728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:26.697728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:29.793406+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":561,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.431981+0000","last_change":"2024-03-21T13:09:13.694824+0000","last_active":"2024-03-21T13:12:06.431981+0000","last_peered":"2024-03-21T13:12:06.431981+0000","last_clean":"2024-03-21T13:12:06.431981+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:12:06.431981+0000","last_undegraded":"2024-03-21T13:12:06.431981+0000","last_fullsized":"2024-03-21T13:12:06.431981+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:09:13.694702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:13.694702+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182697+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'15","reported_seq":484,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429083+0000","last_change":"2024-03-21T13:09:10.698886+0000","last_active":"2024-03-21T13:12:06.429083+0000","last_peered":"2024-03-21T13:12:06.429083+0000","last_clean":"2024-03-21T13:12:06.429083+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:12:06.429083+0000","last_undegraded":"2024-03-21T13:12:06.429083+0000","last_fullsized":"2024-03-21T13:12:06.429083+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:09:10.698706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.698706+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":535,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429974+0000","last_change":"2024-03-21T13:07:52.425295+0000","last_active":"2024-03-21T13:12:06.429974+0000","last_peered":"2024-03-21T13:12:06.429974+0000","last_clean":"2024-03-21T13:12:06.429974+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:12:06.429974+0000","last_undegraded":"2024-03-21T13:12:06.429974+0000","last_fullsized":"2024-03-21T13:12:06.429974+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:52.424928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:07:52.424928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182097+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":530,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429295+0000","last_change":"2024-03-21T13:05:25.466812+0000","last_active":"2024-03-21T13:12:06.429295+0000","last_peered":"2024-03-21T13:12:06.429295+0000","last_clean":"2024-03-21T13:12:06.429295+0000","last_became_active":"2024-03-21T13:02:26.242502+0000","last_became_peered":"2024-03-21T13:02:26.242502+0000","last_unstale":"2024-03-21T13:12:06.429295+0000","last_undegraded":"2024-03-21T13:12:06.429295+0000","last_fullsized":"2024-03-21T13:12:06.429295+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:25.466695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:25.466695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:12.989921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":563,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429357+0000","last_change":"2024-03-21T13:11:34.524448+0000","last_active":"2024-03-21T13:12:06.429357+0000","last_peered":"2024-03-21T13:12:06.429357+0000","last_clean":"2024-03-21T13:12:06.429357+0000","last_became_active":"2024-03-21T13:02:26.252406+0000","last_became_peered":"2024-03-21T13:02:26.252406+0000","last_unstale":"2024-03-21T13:12:06.429357+0000","last_undegraded":"2024-03-21T13:12:06.429357+0000","last_fullsized":"2024-03-21T13:12:06.429357+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:34.524370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:34.524370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:13:00.539197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":605,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429409+0000","last_change":"2024-03-21T13:10:03.647440+0000","last_active":"2024-03-21T13:12:06.429409+0000","last_peered":"2024-03-21T13:12:06.429409+0000","last_clean":"2024-03-21T13:12:06.429409+0000","last_became_active":"2024-03-21T13:10:03.610810+0000","last_became_peered":"2024-03-21T13:10:03.610810+0000","last_unstale":"2024-03-21T13:12:06.429409+0000","last_undegraded":"2024-03-21T13:12:06.429409+0000","last_fullsized":"2024-03-21T13:12:06.429409+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-21T13:09:12.712840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:12.712840+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":434,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430125+0000","last_change":"2024-03-21T13:11:02.758245+0000","last_active":"2024-03-21T13:12:06.430125+0000","last_peered":"2024-03-21T13:12:06.430125+0000","last_clean":"2024-03-21T13:12:06.430125+0000","last_became_active":"2024-03-21T13:10:03.591230+0000","last_became_peered":"2024-03-21T13:10:03.591230+0000","last_unstale":"2024-03-21T13:12:06.430125+0000","last_undegraded":"2024-03-21T13:12:06.430125+0000","last_fullsized":"2024-03-21T13:12:06.430125+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:02.758017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:11:02.758017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:32.083279+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":592,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.431150+0000","last_change":"2024-03-21T13:11:31.040564+0000","last_active":"2024-03-21T13:12:06.431150+0000","last_peered":"2024-03-21T13:12:06.431150+0000","last_clean":"2024-03-21T13:12:06.431150+0000","last_became_active":"2024-03-21T13:06:29.487439+0000","last_became_peered":"2024-03-21T13:06:29.487439+0000","last_unstale":"2024-03-21T13:12:06.431150+0000","last_undegraded":"2024-03-21T13:12:06.431150+0000","last_fullsized":"2024-03-21T13:12:06.431150+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:11:31.040414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:31.040414+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:33.097785+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":500,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430389+0000","last_change":"2024-03-21T13:08:10.436112+0000","last_active":"2024-03-21T13:12:06.430389+0000","last_peered":"2024-03-21T13:12:06.430389+0000","last_clean":"2024-03-21T13:12:06.430389+0000","last_became_active":"2024-03-21T13:02:26.237800+0000","last_became_peered":"2024-03-21T13:02:26.237800+0000","last_unstale":"2024-03-21T13:12:06.430389+0000","last_undegraded":"2024-03-21T13:12:06.430389+0000","last_fullsized":"2024-03-21T13:12:06.430389+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-21T13:08:10.435742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:08:10.435742+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182338+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":715,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.432200+0000","last_change":"2024-03-21T13:08:53.619838+0000","last_active":"2024-03-21T13:12:06.432200+0000","last_peered":"2024-03-21T13:12:06.432200+0000","last_clean":"2024-03-21T13:12:06.432200+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:12:06.432200+0000","last_undegraded":"2024-03-21T13:12:06.432200+0000","last_fullsized":"2024-03-21T13:12:06.432200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:08:53.619709+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:08:53.619709+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182434+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":535,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428842+0000","last_change":"2024-03-21T13:05:43.596914+0000","last_active":"2024-03-21T13:12:06.428842+0000","last_peered":"2024-03-21T13:12:06.428842+0000","last_clean":"2024-03-21T13:12:06.428842+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:12:06.428842+0000","last_undegraded":"2024-03-21T13:12:06.428842+0000","last_fullsized":"2024-03-21T13:12:06.428842+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.596691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":562,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430839+0000","last_change":"2024-03-21T13:09:14.724844+0000","last_active":"2024-03-21T13:12:06.430839+0000","last_peered":"2024-03-21T13:12:06.430839+0000","last_clean":"2024-03-21T13:12:06.430839+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:12:06.430839+0000","last_undegraded":"2024-03-21T13:12:06.430839+0000","last_fullsized":"2024-03-21T13:12:06.430839+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:14.724681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:14.724681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"234'18","reported_seq":489,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428511+0000","last_change":"2024-03-21T13:07:48.778302+0000","last_active":"2024-03-21T13:12:06.428511+0000","last_peered":"2024-03-21T13:12:06.428511+0000","last_clean":"2024-03-21T13:12:06.428511+0000","last_became_active":"2024-03-21T13:02:14.347102+0000","last_became_peered":"2024-03-21T13:02:14.347102+0000","last_unstale":"2024-03-21T13:12:06.428511+0000","last_undegraded":"2024-03-21T13:12:06.428511+0000","last_fullsized":"2024-03-21T13:12:06.428511+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-21T13:07:48.778145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:07:48.778145+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":105,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.385189+0000","last_change":"2024-03-21T13:10:03.605416+0000","last_active":"2024-03-21T13:12:06.385189+0000","last_peered":"2024-03-21T13:12:06.385189+0000","last_clean":"2024-03-21T13:12:06.385189+0000","last_became_active":"2024-03-21T13:10:03.604015+0000","last_became_peered":"2024-03-21T13:10:03.604015+0000","last_unstale":"2024-03-21T13:12:06.385189+0000","last_undegraded":"2024-03-21T13:12:06.385189+0000","last_fullsized":"2024-03-21T13:12:06.385189+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:10.712191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.712191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:07.821855+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":556,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429989+0000","last_change":"2024-03-21T13:10:03.598806+0000","last_active":"2024-03-21T13:12:06.429989+0000","last_peered":"2024-03-21T13:12:06.429989+0000","last_clean":"2024-03-21T13:12:06.429989+0000","last_became_active":"2024-03-21T13:10:03.598366+0000","last_became_peered":"2024-03-21T13:10:03.598366+0000","last_unstale":"2024-03-21T13:12:06.429989+0000","last_undegraded":"2024-03-21T13:12:06.429989+0000","last_fullsized":"2024-03-21T13:12:06.429989+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:08:14.371718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182245+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":113,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.385746+0000","last_change":"2024-03-21T13:11:33.743407+0000","last_active":"2024-03-21T13:12:06.385746+0000","last_peered":"2024-03-21T13:12:06.385746+0000","last_clean":"2024-03-21T13:12:06.385746+0000","last_became_active":"2024-03-21T13:10:03.615067+0000","last_became_peered":"2024-03-21T13:10:03.615067+0000","last_unstale":"2024-03-21T13:12:06.385746+0000","last_undegraded":"2024-03-21T13:12:06.385746+0000","last_fullsized":"2024-03-21T13:12:06.385746+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:33.743269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:33.743269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:45.180207+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":562,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430331+0000","last_change":"2024-03-21T13:11:17.019489+0000","last_active":"2024-03-21T13:12:06.430331+0000","last_peered":"2024-03-21T13:12:06.430331+0000","last_clean":"2024-03-21T13:12:06.430331+0000","last_became_active":"2024-03-21T13:02:26.236780+0000","last_became_peered":"2024-03-21T13:02:26.236780+0000","last_unstale":"2024-03-21T13:12:06.430331+0000","last_undegraded":"2024-03-21T13:12:06.430331+0000","last_fullsized":"2024-03-21T13:12:06.430331+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.019382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.019382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:18.671834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'35","reported_seq":661,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428717+0000","last_change":"2024-03-21T13:10:02.601328+0000","last_active":"2024-03-21T13:12:06.428717+0000","last_peered":"2024-03-21T13:12:06.428717+0000","last_clean":"2024-03-21T13:12:06.428717+0000","last_became_active":"2024-03-21T13:02:26.240350+0000","last_became_peered":"2024-03-21T13:02:26.240350+0000","last_unstale":"2024-03-21T13:12:06.428717+0000","last_undegraded":"2024-03-21T13:12:06.428717+0000","last_fullsized":"2024-03-21T13:12:06.428717+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:10:02.601053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:10:02.601053+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":460,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.431891+0000","last_change":"2024-03-21T13:11:38.166199+0000","last_active":"2024-03-21T13:12:06.431891+0000","last_peered":"2024-03-21T13:12:06.431891+0000","last_clean":"2024-03-21T13:12:06.431891+0000","last_became_active":"2024-03-21T13:10:03.606655+0000","last_became_peered":"2024-03-21T13:10:03.606655+0000","last_unstale":"2024-03-21T13:12:06.431891+0000","last_undegraded":"2024-03-21T13:12:06.431891+0000","last_fullsized":"2024-03-21T13:12:06.431891+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:38.166077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:38.166077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:47.674560+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":105,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.386097+0000","last_change":"2024-03-21T13:10:03.603979+0000","last_active":"2024-03-21T13:12:06.386097+0000","last_peered":"2024-03-21T13:12:06.386097+0000","last_clean":"2024-03-21T13:12:06.386097+0000","last_became_active":"2024-03-21T13:10:03.603646+0000","last_became_peered":"2024-03-21T13:10:03.603646+0000","last_unstale":"2024-03-21T13:12:06.386097+0000","last_undegraded":"2024-03-21T13:12:06.386097+0000","last_fullsized":"2024-03-21T13:12:06.386097+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:12.031099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:12.031099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:07.821739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":414,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428969+0000","last_change":"2024-03-21T13:08:57.570445+0000","last_active":"2024-03-21T13:12:06.428969+0000","last_peered":"2024-03-21T13:12:06.428969+0000","last_clean":"2024-03-21T13:12:06.428969+0000","last_became_active":"2024-03-21T13:02:26.243974+0000","last_became_peered":"2024-03-21T13:02:26.243974+0000","last_unstale":"2024-03-21T13:12:06.428969+0000","last_undegraded":"2024-03-21T13:12:06.428969+0000","last_fullsized":"2024-03-21T13:12:06.428969+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:57.570303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.570303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"234'19","reported_seq":529,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428274+0000","last_change":"2024-03-21T13:11:09.615578+0000","last_active":"2024-03-21T13:12:06.428274+0000","last_peered":"2024-03-21T13:12:06.428274+0000","last_clean":"2024-03-21T13:12:06.428274+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:12:06.428274+0000","last_undegraded":"2024-03-21T13:12:06.428274+0000","last_fullsized":"2024-03-21T13:12:06.428274+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:11:09.615453+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:11:09.615453+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:14.465735+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":530,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429265+0000","last_change":"2024-03-21T13:09:11.661260+0000","last_active":"2024-03-21T13:12:06.429265+0000","last_peered":"2024-03-21T13:12:06.429265+0000","last_clean":"2024-03-21T13:12:06.429265+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:12:06.429265+0000","last_undegraded":"2024-03-21T13:12:06.429265+0000","last_fullsized":"2024-03-21T13:12:06.429265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:11.661130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:09:11.661130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":445,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428580+0000","last_change":"2024-03-21T13:05:43.961181+0000","last_active":"2024-03-21T13:12:06.428580+0000","last_peered":"2024-03-21T13:12:06.428580+0000","last_clean":"2024-03-21T13:12:06.428580+0000","last_became_active":"2024-03-21T13:02:26.243692+0000","last_became_peered":"2024-03-21T13:02:26.243692+0000","last_unstale":"2024-03-21T13:12:06.428580+0000","last_undegraded":"2024-03-21T13:12:06.428580+0000","last_fullsized":"2024-03-21T13:12:06.428580+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:05:43.961057+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":260,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428582+0000","last_change":"2024-03-21T13:10:00.606614+0000","last_active":"2024-03-21T13:12:06.428582+0000","last_peered":"2024-03-21T13:12:06.428582+0000","last_clean":"2024-03-21T13:12:06.428582+0000","last_became_active":"2024-03-21T13:06:45.410017+0000","last_became_peered":"2024-03-21T13:06:45.410017+0000","last_unstale":"2024-03-21T13:12:06.428582+0000","last_undegraded":"2024-03-21T13:12:06.428582+0000","last_fullsized":"2024-03-21T13:12:06.428582+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:10:00.606508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:10:00.606508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"272'43","reported_seq":162,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:07.931273+0000","last_change":"2024-03-21T13:10:04.094557+0000","last_active":"2024-03-21T13:12:07.931273+0000","last_peered":"2024-03-21T13:12:07.931273+0000","last_clean":"2024-03-21T13:12:07.931273+0000","last_became_active":"2024-03-21T13:10:03.603889+0000","last_became_peered":"2024-03-21T13:10:03.603889+0000","last_unstale":"2024-03-21T13:12:07.931273+0000","last_undegraded":"2024-03-21T13:12:07.931273+0000","last_fullsized":"2024-03-21T13:12:07.931273+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'43","last_scrub_stamp":"2024-03-21T13:10:04.094455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:10:04.094455+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":264,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429068+0000","last_change":"2024-03-21T13:09:07.643588+0000","last_active":"2024-03-21T13:12:06.429068+0000","last_peered":"2024-03-21T13:12:06.429068+0000","last_clean":"2024-03-21T13:12:06.429068+0000","last_became_active":"2024-03-21T13:06:45.406734+0000","last_became_peered":"2024-03-21T13:06:45.406734+0000","last_unstale":"2024-03-21T13:12:06.429068+0000","last_undegraded":"2024-03-21T13:12:06.429068+0000","last_fullsized":"2024-03-21T13:12:06.429068+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:07.643482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:09:07.643482+0000","last_clean_scrub_stamp":"2024-03-21T13:09:07.643482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":544,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430480+0000","last_change":"2024-03-21T13:08:58.723773+0000","last_active":"2024-03-21T13:12:06.430480+0000","last_peered":"2024-03-21T13:12:06.430480+0000","last_clean":"2024-03-21T13:12:06.430480+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:12:06.430480+0000","last_undegraded":"2024-03-21T13:12:06.430480+0000","last_fullsized":"2024-03-21T13:12:06.430480+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:58.723653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:58.723653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182548+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":106,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.385238+0000","last_change":"2024-03-21T13:10:03.604686+0000","last_active":"2024-03-21T13:12:06.385238+0000","last_peered":"2024-03-21T13:12:06.385238+0000","last_clean":"2024-03-21T13:12:06.385238+0000","last_became_active":"2024-03-21T13:10:03.604303+0000","last_became_peered":"2024-03-21T13:10:03.604303+0000","last_unstale":"2024-03-21T13:12:06.385238+0000","last_undegraded":"2024-03-21T13:12:06.385238+0000","last_fullsized":"2024-03-21T13:12:06.385238+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:15.590828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:15.590828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:07.821794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":106,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.385459+0000","last_change":"2024-03-21T13:10:03.605166+0000","last_active":"2024-03-21T13:12:06.385459+0000","last_peered":"2024-03-21T13:12:06.385459+0000","last_clean":"2024-03-21T13:12:06.385459+0000","last_became_active":"2024-03-21T13:10:03.604427+0000","last_became_peered":"2024-03-21T13:10:03.604427+0000","last_unstale":"2024-03-21T13:12:06.385459+0000","last_undegraded":"2024-03-21T13:12:06.385459+0000","last_fullsized":"2024-03-21T13:12:06.385459+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:46.263644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:05:46.263644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"243'20","reported_seq":491,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430826+0000","last_change":"2024-03-21T13:09:10.683920+0000","last_active":"2024-03-21T13:12:06.430826+0000","last_peered":"2024-03-21T13:12:06.430826+0000","last_clean":"2024-03-21T13:12:06.430826+0000","last_became_active":"2024-03-21T13:02:26.234230+0000","last_became_peered":"2024-03-21T13:02:26.234230+0000","last_unstale":"2024-03-21T13:12:06.430826+0000","last_undegraded":"2024-03-21T13:12:06.430826+0000","last_fullsized":"2024-03-21T13:12:06.430826+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'20","last_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_deep_scrub":"243'20","last_deep_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.683382+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182716+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":105,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.386386+0000","last_change":"2024-03-21T13:10:03.609881+0000","last_active":"2024-03-21T13:12:06.386386+0000","last_peered":"2024-03-21T13:12:06.386386+0000","last_clean":"2024-03-21T13:12:06.386386+0000","last_became_active":"2024-03-21T13:10:03.609581+0000","last_became_peered":"2024-03-21T13:10:03.609581+0000","last_unstale":"2024-03-21T13:12:06.386386+0000","last_undegraded":"2024-03-21T13:12:06.386386+0000","last_fullsized":"2024-03-21T13:12:06.386386+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:16.750077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:16.750077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:07.821840+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":422,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429437+0000","last_change":"2024-03-21T13:09:57.956198+0000","last_active":"2024-03-21T13:12:06.429437+0000","last_peered":"2024-03-21T13:12:06.429437+0000","last_clean":"2024-03-21T13:12:06.429437+0000","last_became_active":"2024-03-21T13:02:26.235640+0000","last_became_peered":"2024-03-21T13:02:26.235640+0000","last_unstale":"2024-03-21T13:12:06.429437+0000","last_undegraded":"2024-03-21T13:12:06.429437+0000","last_fullsized":"2024-03-21T13:12:06.429437+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:57.956080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:09:57.956080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":806,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430228+0000","last_change":"2024-03-21T13:08:07.454234+0000","last_active":"2024-03-21T13:12:06.430228+0000","last_peered":"2024-03-21T13:12:06.430228+0000","last_clean":"2024-03-21T13:12:06.430228+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:12:06.430228+0000","last_undegraded":"2024-03-21T13:12:06.430228+0000","last_fullsized":"2024-03-21T13:12:06.430228+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:08:07.454016+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182307+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":45,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.385141+0000","last_change":"2024-03-21T13:11:21.699418+0000","last_active":"2024-03-21T13:12:06.385141+0000","last_peered":"2024-03-21T13:12:06.385141+0000","last_clean":"2024-03-21T13:12:06.385141+0000","last_became_active":"2024-03-21T13:11:21.695629+0000","last_became_peered":"2024-03-21T13:11:21.695629+0000","last_unstale":"2024-03-21T13:12:06.385141+0000","last_undegraded":"2024-03-21T13:12:06.385141+0000","last_fullsized":"2024-03-21T13:12:06.385141+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:07:49.758485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:07:49.758485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:07.821723+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"234'36","reported_seq":757,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429247+0000","last_change":"2024-03-21T13:05:42.033620+0000","last_active":"2024-03-21T13:12:06.429247+0000","last_peered":"2024-03-21T13:12:06.429247+0000","last_clean":"2024-03-21T13:12:06.429247+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:12:06.429247+0000","last_undegraded":"2024-03-21T13:12:06.429247+0000","last_fullsized":"2024-03-21T13:12:06.429247+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'24","last_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:05:42.033266+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:12.350983+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":566,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429811+0000","last_change":"2024-03-21T13:11:11.001343+0000","last_active":"2024-03-21T13:12:06.429811+0000","last_peered":"2024-03-21T13:12:06.429811+0000","last_clean":"2024-03-21T13:12:06.429811+0000","last_became_active":"2024-03-21T13:10:03.599048+0000","last_became_peered":"2024-03-21T13:10:03.599048+0000","last_unstale":"2024-03-21T13:12:06.429811+0000","last_undegraded":"2024-03-21T13:12:06.429811+0000","last_fullsized":"2024-03-21T13:12:06.429811+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:11.001239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:11:11.001239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:36.367693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":424,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428742+0000","last_change":"2024-03-21T13:11:36.890901+0000","last_active":"2024-03-21T13:12:06.428742+0000","last_peered":"2024-03-21T13:12:06.428742+0000","last_clean":"2024-03-21T13:12:06.428742+0000","last_became_active":"2024-03-21T13:02:26.242708+0000","last_became_peered":"2024-03-21T13:02:26.242708+0000","last_unstale":"2024-03-21T13:12:06.428742+0000","last_undegraded":"2024-03-21T13:12:06.428742+0000","last_fullsized":"2024-03-21T13:12:06.428742+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:36.890785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:36.890785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:13:03.917278+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":456,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429792+0000","last_change":"2024-03-21T13:11:06.709643+0000","last_active":"2024-03-21T13:12:06.429792+0000","last_peered":"2024-03-21T13:12:06.429792+0000","last_clean":"2024-03-21T13:12:06.429792+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:12:06.429792+0000","last_undegraded":"2024-03-21T13:12:06.429792+0000","last_fullsized":"2024-03-21T13:12:06.429792+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:06.709498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:11:06.709498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:06.783266+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":1372,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:08.616194+0000","last_change":"2024-03-21T13:08:05.448565+0000","last_active":"2024-03-21T13:12:08.616194+0000","last_peered":"2024-03-21T13:12:08.616194+0000","last_clean":"2024-03-21T13:12:08.616194+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:12:08.616194+0000","last_undegraded":"2024-03-21T13:12:08.616194+0000","last_fullsized":"2024-03-21T13:12:08.616194+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:08:05.448403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:08:05.448403+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182276+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":802,"num_read_kb":797,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":673,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.387082+0000","last_change":"2024-03-21T13:11:17.444422+0000","last_active":"2024-03-21T13:12:06.387082+0000","last_peered":"2024-03-21T13:12:06.387082+0000","last_clean":"2024-03-21T13:12:06.387082+0000","last_became_active":"2024-03-21T13:10:03.606672+0000","last_became_peered":"2024-03-21T13:10:03.606672+0000","last_unstale":"2024-03-21T13:12:06.387082+0000","last_undegraded":"2024-03-21T13:12:06.387082+0000","last_fullsized":"2024-03-21T13:12:06.387082+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-21T13:11:17.443750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.443750+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:28.261075+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":202,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428665+0000","last_change":"2024-03-21T13:08:56.630611+0000","last_active":"2024-03-21T13:12:06.428665+0000","last_peered":"2024-03-21T13:12:06.428665+0000","last_clean":"2024-03-21T13:12:06.428665+0000","last_became_active":"2024-03-21T13:08:20.892193+0000","last_became_peered":"2024-03-21T13:08:20.892193+0000","last_unstale":"2024-03-21T13:12:06.428665+0000","last_undegraded":"2024-03-21T13:12:06.428665+0000","last_fullsized":"2024-03-21T13:12:06.428665+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:56.630492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:56.630492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":446,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.431145+0000","last_change":"2024-03-21T13:10:03.591632+0000","last_active":"2024-03-21T13:12:06.431145+0000","last_peered":"2024-03-21T13:12:06.431145+0000","last_clean":"2024-03-21T13:12:06.431145+0000","last_became_active":"2024-03-21T13:10:03.591088+0000","last_became_peered":"2024-03-21T13:10:03.591088+0000","last_unstale":"2024-03-21T13:12:06.431145+0000","last_undegraded":"2024-03-21T13:12:06.431145+0000","last_fullsized":"2024-03-21T13:12:06.431145+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:57.640892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:57.640892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":106,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.385979+0000","last_change":"2024-03-21T13:10:03.606230+0000","last_active":"2024-03-21T13:12:06.385979+0000","last_peered":"2024-03-21T13:12:06.385979+0000","last_clean":"2024-03-21T13:12:06.385979+0000","last_became_active":"2024-03-21T13:10:03.605469+0000","last_became_peered":"2024-03-21T13:10:03.605469+0000","last_unstale":"2024-03-21T13:12:06.385979+0000","last_undegraded":"2024-03-21T13:12:06.385979+0000","last_fullsized":"2024-03-21T13:12:06.385979+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:22.012892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:08:22.012892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":558,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429293+0000","last_change":"2024-03-21T13:10:03.649497+0000","last_active":"2024-03-21T13:12:06.429293+0000","last_peered":"2024-03-21T13:12:06.429293+0000","last_clean":"2024-03-21T13:12:06.429293+0000","last_became_active":"2024-03-21T13:10:03.598221+0000","last_became_peered":"2024-03-21T13:10:03.598221+0000","last_unstale":"2024-03-21T13:12:06.429293+0000","last_undegraded":"2024-03-21T13:12:06.429293+0000","last_fullsized":"2024-03-21T13:12:06.429293+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-21T13:09:15.777140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:15.777140+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.182632+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":568,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428823+0000","last_change":"2024-03-21T13:10:03.592045+0000","last_active":"2024-03-21T13:12:06.428823+0000","last_peered":"2024-03-21T13:12:06.428823+0000","last_clean":"2024-03-21T13:12:06.428823+0000","last_became_active":"2024-03-21T13:10:03.591328+0000","last_became_peered":"2024-03-21T13:10:03.591328+0000","last_unstale":"2024-03-21T13:12:06.428823+0000","last_undegraded":"2024-03-21T13:12:06.428823+0000","last_fullsized":"2024-03-21T13:12:06.428823+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:16.577170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:16.577170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":453,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428238+0000","last_change":"2024-03-21T13:10:03.597778+0000","last_active":"2024-03-21T13:12:06.428238+0000","last_peered":"2024-03-21T13:12:06.428238+0000","last_clean":"2024-03-21T13:12:06.428238+0000","last_became_active":"2024-03-21T13:10:03.595454+0000","last_became_peered":"2024-03-21T13:10:03.595454+0000","last_unstale":"2024-03-21T13:12:06.428238+0000","last_undegraded":"2024-03-21T13:12:06.428238+0000","last_fullsized":"2024-03-21T13:12:06.428238+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:08:55.487977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:08:55.487977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":426,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429019+0000","last_change":"2024-03-21T13:10:03.611967+0000","last_active":"2024-03-21T13:12:06.429019+0000","last_peered":"2024-03-21T13:12:06.429019+0000","last_clean":"2024-03-21T13:12:06.429019+0000","last_became_active":"2024-03-21T13:10:03.611677+0000","last_became_peered":"2024-03-21T13:10:03.611677+0000","last_unstale":"2024-03-21T13:12:06.429019+0000","last_undegraded":"2024-03-21T13:12:06.429019+0000","last_fullsized":"2024-03-21T13:12:06.429019+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:13.324701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:13.324701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"234'25","reported_seq":547,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428957+0000","last_change":"2024-03-21T13:11:37.750033+0000","last_active":"2024-03-21T13:12:06.428957+0000","last_peered":"2024-03-21T13:12:06.428957+0000","last_clean":"2024-03-21T13:12:06.428957+0000","last_became_active":"2024-03-21T13:10:03.594571+0000","last_became_peered":"2024-03-21T13:10:03.594571+0000","last_unstale":"2024-03-21T13:12:06.428957+0000","last_undegraded":"2024-03-21T13:12:06.428957+0000","last_fullsized":"2024-03-21T13:12:06.428957+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-21T13:11:37.749901+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:11:37.749901+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:13:01.612839+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"125'12","reported_seq":610,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430439+0000","last_change":"2024-03-21T13:08:18.149427+0000","last_active":"2024-03-21T13:12:06.430439+0000","last_peered":"2024-03-21T13:12:06.430439+0000","last_clean":"2024-03-21T13:12:06.430439+0000","last_became_active":"2024-03-21T13:02:26.253720+0000","last_became_peered":"2024-03-21T13:02:26.253720+0000","last_unstale":"2024-03-21T13:12:06.430439+0000","last_undegraded":"2024-03-21T13:12:06.430439+0000","last_fullsized":"2024-03-21T13:12:06.430439+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'12","last_scrub_stamp":"2024-03-21T13:08:18.149295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:08:18.149295+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:12.351052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":548,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429387+0000","last_change":"2024-03-21T13:10:03.606897+0000","last_active":"2024-03-21T13:12:06.429387+0000","last_peered":"2024-03-21T13:12:06.429387+0000","last_clean":"2024-03-21T13:12:06.429387+0000","last_became_active":"2024-03-21T13:10:03.606557+0000","last_became_peered":"2024-03-21T13:10:03.606557+0000","last_unstale":"2024-03-21T13:12:06.429387+0000","last_undegraded":"2024-03-21T13:12:06.429387+0000","last_fullsized":"2024-03-21T13:12:06.429387+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:41.951655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:41.951655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.181962+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":600,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.387182+0000","last_change":"2024-03-21T13:11:22.424995+0000","last_active":"2024-03-21T13:12:06.387182+0000","last_peered":"2024-03-21T13:12:06.387182+0000","last_clean":"2024-03-21T13:12:06.387182+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:12:06.387182+0000","last_undegraded":"2024-03-21T13:12:06.387182+0000","last_fullsized":"2024-03-21T13:12:06.387182+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:11:22.424870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:22.424870+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:50.879766+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'35","reported_seq":535,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430769+0000","last_change":"2024-03-21T13:08:54.645581+0000","last_active":"2024-03-21T13:12:06.430769+0000","last_peered":"2024-03-21T13:12:06.430769+0000","last_clean":"2024-03-21T13:12:06.430769+0000","last_became_active":"2024-03-21T13:02:26.227165+0000","last_became_peered":"2024-03-21T13:02:26.227165+0000","last_unstale":"2024-03-21T13:12:06.430769+0000","last_undegraded":"2024-03-21T13:12:06.430769+0000","last_fullsized":"2024-03-21T13:12:06.430769+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:08:54.645360+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:08:54.645360+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"226'10","reported_seq":717,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428149+0000","last_change":"2024-03-21T13:08:16.739276+0000","last_active":"2024-03-21T13:12:06.428149+0000","last_peered":"2024-03-21T13:12:06.428149+0000","last_clean":"2024-03-21T13:12:06.428149+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:12:06.428149+0000","last_undegraded":"2024-03-21T13:12:06.428149+0000","last_fullsized":"2024-03-21T13:12:06.428149+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:08:16.739190+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":467,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428662+0000","last_change":"2024-03-21T13:10:03.610862+0000","last_active":"2024-03-21T13:12:06.428662+0000","last_peered":"2024-03-21T13:12:06.428662+0000","last_clean":"2024-03-21T13:12:06.428662+0000","last_became_active":"2024-03-21T13:10:03.610471+0000","last_became_peered":"2024-03-21T13:10:03.610471+0000","last_unstale":"2024-03-21T13:12:06.428662+0000","last_undegraded":"2024-03-21T13:12:06.428662+0000","last_fullsized":"2024-03-21T13:12:06.428662+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:18.416262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:05:18.416262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":550,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429347+0000","last_change":"2024-03-21T13:09:55.872269+0000","last_active":"2024-03-21T13:12:06.429347+0000","last_peered":"2024-03-21T13:12:06.429347+0000","last_clean":"2024-03-21T13:12:06.429347+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:12:06.429347+0000","last_undegraded":"2024-03-21T13:12:06.429347+0000","last_fullsized":"2024-03-21T13:12:06.429347+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:55.869919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:09:55.869919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"234'30","reported_seq":588,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428563+0000","last_change":"2024-03-21T13:05:12.322522+0000","last_active":"2024-03-21T13:12:06.428563+0000","last_peered":"2024-03-21T13:12:06.428563+0000","last_clean":"2024-03-21T13:12:06.428563+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:12:06.428563+0000","last_undegraded":"2024-03-21T13:12:06.428563+0000","last_fullsized":"2024-03-21T13:12:06.428563+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-21T13:05:12.322284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:12.322284+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":550,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429553+0000","last_change":"2024-03-21T13:11:22.759665+0000","last_active":"2024-03-21T13:12:06.429553+0000","last_peered":"2024-03-21T13:12:06.429553+0000","last_clean":"2024-03-21T13:12:06.429553+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:12:06.429553+0000","last_undegraded":"2024-03-21T13:12:06.429553+0000","last_fullsized":"2024-03-21T13:12:06.429553+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:22.759551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:22.759551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:48.284181+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":105,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.385541+0000","last_change":"2024-03-21T13:10:03.604456+0000","last_active":"2024-03-21T13:12:06.385541+0000","last_peered":"2024-03-21T13:12:06.385541+0000","last_clean":"2024-03-21T13:12:06.385541+0000","last_became_active":"2024-03-21T13:10:03.604092+0000","last_became_peered":"2024-03-21T13:10:03.604092+0000","last_unstale":"2024-03-21T13:12:06.385541+0000","last_undegraded":"2024-03-21T13:12:06.385541+0000","last_fullsized":"2024-03-21T13:12:06.385541+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:47.328925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:47.328925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:07.821666+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"272'12","reported_seq":613,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.387528+0000","last_change":"2024-03-21T13:11:18.400404+0000","last_active":"2024-03-21T13:12:06.387528+0000","last_peered":"2024-03-21T13:12:06.387528+0000","last_clean":"2024-03-21T13:12:06.387528+0000","last_became_active":"2024-03-21T13:10:03.606886+0000","last_became_peered":"2024-03-21T13:10:03.606886+0000","last_unstale":"2024-03-21T13:12:06.387528+0000","last_undegraded":"2024-03-21T13:12:06.387528+0000","last_fullsized":"2024-03-21T13:12:06.387528+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'12","last_scrub_stamp":"2024-03-21T13:11:18.400270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:18.400270+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:29.795823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":556,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428453+0000","last_change":"2024-03-21T13:09:15.640885+0000","last_active":"2024-03-21T13:12:06.428453+0000","last_peered":"2024-03-21T13:12:06.428453+0000","last_clean":"2024-03-21T13:12:06.428453+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:12:06.428453+0000","last_undegraded":"2024-03-21T13:12:06.428453+0000","last_fullsized":"2024-03-21T13:12:06.428453+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:15.640781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:09:15.640781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"234'34","reported_seq":511,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429356+0000","last_change":"2024-03-21T13:08:22.736499+0000","last_active":"2024-03-21T13:12:06.429356+0000","last_peered":"2024-03-21T13:12:06.429356+0000","last_clean":"2024-03-21T13:12:06.429356+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:12:06.429356+0000","last_undegraded":"2024-03-21T13:12:06.429356+0000","last_fullsized":"2024-03-21T13:12:06.429356+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:08:22.736371+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":540,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428839+0000","last_change":"2024-03-21T13:05:39.518051+0000","last_active":"2024-03-21T13:12:06.428839+0000","last_peered":"2024-03-21T13:12:06.428839+0000","last_clean":"2024-03-21T13:12:06.428839+0000","last_became_active":"2024-03-21T13:02:26.243985+0000","last_became_peered":"2024-03-21T13:02:26.243985+0000","last_unstale":"2024-03-21T13:12:06.428839+0000","last_undegraded":"2024-03-21T13:12:06.428839+0000","last_fullsized":"2024-03-21T13:12:06.428839+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:05:39.517591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:39.517591+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":450,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.431523+0000","last_change":"2024-03-21T13:11:27.742914+0000","last_active":"2024-03-21T13:12:06.431523+0000","last_peered":"2024-03-21T13:12:06.431523+0000","last_clean":"2024-03-21T13:12:06.431523+0000","last_became_active":"2024-03-21T13:10:03.594379+0000","last_became_peered":"2024-03-21T13:10:03.594379+0000","last_unstale":"2024-03-21T13:12:06.431523+0000","last_undegraded":"2024-03-21T13:12:06.431523+0000","last_fullsized":"2024-03-21T13:12:06.431523+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:27.742792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:27.742792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:52.061542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":578,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429640+0000","last_change":"2024-03-21T13:10:03.725464+0000","last_active":"2024-03-21T13:12:06.429640+0000","last_peered":"2024-03-21T13:12:06.429640+0000","last_clean":"2024-03-21T13:12:06.429640+0000","last_became_active":"2024-03-21T13:10:03.586593+0000","last_became_peered":"2024-03-21T13:10:03.586593+0000","last_unstale":"2024-03-21T13:12:06.429640+0000","last_undegraded":"2024-03-21T13:12:06.429640+0000","last_fullsized":"2024-03-21T13:12:06.429640+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:10:03.725389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:10:03.725389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:10.770650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":310,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428669+0000","last_change":"2024-03-21T13:11:00.776320+0000","last_active":"2024-03-21T13:12:06.428669+0000","last_peered":"2024-03-21T13:12:06.428669+0000","last_clean":"2024-03-21T13:12:06.428669+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:12:06.428669+0000","last_undegraded":"2024-03-21T13:12:06.428669+0000","last_fullsized":"2024-03-21T13:12:06.428669+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:11:00.775958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:00.775958+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:19.616562+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":113,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.385618+0000","last_change":"2024-03-21T13:11:35.902330+0000","last_active":"2024-03-21T13:12:06.385618+0000","last_peered":"2024-03-21T13:12:06.385618+0000","last_clean":"2024-03-21T13:12:06.385618+0000","last_became_active":"2024-03-21T13:10:03.602752+0000","last_became_peered":"2024-03-21T13:10:03.602752+0000","last_unstale":"2024-03-21T13:12:06.385618+0000","last_undegraded":"2024-03-21T13:12:06.385618+0000","last_fullsized":"2024-03-21T13:12:06.385618+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:35.902195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:11:35.902195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:35.938985+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":450,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.430896+0000","last_change":"2024-03-21T13:11:02.693027+0000","last_active":"2024-03-21T13:12:06.430896+0000","last_peered":"2024-03-21T13:12:06.430896+0000","last_clean":"2024-03-21T13:12:06.430896+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:12:06.430896+0000","last_undegraded":"2024-03-21T13:12:06.430896+0000","last_fullsized":"2024-03-21T13:12:06.430896+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:02.692916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:02.692916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:08.411083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":564,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.429835+0000","last_change":"2024-03-21T13:06:30.274121+0000","last_active":"2024-03-21T13:12:06.429835+0000","last_peered":"2024-03-21T13:12:06.429835+0000","last_clean":"2024-03-21T13:12:06.429835+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:12:06.429835+0000","last_undegraded":"2024-03-21T13:12:06.429835+0000","last_fullsized":"2024-03-21T13:12:06.429835+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:06:30.274033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:06:30.274033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"234'31","reported_seq":123,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.385500+0000","last_change":"2024-03-21T13:10:03.634167+0000","last_active":"2024-03-21T13:12:06.385500+0000","last_peered":"2024-03-21T13:12:06.385500+0000","last_clean":"2024-03-21T13:12:06.385500+0000","last_became_active":"2024-03-21T13:10:03.591832+0000","last_became_peered":"2024-03-21T13:10:03.591832+0000","last_unstale":"2024-03-21T13:12:06.385500+0000","last_undegraded":"2024-03-21T13:12:06.385500+0000","last_fullsized":"2024-03-21T13:12:06.385500+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'31","last_scrub_stamp":"2024-03-21T13:09:03.608302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:09:03.608302+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:07.821825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":105,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.386366+0000","last_change":"2024-03-21T13:10:03.609956+0000","last_active":"2024-03-21T13:12:06.386366+0000","last_peered":"2024-03-21T13:12:06.386366+0000","last_clean":"2024-03-21T13:12:06.386366+0000","last_became_active":"2024-03-21T13:10:03.609620+0000","last_became_peered":"2024-03-21T13:10:03.609620+0000","last_unstale":"2024-03-21T13:12:06.386366+0000","last_undegraded":"2024-03-21T13:12:06.386366+0000","last_fullsized":"2024-03-21T13:12:06.386366+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:34.291369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:05:34.291369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:07.821644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"272'12","reported_seq":135,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.385443+0000","last_change":"2024-03-21T13:10:03.654783+0000","last_active":"2024-03-21T13:12:06.385443+0000","last_peered":"2024-03-21T13:12:06.385443+0000","last_clean":"2024-03-21T13:12:06.385443+0000","last_became_active":"2024-03-21T13:10:03.609698+0000","last_became_peered":"2024-03-21T13:10:03.609698+0000","last_unstale":"2024-03-21T13:12:06.385443+0000","last_undegraded":"2024-03-21T13:12:06.385443+0000","last_fullsized":"2024-03-21T13:12:06.385443+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'10","last_scrub_stamp":"2024-03-21T13:05:30.300605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:05:30.300605+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":542,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428696+0000","last_change":"2024-03-21T13:05:38.492137+0000","last_active":"2024-03-21T13:12:06.428696+0000","last_peered":"2024-03-21T13:12:06.428696+0000","last_clean":"2024-03-21T13:12:06.428696+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:12:06.428696+0000","last_undegraded":"2024-03-21T13:12:06.428696+0000","last_fullsized":"2024-03-21T13:12:06.428696+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:05:38.492015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:05:38.492015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:12.990105+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"234'20","reported_seq":503,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-21T13:12:06.428962+0000","last_change":"2024-03-21T13:05:22.406498+0000","last_active":"2024-03-21T13:12:06.428962+0000","last_peered":"2024-03-21T13:12:06.428962+0000","last_clean":"2024-03-21T13:12:06.428962+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:12:06.428962+0000","last_undegraded":"2024-03-21T13:12:06.428962+0000","last_fullsized":"2024-03-21T13:12:06.428962+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-21T13:05:22.406367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:05:22.406367+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":802,"num_read_kb":797,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1108,"num_read_kb":865,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":285,"num_bytes_recovered":5776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":730,"ondisk_log_size":730,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168743,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":950476,"kb_used_data":220996,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":92794676,"statfs":{"total":95995035648,"available":95021748224,"internally_reserved":0,"allocated":226299904,"data_stored":225799243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":746977738},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463085,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":708884,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93036268,"statfs":{"total":95995035648,"available":95269138432,"internally_reserved":0,"allocated":1646592,"data_stored":1142011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":724236742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575046,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738571,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":665736,"kb_used_data":80640,"kb_used_omap":1,"kb_used_meta":585086,"kb_avail":93079416,"statfs":{"total":95995035648,"available":95313321984,"internally_reserved":0,"allocated":82575360,"data_stored":82078170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":599128517},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411521,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636076,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93109076,"statfs":{"total":95995035648,"available":95343693824,"internally_reserved":0,"allocated":1015808,"data_stored":550320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182588,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":830504,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92914648,"statfs":{"total":95995035648,"available":95144599552,"internally_reserved":0,"allocated":1077248,"data_stored":551997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":849344967},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019062,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":748536,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":747518,"kb_avail":92996616,"statfs":{"total":95995035648,"available":95228534784,"internally_reserved":0,"allocated":1028096,"data_stored":550799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":765458885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724722,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:12:18.116 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-21T13:12:18.117 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:12:18.117 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-21T13:12:18.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd in 5 2024-03-21T13:12:18.195 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:18.218 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:18.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:18.320 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:18.344 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:18.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:18.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:18.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:18.453 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'unique_pool_3' created 2024-03-21T13:12:18.474 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:18.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:18.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:18.603 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:18.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:18.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:18.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:18.730 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:18.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:18.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:18 smithi204 ceph-mon[33153]: pgmap v636: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 935 B/s rd, 0 op/s 2024-03-21T13:12:18.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:18 smithi204 ceph-mon[33153]: 4.c scrub starts 2024-03-21T13:12:18.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:18 smithi204 ceph-mon[33153]: 4.c scrub ok 2024-03-21T13:12:18.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:18 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3714270334' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-21T13:12:18.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:18 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-21T13:12:18.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:18 smithi204 ceph-mon[33153]: 3.13 scrub starts 2024-03-21T13:12:18.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:18 smithi204 ceph-mon[33153]: 3.13 scrub ok 2024-03-21T13:12:18.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[29621]: pgmap v636: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 935 B/s rd, 0 op/s 2024-03-21T13:12:18.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[29621]: 4.c scrub starts 2024-03-21T13:12:18.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[29621]: 4.c scrub ok 2024-03-21T13:12:18.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3714270334' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-21T13:12:18.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-21T13:12:18.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[29621]: 3.13 scrub starts 2024-03-21T13:12:18.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[29621]: 3.13 scrub ok 2024-03-21T13:12:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[40675]: pgmap v636: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 935 B/s rd, 0 op/s 2024-03-21T13:12:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[40675]: 4.c scrub starts 2024-03-21T13:12:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[40675]: 4.c scrub ok 2024-03-21T13:12:18.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3714270334' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-21T13:12:18.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-21T13:12:18.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[40675]: 3.13 scrub starts 2024-03-21T13:12:18.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:18 smithi163 ceph-mon[40675]: 3.13 scrub ok 2024-03-21T13:12:18.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:18.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:18.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:18.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:18.959 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:18.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:18.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:19.086 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:19.111 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:19.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:19.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:19.213 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:19.237 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:19.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:19.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:19.365 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:19.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:19.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:19.415 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:12:19.467 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:19.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:19.491 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:19.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:19.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:19.618 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:19.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:19.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:19.634 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:19 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-21T13:12:19.635 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:19 smithi163 ceph-mon[29621]: osdmap e324: 8 total, 8 up, 6 in 2024-03-21T13:12:19.635 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:19 smithi163 ceph-mon[29621]: 5.d scrub starts 2024-03-21T13:12:19.635 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:19 smithi163 ceph-mon[29621]: 5.d scrub ok 2024-03-21T13:12:19.635 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:19 smithi163 ceph-mon[29621]: 6.3 scrub starts 2024-03-21T13:12:19.635 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:19 smithi163 ceph-mon[29621]: 6.3 scrub ok 2024-03-21T13:12:19.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:19 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-21T13:12:19.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:19 smithi163 ceph-mon[40675]: osdmap e324: 8 total, 8 up, 6 in 2024-03-21T13:12:19.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:19 smithi163 ceph-mon[40675]: 5.d scrub starts 2024-03-21T13:12:19.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:19 smithi163 ceph-mon[40675]: 5.d scrub ok 2024-03-21T13:12:19.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:19 smithi163 ceph-mon[40675]: 6.3 scrub starts 2024-03-21T13:12:19.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:19 smithi163 ceph-mon[40675]: 6.3 scrub ok 2024-03-21T13:12:19.720 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:19.745 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:19.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:19 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-21T13:12:19.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:19 smithi204 ceph-mon[33153]: osdmap e324: 8 total, 8 up, 6 in 2024-03-21T13:12:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:19 smithi204 ceph-mon[33153]: 5.d scrub starts 2024-03-21T13:12:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:19 smithi204 ceph-mon[33153]: 5.d scrub ok 2024-03-21T13:12:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:19 smithi204 ceph-mon[33153]: 6.3 scrub starts 2024-03-21T13:12:19.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:19 smithi204 ceph-mon[33153]: 6.3 scrub ok 2024-03-21T13:12:19.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:19.875 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:19.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:19.976 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:20.001 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:20.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:20.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:20.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:20.130 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:20.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:20.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:20.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:20.258 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:20.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:20.359 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:20.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:20.384 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:20.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:20.485 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:20.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:20.511 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:20.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:20.612 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:20.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:20.639 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:20.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:20.716 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[29621]: pgmap v638: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:20.716 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[29621]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:12:20.716 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[29621]: osdmap e325: 8 total, 8 up, 6 in 2024-03-21T13:12:20.716 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[29621]: 6.1c scrub starts 2024-03-21T13:12:20.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[29621]: 6.1c scrub ok 2024-03-21T13:12:20.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:12:20.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:20.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[40675]: pgmap v638: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:20.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[40675]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:12:20.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[40675]: osdmap e325: 8 total, 8 up, 6 in 2024-03-21T13:12:20.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[40675]: 6.1c scrub starts 2024-03-21T13:12:20.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[40675]: 6.1c scrub ok 2024-03-21T13:12:20.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:12:20.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:20.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:20.743 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-21T13:12:20.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:20 smithi204 ceph-mon[33153]: pgmap v638: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:20 smithi204 ceph-mon[33153]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:12:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:20 smithi204 ceph-mon[33153]: osdmap e325: 8 total, 8 up, 6 in 2024-03-21T13:12:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:20 smithi204 ceph-mon[33153]: 6.1c scrub starts 2024-03-21T13:12:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:20 smithi204 ceph-mon[33153]: 6.1c scrub ok 2024-03-21T13:12:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:12:20.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:20.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:20.766 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:20.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:20.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:20.894 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:20.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:20.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:21.021 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:21.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:21.123 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:21.148 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:21.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:21.249 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:21.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:21.276 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:21.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:21.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:21.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:21.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:21.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:21.505 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:21.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:21.531 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:21.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:21.632 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:21.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:21.658 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:21.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:21 smithi204 ceph-mon[33153]: osdmap e326: 8 total, 8 up, 6 in 2024-03-21T13:12:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:21 smithi204 ceph-mon[33153]: 3.1 scrub starts 2024-03-21T13:12:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:21 smithi204 ceph-mon[33153]: 3.1 scrub ok 2024-03-21T13:12:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:21 smithi204 ceph-mon[33153]: 6.e scrub starts 2024-03-21T13:12:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:21 smithi204 ceph-mon[33153]: 6.e scrub ok 2024-03-21T13:12:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:21 smithi204 ceph-mon[33153]: 6.12 scrub starts 2024-03-21T13:12:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:21 smithi204 ceph-mon[33153]: 6.12 scrub ok 2024-03-21T13:12:21.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:21 smithi204 ceph-mon[33153]: osdmap e327: 8 total, 8 up, 6 in 2024-03-21T13:12:21.759 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[29621]: osdmap e326: 8 total, 8 up, 6 in 2024-03-21T13:12:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[29621]: 3.1 scrub starts 2024-03-21T13:12:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[29621]: 3.1 scrub ok 2024-03-21T13:12:21.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[29621]: 6.e scrub starts 2024-03-21T13:12:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[29621]: 6.e scrub ok 2024-03-21T13:12:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[29621]: 6.12 scrub starts 2024-03-21T13:12:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[29621]: 6.12 scrub ok 2024-03-21T13:12:21.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[29621]: osdmap e327: 8 total, 8 up, 6 in 2024-03-21T13:12:21.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[40675]: osdmap e326: 8 total, 8 up, 6 in 2024-03-21T13:12:21.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[40675]: 3.1 scrub starts 2024-03-21T13:12:21.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[40675]: 3.1 scrub ok 2024-03-21T13:12:21.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[40675]: 6.e scrub starts 2024-03-21T13:12:21.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[40675]: 6.e scrub ok 2024-03-21T13:12:21.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[40675]: 6.12 scrub starts 2024-03-21T13:12:21.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[40675]: 6.12 scrub ok 2024-03-21T13:12:21.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:21 smithi163 ceph-mon[40675]: osdmap e327: 8 total, 8 up, 6 in 2024-03-21T13:12:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:21.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:21.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:21.848 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:21.849 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:21.856 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:21.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:21.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:21.913 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:21.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:22.014 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:22.042 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:22.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:22.143 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:22.169 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:22.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:22.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:12:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:12:22.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:22.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:22.295 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:22.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:22.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:22.420 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:22.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:22.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:22.473 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:12:22.522 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:22.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:22.548 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:22.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:22.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:22.665 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:22 smithi163 ceph-mon[40675]: pgmap v641: 148 pgs: 1 unknown, 147 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:12:22.665 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:22 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/984021026' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:12:22.665 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:22 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:12:22.666 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:22 smithi163 ceph-mon[29621]: pgmap v641: 148 pgs: 1 unknown, 147 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:12:22.666 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:22 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/984021026' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:12:22.666 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:22 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:12:22.678 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:22.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:22.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:22 smithi204 ceph-mon[33153]: pgmap v641: 148 pgs: 1 unknown, 147 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:12:22.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:22 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/984021026' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:12:22.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:22 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:12:22.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:22.810 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:22.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:22.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:22.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:22.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:23.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:23.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:23.071 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:23.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:23.172 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:23.199 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:23.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:23.300 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:23.325 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:23.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:23.427 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:23.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:23.456 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:23.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:23.557 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:23.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:23.583 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:23.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:23.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:23 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:23.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:23 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:12:23.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:23 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:12:23.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:23 smithi163 ceph-mon[40675]: osdmap e328: 8 total, 8 up, 6 in 2024-03-21T13:12:23.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:23 smithi163 ceph-mon[40675]: 5.c scrub starts 2024-03-21T13:12:23.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:23 smithi163 ceph-mon[40675]: 5.c scrub ok 2024-03-21T13:12:23.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:23 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:23.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:23 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:12:23.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:23 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:12:23.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:23 smithi163 ceph-mon[29621]: osdmap e328: 8 total, 8 up, 6 in 2024-03-21T13:12:23.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:23 smithi163 ceph-mon[29621]: 5.c scrub starts 2024-03-21T13:12:23.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:23 smithi163 ceph-mon[29621]: 5.c scrub ok 2024-03-21T13:12:23.684 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:23.709 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:23.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:23.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:23 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:23.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:23 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:12:23.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:23 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:12:23.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:23 smithi204 ceph-mon[33153]: osdmap e328: 8 total, 8 up, 6 in 2024-03-21T13:12:23.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:23 smithi204 ceph-mon[33153]: 5.c scrub starts 2024-03-21T13:12:23.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:23 smithi204 ceph-mon[33153]: 5.c scrub ok 2024-03-21T13:12:23.810 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:23.836 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:23.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:23.937 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:23.961 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:23.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:23.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:24.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:24.091 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:24.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:24.193 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:24.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:24.218 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:24.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:24.319 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:24.343 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:24.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:24.445 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:24.468 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:24.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:24.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:24.571 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:24.597 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:24.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:24.699 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:24.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:24.723 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:24.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:24 smithi204 ceph-mon[33153]: pgmap v644: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:24.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:24 smithi204 ceph-mon[33153]: 3.17 deep-scrub starts 2024-03-21T13:12:24.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:24 smithi204 ceph-mon[33153]: 3.17 deep-scrub ok 2024-03-21T13:12:24.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:24 smithi204 ceph-mon[33153]: 6.1e deep-scrub starts 2024-03-21T13:12:24.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:24 smithi204 ceph-mon[33153]: 6.1e deep-scrub ok 2024-03-21T13:12:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:24 smithi163 ceph-mon[29621]: pgmap v644: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:24 smithi163 ceph-mon[29621]: 3.17 deep-scrub starts 2024-03-21T13:12:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:24 smithi163 ceph-mon[29621]: 3.17 deep-scrub ok 2024-03-21T13:12:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:24 smithi163 ceph-mon[29621]: 6.1e deep-scrub starts 2024-03-21T13:12:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:24 smithi163 ceph-mon[29621]: 6.1e deep-scrub ok 2024-03-21T13:12:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:24 smithi163 ceph-mon[40675]: pgmap v644: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:24 smithi163 ceph-mon[40675]: 3.17 deep-scrub starts 2024-03-21T13:12:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:24 smithi163 ceph-mon[40675]: 3.17 deep-scrub ok 2024-03-21T13:12:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:24 smithi163 ceph-mon[40675]: 6.1e deep-scrub starts 2024-03-21T13:12:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:24 smithi163 ceph-mon[40675]: 6.1e deep-scrub ok 2024-03-21T13:12:24.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:24.854 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:24.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:24.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:24.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:24.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:24.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:25.085 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:25.110 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:25.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:25.211 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:25.236 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:25.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:25.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:25.365 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:25.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:25.468 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:25.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:25.492 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:25.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:25.593 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:25.618 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:25.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:25.663 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:12:25.719 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:25.744 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:25.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:25.845 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:25.871 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:25.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:25.955 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:12:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:12:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:12:25.973 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:25.997 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:25.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:25.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:26.099 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:26.124 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:26.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:26.198 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:12:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-03-21T13:12:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-03-21T13:12:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:12:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175770 7.e 7:78fafcef:::benchmark_data_smithi163_80312_object58329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T13:12:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.228931+0000", 2024-03-21T13:12:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 541.96758188199999, 2024-03-21T13:12:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0743425600000001, 2024-03-21T13:12:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:12:26.199 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:12:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:12:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:12:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175770 2024-03-21T13:12:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.228931+0000", 2024-03-21T13:12:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.200 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.228931+0000", 2024-03-21T13:12:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.228933+0000", 2024-03-21T13:12:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.643e-06 2024-03-21T13:12:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.228949+0000", 2024-03-21T13:12:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5934e-05 2024-03-21T13:12:26.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.228970+0000", 2024-03-21T13:12:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0761e-05 2024-03-21T13:12:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.228979+0000", 2024-03-21T13:12:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0000000000000002e-06 2024-03-21T13:12:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.202 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.229178+0000", 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019924100000000001 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.229299+0000", 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000121453 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,5", 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.229415+0000", 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011579 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.203 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.232533+0000", 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031179269999999999 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.232583+0000", 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9795e-05 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.240425+0000", 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078425779999999994 2024-03-21T13:12:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.284990+0000", 2024-03-21T13:12:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044564430000000002 2024-03-21T13:12:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:12:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285004+0000", 2024-03-21T13:12:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4148000000000001e-05 2024-03-21T13:12:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.303201+0000", 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0181971030000001 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.303274+0000", 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2756999999999995e-05 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175774 7.e 7:7df3ee0e:::benchmark_data_smithi163_80312_object58333:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.242901+0000", 2024-03-21T13:12:26.206 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 541.95361253299995, 2024-03-21T13:12:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.061355786, 2024-03-21T13:12:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:12:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:12:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:12:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:12:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175774 2024-03-21T13:12:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.207 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.242901+0000", 2024-03-21T13:12:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.242901+0000", 2024-03-21T13:12:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.208 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.242902+0000", 2024-03-21T13:12:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5069999999999999e-06 2024-03-21T13:12:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.242909+0000", 2024-03-21T13:12:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.99e-06 2024-03-21T13:12:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.242922+0000", 2024-03-21T13:12:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2479e-05 2024-03-21T13:12:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.209 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.242931+0000", 2024-03-21T13:12:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9199999999999993e-06 2024-03-21T13:12:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.244240+0000", 2024-03-21T13:12:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00130948 2024-03-21T13:12:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.244703+0000", 2024-03-21T13:12:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00046305100000000002 2024-03-21T13:12:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,5", 2024-03-21T13:12:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.245050+0000", 2024-03-21T13:12:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00034666999999999999 2024-03-21T13:12:26.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.249144+0000", 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004094019 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265019+0000", 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015874935999999999 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265037+0000", 2024-03-21T13:12:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8360999999999999e-05 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285302+0000", 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020265004999999999 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285312+0000", 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7899999999999994e-06 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304189+0000", 2024-03-21T13:12:26.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0188767400000001 2024-03-21T13:12:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304256+0000", 2024-03-21T13:12:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7837999999999997e-05 2024-03-21T13:12:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.214 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175778 7.e 7:7258fe02:::benchmark_data_smithi163_80312_object58337:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.252579+0000", 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 541.94393433000005, 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.051759933, 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175778 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252579+0000", 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252579+0000", 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252580+0000", 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.573e-06 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252588+0000", 2024-03-21T13:12:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9950000000000005e-06 2024-03-21T13:12:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252605+0000", 2024-03-21T13:12:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6659000000000001e-05 2024-03-21T13:12:26.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252615+0000", 2024-03-21T13:12:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0421000000000001e-05 2024-03-21T13:12:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252739+0000", 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012360100000000001 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.252887+0000", 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000148133 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,5", 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.253037+0000", 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000149357 2024-03-21T13:12:26.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.256167+0000", 2024-03-21T13:12:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031299349999999999 2024-03-21T13:12:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:12:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265042+0000", 2024-03-21T13:12:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0088755320000000002 2024-03-21T13:12:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265074+0000", 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2274999999999998e-05 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285342+0000", 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020267252999999999 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285351+0000", 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9260000000000006e-06 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304289+0000", 2024-03-21T13:12:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0189389820000001 2024-03-21T13:12:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304339+0000", 2024-03-21T13:12:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9291e-05 2024-03-21T13:12:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.222 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175781 7.e 7:70cd84ea:::benchmark_data_smithi163_80312_object58340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T13:12:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.262490+0000", 2024-03-21T13:12:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 541.93402310600004, 2024-03-21T13:12:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.042246081, 2024-03-21T13:12:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:12:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:12:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:12:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:12:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175781 2024-03-21T13:12:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.262490+0000", 2024-03-21T13:12:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.262490+0000", 2024-03-21T13:12:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.262492+0000", 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0430000000000002e-06 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.262498+0000", 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3239999999999999e-06 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.262511+0000", 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2934e-05 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.262520+0000", 2024-03-21T13:12:26.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4519999999999997e-06 2024-03-21T13:12:26.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265104+0000", 2024-03-21T13:12:26.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002584392 2024-03-21T13:12:26.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265233+0000", 2024-03-21T13:12:26.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000128823 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,5", 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.265363+0000", 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013014999999999999 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285381+0000", 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020017521999999999 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.285389+0000", 2024-03-21T13:12:26.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5459999999999992e-06 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304470+0000", 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.019080553 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304736+0000", 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00026634199999999998 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.229 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175796 7.e 7:7e6aa0fe:::benchmark_data_smithi163_80312_object58355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T13:12:26.229 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.292110+0000", 2024-03-21T13:12:26.229 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 541.90440309799999, 2024-03-21T13:12:26.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.031751093, 2024-03-21T13:12:26.229 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.229 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:12:26.229 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:12:26.229 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:12:26.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:26.231 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:12:26.231 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175796 2024-03-21T13:12:26.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.231 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.292110+0000", 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.292110+0000", 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.292112+0000", 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.826e-06 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.292120+0000", 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8499999999999994e-06 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295305+0000", 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031855289999999999 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295317+0000", 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1387e-05 2024-03-21T13:12:26.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295911+0000", 2024-03-21T13:12:26.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059404100000000001 2024-03-21T13:12:26.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:12:26.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295919+0000", 2024-03-21T13:12:26.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3659999999999992e-06 2024-03-21T13:12:26.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304767+0000", 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0088483269999999 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304886+0000", 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119027 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.304966+0000", 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9462999999999995e-05 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:12:26.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.316206+0000", 2024-03-21T13:12:26.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011240529000000001 2024-03-21T13:12:26.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.319434+0000", 2024-03-21T13:12:26.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003227415 2024-03-21T13:12:26.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.323761+0000", 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0043271450000000001 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.323816+0000", 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5157000000000001e-05 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.323861+0000", 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5031000000000003e-05 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.237 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.238 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175802 7.e 7:76aff798:::benchmark_data_smithi163_80312_object58361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T13:12:26.238 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.295348+0000", 2024-03-21T13:12:26.238 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 541.90116478899995, 2024-03-21T13:12:26.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0286435650000001, 2024-03-21T13:12:26.238 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.238 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:12:26.238 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:12:26.238 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:12:26.238 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:12:26.238 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175802 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295348+0000", 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295348+0000", 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295350+0000", 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2839999999999999e-06 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295356+0000", 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9959999999999999e-06 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295363+0000", 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3549999999999999e-06 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295369+0000", 2024-03-21T13:12:26.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4979999999999997e-06 2024-03-21T13:12:26.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295939+0000", 2024-03-21T13:12:26.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057073699999999998 2024-03-21T13:12:26.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:12:26.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.295945+0000", 2024-03-21T13:12:26.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3079999999999998e-06 2024-03-21T13:12:26.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.307185+0000", 2024-03-21T13:12:26.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0112405719999999 2024-03-21T13:12:26.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.307303+0000", 2024-03-21T13:12:26.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118151 2024-03-21T13:12:26.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.307426+0000", 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000122302 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.319353+0000", 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011927713 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.319478+0000", 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012420500000000001 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.244 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.323906+0000", 2024-03-21T13:12:26.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044284820000000001 2024-03-21T13:12:26.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.244 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:12:26.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.323955+0000", 2024-03-21T13:12:26.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9020999999999997e-05 2024-03-21T13:12:26.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.323992+0000", 2024-03-21T13:12:26.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6940999999999997e-05 2024-03-21T13:12:26.245 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.245 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.246 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.246 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175826 7.e 7:707dd3cd:::benchmark_data_smithi163_80312_object58385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T13:12:26.246 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.326877+0000", 2024-03-21T13:12:26.246 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 541.86963597600004, 2024-03-21T13:12:26.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.011345951, 2024-03-21T13:12:26.246 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.246 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:12:26.246 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:12:26.246 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:12:26.246 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:12:26.246 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175826 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.326877+0000", 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.326877+0000", 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.326879+0000", 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1969999999999999e-06 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.326887+0000", 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6639999999999998e-06 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.326907+0000", 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0026000000000001e-05 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.326918+0000", 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1051e-05 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.327127+0000", 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020895399999999999 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.327139+0000", 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1640999999999999e-05 2024-03-21T13:12:26.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.313333+0000", 2024-03-21T13:12:26.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98619378800000002 2024-03-21T13:12:26.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.313440+0000", 2024-03-21T13:12:26.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107896 2024-03-21T13:12:26.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:12:26.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.313556+0000", 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115229 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.319392+0000", 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0058364580000000001 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.324041+0000", 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0046487569999999999 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338129+0000", 2024-03-21T13:12:26.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01408769 2024-03-21T13:12:26.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:12:26.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338181+0000", 2024-03-21T13:12:26.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1984000000000002e-05 2024-03-21T13:12:26.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338223+0000", 2024-03-21T13:12:26.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2616e-05 2024-03-21T13:12:26.252 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.252 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175840 7.e 7:7d14a16e:::benchmark_data_smithi163_80312_object58399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.351435+0000", 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 541.84507793199998, 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98690400499999997, 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175840 2024-03-21T13:12:26.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.254 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351435+0000", 2024-03-21T13:12:26.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351435+0000", 2024-03-21T13:12:26.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351437+0000", 2024-03-21T13:12:26.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4699999999999999e-06 2024-03-21T13:12:26.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351444+0000", 2024-03-21T13:12:26.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9489999999999999e-06 2024-03-21T13:12:26.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351459+0000", 2024-03-21T13:12:26.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5057e-05 2024-03-21T13:12:26.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351468+0000", 2024-03-21T13:12:26.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2699999999999993e-06 2024-03-21T13:12:26.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351526+0000", 2024-03-21T13:12:26.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7726999999999998e-05 2024-03-21T13:12:26.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.351532+0000", 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9699999999999996e-06 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.316235+0000", 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.96470319000000004 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.316345+0000", 2024-03-21T13:12:26.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011040900000000001 2024-03-21T13:12:26.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:12:26.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.316473+0000", 2024-03-21T13:12:26.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012779800000000001 2024-03-21T13:12:26.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:12:26.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.319490+0000", 2024-03-21T13:12:26.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0030170319999999998 2024-03-21T13:12:26.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.324867+0000", 2024-03-21T13:12:26.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0053769639999999997 2024-03-21T13:12:26.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338262+0000", 2024-03-21T13:12:26.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0133945 2024-03-21T13:12:26.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:12:26.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338306+0000", 2024-03-21T13:12:26.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4622000000000001e-05 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338339+0000", 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3046999999999997e-05 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14820.0:175843 7.e 7:7008cba1:::benchmark_data_smithi163_80312_object58402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:03:24.360203+0000", 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 541.83631040900002, 2024-03-21T13:12:26.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.97825061400000002, 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14820", 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/145843339", 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 175843 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360203+0000", 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360203+0000", 2024-03-21T13:12:26.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360204+0000", 2024-03-21T13:12:26.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5030000000000001e-06 2024-03-21T13:12:26.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360210+0000", 2024-03-21T13:12:26.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0669999999999997e-06 2024-03-21T13:12:26.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360225+0000", 2024-03-21T13:12:26.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4739000000000001e-05 2024-03-21T13:12:26.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360234+0000", 2024-03-21T13:12:26.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2119999999999992e-06 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360303+0000", 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8526000000000001e-05 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:24.360312+0000", 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4539999999999996e-06 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.320443+0000", 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.96013029599999999 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.320547+0000", 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010440700000000001 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.320898+0000", 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035098699999999998 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:12:26.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.324884+0000", 2024-03-21T13:12:26.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039858139999999999 2024-03-21T13:12:26.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.325424+0000", 2024-03-21T13:12:26.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053980900000000004 2024-03-21T13:12:26.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:12:26.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338376+0000", 2024-03-21T13:12:26.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012952823 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338421+0000", 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4348999999999998e-05 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:03:25.338453+0000", 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2628e-05 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.268 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15078.0:180995 8.f e204/196 8:f71cfc61:::benchmark_data_smithi163_107079_object45122:head v 204'11180, mlcod=204'11178)", 2024-03-21T13:12:26.268 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:08.424208+0000", 2024-03-21T13:12:26.268 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 317.772305136, 2024-03-21T13:12:26.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.147111875, 2024-03-21T13:12:26.269 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.270 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:12:26.270 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.424208+0000", 2024-03-21T13:12:26.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.424208+0000", 2024-03-21T13:12:26.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.424209+0000", 2024-03-21T13:12:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7899999999999998e-07 2024-03-21T13:12:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.424216+0000", 2024-03-21T13:12:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2289999999999997e-06 2024-03-21T13:12:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.424226+0000", 2024-03-21T13:12:26.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0028e-05 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.424241+0000", 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4575e-05 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.424297+0000", 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6261999999999998e-05 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.424322+0000", 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5041999999999998e-05 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.571262+0000", 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.14694052099999999 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.571320+0000", 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7538999999999999e-05 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.273 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.274 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15078.0:180996 8.e e204/196 8:76cf5fdb:::benchmark_data_smithi163_107079_object45123:head v 204'11396, mlcod=204'11394)", 2024-03-21T13:12:26.274 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:08.425114+0000", 2024-03-21T13:12:26.274 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 317.77139899899998, 2024-03-21T13:12:26.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.14630836, 2024-03-21T13:12:26.274 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.274 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:12:26.274 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.425114+0000", 2024-03-21T13:12:26.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.425114+0000", 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.425115+0000", 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1100000000000005e-07 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.425121+0000", 2024-03-21T13:12:26.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4930000000000003e-06 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.425132+0000", 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0154e-05 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.425138+0000", 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7939999999999997e-06 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.425179+0000", 2024-03-21T13:12:26.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0429999999999997e-05 2024-03-21T13:12:26.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.425195+0000", 2024-03-21T13:12:26.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6569e-05 2024-03-21T13:12:26.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:12:26.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.571386+0000", 2024-03-21T13:12:26.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.14619043200000001 2024-03-21T13:12:26.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:08.571423+0000", 2024-03-21T13:12:26.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6677000000000001e-05 2024-03-21T13:12:26.278 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.278 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.278 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.278 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24325.0:2726 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94483560961536 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:12:26.278 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:02.908756+0000", 2024-03-21T13:12:26.278 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 143.28775678599999, 2024-03-21T13:12:26.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.72060023299999998, 2024-03-21T13:12:26.278 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24325", 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/1345925145", 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2726 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908756+0000", 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908756+0000", 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908758+0000", 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4410000000000001e-06 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908765+0000", 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0600000000000002e-06 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908782+0000", 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7317999999999999e-05 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908793+0000", 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0492000000000001e-05 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908855+0000", 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2321000000000002e-05 2024-03-21T13:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908863+0000", 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6690000000000006e-06 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.609271+0000", 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.70040877999999995 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for missing object", 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.609413+0000", 2024-03-21T13:12:26.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141738 2024-03-21T13:12:26.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.627023+0000", 2024-03-21T13:12:26.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017609763 2024-03-21T13:12:26.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-03-21T13:12:26.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.627177+0000", 2024-03-21T13:12:26.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015424699999999999 2024-03-21T13:12:26.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.627685+0000", 2024-03-21T13:12:26.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050788099999999996 2024-03-21T13:12:26.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.627782+0000", 2024-03-21T13:12:26.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6812999999999996e-05 2024-03-21T13:12:26.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.629357+0000", 2024-03-21T13:12:26.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00157471 2024-03-21T13:12:26.285 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.285 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.285 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.285 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24325.0:2727 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94483560959232 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:12:26.285 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:02.908815+0000", 2024-03-21T13:12:26.285 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 143.28769812799999, 2024-03-21T13:12:26.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.71802334999999995, 2024-03-21T13:12:26.285 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.285 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:12:26.285 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:12:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24325", 2024-03-21T13:12:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/1345925145", 2024-03-21T13:12:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2727 2024-03-21T13:12:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908815+0000", 2024-03-21T13:12:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908815+0000", 2024-03-21T13:12:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908816+0000", 2024-03-21T13:12:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.89e-07 2024-03-21T13:12:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908819+0000", 2024-03-21T13:12:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7129999999999999e-06 2024-03-21T13:12:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908826+0000", 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3790000000000003e-06 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908831+0000", 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8239999999999996e-06 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908877+0000", 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5116e-05 2024-03-21T13:12:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908883+0000", 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1090000000000003e-06 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.610133+0000", 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.701249868 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for missing object", 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.610247+0000", 2024-03-21T13:12:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114774 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.622278+0000", 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012030394999999999 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.622383+0000", 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105176 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.623065+0000", 2024-03-21T13:12:26.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00068221899999999997 2024-03-21T13:12:26.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.623139+0000", 2024-03-21T13:12:26.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3801999999999996e-05 2024-03-21T13:12:26.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.626838+0000", 2024-03-21T13:12:26.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0036994860000000001 2024-03-21T13:12:26.291 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.291 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24325.0:3004 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94483560961536 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:12:22.933118+0000", 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.2633953309999999, 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00041652900000000001, 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24325", 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/1345925145", 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3004 2024-03-21T13:12:26.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.293 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933118+0000", 2024-03-21T13:12:26.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933118+0000", 2024-03-21T13:12:26.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933120+0000", 2024-03-21T13:12:26.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6959999999999999e-06 2024-03-21T13:12:26.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933126+0000", 2024-03-21T13:12:26.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.601e-06 2024-03-21T13:12:26.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933142+0000", 2024-03-21T13:12:26.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6218e-05 2024-03-21T13:12:26.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933150+0000", 2024-03-21T13:12:26.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9559999999999997e-06 2024-03-21T13:12:26.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933237+0000", 2024-03-21T13:12:26.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6253000000000002e-05 2024-03-21T13:12:26.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933338+0000", 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000101594 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933534+0000", 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019621100000000001 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24325.0:3005 5.d 5:b4812045:::notify.4:head [watch ping cookie 94483560959232 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-03-21T13:12:26.296 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:12:22.933173+0000", 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.2633403940000001, 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029761499999999999, 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24325", 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/1345925145", 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3005 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933173+0000", 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933173+0000", 2024-03-21T13:12:26.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933173+0000", 2024-03-21T13:12:26.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0200000000000002e-07 2024-03-21T13:12:26.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933176+0000", 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9359999999999999e-06 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933190+0000", 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3524e-05 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933196+0000", 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4899999999999997e-06 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933257+0000", 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0406999999999999e-05 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933338+0000", 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0882000000000003e-05 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:22.933470+0000", 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000132874 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.300 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.301 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDScrubReserve(6.d,GRANT,e:328,from:0,reservation_nonce:0)", 2024-03-21T13:12:26.301 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:12:25.810864+0000", 2024-03-21T13:12:26.301 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.38564928199999998, 2024-03-21T13:12:26.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000101297, 2024-03-21T13:12:26.301 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.301 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:12:26.301 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.810864+0000", 2024-03-21T13:12:26.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.810864+0000", 2024-03-21T13:12:26.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.810865+0000", 2024-03-21T13:12:26.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0800000000000004e-07 2024-03-21T13:12:26.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.810868+0000", 2024-03-21T13:12:26.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9610000000000001e-06 2024-03-21T13:12:26.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.810877+0000", 2024-03-21T13:12:26.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3619999999999995e-06 2024-03-21T13:12:26.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.810885+0000", 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7309999999999993e-06 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.810923+0000", 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8306000000000002e-05 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.810935+0000", 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1718e-05 2024-03-21T13:12:26.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.810965+0000", 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0511e-05 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDScrubReserve(6.d,GRANT,e:328,from:6,reservation_nonce:0)", 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:12:25.811376+0000", 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.385136963, 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018006399999999999, 2024-03-21T13:12:26.305 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.306 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:12:26.306 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.811376+0000", 2024-03-21T13:12:26.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.811376+0000", 2024-03-21T13:12:26.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.811377+0000", 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6000000000000004e-07 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.811379+0000", 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3609999999999999e-06 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.811390+0000", 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0407e-05 2024-03-21T13:12:26.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.811395+0000", 2024-03-21T13:12:26.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0579999999999997e-06 2024-03-21T13:12:26.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.811431+0000", 2024-03-21T13:12:26.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6822000000000001e-05 2024-03-21T13:12:26.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.811441+0000", 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4739999999999994e-06 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.811556+0000", 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115382 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "rep_scrubmap(6.d e271 from shard 6)", 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:12:25.859442+0000", 2024-03-21T13:12:26.309 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.33707152200000001, 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031631989999999998, 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.859442+0000", 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.859442+0000", 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.859443+0000", 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.652e-06 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.859453+0000", 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7240000000000004e-06 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.859474+0000", 2024-03-21T13:12:26.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0611e-05 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.859485+0000", 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1175000000000001e-05 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.862565+0000", 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0030804790000000001 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.862582+0000", 2024-03-21T13:12:26.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6629e-05 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.862605+0000", 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2929e-05 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "rep_scrubmap(6.d e271 from shard 0)", 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:12:25.877995+0000", 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.31851805399999999, 2024-03-21T13:12:26.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000139608, 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.877995+0000", 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.877995+0000", 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.877997+0000", 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.81e-06 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.878005+0000", 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0190000000000006e-06 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.878019+0000", 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4378e-05 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.878041+0000", 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1843000000000001e-05 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.878101+0000", 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9823e-05 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.878111+0000", 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6050000000000004e-06 2024-03-21T13:12:26.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:26.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:26.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:12:26.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:12:25.878135+0000", 2024-03-21T13:12:26.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4130000000000001e-05 2024-03-21T13:12:26.317 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.317 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.317 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.317 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:26.317 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:26.317 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:12:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:26.345 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:26.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:26.446 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:26.472 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:26.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:26.573 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:26.603 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:26.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:26.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:26.728 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:26.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:26.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[29621]: pgmap v645: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 944 B/s rd, 0 op/s 2024-03-21T13:12:26.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[29621]: 5.1f scrub starts 2024-03-21T13:12:26.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[29621]: 5.1f scrub ok 2024-03-21T13:12:26.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[29621]: 6.d scrub starts 2024-03-21T13:12:26.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[29621]: 6.d scrub ok 2024-03-21T13:12:26.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[29621]: 4.0 scrub starts 2024-03-21T13:12:26.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[29621]: 4.0 scrub ok 2024-03-21T13:12:26.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[29621]: osdmap e329: 8 total, 8 up, 6 in 2024-03-21T13:12:26.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[40675]: pgmap v645: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 944 B/s rd, 0 op/s 2024-03-21T13:12:26.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[40675]: 5.1f scrub starts 2024-03-21T13:12:26.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[40675]: 5.1f scrub ok 2024-03-21T13:12:26.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[40675]: 6.d scrub starts 2024-03-21T13:12:26.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[40675]: 6.d scrub ok 2024-03-21T13:12:26.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[40675]: 4.0 scrub starts 2024-03-21T13:12:26.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[40675]: 4.0 scrub ok 2024-03-21T13:12:26.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:26 smithi163 ceph-mon[40675]: osdmap e329: 8 total, 8 up, 6 in 2024-03-21T13:12:26.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:26 smithi204 ceph-mon[33153]: pgmap v645: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 944 B/s rd, 0 op/s 2024-03-21T13:12:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:26 smithi204 ceph-mon[33153]: 5.1f scrub starts 2024-03-21T13:12:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:26 smithi204 ceph-mon[33153]: 5.1f scrub ok 2024-03-21T13:12:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:26 smithi204 ceph-mon[33153]: 6.d scrub starts 2024-03-21T13:12:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:26 smithi204 ceph-mon[33153]: 6.d scrub ok 2024-03-21T13:12:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:26 smithi204 ceph-mon[33153]: 4.0 scrub starts 2024-03-21T13:12:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:26 smithi204 ceph-mon[33153]: 4.0 scrub ok 2024-03-21T13:12:26.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:26 smithi204 ceph-mon[33153]: osdmap e329: 8 total, 8 up, 6 in 2024-03-21T13:12:26.829 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:26.857 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:26.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:26.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:26.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:26.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:27.089 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:27.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:27.113 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:27.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:27.214 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:27.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:27.239 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:27.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:27.339 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:27.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:27.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:27.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:27.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:27.494 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:27.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:27.567 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:27 smithi163 ceph-mon[29621]: 4.5 scrub starts 2024-03-21T13:12:27.567 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:27 smithi163 ceph-mon[29621]: 4.5 scrub ok 2024-03-21T13:12:27.567 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:27 smithi163 ceph-mon[40675]: 4.5 scrub starts 2024-03-21T13:12:27.567 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:27 smithi163 ceph-mon[40675]: 4.5 scrub ok 2024-03-21T13:12:27.595 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:27.622 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:27.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:27.697 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:12:27.716 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:12:27.723 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:27.736 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:12:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:27.748 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:27.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:27 smithi204 ceph-mon[33153]: 4.5 scrub starts 2024-03-21T13:12:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:27 smithi204 ceph-mon[33153]: 4.5 scrub ok 2024-03-21T13:12:27.756 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:12:27.777 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:12:27.797 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:12:27.818 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:12:27.840 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:12:27.840 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:12:27.841 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:12:27.841 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:12:27.841 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:12:27.841 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:12:27.841 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:12:27.841 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:12:27.841 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:12:27.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:27.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:27.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:27.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:27.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:28.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:28.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:28.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:28.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:28.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:28.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:28.235 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:28.259 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:28.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:28.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:28.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:28.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:28.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a[29617]: 2024-03-21T13:12:28.135+0000 7f8e1b000700 -1 mon.a@0(leader).osd e329 definitely_dead 0 2024-03-21T13:12:28.487 INFO:teuthology.orchestra.run.smithi163.stderr:marked in osd.5. 2024-03-21T13:12:28.491 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:28.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:28.621 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:28.644 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:28.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:28.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:28.747 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:28 smithi204 ceph-mon[33153]: pgmap v647: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:28 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1119080211' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:12:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:28 smithi204 ceph-mon[33153]: 2.1 scrub starts 2024-03-21T13:12:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:28 smithi204 ceph-mon[33153]: 2.1 scrub ok 2024-03-21T13:12:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:28 smithi204 ceph-mon[33153]: 4.b scrub starts 2024-03-21T13:12:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:28 smithi204 ceph-mon[33153]: 4.b scrub ok 2024-03-21T13:12:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:28 smithi204 ceph-mon[33153]: 3.5 deep-scrub starts 2024-03-21T13:12:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:28 smithi204 ceph-mon[33153]: 3.5 deep-scrub ok 2024-03-21T13:12:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:28 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/683152910' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-21T13:12:28.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:28 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-21T13:12:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[29621]: pgmap v647: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1119080211' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:12:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[29621]: 2.1 scrub starts 2024-03-21T13:12:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[29621]: 2.1 scrub ok 2024-03-21T13:12:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[29621]: 4.b scrub starts 2024-03-21T13:12:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[29621]: 4.b scrub ok 2024-03-21T13:12:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[29621]: 3.5 deep-scrub starts 2024-03-21T13:12:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[29621]: 3.5 deep-scrub ok 2024-03-21T13:12:28.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/683152910' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-21T13:12:28.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-21T13:12:28.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[40675]: pgmap v647: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:28.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1119080211' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:12:28.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[40675]: 2.1 scrub starts 2024-03-21T13:12:28.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[40675]: 2.1 scrub ok 2024-03-21T13:12:28.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[40675]: 4.b scrub starts 2024-03-21T13:12:28.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[40675]: 4.b scrub ok 2024-03-21T13:12:28.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[40675]: 3.5 deep-scrub starts 2024-03-21T13:12:28.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[40675]: 3.5 deep-scrub ok 2024-03-21T13:12:28.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/683152910' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-21T13:12:28.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:28 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-21T13:12:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:28.775 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:28.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:28.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:28.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:28.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:29.005 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:29.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:29.031 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:29.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:29.132 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:29.157 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:29.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:29.218 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-21T13:12:29.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:29.286 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:29.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:29.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:29.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:29.416 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:29.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:29.519 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:29.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:29.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:29.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:29.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:29.682 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:29.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:29.724 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:29.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:29 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-21T13:12:29.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:29 smithi204 ceph-mon[33153]: osdmap e330: 8 total, 8 up, 7 in 2024-03-21T13:12:29.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:29 smithi204 ceph-mon[33153]: 6.2 scrub starts 2024-03-21T13:12:29.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:29 smithi204 ceph-mon[33153]: 6.2 scrub ok 2024-03-21T13:12:29.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:29 smithi204 ceph-mon[33153]: osdmap e331: 8 total, 8 up, 7 in 2024-03-21T13:12:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:29 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-21T13:12:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:29 smithi163 ceph-mon[29621]: osdmap e330: 8 total, 8 up, 7 in 2024-03-21T13:12:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:29 smithi163 ceph-mon[29621]: 6.2 scrub starts 2024-03-21T13:12:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:29 smithi163 ceph-mon[29621]: 6.2 scrub ok 2024-03-21T13:12:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:29 smithi163 ceph-mon[29621]: osdmap e331: 8 total, 8 up, 7 in 2024-03-21T13:12:29.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:29 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-21T13:12:29.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:29 smithi163 ceph-mon[40675]: osdmap e330: 8 total, 8 up, 7 in 2024-03-21T13:12:29.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:29 smithi163 ceph-mon[40675]: 6.2 scrub starts 2024-03-21T13:12:29.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:29 smithi163 ceph-mon[40675]: 6.2 scrub ok 2024-03-21T13:12:29.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:29 smithi163 ceph-mon[40675]: osdmap e331: 8 total, 8 up, 7 in 2024-03-21T13:12:29.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:29.807 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:29.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:29.909 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:29.935 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:29.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:30.036 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:30.061 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:30.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:30.162 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:30.188 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:30.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:30.269 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-21T13:12:30.288 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:30.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:30.338 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:30.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:30.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:30.466 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:30.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:30.499 INFO:teuthology.orchestra.run.smithi163.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-21T13:12:30.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:30.593 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:30.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:30.601 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:12:30.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:30.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:30 smithi163 ceph-mon[29621]: pgmap v649: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 798 B/s rd, 0 op/s 2024-03-21T13:12:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:30 smithi163 ceph-mon[29621]: 4.6 scrub starts 2024-03-21T13:12:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:30 smithi163 ceph-mon[29621]: 4.6 scrub ok 2024-03-21T13:12:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:30 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/798365519' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:12:30.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:30 smithi163 ceph-mon[40675]: pgmap v649: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 798 B/s rd, 0 op/s 2024-03-21T13:12:30.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:30 smithi163 ceph-mon[40675]: 4.6 scrub starts 2024-03-21T13:12:30.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:30 smithi163 ceph-mon[40675]: 4.6 scrub ok 2024-03-21T13:12:30.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:30 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/798365519' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:12:30.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:30.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:30.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:30.945 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:30.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:30.970 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:30.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:30 smithi204 ceph-mon[33153]: pgmap v649: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 798 B/s rd, 0 op/s 2024-03-21T13:12:31.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:30 smithi204 ceph-mon[33153]: 4.6 scrub starts 2024-03-21T13:12:31.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:30 smithi204 ceph-mon[33153]: 4.6 scrub ok 2024-03-21T13:12:31.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:30 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/798365519' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:12:31.071 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:31.096 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:31.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:31.197 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:31.222 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:31.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:31.323 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:31.348 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:31.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:31.449 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:31.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:31.474 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:31.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:31.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:31.601 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:31.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:31.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:31.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:31.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:31.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:12:31.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:12:31.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[29621]: 4.1f deep-scrub starts 2024-03-21T13:12:31.756 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[29621]: 4.1f deep-scrub ok 2024-03-21T13:12:31.756 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/798365519' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:12:31.756 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[29621]: osdmap e332: 8 total, 8 up, 7 in 2024-03-21T13:12:31.756 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[29621]: osdmap e333: 8 total, 8 up, 7 in 2024-03-21T13:12:31.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:12:31.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:12:31.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[40675]: 4.1f deep-scrub starts 2024-03-21T13:12:31.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[40675]: 4.1f deep-scrub ok 2024-03-21T13:12:31.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/798365519' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:12:31.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[40675]: osdmap e332: 8 total, 8 up, 7 in 2024-03-21T13:12:31.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:31 smithi163 ceph-mon[40675]: osdmap e333: 8 total, 8 up, 7 in 2024-03-21T13:12:31.829 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:31.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:31.854 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:31.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:31.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:31.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:32.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:31 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:12:32.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:31 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:12:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:31 smithi204 ceph-mon[33153]: 4.1f deep-scrub starts 2024-03-21T13:12:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:31 smithi204 ceph-mon[33153]: 4.1f deep-scrub ok 2024-03-21T13:12:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:31 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/798365519' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:12:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:31 smithi204 ceph-mon[33153]: osdmap e332: 8 total, 8 up, 7 in 2024-03-21T13:12:32.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:31 smithi204 ceph-mon[33153]: osdmap e333: 8 total, 8 up, 7 in 2024-03-21T13:12:32.081 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:32.105 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:32.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:32.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:32.185 DEBUG:teuthology.orchestra.run.smithi163:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-21T13:12:32.188 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-21T13:12:32.188 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-21T13:12:32.207 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:32.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:32.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:32.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:32.335 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:32.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:32.359 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:32.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:32.387 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:hints = 1 2024-03-21T13:12:32.387 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-21T13:12:32.387 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Object prefix: benchmark_data_smithi163_154138 2024-03-21T13:12:32.387 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:12:32.387 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 0 0 0 0 0 0 - 0 2024-03-21T13:12:32.460 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:32.500 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:32.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:32.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:32.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:12:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:12:32.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:32.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:32.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:32.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:32.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[29621]: pgmap v652: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 175 B/s, 9 objects/s recovering 2024-03-21T13:12:32.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:12:32.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:12:32.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[29621]: 3.a scrub starts 2024-03-21T13:12:32.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[29621]: 3.a scrub ok 2024-03-21T13:12:32.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[29621]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:12:32.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:12:32.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[40675]: pgmap v652: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 175 B/s, 9 objects/s recovering 2024-03-21T13:12:32.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:12:32.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:12:32.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[40675]: 3.a scrub starts 2024-03-21T13:12:32.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[40675]: 3.a scrub ok 2024-03-21T13:12:32.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[40675]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:12:32.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:32 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:12:32.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:32.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:32.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:32.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:32.897 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:32.901 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:32.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:32.920 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:33.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:32 smithi204 ceph-mon[33153]: pgmap v652: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 175 B/s, 9 objects/s recovering 2024-03-21T13:12:33.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:32 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:12:33.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:32 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:12:33.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:32 smithi204 ceph-mon[33153]: 3.a scrub starts 2024-03-21T13:12:33.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:32 smithi204 ceph-mon[33153]: 3.a scrub ok 2024-03-21T13:12:33.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:32 smithi204 ceph-mon[33153]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:12:33.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:32 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:12:33.003 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:33.043 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:33.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:33.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:33.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:33.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:33.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:33.288 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:33.331 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:33.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:33.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:33.387 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 1 16 3439 3423 213.926 213.938 0.00246806 0.00465847 2024-03-21T13:12:33.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:33.471 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:33.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:33.540 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:12:33.573 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:33.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:33.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:33.713 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:33.753 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:33.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:33.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[29621]: pgmap v654: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 182 B/s, 9 objects/s recovering 2024-03-21T13:12:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[29621]: 6.c scrub starts 2024-03-21T13:12:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[29621]: 6.c scrub ok 2024-03-21T13:12:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[29621]: 3.1b scrub starts 2024-03-21T13:12:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[29621]: 3.1b scrub ok 2024-03-21T13:12:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4212205561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:12:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:12:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[40675]: pgmap v654: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 182 B/s, 9 objects/s recovering 2024-03-21T13:12:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[40675]: 6.c scrub starts 2024-03-21T13:12:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[40675]: 6.c scrub ok 2024-03-21T13:12:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[40675]: 3.1b scrub starts 2024-03-21T13:12:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[40675]: 3.1b scrub ok 2024-03-21T13:12:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4212205561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:12:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:33 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:12:33.855 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:33.892 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:33.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:33.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:33.994 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:33 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:33 smithi204 ceph-mon[33153]: pgmap v654: 148 pgs: 148 active+clean; 582 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 182 B/s, 9 objects/s recovering 2024-03-21T13:12:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:33 smithi204 ceph-mon[33153]: 6.c scrub starts 2024-03-21T13:12:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:33 smithi204 ceph-mon[33153]: 6.c scrub ok 2024-03-21T13:12:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:33 smithi204 ceph-mon[33153]: 3.1b scrub starts 2024-03-21T13:12:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:33 smithi204 ceph-mon[33153]: 3.1b scrub ok 2024-03-21T13:12:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:33 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4212205561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:12:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:33 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:12:34.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:34.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:34.135 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:34.172 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:34.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:34.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:34.219 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 2, 5] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:12:34.219 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:12:34.219 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-21T13:12:34.219 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-21T13:12:34.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:34.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:34.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:34.387 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 2 16 6523 6507 203.33 192.75 0.00926296 0.00490669 2024-03-21T13:12:34.421 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:34.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:34.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:34.566 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:34.604 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:34.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:34.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:34.706 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:34.744 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:34.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:34.846 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:34.883 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:34.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:34.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:34 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:12:34.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:34 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4212205561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:12:34.960 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:34 smithi163 ceph-mon[29621]: osdmap e334: 8 total, 8 up, 7 in 2024-03-21T13:12:34.960 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:34 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:12:34.960 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:34 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4212205561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:12:34.960 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:34 smithi163 ceph-mon[40675]: osdmap e334: 8 total, 8 up, 7 in 2024-03-21T13:12:34.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:35.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:34 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:12:35.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:34 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4212205561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:12:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:34 smithi204 ceph-mon[33153]: osdmap e334: 8 total, 8 up, 7 in 2024-03-21T13:12:35.022 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:35.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:35.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:35.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:35.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:35.264 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:35.297 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:35.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:35.387 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 3 16 9751 9735 202.797 201.75 0.0037445 0.00492078 2024-03-21T13:12:35.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:35.437 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:35.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:35.539 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:35.586 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:35.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:35.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:35.688 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:35.726 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:35.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:35.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:12:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:12:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:12:35.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:35.867 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:35.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:35.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:35.969 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:36.008 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:36.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:36.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:36.146 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:36.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:36.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:36.248 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:36.293 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:36.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:36.387 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 4 16 12722 12706 198.514 185.688 0.0060462 0.00502939 2024-03-21T13:12:36.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:36.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:36.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:36.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:36.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:36 smithi204 ceph-mon[33153]: pgmap v656: 148 pgs: 148 active+clean; 61 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 11 MiB/s wr, 179 op/s; 148 B/s, 7 objects/s recovering 2024-03-21T13:12:36.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:36 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:12:36.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:36 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:36.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:36 smithi163 ceph-mon[29621]: pgmap v656: 148 pgs: 148 active+clean; 61 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 11 MiB/s wr, 179 op/s; 148 B/s, 7 objects/s recovering 2024-03-21T13:12:36.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:36 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:12:36.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:36 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:36.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:36 smithi163 ceph-mon[40675]: pgmap v656: 148 pgs: 148 active+clean; 61 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 11 MiB/s wr, 179 op/s; 148 B/s, 7 objects/s recovering 2024-03-21T13:12:36.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:36 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:12:36.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:36 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:36.550 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:36.592 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:36.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:36.693 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:36.729 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:36.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:36.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:36.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:36.888 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:36.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:36.989 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:37.044 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:37.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:37.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:37.103 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:37 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:12:37.103 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:37 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:12:37.103 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:37 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:12:37.103 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:37 smithi163 ceph-mon[29621]: osdmap e335: 8 total, 8 up, 7 in 2024-03-21T13:12:37.104 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:37 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:12:37.104 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:37 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:12:37.104 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:37 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:12:37.104 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:37 smithi163 ceph-mon[40675]: osdmap e335: 8 total, 8 up, 7 in 2024-03-21T13:12:37.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:37.184 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:37.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:37.212 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:12:37.212 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:12:37.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:37.325 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:37.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:37.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:37.388 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 5 16 15430 15414 192.658 169.25 0.00381961 0.00518308 2024-03-21T13:12:37.426 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:37.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:37.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:37 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:12:37.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:37 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:12:37.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:37 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:12:37.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:37 smithi204 ceph-mon[33153]: osdmap e335: 8 total, 8 up, 7 in 2024-03-21T13:12:37.557 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:37.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:37.597 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:37.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:37.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:37.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:37.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:37.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:37.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:37.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:37.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:37.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:37.976 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:38.014 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:38.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:38.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:38.116 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:38.153 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:38.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:38.255 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:38 smithi163 ceph-mon[29621]: pgmap v658: 148 pgs: 148 active+clean; 551 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s; 2 B/s, 0 objects/s recovering 2024-03-21T13:12:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:38 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:12:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:38 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:12:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:38 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:12:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:38 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:12:38.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:38 smithi163 ceph-mon[40675]: pgmap v658: 148 pgs: 148 active+clean; 551 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s; 2 B/s, 0 objects/s recovering 2024-03-21T13:12:38.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:38 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:12:38.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:38 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:12:38.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:38 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:12:38.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:38 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:12:38.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:38.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:38.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:38.387 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 6 15 18149 18134 188.879 170 0.00492533 0.00528819 2024-03-21T13:12:38.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:38.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:38.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:38.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:38.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-21T13:12:38.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:38 smithi204 ceph-mon[33153]: pgmap v658: 148 pgs: 148 active+clean; 551 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s; 2 B/s, 0 objects/s recovering 2024-03-21T13:12:38.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:38 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:12:38.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:38 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:12:38.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:38 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:12:38.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:38 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:12:38.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:38.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:38.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:38.610 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:12:38.672 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:38.721 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:38.747 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:38.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:38.815 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:12:38.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:38.854 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:12:38.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:38.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:38.895 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:12:38.937 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:12:38.979 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:12:38.987 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:39.009 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:12:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:39.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:39.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:39.030 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:12:39.068 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:12:39.069 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:12:39.069 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:12:39.070 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:12:39.070 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:12:39.070 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:12:39.070 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:12:39.070 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:12:39.070 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:12:39.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:39.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:39.155 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:39.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:39.256 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:39.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:39 smithi163 ceph-mon[40675]: osdmap e336: 8 total, 8 up, 7 in 2024-03-21T13:12:39.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:39 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/358942371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:12:39.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:39 smithi163 ceph-mon[29621]: osdmap e336: 8 total, 8 up, 7 in 2024-03-21T13:12:39.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:39 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/358942371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:12:39.291 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:39.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:39.387 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 7 16 20353 20337 181.564 137.688 0.00181109 0.00548956 2024-03-21T13:12:39.393 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:39.422 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:39.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:39.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:39 smithi204 ceph-mon[33153]: osdmap e336: 8 total, 8 up, 7 in 2024-03-21T13:12:39.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:39 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/358942371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:12:39.523 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:39.560 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:39.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:39.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:39.661 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:39.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:39.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:39.801 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:39.842 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:39.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:39.943 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:39.979 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:39.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:39.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:40.081 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:40.121 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:40.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:40.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:40.223 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:40.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:40.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:40.362 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:40.388 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 8 16 22910 22894 178.843 159.812 0.00566796 0.00558514 2024-03-21T13:12:40.398 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:40.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:40.500 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:40.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:40 smithi204 ceph-mon[33153]: pgmap v660: 148 pgs: 148 active+clean; 551 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T13:12:40.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:40 smithi163 ceph-mon[29621]: pgmap v660: 148 pgs: 148 active+clean; 551 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T13:12:40.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:40 smithi163 ceph-mon[40675]: pgmap v660: 148 pgs: 148 active+clean; 551 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T13:12:40.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:40.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:40.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:40.640 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:40.674 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:40.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:40.776 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:40.815 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:40.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:40.916 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:40.957 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:40.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:41.058 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:41.099 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:41.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:41.201 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:41.243 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:41.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:41.344 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:41.382 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:41.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:41.387 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 9 16 25285 25269 175.463 148.438 0.00247216 0.00569291 2024-03-21T13:12:41.485 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:41.522 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:41.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:41.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:41.662 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:41.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:41.763 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:41.803 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:41.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:41.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:41.942 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:41.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:41.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:42.044 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:42.078 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:42.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:42.138 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:42.145 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:42.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:42.216 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:42.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:42.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:12:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:12:42.319 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:42.356 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:42.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:42.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:42.388 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 10 16 27590 27574 172.322 144.062 0.00540267 0.00579721 2024-03-21T13:12:42.458 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:42.489 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:42.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:42.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:42.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:42 smithi163 ceph-mon[40675]: pgmap v661: 148 pgs: 148 active+clean; 1.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.4 KiB/s rd, 182 MiB/s wr, 2.91k op/s 2024-03-21T13:12:42.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:42 smithi163 ceph-mon[40675]: osdmap e337: 8 total, 8 up, 7 in 2024-03-21T13:12:42.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:42 smithi163 ceph-mon[29621]: pgmap v661: 148 pgs: 148 active+clean; 1.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.4 KiB/s rd, 182 MiB/s wr, 2.91k op/s 2024-03-21T13:12:42.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:42 smithi163 ceph-mon[29621]: osdmap e337: 8 total, 8 up, 7 in 2024-03-21T13:12:42.591 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:42.627 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:42.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:42.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:42.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:42 smithi204 ceph-mon[33153]: pgmap v661: 148 pgs: 148 active+clean; 1.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.4 KiB/s rd, 182 MiB/s wr, 2.91k op/s 2024-03-21T13:12:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:42 smithi204 ceph-mon[33153]: osdmap e337: 8 total, 8 up, 7 in 2024-03-21T13:12:42.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:42.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:42.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:42.907 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:42.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:43.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:43.046 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:43.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:43.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:43.184 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:43.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:43.286 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:43.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:43.311 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:43.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:43.388 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 11 16 29514 29498 167.587 120.25 0.00735823 0.0059618 2024-03-21T13:12:43.411 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:43.452 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:12:43.453 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:12:43.453 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 337, 2024-03-21T13:12:43.453 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "d4bf79c0-e781-11ee-95cd-87774f69a715", 2024-03-21T13:12:43.453 INFO:teuthology.orchestra.run.smithi163.stdout: "created": "2024-03-21T12:52:56.031419+0000", 2024-03-21T13:12:43.453 INFO:teuthology.orchestra.run.smithi163.stdout: "modified": "2024-03-21T13:12:41.389506+0000", 2024-03-21T13:12:43.453 INFO:teuthology.orchestra.run.smithi163.stdout: "last_up_change": "2024-03-21T12:58:23.775302+0000", 2024-03-21T13:12:43.453 INFO:teuthology.orchestra.run.smithi163.stdout: "last_in_change": "2024-03-21T13:12:28.474723+0000", 2024-03-21T13:12:43.453 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-21T13:12:43.453 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_num": 5801984, 2024-03-21T13:12:43.453 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_set": [ 2024-03-21T13:12:43.453 INFO:teuthology.orchestra.run.smithi163.stdout: "noscrub", 2024-03-21T13:12:43.454 INFO:teuthology.orchestra.run.smithi163.stdout: "pglog_hardlimit", 2024-03-21T13:12:43.454 INFO:teuthology.orchestra.run.smithi163.stdout: "purged_snapdirs", 2024-03-21T13:12:43.454 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_deletes", 2024-03-21T13:12:43.454 INFO:teuthology.orchestra.run.smithi163.stdout: "sortbitwise" 2024-03-21T13:12:43.454 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:12:43.454 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_version": 18, 2024-03-21T13:12:43.454 INFO:teuthology.orchestra.run.smithi163.stdout: "full_ratio": 0.94999998807907104, 2024-03-21T13:12:43.454 INFO:teuthology.orchestra.run.smithi163.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-21T13:12:43.454 INFO:teuthology.orchestra.run.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-21T13:12:43.454 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_snapshot": "", 2024-03-21T13:12:43.454 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_max": 10, 2024-03-21T13:12:43.454 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd": 8, 2024-03-21T13:12:43.454 INFO:teuthology.orchestra.run.smithi163.stdout: "require_min_compat_client": "luminous", 2024-03-21T13:12:43.455 INFO:teuthology.orchestra.run.smithi163.stdout: "min_compat_client": "luminous", 2024-03-21T13:12:43.455 INFO:teuthology.orchestra.run.smithi163.stdout: "require_osd_release": "squid", 2024-03-21T13:12:43.455 INFO:teuthology.orchestra.run.smithi163.stdout: "allow_crimson": false, 2024-03-21T13:12:43.455 INFO:teuthology.orchestra.run.smithi163.stdout: "pools": [ 2024-03-21T13:12:43.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.455 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 1, 2024-03-21T13:12:43.455 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".mgr", 2024-03-21T13:12:43.455 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:56:20.557490+0000", 2024-03-21T13:12:43.455 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:12:43.455 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:12:43.455 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:12:43.455 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:12:43.455 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:12:43.456 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:12:43.456 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:12:43.456 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:12:43.456 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:12:43.456 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:12:43.456 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:12:43.456 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:12:43.456 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 1, 2024-03-21T13:12:43.456 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 1, 2024-03-21T13:12:43.456 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 1, 2024-03-21T13:12:43.456 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 1, 2024-03-21T13:12:43.456 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 1, 2024-03-21T13:12:43.457 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:12:43.457 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:12:43.457 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:12:43.457 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:12:43.457 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:12:43.457 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:12:43.457 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:12:43.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.458 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "264", 2024-03-21T13:12:43.458 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:12:43.458 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:12:43.458 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:12:43.458 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:12:43.458 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:12:43.458 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:12:43.458 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:12:43.458 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:12:43.458 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:12:43.459 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:12:43.459 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:12:43.459 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:12:43.459 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:12:43.459 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:12:43.459 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:12:43.459 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:12:43.459 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:12:43.459 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:12:43.459 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:12:43.459 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:12:43.459 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:12:43.460 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:12:43.460 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:12:43.460 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:12:43.460 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:12:43.460 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:12:43.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.460 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:12:43.460 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:12:43.460 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:12:43.460 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:12:43.460 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:12:43.460 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:12:43.460 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:12:43.461 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:12:43.461 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:12:43.461 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:12:43.461 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:12:43.461 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-03-21T13:12:43.461 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_max": 32, 2024-03-21T13:12:43.461 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_min": 1 2024-03-21T13:12:43.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.461 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:12:43.461 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": {} 2024-03-21T13:12:43.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.461 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:12:43.462 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:12:43.462 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 7.8899998664855957, 2024-03-21T13:12:43.462 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 7.8899998664855957, 2024-03-21T13:12:43.462 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.37999999523162842, 2024-03-21T13:12:43.462 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 3, 2024-03-21T13:12:43.462 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 3, 2024-03-21T13:12:43.462 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:12:43.462 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:12:43.462 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:12:43.462 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.462 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 2, 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "datapool", 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:35.713383+0000", 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:12:43.463 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 3, 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 3, 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 3, 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 3, 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 3, 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.464 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "67", 2024-03-21T13:12:43.465 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:12:43.465 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:12:43.465 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:12:43.465 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:12:43.465 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:12:43.465 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 2, 2024-03-21T13:12:43.465 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 67, 2024-03-21T13:12:43.465 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:12:43.465 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:12:43.465 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:12:43.465 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:12:43.465 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:12:43.465 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:12:43.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.467 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:12:43.467 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:12:43.467 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:12:43.467 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:12:43.467 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:12:43.467 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:12:43.467 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:12:43.467 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:12:43.467 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:12:43.467 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:12:43.467 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:12:43.467 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:12:43.467 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:12:43.468 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd": {} 2024-03-21T13:12:43.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.468 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:12:43.468 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:12:43.468 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.6700000762939453, 2024-03-21T13:12:43.468 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.6700000762939453, 2024-03-21T13:12:43.468 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.75, 2024-03-21T13:12:43.468 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 2, 2024-03-21T13:12:43.468 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 2, 2024-03-21T13:12:43.468 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:12:43.468 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:12:43.468 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 3, 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".rgw.root", 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:36.436384+0000", 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:12:43.469 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:12:43.470 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:12:43.470 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:12:43.470 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:12:43.470 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:12:43.470 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:12:43.470 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:12:43.470 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:12:43.470 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:12:43.470 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:12:43.470 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:12:43.470 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:12:43.470 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:12:43.470 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:12:43.471 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:12:43.471 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:12:43.471 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:12:43.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.471 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "65", 2024-03-21T13:12:43.471 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:12:43.471 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:12:43.471 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:12:43.471 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:12:43.471 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:12:43.471 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:12:43.471 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:12:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:12:43.473 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:12:43.474 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:12:43.474 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:12:43.474 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:12:43.474 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:12:43.474 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:12:43.474 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:12:43.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.474 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:12:43.474 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:12:43.474 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.2400000095367432, 2024-03-21T13:12:43.474 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.2400000095367432, 2024-03-21T13:12:43.474 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.87999999523162842, 2024-03-21T13:12:43.474 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 4, 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.log", 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:38.049304+0000", 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:12:43.475 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:12:43.476 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "67", 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:12:43.477 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:12:43.478 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:12:43.478 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:12:43.478 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:12:43.478 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:12:43.478 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:12:43.478 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:12:43.478 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:12:43.478 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:12:43.478 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:12:43.478 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:12:43.478 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:12:43.478 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:12:43.479 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:12:43.479 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:12:43.479 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:12:43.479 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:12:43.479 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:12:43.479 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:12:43.479 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:12:43.479 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:12:43.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.479 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:12:43.479 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:12:43.479 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:12:43.479 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.9900000095367432, 2024-03-21T13:12:43.480 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.9900000095367432, 2024-03-21T13:12:43.481 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.87999999523162842, 2024-03-21T13:12:43.481 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 2.630000114440918, 2024-03-21T13:12:43.481 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 2.630000114440918, 2024-03-21T13:12:43.481 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:12:43.481 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:12:43.481 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:12:43.481 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.481 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 5, 2024-03-21T13:12:43.481 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.control", 2024-03-21T13:12:43.481 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:39.990512+0000", 2024-03-21T13:12:43.481 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:12:43.482 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:12:43.482 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:12:43.482 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:12:43.482 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:12:43.482 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:12:43.482 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:12:43.482 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:12:43.482 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:12:43.482 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:12:43.482 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:12:43.482 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:12:43.482 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:12:43.482 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:12:43.483 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:12:43.483 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:12:43.483 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:12:43.483 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:12:43.483 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:12:43.483 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:12:43.483 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:12:43.483 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:12:43.483 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:12:43.483 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:12:43.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.483 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "69", 2024-03-21T13:12:43.483 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:12:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:12:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:12:43.486 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:12:43.486 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:12:43.486 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:12:43.486 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:12:43.486 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:12:43.486 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:12:43.486 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:12:43.486 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:12:43.486 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:12:43.486 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:12:43.486 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:12:43.486 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:12:43.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.487 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:12:43.487 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:12:43.487 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 1.9900000095367432, 2024-03-21T13:12:43.487 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 1.9900000095367432, 2024-03-21T13:12:43.487 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.87999999523162842, 2024-03-21T13:12:43.487 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.75, 2024-03-21T13:12:43.487 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.75, 2024-03-21T13:12:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:43.488 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:43.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:43.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:43.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:43.490 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:12:43.490 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:12:43.490 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:12:43.490 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.490 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 6, 2024-03-21T13:12:43.490 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.meta", 2024-03-21T13:12:43.490 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:41.463686+0000", 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:12:43.491 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:12:43.492 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:12:43.492 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:12:43.492 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:12:43.492 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:12:43.492 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:12:43.492 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:12:43.492 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:12:43.492 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:12:43.492 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:12:43.492 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "71", 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:12:43.493 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:12:43.494 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:12:43.494 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:12:43.494 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:12:43.494 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:12:43.494 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:12:43.494 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:12:43.494 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:12:43.494 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:12:43.494 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:12:43.494 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:12:43.494 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:12:43.494 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:12:43.494 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:12:43.495 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:12:43.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.495 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:12:43.495 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:12:43.495 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:12:43.495 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:12:43.495 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:12:43.497 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:12:43.497 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:12:43.497 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:12:43.497 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:12:43.497 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:12:43.497 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_bias": 4 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.2400000095367432, 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.2400000095367432, 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.87999999523162842, 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-21T13:12:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 10, 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "unique_pool_3", 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T13:12:18.124611+0000", 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:12:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 16, 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 16, 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 16, 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 16, 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 16, 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:12:43.500 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "332", 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:12:43.501 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:12:43.502 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:12:43.503 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:12:43.504 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:12:43.504 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:12:43.504 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:12:43.504 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:12:43.504 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-03-21T13:12:43.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.504 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:12:43.504 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:12:43.504 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 3.4800000190734863, 2024-03-21T13:12:43.504 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 3.4800000190734863, 2024-03-21T13:12:43.504 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.87999999523162842, 2024-03-21T13:12:43.504 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 3.059999942779541, 2024-03-21T13:12:43.504 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 3.059999942779541, 2024-03-21T13:12:43.505 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.83299827575683594, 2024-03-21T13:12:43.505 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-21T13:12:43.505 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-21T13:12:43.505 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.505 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.505 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:12:43.505 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:12:43.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.505 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-03-21T13:12:43.505 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f", 2024-03-21T13:12:43.505 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:12:43.505 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:12:43.506 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:12:43.506 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0.0852813720703125, 2024-03-21T13:12:43.506 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:12:43.506 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:12:43.506 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 8, 2024-03-21T13:12:43.506 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 299, 2024-03-21T13:12:43.506 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:12:43.506 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:12:43.506 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:12:43.506 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.506 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.506 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6801", 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6802", 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.507 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:12:43.508 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.508 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.508 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6804", 2024-03-21T13:12:43.508 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:12:43.508 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.508 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.508 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:12:43.508 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.508 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.508 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6803", 2024-03-21T13:12:43.509 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:12:43.509 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.509 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.509 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6801/3086410981", 2024-03-21T13:12:43.509 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6802/3086410981", 2024-03-21T13:12:43.509 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6804/3086410981", 2024-03-21T13:12:43.509 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6803/3086410981", 2024-03-21T13:12:43.509 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:12:43.509 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:12:43.509 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:12:43.509 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.510 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-03-21T13:12:43.510 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "36e05f89-c238-41f7-be65-1553b529e8ce", 2024-03-21T13:12:43.510 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:12:43.510 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 0, 2024-03-21T13:12:43.510 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 0, 2024-03-21T13:12:43.510 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:12:43.510 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:12:43.510 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:12:43.510 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 13, 2024-03-21T13:12:43.510 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 110, 2024-03-21T13:12:43.510 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:12:43.510 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6805", 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.511 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6806", 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6808", 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.512 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:12:43.513 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.513 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.513 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6807", 2024-03-21T13:12:43.513 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:12:43.513 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.513 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.513 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6805/3666062784", 2024-03-21T13:12:43.513 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6806/3666062784", 2024-03-21T13:12:43.513 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6808/3666062784", 2024-03-21T13:12:43.513 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6807/3666062784", 2024-03-21T13:12:43.513 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f", 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:12:43.514 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 18, 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 330, 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6809", 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:12:43.515 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6810", 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6812", 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:12:43.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.517 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.517 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:12:43.517 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.517 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.517 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6811", 2024-03-21T13:12:43.517 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:12:43.517 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.517 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.517 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6809/4009213507", 2024-03-21T13:12:43.517 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6810/4009213507", 2024-03-21T13:12:43.518 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6812/4009213507", 2024-03-21T13:12:43.518 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6811/4009213507", 2024-03-21T13:12:43.518 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:12:43.518 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:12:43.518 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:12:43.518 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.518 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-03-21T13:12:43.518 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d", 2024-03-21T13:12:43.518 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:12:43.518 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:12:43.519 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:12:43.519 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:12:43.519 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:12:43.519 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:12:43.519 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 25, 2024-03-21T13:12:43.519 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 330, 2024-03-21T13:12:43.519 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:12:43.519 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:12:43.519 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:12:43.519 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.519 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6813", 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6814", 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.520 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6816", 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6815", 2024-03-21T13:12:43.521 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:12:43.522 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.522 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.522 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6813/3981878189", 2024-03-21T13:12:43.522 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6814/3981878189", 2024-03-21T13:12:43.522 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6816/3981878189", 2024-03-21T13:12:43.522 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6815/3981878189", 2024-03-21T13:12:43.522 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:12:43.522 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471", 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0.578704833984375, 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 30, 2024-03-21T13:12:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 330, 2024-03-21T13:12:43.524 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:12:43.524 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:12:43.524 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:12:43.524 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.524 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.524 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6800", 2024-03-21T13:12:43.524 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:12:43.524 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.524 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.524 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:12:43.524 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.525 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.525 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6801", 2024-03-21T13:12:43.525 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:12:43.525 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.525 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.525 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:12:43.525 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.525 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.525 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6803", 2024-03-21T13:12:43.525 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6802", 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6800/719276054", 2024-03-21T13:12:43.526 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6801/719276054", 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6803/719276054", 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6802/719276054", 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1", 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:12:43.527 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:12:43.528 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:12:43.528 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 39, 2024-03-21T13:12:43.528 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 330, 2024-03-21T13:12:43.528 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:12:43.528 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:12:43.528 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:12:43.528 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.528 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.528 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6804", 2024-03-21T13:12:43.528 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:12:43.528 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.528 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.529 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:12:43.529 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.530 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.530 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6805", 2024-03-21T13:12:43.530 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:12:43.530 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.530 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.530 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:12:43.530 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.530 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.530 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6807", 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6806", 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.531 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6804/1346947668", 2024-03-21T13:12:43.532 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6805/1346947668", 2024-03-21T13:12:43.532 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6807/1346947668", 2024-03-21T13:12:43.532 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6806/1346947668", 2024-03-21T13:12:43.532 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:12:43.532 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:12:43.532 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:12:43.532 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.532 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-03-21T13:12:43.532 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b", 2024-03-21T13:12:43.532 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:12:43.532 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 47, 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 330, 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6808", 2024-03-21T13:12:43.533 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:12:43.534 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.534 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.534 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:12:43.534 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.534 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.534 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6809", 2024-03-21T13:12:43.534 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:12:43.534 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.534 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.534 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6811", 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6810", 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:12:43.535 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6808/200077988", 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6809/200077988", 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6811/200077988", 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6810/200077988", 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49", 2024-03-21T13:12:43.536 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 56, 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 330, 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6812", 2024-03-21T13:12:43.537 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:12:43.538 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.538 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.538 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:12:43.538 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.538 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.538 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6813", 2024-03-21T13:12:43.538 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:12:43.538 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.538 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6815", 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:12:43.539 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6814", 2024-03-21T13:12:43.540 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:12:43.540 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.540 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.540 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6812/315664888", 2024-03-21T13:12:43.540 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6813/315664888", 2024-03-21T13:12:43.540 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6815/315664888", 2024-03-21T13:12:43.540 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6814/315664888", 2024-03-21T13:12:43.540 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:12:43.540 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:12:43.540 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:12:43.540 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.541 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.541 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:12:43.541 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_xinfo": [ 2024-03-21T13:12:43.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.541 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-03-21T13:12:43.541 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:12:43.541 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:12:43.541 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:12:43.541 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:12:43.541 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:12:43.541 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:11:49.779903+0000", 2024-03-21T13:12:43.541 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:12:43.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.542 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-03-21T13:12:43.542 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:12:43.542 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:12:43.542 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:12:43.542 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:12:43.542 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 65536, 2024-03-21T13:12:43.542 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:12:37.270978+0000", 2024-03-21T13:12:43.542 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:12:43.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.542 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-03-21T13:12:43.542 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:12:37.219676+0000", 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:12:43.543 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:12:13.849583+0000", 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:12:31.173973+0000", 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:12:29.765954+0000", 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:12:43.545 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:12:11.449032+0000", 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:11:34.120926+0000", 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:12:43.546 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.547 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:12:43.547 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap": [ 2024-03-21T13:12:43.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.547 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.13", 2024-03-21T13:12:43.547 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:12:43.547 INFO:teuthology.orchestra.run.smithi163.stdout: 0, 2024-03-21T13:12:43.547 INFO:teuthology.orchestra.run.smithi163.stdout: 5, 2024-03-21T13:12:43.547 INFO:teuthology.orchestra.run.smithi163.stdout: 6 2024-03-21T13:12:43.547 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.547 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "6.11", 2024-03-21T13:12:43.547 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:12:43.548 INFO:teuthology.orchestra.run.smithi163.stdout: 7, 2024-03-21T13:12:43.548 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:12:43.548 INFO:teuthology.orchestra.run.smithi163.stdout: 4 2024-03-21T13:12:43.548 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.548 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.548 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:12:43.548 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_items": [ 2024-03-21T13:12:43.548 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.548 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "3.2", 2024-03-21T13:12:43.548 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:12:43.548 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.548 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 3, 2024-03-21T13:12:43.548 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "3.19", 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 4, 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 7 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.549 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.19", 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 3, 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 7 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "5.7", 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 0, 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 6 2024-03-21T13:12:43.550 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "5.10", 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 7, 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 0 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "6.1c", 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:12:43.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 7, 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 0 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_primaries": [], 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_temp": [], 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_temp": [], 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: "blocklist": { 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3453855427": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/2571244897": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/3073378854": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:12:43.552 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1334591766": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/116934309": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3890775846": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1783921524": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2553440211": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1812694077": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1070540283": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/2354157786": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1591817392": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/4152198951": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/1798115633": "2024-03-22T12:54:00.224135+0000" 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: "range_blocklist": {}, 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profiles": { 2024-03-21T13:12:43.553 INFO:teuthology.orchestra.run.smithi163.stdout: "default": { 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van" 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps_queue": [], 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: "new_removed_snaps": [], 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: "new_purged_snaps": [], 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_node_flags": {}, 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: "device_class_flags": {}, 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode": { 2024-03-21T13:12:43.554 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_enabled": false, 2024-03-21T13:12:43.555 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_bucket_count": 0, 2024-03-21T13:12:43.555 INFO:teuthology.orchestra.run.smithi163.stdout: "degraded_stretch_mode": 0, 2024-03-21T13:12:43.555 INFO:teuthology.orchestra.run.smithi163.stdout: "recovering_stretch_mode": 0, 2024-03-21T13:12:43.555 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_bucket": 0 2024-03-21T13:12:43.555 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:12:43.555 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:12:43.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:43.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:43.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:43.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:43.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:43.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:43.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:43.855 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:43.892 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:43.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:43.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:43.994 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:44.031 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:44.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:44.122 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:12:44.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:44.170 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:44.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:44.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:44.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:44.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:44.388 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 12 16 31500 31484 163.964 124.125 0.00355212 0.00609364 2024-03-21T13:12:44.412 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:44.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:44.448 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:44.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:44 smithi163 ceph-mon[29621]: pgmap v663: 148 pgs: 148 active+clean; 1.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 789 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-03-21T13:12:44.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3693149831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:12:44.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:44 smithi163 ceph-mon[40675]: pgmap v663: 148 pgs: 148 active+clean; 1.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 789 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-03-21T13:12:44.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3693149831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:12:44.549 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:44.588 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:44.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:44.690 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:44.724 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:44.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:44.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:44 smithi204 ceph-mon[33153]: pgmap v663: 148 pgs: 148 active+clean; 1.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 789 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-03-21T13:12:44.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3693149831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:12:44.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:44.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:44.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:44.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:44.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:45.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:45.101 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:45.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:45.138 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:45.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:45.239 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:45.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:45.275 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:45.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:45.376 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:45.388 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 13 16 33637 33621 161.624 133.562 0.00663553 0.00618223 2024-03-21T13:12:45.413 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:45.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:45.514 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:45.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:45.552 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:45.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:45.653 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:45.691 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:45.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:45.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:45.793 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:45.827 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:45.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:45.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:45.929 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:45.966 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:45.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:45.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:46.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:12:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:12:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:12:46.068 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:46.105 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:46.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:46.206 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:46.248 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:46.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:46.350 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:46.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:46.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:46.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:46.389 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 14 16 35673 35657 159.168 127.25 0.00918131 0.00627691 2024-03-21T13:12:46.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:46.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:46.519 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:46.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:46.620 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:46.657 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:46.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:46.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:46.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:46 smithi204 ceph-mon[33153]: pgmap v664: 148 pgs: 148 active+clean; 1.4 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 115 MiB/s wr, 1.85k op/s 2024-03-21T13:12:46.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:46.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:46 smithi163 ceph-mon[29621]: pgmap v664: 148 pgs: 148 active+clean; 1.4 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 115 MiB/s wr, 1.85k op/s 2024-03-21T13:12:46.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:46 smithi163 ceph-mon[40675]: pgmap v664: 148 pgs: 148 active+clean; 1.4 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 115 MiB/s wr, 1.85k op/s 2024-03-21T13:12:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:46.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:46.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:46.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:46.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:46.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:46.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:47.039 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:47.063 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:47.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:47.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:47.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:47.203 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:47.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:47.305 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:47.343 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:47.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:47.388 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 15 16 37419 37403 155.831 109.125 0.0111113 0.00641194 2024-03-21T13:12:47.445 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:47.484 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:47.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:47.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:47.586 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:47.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:47.625 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:47.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:47.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:47.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:47.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:47.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:47.905 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:47.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:48.007 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:48.045 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:48.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:48.146 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:48.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:48.177 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:48.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:48.278 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:48.303 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:48.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:48.388 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 16 16 39024 39008 152.361 100.312 0.0043905 0.00655504 2024-03-21T13:12:48.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:48.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:48.444 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:48.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:48.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:48.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:48.584 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:48.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:48.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:48.724 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:48.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:48.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:48 smithi204 ceph-mon[33153]: pgmap v665: 148 pgs: 148 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 172 MiB/s wr, 2.75k op/s 2024-03-21T13:12:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:48 smithi163 ceph-mon[29621]: pgmap v665: 148 pgs: 148 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 172 MiB/s wr, 2.75k op/s 2024-03-21T13:12:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:48 smithi163 ceph-mon[40675]: pgmap v665: 148 pgs: 148 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 172 MiB/s wr, 2.75k op/s 2024-03-21T13:12:48.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:48.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:48.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:48.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:48.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:49.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:49.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:49.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:49.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:49.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:49.143 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:49.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:49.244 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:49.283 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:49.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:49.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:49.385 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:49.388 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 17 16 40974 40958 150.567 121.875 0.00587292 0.00663688 2024-03-21T13:12:49.421 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:49.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:49.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:49.523 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:49.563 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:49.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:49.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:49.665 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:49.702 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:49.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:49.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:49.841 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:49.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:49.942 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:49.976 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:49.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:49.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:50.079 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:50.123 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:50.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:50.225 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:50.270 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:50.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:50.372 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:50.389 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 18 16 42841 42825 148.684 116.688 0.00226152 0.00671085 2024-03-21T13:12:50.395 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:50.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:50.497 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:50.534 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:50.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:50.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:50.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:50 smithi163 ceph-mon[29621]: pgmap v666: 148 pgs: 148 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-21T13:12:50.672 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:50.672 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:50 smithi163 ceph-mon[40675]: pgmap v666: 148 pgs: 148 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-21T13:12:50.672 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:50.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:50.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:50.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:50 smithi204 ceph-mon[33153]: pgmap v666: 148 pgs: 148 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-21T13:12:50.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:50.775 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:50.812 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:50.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:50.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:50.914 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:50.975 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:50.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:51.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:51.113 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:51.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:51.214 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:51.253 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:51.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:51.354 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:51.389 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 19 16 44592 44576 146.618 109.438 0.010776 0.00681554 2024-03-21T13:12:51.389 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:51.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:51.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:51.491 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:51.525 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:51.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:51.627 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:51.663 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:51.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:51.690 INFO:tasks.thrashosds.thrasher:j is {'epoch': 337, 'fsid': 'd4bf79c0-e781-11ee-95cd-87774f69a715', 'created': '2024-03-21T12:52:56.031419+0000', 'modified': '2024-03-21T13:12:41.389506+0000', 'last_up_change': '2024-03-21T12:58:23.775302+0000', 'last_in_change': '2024-03-21T13:12:28.474723+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T12:56:20.557490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-21T12:58:35.713383+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-21T12:58:36.436384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-21T12:58:38.049304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-21T12:58:39.990512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-21T12:58:41.463686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-21T13:12:18.124611+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '332', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.4800000190734863, 'score_stable': 3.4800000190734863, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 3.059999942779541, 'raw_score_stable': 3.059999942779541, 'primary_affinity_weighted': 0.8329982757568359, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'c34a3b25-9e02-46e6-8f9a-d0af2615258f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.0852813720703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 299, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6801', 'nonce': 3086410981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6802', 'nonce': 3086410981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6804', 'nonce': 3086410981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6803', 'nonce': 3086410981}]}, 'public_addr': '172.21.15.163:6801/3086410981', 'cluster_addr': '172.21.15.163:6802/3086410981', 'heartbeat_back_addr': '172.21.15.163:6804/3086410981', 'heartbeat_front_addr': '172.21.15.163:6803/3086410981', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '36e05f89-c238-41f7-be65-1553b529e8ce', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6805', 'nonce': 3666062784}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6806', 'nonce': 3666062784}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6808', 'nonce': 3666062784}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6807', 'nonce': 3666062784}]}, 'public_addr': '172.21.15.163:6805/3666062784', 'cluster_addr': '172.21.15.163:6806/3666062784', 'heartbeat_back_addr': '172.21.15.163:6808/3666062784', 'heartbeat_front_addr': '172.21.15.163:6807/3666062784', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '968ec78d-25b8-4905-b328-d490c38ca85f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6809', 'nonce': 4009213507}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6810', 'nonce': 4009213507}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6812', 'nonce': 4009213507}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6811', 'nonce': 4009213507}]}, 'public_addr': '172.21.15.163:6809/4009213507', 'cluster_addr': '172.21.15.163:6810/4009213507', 'heartbeat_back_addr': '172.21.15.163:6812/4009213507', 'heartbeat_front_addr': '172.21.15.163:6811/4009213507', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '67dfa4f6-1709-48f6-971d-c6287233492d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6813', 'nonce': 3981878189}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6814', 'nonce': 3981878189}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6816', 'nonce': 3981878189}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6815', 'nonce': 3981878189}]}, 'public_addr': '172.21.15.163:6813/3981878189', 'cluster_addr': '172.21.15.163:6814/3981878189', 'heartbeat_back_addr': '172.21.15.163:6816/3981878189', 'heartbeat_front_addr': '172.21.15.163:6815/3981878189', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7fd311a9-f3c2-456d-8ab1-341cf5b33471', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.578704833984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 719276054}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6801', 'nonce': 719276054}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6803', 'nonce': 719276054}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 719276054}]}, 'public_addr': '172.21.15.204:6800/719276054', 'cluster_addr': '172.21.15.204:6801/719276054', 'heartbeat_back_addr': '172.21.15.204:6803/719276054', 'heartbeat_front_addr': '172.21.15.204:6802/719276054', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a87759c8-251c-415d-9387-a49f7d2f36d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 1346947668}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6805', 'nonce': 1346947668}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6807', 'nonce': 1346947668}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 1346947668}]}, 'public_addr': '172.21.15.204:6804/1346947668', 'cluster_addr': '172.21.15.204:6805/1346947668', 'heartbeat_back_addr': '172.21.15.204:6807/1346947668', 'heartbeat_front_addr': '172.21.15.204:6806/1346947668', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5eb135cd-f677-4d0c-b23a-738f31615b6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 200077988}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6809', 'nonce': 200077988}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6811', 'nonce': 200077988}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 200077988}]}, 'public_addr': '172.21.15.204:6808/200077988', 'cluster_addr': '172.21.15.204:6809/200077988', 'heartbeat_back_addr': '172.21.15.204:6811/200077988', 'heartbeat_front_addr': '172.21.15.204:6810/200077988', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4f528e10-2c93-4bc2-a54c-06ca4132ed49', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 315664888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6813', 'nonce': 315664888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6815', 'nonce': 315664888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 315664888}]}, 'public_addr': '172.21.15.204:6812/315664888', 'cluster_addr': '172.21.15.204:6813/315664888', 'heartbeat_back_addr': '172.21.15.204:6815/315664888', 'heartbeat_front_addr': '172.21.15.204:6814/315664888', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:11:49.779903+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-21T13:12:37.270978+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:12:37.219676+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:12:13.849583+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:12:31.173973+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:12:29.765954+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:12:11.449032+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:11:34.120926+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.13', 'osds': [0, 5, 6]}, {'pgid': '6.11', 'osds': [7, 3, 4]}], 'pg_upmap_items': [{'pgid': '3.2', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.19', 'mappings': [{'from': 4, 'to': 7}]}, {'pgid': '4.19', 'mappings': [{'from': 3, 'to': 7}]}, {'pgid': '5.7', 'mappings': [{'from': 0, 'to': 6}]}, {'pgid': '5.10', 'mappings': [{'from': 7, 'to': 0}]}, {'pgid': '6.1c', 'mappings': [{'from': 7, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.163:0/3453855427': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/2571244897': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:6800/3073378854': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/1334591766': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/116934309': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:0/3890775846': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/1783921524': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:6800/2553440211': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1812694077': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1070540283': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/2354157786': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1591817392': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/4152198951': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:6800/1798115633': '2024-03-22T12:54:00.224135+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-21T13:12:51.690 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 6.1c 2024-03-21T13:12:51.691 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd rm-pg-upmap-items 6.1c 2024-03-21T13:12:51.765 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:51.811 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:51.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:51.909 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:51.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:51.943 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:51.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:51.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:52.045 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:52.085 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:52.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:52.187 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:52.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:52.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:52.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:52.327 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:52.364 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:52.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:52.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:52.388 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:12:52.389526+0000 min lat: 0.00117124 max lat: 0.125378 avg lat: 0.00691838 2024-03-21T13:12:52.389 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:12:52.389 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 20 16 46239 46223 144.434 102.938 0.0121685 0.00691838 2024-03-21T13:12:52.426 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:12:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:12:52.453 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:12:52.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:52.501 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:52.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:52.569 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:52 smithi163 ceph-mon[29621]: pgmap v667: 148 pgs: 148 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 128 MiB/s wr, 2.04k op/s 2024-03-21T13:12:52.569 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:52 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3287997824' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:12:52.569 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:52 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:12:52.569 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:52 smithi163 ceph-mon[40675]: pgmap v667: 148 pgs: 148 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 128 MiB/s wr, 2.04k op/s 2024-03-21T13:12:52.569 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:52 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3287997824' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:12:52.569 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:52 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:12:52.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:52.630 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:52.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:52.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:52.732 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:52 smithi204 ceph-mon[33153]: pgmap v667: 148 pgs: 148 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 128 MiB/s wr, 2.04k op/s 2024-03-21T13:12:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:52 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3287997824' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:12:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:52 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:12:52.768 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:52.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:52.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:52.906 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:52.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:53.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:53.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:53.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:53.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:53.143 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:12:53.143 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:12:53.144 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-03-21T13:12:53.144 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-03-21T13:12:53.144 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:12:53.185 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:53.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:53.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:53.286 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:53.325 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:53.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:53.389 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 21 16 47949 47933 142.645 106.875 0.00413754 0.00700522 2024-03-21T13:12:53.426 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:53.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:53.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:53 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:12:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:53 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:12:53.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:53 smithi163 ceph-mon[29621]: osdmap e338: 8 total, 8 up, 7 in 2024-03-21T13:12:53.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:53.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:53 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:12:53.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:53 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:12:53.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:53 smithi163 ceph-mon[40675]: osdmap e338: 8 total, 8 up, 7 in 2024-03-21T13:12:53.564 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:53.599 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:53.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:53.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:53.740 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:53.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:53.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:53.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:53 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:12:53.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:53 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:12:53.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:53 smithi204 ceph-mon[33153]: osdmap e338: 8 total, 8 up, 7 in 2024-03-21T13:12:53.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:53.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:53.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:53.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:53.981 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:54.019 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:54.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:54.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:54.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:54.156 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:54.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:54.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:54.257 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:54.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:54.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:54.389 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 22 16 49540 49524 140.681 99.4375 0.00814777 0.00710325 2024-03-21T13:12:54.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:54.432 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:54.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:54.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:54.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:54 smithi163 ceph-mon[29621]: pgmap v669: 148 pgs: 148 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-21T13:12:54.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:54 smithi163 ceph-mon[40675]: pgmap v669: 148 pgs: 148 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-21T13:12:54.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:54.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:54.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:54.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:54.671 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:54.708 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:54.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:54.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:54 smithi204 ceph-mon[33153]: pgmap v669: 148 pgs: 148 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-21T13:12:54.810 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:54.844 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:54.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:54.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:54.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:54.985 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:54.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:55.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:55.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:55.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:55.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:55.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:55.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:55.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:55.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:55.389 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 23 16 51210 51194 139.102 104.375 0.00918687 0.0071842 2024-03-21T13:12:55.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:55.399 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:55.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:55.500 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:55.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:55.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:55.639 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:55.680 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:55.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:55.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:55 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:12:55.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:55 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:12:55.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:55 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:12:55.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:55 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:12:55.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:12:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:12:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:12:55.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:55 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:12:55.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:55 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:12:55.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:55.820 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:55.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:55.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:55.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:55.952 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:55.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:56.053 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:56.077 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:56.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:56.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:56.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:56.213 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:56.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:56.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:56.315 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:56.348 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:56.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:56.389 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 24 16 52766 52750 137.358 97.25 0.011626 0.00727517 2024-03-21T13:12:56.450 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:56.485 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:56.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:56.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:56.587 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:56.620 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:56.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:56.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:56.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:56 smithi204 ceph-mon[33153]: pgmap v670: 148 pgs: 148 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-21T13:12:56.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:56 smithi204 ceph-mon[33153]: osdmap e339: 8 total, 8 up, 7 in 2024-03-21T13:12:56.759 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:56.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:56 smithi163 ceph-mon[29621]: pgmap v670: 148 pgs: 148 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-21T13:12:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:56 smithi163 ceph-mon[29621]: osdmap e339: 8 total, 8 up, 7 in 2024-03-21T13:12:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:56 smithi163 ceph-mon[40675]: pgmap v670: 148 pgs: 148 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-21T13:12:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:56 smithi163 ceph-mon[40675]: osdmap e339: 8 total, 8 up, 7 in 2024-03-21T13:12:56.861 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:12:56.897 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:56.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:56.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:12:56.998 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:57.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:57.040 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:57.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:57.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:57.178 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:57.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:57.245 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:12:57.278 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-21T13:12:57.279 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:12:57.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:57.372 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:57.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:12:57.389 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 25 16 54397 54381 135.94 101.938 0.00669319 0.00735099 2024-03-21T13:12:57.474 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:57.509 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:57.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:57.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:12:57.646 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:57.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:57.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:12:57.748 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:57.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:57.802 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:57.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:57.838 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:57.872 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:57.894 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:57.902 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:57.961 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:57.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:58.063 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:58.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:58.108 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:58.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:58.209 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:58.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:58.251 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:58.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:58.352 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:58.389 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 26 16 55922 55906 134.377 95.3125 0.00858812 0.0074369 2024-03-21T13:12:58.390 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:58.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:58.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:58.492 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:58.532 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:58.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:58.603 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:12:58 smithi163 ceph-mon[29621]: pgmap v672: 148 pgs: 148 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-03-21T13:12:58.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:12:58 smithi163 ceph-mon[40675]: pgmap v672: 148 pgs: 148 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-03-21T13:12:58.634 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:58.671 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:58.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:12:58 smithi204 ceph-mon[33153]: pgmap v672: 148 pgs: 148 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-03-21T13:12:58.773 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:12:58.813 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:58.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:12:58.915 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:58.952 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:58.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:59.053 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:12:59.106 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:59.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:12:59.208 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:59.247 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:59.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:59.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:59.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:59.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:59.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:59.389 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 27 16 57511 57495 133.078 99.3125 0.00631414 0.00750865 2024-03-21T13:12:59.490 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:59.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:59.530 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:59.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:12:59.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:12:59.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:12:59.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:59.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:12:59.773 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:12:59.814 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:59.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:59.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:12:59.916 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:12:59.957 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:12:59.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:59.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:00.060 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:00.097 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:00.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:00.198 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:00.236 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:00.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:00.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:00.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:00.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:00.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:00.389 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 28 16 59025 59009 131.705 94.625 0.0133337 0.00758811 2024-03-21T13:13:00.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:00.486 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:00.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:00.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:00 smithi163 ceph-mon[29621]: pgmap v673: 148 pgs: 148 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T13:13:00.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:00 smithi163 ceph-mon[40675]: pgmap v673: 148 pgs: 148 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T13:13:00.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:00.623 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:00.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:00.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:00.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:00 smithi204 ceph-mon[33153]: pgmap v673: 148 pgs: 148 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T13:13:00.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:00.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:00.864 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:00.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:00.905 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:00.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:01.007 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:01.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:01.047 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:01.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:01.148 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:01.187 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:01.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:01.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:01.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:01.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:01.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:01.390 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 29 16 60422 60406 130.174 87.3125 0.00588523 0.00767756 2024-03-21T13:13:01.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:01.465 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:01.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:01.504 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:01 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:13:01.505 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:01 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:13:01.505 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:01 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2790802161' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:01.505 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:01 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:13:01.505 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:01 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:13:01.505 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:01 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2790802161' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:01.566 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:01.603 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:01.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:01.705 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:01.743 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:01.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:01.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:01 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:13:01.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:01 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:13:01.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:01 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2790802161' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:01.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:01.878 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:13:01.878 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:13:01.878 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:13:01.878 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:13:01.878 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:13:01.878 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:13:01.878 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:13:01.878 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:13:01.879 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:13:01.879 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:13:01.879 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:13:01.879 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:13:01.879 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:13:01.879 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:13:01.879 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:13:01.879 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:13:01.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:01.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:01.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:02.024 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:02.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:02.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:02.165 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:02.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:02.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:02.302 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:02.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:02.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:02.390 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 30 16 61878 61862 128.868 91 0.00345584 0.00775419 2024-03-21T13:13:02.404 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:02.440 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:02.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:02.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:02.473 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:13:02.481 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:13:02 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:13:02.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:02 smithi163 ceph-mon[29621]: pgmap v674: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-21T13:13:02.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:02 smithi163 ceph-mon[29621]: osdmap e340: 8 total, 8 up, 7 in 2024-03-21T13:13:02.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:02 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/856315191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:13:02.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:02 smithi163 ceph-mon[40675]: pgmap v674: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-21T13:13:02.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:02 smithi163 ceph-mon[40675]: osdmap e340: 8 total, 8 up, 7 in 2024-03-21T13:13:02.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:02 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/856315191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:13:02.543 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:02.580 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:02.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:02.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:02.682 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:02.724 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:02.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:02 smithi204 ceph-mon[33153]: pgmap v674: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-21T13:13:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:02 smithi204 ceph-mon[33153]: osdmap e340: 8 total, 8 up, 7 in 2024-03-21T13:13:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:02 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/856315191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:13:02.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:02.864 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:02.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:02.965 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:03.003 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:03.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:03.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:03.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:03.142 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:03.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:03.243 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:03.280 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:03.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:03.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:03.390 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 31 16 63368 63352 127.714 93.125 0.00768899 0.00782555 2024-03-21T13:13:03.418 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:03.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:03.520 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:03.555 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:03.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:03.657 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:03.688 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:03.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:03.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:03 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:13:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:03 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/856315191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:13:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:03 smithi204 ceph-mon[33153]: osdmap e341: 8 total, 8 up, 7 in 2024-03-21T13:13:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:03 smithi204 ceph-mon[33153]: 3.2 scrub starts 2024-03-21T13:13:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:03 smithi204 ceph-mon[33153]: 3.2 scrub ok 2024-03-21T13:13:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:03 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:13:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:03 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/856315191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:13:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:03 smithi163 ceph-mon[29621]: osdmap e341: 8 total, 8 up, 7 in 2024-03-21T13:13:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:03 smithi163 ceph-mon[29621]: 3.2 scrub starts 2024-03-21T13:13:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:03 smithi163 ceph-mon[29621]: 3.2 scrub ok 2024-03-21T13:13:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:03 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:13:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:03 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/856315191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:13:03.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:03 smithi163 ceph-mon[40675]: osdmap e341: 8 total, 8 up, 7 in 2024-03-21T13:13:03.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:03 smithi163 ceph-mon[40675]: 3.2 scrub starts 2024-03-21T13:13:03.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:03 smithi163 ceph-mon[40675]: 3.2 scrub ok 2024-03-21T13:13:03.790 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:03.829 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:03.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:03.931 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:03.970 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:03.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:04.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:04.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:04.109 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:04.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:04.210 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:04.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:04.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:04.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:04.350 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:04.389 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:04.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:04.390 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 32 16 64798 64782 126.516 89.375 0.0127411 0.00789876 2024-03-21T13:13:04.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:04.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:04.530 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:04.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:04.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:04.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:04.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:04.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:04 smithi163 ceph-mon[29621]: pgmap v677: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 688 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-21T13:13:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:04 smithi163 ceph-mon[29621]: 5.1a scrub starts 2024-03-21T13:13:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:04 smithi163 ceph-mon[29621]: 5.1a scrub ok 2024-03-21T13:13:04.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:04 smithi163 ceph-mon[40675]: pgmap v677: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 688 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-21T13:13:04.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:04 smithi163 ceph-mon[40675]: 5.1a scrub starts 2024-03-21T13:13:04.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:04 smithi163 ceph-mon[40675]: 5.1a scrub ok 2024-03-21T13:13:04.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:04.805 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:04.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:04.907 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:04.943 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:04.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:04 smithi204 ceph-mon[33153]: pgmap v677: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 688 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-21T13:13:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:04 smithi204 ceph-mon[33153]: 5.1a scrub starts 2024-03-21T13:13:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:04 smithi204 ceph-mon[33153]: 5.1a scrub ok 2024-03-21T13:13:05.045 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:05.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:05.098 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:05.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:05.166 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:13:05.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:05.236 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:05.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:05.338 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:05.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:05.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:05.390 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 33 16 66233 66217 125.4 89.6875 0.0121322 0.00797 2024-03-21T13:13:05.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:05.523 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:05.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:05.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:clear 6.1c pg_upmap_items mapping 2024-03-21T13:13:05.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:05.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:05.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:05 smithi163 ceph-mon[29621]: 4.14 scrub starts 2024-03-21T13:13:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:05 smithi163 ceph-mon[29621]: 4.14 scrub ok 2024-03-21T13:13:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:05 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2881526160' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.1c"}]: dispatch 2024-03-21T13:13:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:05 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.1c"}]: dispatch 2024-03-21T13:13:05.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:05.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:05 smithi163 ceph-mon[40675]: 4.14 scrub starts 2024-03-21T13:13:05.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:05 smithi163 ceph-mon[40675]: 4.14 scrub ok 2024-03-21T13:13:05.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:05 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2881526160' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.1c"}]: dispatch 2024-03-21T13:13:05.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:05 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.1c"}]: dispatch 2024-03-21T13:13:05.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:05.762 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:13:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:13:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:13:05.765 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:05.801 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:05.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:05.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:05.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:05.956 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:05.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:05.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:05 smithi204 ceph-mon[33153]: 4.14 scrub starts 2024-03-21T13:13:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:05 smithi204 ceph-mon[33153]: 4.14 scrub ok 2024-03-21T13:13:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:05 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2881526160' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.1c"}]: dispatch 2024-03-21T13:13:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:05 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.1c"}]: dispatch 2024-03-21T13:13:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:06.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:06.095 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:06.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:06.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:06.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:06.259 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:06.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:06.360 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:06.390 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 34 16 67659 67643 124.332 89.125 0.00550647 0.00803776 2024-03-21T13:13:06.401 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:06.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:06.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:06.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:06.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:06.641 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:06.673 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:06.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:06.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:06 smithi163 ceph-mon[29621]: pgmap v678: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-21T13:13:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:06 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.1c"}]': finished 2024-03-21T13:13:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:06 smithi163 ceph-mon[29621]: osdmap e342: 8 total, 8 up, 7 in 2024-03-21T13:13:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:06 smithi163 ceph-mon[29621]: 5.10 scrub starts 2024-03-21T13:13:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:06 smithi163 ceph-mon[29621]: 5.10 scrub ok 2024-03-21T13:13:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:06 smithi163 ceph-mon[29621]: osdmap e343: 8 total, 8 up, 7 in 2024-03-21T13:13:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:06 smithi163 ceph-mon[40675]: pgmap v678: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-21T13:13:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:06 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.1c"}]': finished 2024-03-21T13:13:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:06 smithi163 ceph-mon[40675]: osdmap e342: 8 total, 8 up, 7 in 2024-03-21T13:13:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:06 smithi163 ceph-mon[40675]: 5.10 scrub starts 2024-03-21T13:13:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:06 smithi163 ceph-mon[40675]: 5.10 scrub ok 2024-03-21T13:13:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:06 smithi163 ceph-mon[40675]: osdmap e343: 8 total, 8 up, 7 in 2024-03-21T13:13:06.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:06.820 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:06.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:06.922 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:06.959 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:06.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:06.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:06 smithi204 ceph-mon[33153]: pgmap v678: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-21T13:13:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:06 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.1c"}]': finished 2024-03-21T13:13:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:06 smithi204 ceph-mon[33153]: osdmap e342: 8 total, 8 up, 7 in 2024-03-21T13:13:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:06 smithi204 ceph-mon[33153]: 5.10 scrub starts 2024-03-21T13:13:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:06 smithi204 ceph-mon[33153]: 5.10 scrub ok 2024-03-21T13:13:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:06 smithi204 ceph-mon[33153]: osdmap e343: 8 total, 8 up, 7 in 2024-03-21T13:13:07.061 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:07.104 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:07.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:07.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:07.184 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:13:07.185 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:13:07.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:07.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:07.242 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:07.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:07.343 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:07.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:07.388 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:07.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:07.390 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 35 16 68979 68963 123.137 82.5 0.0096115 0.00811551 2024-03-21T13:13:07.472 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:07.489 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:07.538 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:07.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:07.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:07.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:07.682 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:07.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:07.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:07.686 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:13:07.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:07 smithi163 ceph-mon[40675]: 5.a scrub starts 2024-03-21T13:13:07.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:07 smithi163 ceph-mon[40675]: 5.a scrub ok 2024-03-21T13:13:07.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:07 smithi163 ceph-mon[40675]: pgmap v681: 148 pgs: 148 active+clean; 4.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 944 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-21T13:13:07.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:07 smithi163 ceph-mon[40675]: 4.18 scrub starts 2024-03-21T13:13:07.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:07 smithi163 ceph-mon[40675]: 4.18 scrub ok 2024-03-21T13:13:07.726 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:07 smithi163 ceph-mon[29621]: 5.a scrub starts 2024-03-21T13:13:07.726 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:07 smithi163 ceph-mon[29621]: 5.a scrub ok 2024-03-21T13:13:07.726 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:07 smithi163 ceph-mon[29621]: pgmap v681: 148 pgs: 148 active+clean; 4.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 944 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-21T13:13:07.726 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:07 smithi163 ceph-mon[29621]: 4.18 scrub starts 2024-03-21T13:13:07.726 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:07 smithi163 ceph-mon[29621]: 4.18 scrub ok 2024-03-21T13:13:07.784 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:07.825 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:07.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:07.927 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:07.972 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:07.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:07 smithi204 ceph-mon[33153]: 5.a scrub starts 2024-03-21T13:13:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:07 smithi204 ceph-mon[33153]: 5.a scrub ok 2024-03-21T13:13:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:07 smithi204 ceph-mon[33153]: pgmap v681: 148 pgs: 148 active+clean; 4.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 944 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-21T13:13:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:07 smithi204 ceph-mon[33153]: 4.18 scrub starts 2024-03-21T13:13:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:07 smithi204 ceph-mon[33153]: 4.18 scrub ok 2024-03-21T13:13:08.073 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:08.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-21T13:13:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:08.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:08.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:08.215 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:08.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:08.247 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:08.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:08.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:08.382 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:08.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:08.390 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 36 16 70299 70283 122.008 82.5 0.00810331 0.0081906 2024-03-21T13:13:08.484 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:08.522 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:08.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:08.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:08.562 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:08 smithi163 ceph-mon[29621]: 5.16 scrub starts 2024-03-21T13:13:08.563 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:08 smithi163 ceph-mon[29621]: 5.16 scrub ok 2024-03-21T13:13:08.563 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:08 smithi163 ceph-mon[29621]: 3.1a scrub starts 2024-03-21T13:13:08.563 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:08 smithi163 ceph-mon[29621]: 3.1a scrub ok 2024-03-21T13:13:08.563 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:08 smithi163 ceph-mon[29621]: 5.13 scrub starts 2024-03-21T13:13:08.563 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:08 smithi163 ceph-mon[29621]: 5.13 scrub ok 2024-03-21T13:13:08.563 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:08 smithi163 ceph-mon[40675]: 5.16 scrub starts 2024-03-21T13:13:08.563 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:08 smithi163 ceph-mon[40675]: 5.16 scrub ok 2024-03-21T13:13:08.563 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:08 smithi163 ceph-mon[40675]: 3.1a scrub starts 2024-03-21T13:13:08.563 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:08 smithi163 ceph-mon[40675]: 3.1a scrub ok 2024-03-21T13:13:08.563 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:08 smithi163 ceph-mon[40675]: 5.13 scrub starts 2024-03-21T13:13:08.563 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:08 smithi163 ceph-mon[40675]: 5.13 scrub ok 2024-03-21T13:13:08.624 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:08.662 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:08.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:08.764 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:08.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:08.799 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:08.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:08.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:08.935 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:08.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:08 smithi204 ceph-mon[33153]: 5.16 scrub starts 2024-03-21T13:13:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:08 smithi204 ceph-mon[33153]: 5.16 scrub ok 2024-03-21T13:13:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:08 smithi204 ceph-mon[33153]: 3.1a scrub starts 2024-03-21T13:13:09.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:08 smithi204 ceph-mon[33153]: 3.1a scrub ok 2024-03-21T13:13:09.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:08 smithi204 ceph-mon[33153]: 5.13 scrub starts 2024-03-21T13:13:09.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:08 smithi204 ceph-mon[33153]: 5.13 scrub ok 2024-03-21T13:13:09.037 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:09.068 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:09.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:09.170 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:09.203 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:09.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:09.305 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:09.342 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:09.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:09.390 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 37 16 71655 71639 121 84.75 0.0161899 0.00825956 2024-03-21T13:13:09.444 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:09.460 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:09.496 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:09.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:09.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:09.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:09.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:09 smithi163 ceph-mon[29621]: 3.4 scrub starts 2024-03-21T13:13:09.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:09 smithi163 ceph-mon[29621]: 3.4 scrub ok 2024-03-21T13:13:09.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:09 smithi163 ceph-mon[29621]: pgmap v682: 148 pgs: 148 active+clean; 4.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 797 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T13:13:09.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:09 smithi163 ceph-mon[29621]: 4.11 scrub starts 2024-03-21T13:13:09.620 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:09 smithi163 ceph-mon[29621]: 4.11 scrub ok 2024-03-21T13:13:09.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:09 smithi163 ceph-mon[40675]: 3.4 scrub starts 2024-03-21T13:13:09.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:09 smithi163 ceph-mon[40675]: 3.4 scrub ok 2024-03-21T13:13:09.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:09 smithi163 ceph-mon[40675]: pgmap v682: 148 pgs: 148 active+clean; 4.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 797 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T13:13:09.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:09 smithi163 ceph-mon[40675]: 4.11 scrub starts 2024-03-21T13:13:09.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:09 smithi163 ceph-mon[40675]: 4.11 scrub ok 2024-03-21T13:13:09.633 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:09.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:09.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:09.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:09.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:09.779 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:09.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:09.880 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:09.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:09.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:09.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:10.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:09 smithi204 ceph-mon[33153]: 3.4 scrub starts 2024-03-21T13:13:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:09 smithi204 ceph-mon[33153]: 3.4 scrub ok 2024-03-21T13:13:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:09 smithi204 ceph-mon[33153]: pgmap v682: 148 pgs: 148 active+clean; 4.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 797 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T13:13:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:09 smithi204 ceph-mon[33153]: 4.11 scrub starts 2024-03-21T13:13:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:09 smithi204 ceph-mon[33153]: 4.11 scrub ok 2024-03-21T13:13:10.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:10.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:10.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:10.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:10.163 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:10.200 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:10.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:10.302 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:10.310 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:13:10.310 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:13:10.310 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:13:10.310 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:13:10.310 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:13:10.321 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:13:10.330 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:13:10.347 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:10.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:10.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:10.367 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:13:10.390 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 38 16 72866 72850 119.808 75.6875 0.0186407 0.00834154 2024-03-21T13:13:10.401 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:13:10.406 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:13:10.406 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:13:10.406 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:13:10.406 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:13:10.406 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:13:10.406 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:13:10.406 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:13:10.406 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:13:10.449 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:10.483 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:10.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:10.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:10.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:10.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:10.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:10.718 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:10.755 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:10.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:10.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:13:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:13:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[40675]: 3.0 scrub starts 2024-03-21T13:13:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[40675]: 3.0 scrub ok 2024-03-21T13:13:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[40675]: 5.1 scrub starts 2024-03-21T13:13:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[40675]: 5.1 scrub ok 2024-03-21T13:13:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[40675]: 5.3 scrub starts 2024-03-21T13:13:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[40675]: 5.3 scrub ok 2024-03-21T13:13:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2008349300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:10.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:13:10.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:13:10.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[29621]: 3.0 scrub starts 2024-03-21T13:13:10.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[29621]: 3.0 scrub ok 2024-03-21T13:13:10.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[29621]: 5.1 scrub starts 2024-03-21T13:13:10.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[29621]: 5.1 scrub ok 2024-03-21T13:13:10.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[29621]: 5.3 scrub starts 2024-03-21T13:13:10.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[29621]: 5.3 scrub ok 2024-03-21T13:13:10.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:10 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2008349300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:10.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:10.900 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:10.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:10.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:11.002 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:10 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:13:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:10 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:13:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:10 smithi204 ceph-mon[33153]: 3.0 scrub starts 2024-03-21T13:13:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:10 smithi204 ceph-mon[33153]: 3.0 scrub ok 2024-03-21T13:13:11.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:10 smithi204 ceph-mon[33153]: 5.1 scrub starts 2024-03-21T13:13:11.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:10 smithi204 ceph-mon[33153]: 5.1 scrub ok 2024-03-21T13:13:11.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:10 smithi204 ceph-mon[33153]: 5.3 scrub starts 2024-03-21T13:13:11.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:10 smithi204 ceph-mon[33153]: 5.3 scrub ok 2024-03-21T13:13:11.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:10 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2008349300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:11.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:11.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:11.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:11.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:11.188 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:11.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:11.289 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:11.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:11.325 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:11.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:11.390 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 39 16 74135 74119 118.769 79.3125 0.00843027 0.00841481 2024-03-21T13:13:11.426 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:11.455 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:11.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:11.511 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 2, 5] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:13:11.511 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-21T13:13:11.558 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:11.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:11.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:11.696 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:11.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:11.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:11.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:11 smithi163 ceph-mon[29621]: pgmap v683: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-21T13:13:11.829 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:11 smithi163 ceph-mon[29621]: 5.1d scrub starts 2024-03-21T13:13:11.829 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:11 smithi163 ceph-mon[29621]: 5.1d scrub ok 2024-03-21T13:13:11.829 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:11 smithi163 ceph-mon[40675]: pgmap v683: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-21T13:13:11.829 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:11 smithi163 ceph-mon[40675]: 5.1d scrub starts 2024-03-21T13:13:11.829 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:11 smithi163 ceph-mon[40675]: 5.1d scrub ok 2024-03-21T13:13:11.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:11.866 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:11.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:11.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:11 smithi204 ceph-mon[33153]: pgmap v683: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-21T13:13:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:11 smithi204 ceph-mon[33153]: 5.1d scrub starts 2024-03-21T13:13:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:11 smithi204 ceph-mon[33153]: 5.1d scrub ok 2024-03-21T13:13:12.006 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:12.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:12.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:12.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:12.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:12.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:12.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:12.295 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:12.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:12.392 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:13:12.391472+0000 min lat: 0.00117124 max lat: 0.125378 avg lat: 0.00848564 2024-03-21T13:13:12.392 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:13:12.392 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 40 16 75402 75386 117.78 79.1875 0.0137192 0.00848564 2024-03-21T13:13:12.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:12.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:12.429 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:12.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:12.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:13:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:13:12.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:12.564 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:12.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:12.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:12.666 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:12.704 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:12.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:12.806 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:12.836 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:12.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:12.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:12.937 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:12.972 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:12 smithi204 ceph-mon[33153]: 6.15 scrub starts 2024-03-21T13:13:12.972 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:12 smithi204 ceph-mon[33153]: 3.1f scrub starts 2024-03-21T13:13:12.972 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:12 smithi204 ceph-mon[33153]: 3.1f scrub ok 2024-03-21T13:13:12.972 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:12 smithi204 ceph-mon[33153]: 6.15 scrub ok 2024-03-21T13:13:12.972 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:12 smithi204 ceph-mon[33153]: 6.14 scrub starts 2024-03-21T13:13:12.972 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:12 smithi204 ceph-mon[33153]: 6.14 scrub ok 2024-03-21T13:13:12.972 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:12 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:12.972 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:12 smithi204 ceph-mon[33153]: 3.b scrub starts 2024-03-21T13:13:12.972 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:12 smithi204 ceph-mon[33153]: 3.b scrub ok 2024-03-21T13:13:12.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:12.980 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:12.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[29621]: 6.15 scrub starts 2024-03-21T13:13:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[29621]: 3.1f scrub starts 2024-03-21T13:13:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[29621]: 3.1f scrub ok 2024-03-21T13:13:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[29621]: 6.15 scrub ok 2024-03-21T13:13:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[29621]: 6.14 scrub starts 2024-03-21T13:13:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[29621]: 6.14 scrub ok 2024-03-21T13:13:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[29621]: 3.b scrub starts 2024-03-21T13:13:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[29621]: 3.b scrub ok 2024-03-21T13:13:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[40675]: 6.15 scrub starts 2024-03-21T13:13:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[40675]: 3.1f scrub starts 2024-03-21T13:13:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[40675]: 3.1f scrub ok 2024-03-21T13:13:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[40675]: 6.15 scrub ok 2024-03-21T13:13:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[40675]: 6.14 scrub starts 2024-03-21T13:13:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[40675]: 6.14 scrub ok 2024-03-21T13:13:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[40675]: 3.b scrub starts 2024-03-21T13:13:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:12 smithi163 ceph-mon[40675]: 3.b scrub ok 2024-03-21T13:13:13.081 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:13.121 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:13.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:13.214 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:13.222 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:13.270 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:13.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:13.372 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:13.391 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 41 16 76696 76680 116.877 80.875 0.0059851 0.00855154 2024-03-21T13:13:13.407 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:13.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:13.509 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:13.546 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:13.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:13.648 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:13.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:13.685 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:13.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:13.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:13:13.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[29621]: 5.8 scrub starts 2024-03-21T13:13:13.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[29621]: 5.8 scrub ok 2024-03-21T13:13:13.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[29621]: pgmap v684: 148 pgs: 148 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-21T13:13:13.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[29621]: 5.11 scrub starts 2024-03-21T13:13:13.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[29621]: 5.11 scrub ok 2024-03-21T13:13:13.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[29621]: 4.1a scrub starts 2024-03-21T13:13:13.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[29621]: 4.1a scrub ok 2024-03-21T13:13:13.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[40675]: 5.8 scrub starts 2024-03-21T13:13:13.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[40675]: 5.8 scrub ok 2024-03-21T13:13:13.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[40675]: pgmap v684: 148 pgs: 148 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-21T13:13:13.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[40675]: 5.11 scrub starts 2024-03-21T13:13:13.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[40675]: 5.11 scrub ok 2024-03-21T13:13:13.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[40675]: 4.1a scrub starts 2024-03-21T13:13:13.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:13 smithi163 ceph-mon[40675]: 4.1a scrub ok 2024-03-21T13:13:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:13 smithi204 ceph-mon[33153]: 5.8 scrub starts 2024-03-21T13:13:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:13 smithi204 ceph-mon[33153]: 5.8 scrub ok 2024-03-21T13:13:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:13 smithi204 ceph-mon[33153]: pgmap v684: 148 pgs: 148 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-21T13:13:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:13 smithi204 ceph-mon[33153]: 5.11 scrub starts 2024-03-21T13:13:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:13 smithi204 ceph-mon[33153]: 5.11 scrub ok 2024-03-21T13:13:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:13 smithi204 ceph-mon[33153]: 4.1a scrub starts 2024-03-21T13:13:13.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:13 smithi204 ceph-mon[33153]: 4.1a scrub ok 2024-03-21T13:13:13.786 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:13.822 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:13.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:13.923 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:13.961 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:13.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:13.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:14.063 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:14.118 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:14.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:14.219 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:14.253 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:14.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:14.355 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:14.391 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:14.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:14.392 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 42 16 77847 77831 115.807 71.9375 0.0109636 0.0086298 2024-03-21T13:13:14.493 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:14.527 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:14.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:13:14.631 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:14.672 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:14.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:14 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2107300960' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:13:14.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:14 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:13:14.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:14.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:14.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:14.853 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:14 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2107300960' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:13:14.853 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:14 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:13:14.853 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:14 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2107300960' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:13:14.853 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:14 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:13:14.911 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:14.947 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:14.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:15.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:15.086 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:15.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:15.188 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:15.227 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:15.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:15.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:15.329 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:15.362 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:15.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:15.392 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 43 16 79033 79017 114.838 74.125 0.0272363 0.00870242 2024-03-21T13:13:15.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:15.505 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:15.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:15.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:15.607 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:15.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:15.643 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:15.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:15.678 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:15 smithi204 ceph-mon[33153]: 5.19 scrub starts 2024-03-21T13:13:15.678 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:15 smithi204 ceph-mon[33153]: 5.19 scrub ok 2024-03-21T13:13:15.678 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:15 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:13:15.678 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:15 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:13:15.678 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:15 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:13:15.679 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:15 smithi204 ceph-mon[33153]: osdmap e344: 8 total, 8 up, 7 in 2024-03-21T13:13:15.679 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:15 smithi204 ceph-mon[33153]: pgmap v686: 148 pgs: 148 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 603 B/s rd, 55 MiB/s wr, 887 op/s 2024-03-21T13:13:15.679 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:15 smithi204 ceph-mon[33153]: 6.1 scrub starts 2024-03-21T13:13:15.679 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:15 smithi204 ceph-mon[33153]: 6.1 scrub ok 2024-03-21T13:13:15.679 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:15 smithi204 ceph-mon[33153]: 6.b scrub starts 2024-03-21T13:13:15.679 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:15 smithi204 ceph-mon[33153]: 6.b scrub ok 2024-03-21T13:13:15.744 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:15.776 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:15.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:15.878 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:15.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:15.913 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:15.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:16.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:13:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:13:16.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[29621]: 5.19 scrub starts 2024-03-21T13:13:16.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[29621]: 5.19 scrub ok 2024-03-21T13:13:16.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:13:16.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:13:16.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:13:16.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[29621]: osdmap e344: 8 total, 8 up, 7 in 2024-03-21T13:13:16.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[29621]: pgmap v686: 148 pgs: 148 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 603 B/s rd, 55 MiB/s wr, 887 op/s 2024-03-21T13:13:16.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[29621]: 6.1 scrub starts 2024-03-21T13:13:16.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[29621]: 6.1 scrub ok 2024-03-21T13:13:16.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[29621]: 6.b scrub starts 2024-03-21T13:13:16.013 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[29621]: 6.b scrub ok 2024-03-21T13:13:16.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[40675]: 5.19 scrub starts 2024-03-21T13:13:16.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[40675]: 5.19 scrub ok 2024-03-21T13:13:16.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:13:16.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:13:16.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:13:16.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[40675]: osdmap e344: 8 total, 8 up, 7 in 2024-03-21T13:13:16.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[40675]: pgmap v686: 148 pgs: 148 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 603 B/s rd, 55 MiB/s wr, 887 op/s 2024-03-21T13:13:16.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[40675]: 6.1 scrub starts 2024-03-21T13:13:16.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[40675]: 6.1 scrub ok 2024-03-21T13:13:16.016 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[40675]: 6.b scrub starts 2024-03-21T13:13:16.016 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:15 smithi163 ceph-mon[40675]: 6.b scrub ok 2024-03-21T13:13:16.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:16.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:16.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:16.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:16.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:16.205 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:16.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:16.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:16.341 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:16.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:16.392 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 44 16 80233 80217 113.932 75 0.0321726 0.00877222 2024-03-21T13:13:16.443 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:16.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:16.483 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:16.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:16.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:16.609 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:16.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:16.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:16.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:16.745 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:16.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:16.772 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:16.846 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:16.899 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:16.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:16.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:17.001 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:17.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:16 smithi204 ceph-mon[33153]: 3.16 scrub starts 2024-03-21T13:13:17.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:16 smithi204 ceph-mon[33153]: 3.16 scrub ok 2024-03-21T13:13:17.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:16 smithi204 ceph-mon[33153]: 4.d scrub starts 2024-03-21T13:13:17.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:16 smithi204 ceph-mon[33153]: 4.d scrub ok 2024-03-21T13:13:17.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:16 smithi204 ceph-mon[33153]: 5.5 deep-scrub starts 2024-03-21T13:13:17.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:16 smithi204 ceph-mon[33153]: 5.5 deep-scrub ok 2024-03-21T13:13:17.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:16 smithi204 ceph-mon[33153]: osdmap e345: 8 total, 8 up, 7 in 2024-03-21T13:13:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[29621]: 3.16 scrub starts 2024-03-21T13:13:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[29621]: 3.16 scrub ok 2024-03-21T13:13:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[29621]: 4.d scrub starts 2024-03-21T13:13:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[29621]: 4.d scrub ok 2024-03-21T13:13:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[29621]: 5.5 deep-scrub starts 2024-03-21T13:13:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[29621]: 5.5 deep-scrub ok 2024-03-21T13:13:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[29621]: osdmap e345: 8 total, 8 up, 7 in 2024-03-21T13:13:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[40675]: 3.16 scrub starts 2024-03-21T13:13:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[40675]: 3.16 scrub ok 2024-03-21T13:13:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[40675]: 4.d scrub starts 2024-03-21T13:13:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[40675]: 4.d scrub ok 2024-03-21T13:13:17.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[40675]: 5.5 deep-scrub starts 2024-03-21T13:13:17.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[40675]: 5.5 deep-scrub ok 2024-03-21T13:13:17.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:16 smithi163 ceph-mon[40675]: osdmap e345: 8 total, 8 up, 7 in 2024-03-21T13:13:17.046 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:17.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:17.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:17.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:17.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:17.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:17.289 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:17.318 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:17.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:17.392 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 45 16 81435 81419 113.07 75.125 0.00710775 0.00883915 2024-03-21T13:13:17.420 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:17.451 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:17.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:17.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:17.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:17.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:17.597 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:17.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:17.645 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:13:17.645 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:13:17.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.645 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25481.0:81701 10.6 e345/330 10:67915a00:::benchmark_data_smithi163_154138_object81700:head v 345'5096, mlcod=345'5094)", 2024-03-21T13:13:17.646 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:17.628187+0000", 2024-03-21T13:13:17.646 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.014521322, 2024-03-21T13:13:17.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01455479, 2024-03-21T13:13:17.646 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:17.646 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:13:17.646 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:17.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:17.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.628187+0000", 2024-03-21T13:13:17.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:17.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.628187+0000", 2024-03-21T13:13:17.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:17.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.628187+0000", 2024-03-21T13:13:17.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8299999999999997e-07 2024-03-21T13:13:17.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:17.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.628212+0000", 2024-03-21T13:13:17.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4916e-05 2024-03-21T13:13:17.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:17.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.628219+0000", 2024-03-21T13:13:17.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5800000000000003e-06 2024-03-21T13:13:17.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:17.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.628226+0000", 2024-03-21T13:13:17.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1489999999999999e-06 2024-03-21T13:13:17.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:17.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.631289+0000", 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030632979999999999 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.631310+0000", 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0735000000000001e-05 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25481.0:81705 10.6 e345/330 10:6c396776:::benchmark_data_smithi163_154138_object81704:head v 345'5097, mlcod=345'5094)", 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:17.631703+0000", 2024-03-21T13:13:17.649 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.011005149000000001, 2024-03-21T13:13:17.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011115781, 2024-03-21T13:13:17.650 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:17.650 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:13:17.650 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:17.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:17.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.631703+0000", 2024-03-21T13:13:17.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:17.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.631703+0000", 2024-03-21T13:13:17.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:17.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.631704+0000", 2024-03-21T13:13:17.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0210000000000001e-06 2024-03-21T13:13:17.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:17.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.631733+0000", 2024-03-21T13:13:17.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9026999999999998e-05 2024-03-21T13:13:17.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:17.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.631744+0000", 2024-03-21T13:13:17.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1178e-05 2024-03-21T13:13:17.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:17.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.631753+0000", 2024-03-21T13:13:17.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7110000000000001e-06 2024-03-21T13:13:17.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:17.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.634579+0000", 2024-03-21T13:13:17.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028266810000000002 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.634603+0000", 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4079000000000001e-05 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25481.0:81706 10.0 e345/324 10:04d727f1:::benchmark_data_smithi163_154138_object81705:head v 345'5163, mlcod=345'5160)", 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:17.633267+0000", 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0094409769999999997, 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096337430000000002, 2024-03-21T13:13:17.653 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:17.654 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:13:17.654 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:17.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:17.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.633267+0000", 2024-03-21T13:13:17.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:17.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.633267+0000", 2024-03-21T13:13:17.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.633267+0000", 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.96e-07 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.633288+0000", 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1195999999999998e-05 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.633295+0000", 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5649999999999998e-06 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.633300+0000", 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2630000000000003e-06 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.634454+0000", 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001153334 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.634473+0000", 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9511999999999999e-05 2024-03-21T13:13:17.656 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25481.0:81713 10.4 e345/324 10:2013953e:::benchmark_data_smithi163_154138_object81712:head v 345'5120, mlcod=345'5118)", 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:17.638019+0000", 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0046886649999999998, 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0049516559999999996, 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:17.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638019+0000", 2024-03-21T13:13:17.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:17.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638019+0000", 2024-03-21T13:13:17.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:17.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638020+0000", 2024-03-21T13:13:17.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7000000000000003e-07 2024-03-21T13:13:17.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638054+0000", 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3979000000000001e-05 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638065+0000", 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1112e-05 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638073+0000", 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7060000000000004e-06 2024-03-21T13:13:17.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:17.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638182+0000", 2024-03-21T13:13:17.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108966 2024-03-21T13:13:17.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:17.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638201+0000", 2024-03-21T13:13:17.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.95e-05 2024-03-21T13:13:17.660 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.660 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:17.660 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25481.0:81710 10.6 e345/330 10:6445316a:::benchmark_data_smithi163_154138_object81709:head v 345'5098, mlcod=345'5094)", 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:17.638110+0000", 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.004598095, 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0049333700000000003, 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638110+0000", 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:17.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638110+0000", 2024-03-21T13:13:17.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:17.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638110+0000", 2024-03-21T13:13:17.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1800000000000001e-07 2024-03-21T13:13:17.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:17.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638133+0000", 2024-03-21T13:13:17.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3180000000000002e-05 2024-03-21T13:13:17.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:17.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638140+0000", 2024-03-21T13:13:17.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4590000000000003e-06 2024-03-21T13:13:17.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:17.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638145+0000", 2024-03-21T13:13:17.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0180000000000001e-06 2024-03-21T13:13:17.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638570+0000", 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00042552699999999998 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638589+0000", 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9032e-05 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:81716 10.2 10:4bb4ee4e:::benchmark_data_smithi163_154138_object81715:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:17.638263+0000", 2024-03-21T13:13:17.664 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.004444351, 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0048569640000000001, 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 81716 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638263+0000", 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:17.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638263+0000", 2024-03-21T13:13:17.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:17.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638265+0000", 2024-03-21T13:13:17.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4899999999999999e-06 2024-03-21T13:13:17.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638299+0000", 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4143999999999997e-05 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638311+0000", 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1697e-05 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638319+0000", 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7970000000000001e-06 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.639350+0000", 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001031059 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.639479+0000", 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012967799999999999 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.639619+0000", 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013934700000000001 2024-03-21T13:13:17.668 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25481.0:81715 10.c e345/324 10:31fde5f0:::benchmark_data_smithi163_154138_object81714:head v 345'5098, mlcod=345'5096)", 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:17.638990+0000", 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.003717555, 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042185130000000001, 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638990+0000", 2024-03-21T13:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:17.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638990+0000", 2024-03-21T13:13:17.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:17.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.638991+0000", 2024-03-21T13:13:17.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1799999999999995e-07 2024-03-21T13:13:17.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.639014+0000", 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3150999999999999e-05 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.639025+0000", 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0794e-05 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.639031+0000", 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1759999999999998e-06 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.639183+0000", 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000152294 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.639206+0000", 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2772000000000002e-05 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25481.0:81719 10.e e345/324)", 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:17.642536+0000", 2024-03-21T13:13:17.672 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.00017139, 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00073092299999999997, 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "queued for pg", 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.642536+0000", 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.642536+0000", 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:17.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.642537+0000", 2024-03-21T13:13:17.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1500000000000004e-07 2024-03-21T13:13:17.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:17.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.642561+0000", 2024-03-21T13:13:17.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3944999999999999e-05 2024-03-21T13:13:17.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:17.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.642573+0000", 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2397e-05 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.642582+0000", 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3189999999999995e-06 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25481.0:81693 10.0 e345/324 10:01aef9ec:::benchmark_data_smithi163_154138_object81692:head v 345'5162, mlcod=345'5160)", 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:17.616018+0000", 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.026690157999999999, 2024-03-21T13:13:17.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.027306538000000002, 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.616018+0000", 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.616018+0000", 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.616018+0000", 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.75e-07 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.616041+0000", 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2289e-05 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.616053+0000", 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2296000000000001e-05 2024-03-21T13:13:17.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.616059+0000", 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4810000000000003e-06 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.620828+0000", 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047686530000000003 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:17.620850+0000", 2024-03-21T13:13:17.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1540999999999999e-05 2024-03-21T13:13:17.679 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.679 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:17.679 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.679 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:17.679 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-03-21T13:13:17.679 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 9 2024-03-21T13:13:17.679 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:13:17.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:17.725 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:17 smithi204 ceph-mon[33153]: pgmap v688: 148 pgs: 148 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-03-21T13:13:17.728 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:17.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:17.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:17.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:17.848 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:17 smithi163 ceph-mon[29621]: pgmap v688: 148 pgs: 148 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-03-21T13:13:17.849 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:17 smithi163 ceph-mon[40675]: pgmap v688: 148 pgs: 148 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-03-21T13:13:17.875 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:17.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:17.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:17.977 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:18.016 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:18.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:18.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:18.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:18.151 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:18.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:18.253 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:18.276 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:18.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:18.377 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:18.392 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 46 16 82562 82546 112.143 70.4375 0.013178 0.00891203 2024-03-21T13:13:18.415 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:18.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:18.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:18.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:18.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:18.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:18.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:18.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:18.696 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:18.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:18.797 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:18.828 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:18.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:18.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:18.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:18.966 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:18.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:19.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:18 smithi163 ceph-mon[29621]: 4.15 scrub starts 2024-03-21T13:13:19.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:18 smithi163 ceph-mon[29621]: 4.15 scrub ok 2024-03-21T13:13:19.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:18 smithi163 ceph-mon[40675]: 4.15 scrub starts 2024-03-21T13:13:19.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:18 smithi163 ceph-mon[40675]: 4.15 scrub ok 2024-03-21T13:13:19.067 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:19.103 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:19.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:19.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:19.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:19.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:19.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:19.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:18 smithi204 ceph-mon[33153]: 4.15 scrub starts 2024-03-21T13:13:19.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:18 smithi204 ceph-mon[33153]: 4.15 scrub ok 2024-03-21T13:13:19.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:19.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:19.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:19.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:19.393 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 47 16 83643 83627 111.194 67.5625 0.0790514 0.00897855 2024-03-21T13:13:19.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:19.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:19.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:19.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:19.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:19.639 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:19.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:19.741 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:19.780 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:19.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:19.881 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:19.893 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:13:19.921 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:19.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:19.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:20.022 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:20.055 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:20.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:20.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:20.184 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:19 smithi163 ceph-mon[40675]: pgmap v689: 148 pgs: 148 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 824 op/s 2024-03-21T13:13:20.184 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:19 smithi163 ceph-mon[29621]: pgmap v689: 148 pgs: 148 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 824 op/s 2024-03-21T13:13:20.189 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:20.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:20.229 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:19 smithi204 ceph-mon[33153]: pgmap v689: 148 pgs: 148 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 824 op/s 2024-03-21T13:13:20.291 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:20.345 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:20.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:20.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:20.392 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 48 16 84787 84771 110.367 71.5 0.00974806 0.00905585 2024-03-21T13:13:20.447 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:20.486 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:20.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:20.587 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:20.610 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:20.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:20.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:20.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:20.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:20.743 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:20.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:20.790 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.0 to 1 (10000) 2024-03-21T13:13:20.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:20.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:20.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:20.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:20.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: 3.15 deep-scrub starts 2024-03-21T13:13:20.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: 3.15 deep-scrub ok 2024-03-21T13:13:20.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:13:20.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:13:20.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: 5.6 scrub starts 2024-03-21T13:13:20.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: 5.6 scrub ok 2024-03-21T13:13:20.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: osdmap e346: 8 total, 8 up, 7 in 2024-03-21T13:13:20.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/766182734' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:13:20.900 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:20.900 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:20.900 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: 4.9 deep-scrub starts 2024-03-21T13:13:20.900 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: 4.9 deep-scrub ok 2024-03-21T13:13:20.900 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]: dispatch 2024-03-21T13:13:20.900 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.12", "id": [7, 0]}]: dispatch 2024-03-21T13:13:20.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: 3.15 deep-scrub starts 2024-03-21T13:13:20.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: 3.15 deep-scrub ok 2024-03-21T13:13:20.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:13:20.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:13:20.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: 5.6 scrub starts 2024-03-21T13:13:20.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: 5.6 scrub ok 2024-03-21T13:13:20.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: osdmap e346: 8 total, 8 up, 7 in 2024-03-21T13:13:20.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/766182734' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:13:20.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:20.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:20.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: 4.9 deep-scrub starts 2024-03-21T13:13:20.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: 4.9 deep-scrub ok 2024-03-21T13:13:20.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]: dispatch 2024-03-21T13:13:20.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.12", "id": [7, 0]}]: dispatch 2024-03-21T13:13:20.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-21T13:13:20.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:21.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:21.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:21.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:21.213 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:21.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:21.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: 3.15 deep-scrub starts 2024-03-21T13:13:21.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: 3.15 deep-scrub ok 2024-03-21T13:13:21.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:13:21.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:13:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: 5.6 scrub starts 2024-03-21T13:13:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: 5.6 scrub ok 2024-03-21T13:13:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: osdmap e346: 8 total, 8 up, 7 in 2024-03-21T13:13:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/766182734' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:13:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: 4.9 deep-scrub starts 2024-03-21T13:13:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: 4.9 deep-scrub ok 2024-03-21T13:13:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]: dispatch 2024-03-21T13:13:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.12", "id": [7, 0]}]: dispatch 2024-03-21T13:13:21.315 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:21.344 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:21.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:21.392 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 49 15 85770 85755 109.37 61.5 0.0183392 0.00913621 2024-03-21T13:13:21.417 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:13:21.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:21.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:21.520 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:21.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:21.621 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:21.657 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:21.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:21.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:21.759 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:21.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:21.794 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:21.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:21.813 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:21 smithi204 ceph-mon[33153]: 3.14 scrub starts 2024-03-21T13:13:21.866 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-21T13:13:21.895 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:21.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:21.933 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:21.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:22.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:22.067 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:22.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:22.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:22.153 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[29621]: 3.14 scrub starts 2024-03-21T13:13:22.153 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[29621]: 3.14 scrub ok 2024-03-21T13:13:22.153 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[29621]: 4.4 scrub starts 2024-03-21T13:13:22.153 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[29621]: 4.4 scrub ok 2024-03-21T13:13:22.153 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/766182734' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:13:22.153 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]': finished 2024-03-21T13:13:22.153 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.12", "id": [7, 0]}]': finished 2024-03-21T13:13:22.153 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[29621]: osdmap e347: 8 total, 8 up, 7 in 2024-03-21T13:13:22.153 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[29621]: pgmap v692: 148 pgs: 148 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-21T13:13:22.153 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[29621]: 6.1f scrub starts 2024-03-21T13:13:22.154 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[29621]: 6.1f scrub ok 2024-03-21T13:13:22.154 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[29621]: osdmap e348: 8 total, 8 up, 7 in 2024-03-21T13:13:22.154 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[40675]: 3.14 scrub starts 2024-03-21T13:13:22.154 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[40675]: 3.14 scrub ok 2024-03-21T13:13:22.154 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[40675]: 4.4 scrub starts 2024-03-21T13:13:22.154 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[40675]: 4.4 scrub ok 2024-03-21T13:13:22.154 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/766182734' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:13:22.154 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]': finished 2024-03-21T13:13:22.154 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.12", "id": [7, 0]}]': finished 2024-03-21T13:13:22.154 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[40675]: osdmap e347: 8 total, 8 up, 7 in 2024-03-21T13:13:22.154 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[40675]: pgmap v692: 148 pgs: 148 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-21T13:13:22.155 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[40675]: 6.1f scrub starts 2024-03-21T13:13:22.155 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[40675]: 6.1f scrub ok 2024-03-21T13:13:22.155 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:21 smithi163 ceph-mon[40675]: osdmap e348: 8 total, 8 up, 7 in 2024-03-21T13:13:22.168 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:22.228 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:22.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:21 smithi204 ceph-mon[33153]: 3.14 scrub ok 2024-03-21T13:13:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:21 smithi204 ceph-mon[33153]: 4.4 scrub starts 2024-03-21T13:13:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:21 smithi204 ceph-mon[33153]: 4.4 scrub ok 2024-03-21T13:13:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:21 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/766182734' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:13:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]': finished 2024-03-21T13:13:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.12", "id": [7, 0]}]': finished 2024-03-21T13:13:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:21 smithi204 ceph-mon[33153]: osdmap e347: 8 total, 8 up, 7 in 2024-03-21T13:13:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:21 smithi204 ceph-mon[33153]: pgmap v692: 148 pgs: 148 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-21T13:13:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:21 smithi204 ceph-mon[33153]: 6.1f scrub starts 2024-03-21T13:13:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:21 smithi204 ceph-mon[33153]: 6.1f scrub ok 2024-03-21T13:13:22.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:21 smithi204 ceph-mon[33153]: osdmap e348: 8 total, 8 up, 7 in 2024-03-21T13:13:22.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:13:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:13:22.268 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:13:22.291 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:13:22.310 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:13:22.329 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:22.330 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:13:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:22.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:22.369 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:13:22.392 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 50 16 86776 86760 108.438 62.8125 0.00236503 0.00921144 2024-03-21T13:13:22.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:22.480 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:22.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:22.523 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:13:22.562 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:13:22.581 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:22.600 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:13:22.601 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:13:22.601 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:13:22.601 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:13:22.601 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:13:22.601 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:13:22.602 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:13:22.602 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:13:22.602 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:13:22.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:22.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:22.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:22.758 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:22.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:22.860 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:22.891 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:22 smithi204 ceph-mon[33153]: 3.12 scrub starts 2024-03-21T13:13:22.891 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:22 smithi204 ceph-mon[33153]: 3.12 scrub ok 2024-03-21T13:13:22.891 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:22 smithi204 ceph-mon[33153]: 5.b scrub starts 2024-03-21T13:13:22.891 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:22 smithi204 ceph-mon[33153]: 5.b scrub ok 2024-03-21T13:13:22.891 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:22 smithi204 ceph-mon[33153]: 5.2 scrub starts 2024-03-21T13:13:22.891 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:22 smithi204 ceph-mon[33153]: 5.2 scrub ok 2024-03-21T13:13:22.891 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:22 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:22.891 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:22 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3460534464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:22.891 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:22 smithi204 ceph-mon[33153]: 5.12 deep-scrub starts 2024-03-21T13:13:22.892 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:22 smithi204 ceph-mon[33153]: 5.12 deep-scrub ok 2024-03-21T13:13:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:22.894 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:22.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:22.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:23.031 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:23.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:23.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:23.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:23.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:23.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[29621]: 3.12 scrub starts 2024-03-21T13:13:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[29621]: 3.12 scrub ok 2024-03-21T13:13:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[29621]: 5.b scrub starts 2024-03-21T13:13:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[29621]: 5.b scrub ok 2024-03-21T13:13:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[29621]: 5.2 scrub starts 2024-03-21T13:13:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[29621]: 5.2 scrub ok 2024-03-21T13:13:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3460534464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[29621]: 5.12 deep-scrub starts 2024-03-21T13:13:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[29621]: 5.12 deep-scrub ok 2024-03-21T13:13:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[40675]: 3.12 scrub starts 2024-03-21T13:13:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[40675]: 3.12 scrub ok 2024-03-21T13:13:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[40675]: 5.b scrub starts 2024-03-21T13:13:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[40675]: 5.b scrub ok 2024-03-21T13:13:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[40675]: 5.2 scrub starts 2024-03-21T13:13:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[40675]: 5.2 scrub ok 2024-03-21T13:13:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3460534464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[40675]: 5.12 deep-scrub starts 2024-03-21T13:13:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:22 smithi163 ceph-mon[40675]: 5.12 deep-scrub ok 2024-03-21T13:13:23.271 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:23.310 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:23.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:23.392 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 51 16 87900 87884 107.69 70.25 0.0122792 0.00928195 2024-03-21T13:13:23.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:23.457 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:23.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:23.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:23.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:23.597 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:23.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:23.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:23.741 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:23.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:23.843 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:23.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:23.879 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:23.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:23.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:24.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:24.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:24.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:24.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:24.136 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:23 smithi204 ceph-mon[33153]: 4.12 scrub starts 2024-03-21T13:13:24.136 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:23 smithi204 ceph-mon[33153]: 4.12 scrub ok 2024-03-21T13:13:24.137 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:23 smithi204 ceph-mon[33153]: pgmap v694: 148 pgs: 148 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 59 MiB/s wr, 949 op/s 2024-03-21T13:13:24.137 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:23 smithi204 ceph-mon[33153]: 5.15 scrub starts 2024-03-21T13:13:24.137 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:23 smithi204 ceph-mon[33153]: 5.15 scrub ok 2024-03-21T13:13:24.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:24.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:24.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:24.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:23 smithi163 ceph-mon[29621]: 4.12 scrub starts 2024-03-21T13:13:24.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:23 smithi163 ceph-mon[29621]: 4.12 scrub ok 2024-03-21T13:13:24.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:23 smithi163 ceph-mon[29621]: pgmap v694: 148 pgs: 148 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 59 MiB/s wr, 949 op/s 2024-03-21T13:13:24.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:23 smithi163 ceph-mon[29621]: 5.15 scrub starts 2024-03-21T13:13:24.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:23 smithi163 ceph-mon[29621]: 5.15 scrub ok 2024-03-21T13:13:24.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:23 smithi163 ceph-mon[40675]: 4.12 scrub starts 2024-03-21T13:13:24.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:23 smithi163 ceph-mon[40675]: 4.12 scrub ok 2024-03-21T13:13:24.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:23 smithi163 ceph-mon[40675]: pgmap v694: 148 pgs: 148 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 59 MiB/s wr, 949 op/s 2024-03-21T13:13:24.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:23 smithi163 ceph-mon[40675]: 5.15 scrub starts 2024-03-21T13:13:24.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:23 smithi163 ceph-mon[40675]: 5.15 scrub ok 2024-03-21T13:13:24.277 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:24.314 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:24.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:24.393 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 52 16 89107 89091 107.069 75.4375 0.00695392 0.00933514 2024-03-21T13:13:24.416 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:24.457 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:24.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:24.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:24.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:24.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:24.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:24.696 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:24.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:24.735 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:24.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:24.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:24.864 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:13:24.864 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:13:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:24.865 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:24.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:24.966 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:25.003 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:25.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:25.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:25.144 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:25.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:25.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:24 smithi204 ceph-mon[33153]: 6.8 scrub starts 2024-03-21T13:13:25.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:24 smithi204 ceph-mon[33153]: 6.8 scrub ok 2024-03-21T13:13:25.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:24 smithi204 ceph-mon[33153]: 3.1 scrub starts 2024-03-21T13:13:25.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:24 smithi204 ceph-mon[33153]: 3.1 scrub ok 2024-03-21T13:13:25.246 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:25.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:24 smithi163 ceph-mon[29621]: 6.8 scrub starts 2024-03-21T13:13:25.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:24 smithi163 ceph-mon[29621]: 6.8 scrub ok 2024-03-21T13:13:25.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:24 smithi163 ceph-mon[29621]: 3.1 scrub starts 2024-03-21T13:13:25.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:24 smithi163 ceph-mon[29621]: 3.1 scrub ok 2024-03-21T13:13:25.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:24 smithi163 ceph-mon[40675]: 6.8 scrub starts 2024-03-21T13:13:25.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:24 smithi163 ceph-mon[40675]: 6.8 scrub ok 2024-03-21T13:13:25.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:24 smithi163 ceph-mon[40675]: 3.1 scrub starts 2024-03-21T13:13:25.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:24 smithi163 ceph-mon[40675]: 3.1 scrub ok 2024-03-21T13:13:25.284 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:25.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:25.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:25.394 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 53 16 90207 90191 106.345 68.75 0.00882259 0.00939844 2024-03-21T13:13:25.439 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:25.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:25.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:25.577 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:25.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:25.679 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:25.725 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:25.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:25.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:13:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:13:25.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:25.860 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:25.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:25.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:25.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:25.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:26.098 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:26.126 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:25 smithi204 ceph-mon[33153]: 2.0 scrub starts 2024-03-21T13:13:26.126 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:25 smithi204 ceph-mon[33153]: 2.0 scrub ok 2024-03-21T13:13:26.126 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:25 smithi204 ceph-mon[33153]: 6.f scrub starts 2024-03-21T13:13:26.126 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:25 smithi204 ceph-mon[33153]: 4.a scrub starts 2024-03-21T13:13:26.126 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:25 smithi204 ceph-mon[33153]: 4.a scrub ok 2024-03-21T13:13:26.126 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:25 smithi204 ceph-mon[33153]: 6.f scrub ok 2024-03-21T13:13:26.126 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:25 smithi204 ceph-mon[33153]: pgmap v695: 148 pgs: 148 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:13:26.126 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:25 smithi204 ceph-mon[33153]: 5.14 scrub starts 2024-03-21T13:13:26.127 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:25 smithi204 ceph-mon[33153]: 5.14 scrub ok 2024-03-21T13:13:26.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[29621]: 2.0 scrub starts 2024-03-21T13:13:26.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[29621]: 2.0 scrub ok 2024-03-21T13:13:26.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[29621]: 6.f scrub starts 2024-03-21T13:13:26.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[29621]: 4.a scrub starts 2024-03-21T13:13:26.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[29621]: 4.a scrub ok 2024-03-21T13:13:26.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[29621]: 6.f scrub ok 2024-03-21T13:13:26.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[29621]: pgmap v695: 148 pgs: 148 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:13:26.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[29621]: 5.14 scrub starts 2024-03-21T13:13:26.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[29621]: 5.14 scrub ok 2024-03-21T13:13:26.138 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[40675]: 2.0 scrub starts 2024-03-21T13:13:26.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[40675]: 2.0 scrub ok 2024-03-21T13:13:26.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[40675]: 6.f scrub starts 2024-03-21T13:13:26.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[40675]: 4.a scrub starts 2024-03-21T13:13:26.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[40675]: 4.a scrub ok 2024-03-21T13:13:26.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[40675]: 6.f scrub ok 2024-03-21T13:13:26.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[40675]: pgmap v695: 148 pgs: 148 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:13:26.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[40675]: 5.14 scrub starts 2024-03-21T13:13:26.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:25 smithi163 ceph-mon[40675]: 5.14 scrub ok 2024-03-21T13:13:26.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:26.147 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:26.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:26.175 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-21T13:13:26.248 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:26.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:26.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:26.393 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 54 16 91376 91360 105.729 73.0625 0.0163811 0.00945344 2024-03-21T13:13:26.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:26.400 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:26.437 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:26.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:26.539 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:26.559 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:13:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:26.580 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:26.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:26.681 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:26.730 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:26.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:26.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:26.875 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:26.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:26.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:26.977 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:26 smithi204 ceph-mon[33153]: 10.5 scrub starts 2024-03-21T13:13:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:26 smithi204 ceph-mon[33153]: 4.10 scrub starts 2024-03-21T13:13:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:26 smithi204 ceph-mon[33153]: 4.10 scrub ok 2024-03-21T13:13:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:26 smithi204 ceph-mon[33153]: osdmap e349: 8 total, 8 up, 7 in 2024-03-21T13:13:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:26 smithi163 ceph-mon[29621]: 10.5 scrub starts 2024-03-21T13:13:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:26 smithi163 ceph-mon[29621]: 4.10 scrub starts 2024-03-21T13:13:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:26 smithi163 ceph-mon[29621]: 4.10 scrub ok 2024-03-21T13:13:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:26 smithi163 ceph-mon[29621]: osdmap e349: 8 total, 8 up, 7 in 2024-03-21T13:13:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:26 smithi163 ceph-mon[40675]: 10.5 scrub starts 2024-03-21T13:13:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:26 smithi163 ceph-mon[40675]: 4.10 scrub starts 2024-03-21T13:13:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:26 smithi163 ceph-mon[40675]: 4.10 scrub ok 2024-03-21T13:13:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:26 smithi163 ceph-mon[40675]: osdmap e349: 8 total, 8 up, 7 in 2024-03-21T13:13:27.018 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:27.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:27.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:27.120 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:27.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:27.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:27.258 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:27.307 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:27.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:27.393 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 55 16 92499 92483 105.082 70.1875 0.00819848 0.00951163 2024-03-21T13:13:27.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:27.452 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:27.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:27.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:27.554 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:27.588 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:27.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:27.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:27.690 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:27.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:27.733 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:27.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:27.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:27.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:27.871 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:27.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:27.875 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:13:27.898 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:13:27.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:28.023 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:28.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:28.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:28.101 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:27 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1426843126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:13:28.101 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:27 smithi204 ceph-mon[33153]: pgmap v697: 148 pgs: 148 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 837 B/s rd, 59 MiB/s wr, 949 op/s; 35 B/s, 0 objects/s recovering 2024-03-21T13:13:28.101 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:27 smithi204 ceph-mon[33153]: 5.0 scrub starts 2024-03-21T13:13:28.101 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:27 smithi204 ceph-mon[33153]: 5.0 scrub ok 2024-03-21T13:13:28.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:28.171 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:28.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:28.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:28.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:27 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1426843126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:13:28.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:27 smithi163 ceph-mon[29621]: pgmap v697: 148 pgs: 148 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 837 B/s rd, 59 MiB/s wr, 949 op/s; 35 B/s, 0 objects/s recovering 2024-03-21T13:13:28.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:27 smithi163 ceph-mon[29621]: 5.0 scrub starts 2024-03-21T13:13:28.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:27 smithi163 ceph-mon[29621]: 5.0 scrub ok 2024-03-21T13:13:28.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:27 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1426843126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:13:28.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:27 smithi163 ceph-mon[40675]: pgmap v697: 148 pgs: 148 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 837 B/s rd, 59 MiB/s wr, 949 op/s; 35 B/s, 0 objects/s recovering 2024-03-21T13:13:28.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:27 smithi163 ceph-mon[40675]: 5.0 scrub starts 2024-03-21T13:13:28.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:27 smithi163 ceph-mon[40675]: 5.0 scrub ok 2024-03-21T13:13:28.273 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:28.309 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:28.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:28.393 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 56 15 93684 93669 104.529 74.125 0.00535327 0.0095617 2024-03-21T13:13:28.411 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:28.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:28.456 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:28.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:28.557 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:28.591 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:28.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:28.660 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:13:28.660 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:13:28.693 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:28.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:28.734 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:28.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:28.835 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:28.867 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:28.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:28.969 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:28 smithi163 ceph-mon[29621]: 10.5 scrub ok 2024-03-21T13:13:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:28 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:13:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:28 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1426843126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:13:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:28 smithi163 ceph-mon[29621]: osdmap e350: 8 total, 8 up, 7 in 2024-03-21T13:13:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:28 smithi163 ceph-mon[40675]: 10.5 scrub ok 2024-03-21T13:13:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:28 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:13:29.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:28 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1426843126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:13:29.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:28 smithi163 ceph-mon[40675]: osdmap e350: 8 total, 8 up, 7 in 2024-03-21T13:13:29.014 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:29.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:29.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:29.116 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:29.157 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:29.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:29.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:28 smithi204 ceph-mon[33153]: 10.5 scrub ok 2024-03-21T13:13:29.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:28 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:13:29.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:28 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1426843126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:13:29.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:28 smithi204 ceph-mon[33153]: osdmap e350: 8 total, 8 up, 7 in 2024-03-21T13:13:29.258 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:29.299 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:29.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:29.394 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 57 16 94904 94888 104.032 76.1875 0.0118645 0.0096075 2024-03-21T13:13:29.394 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:29.400 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:29.430 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-21T13:13:29.450 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:29.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:29.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:29.552 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:29.590 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:29.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:29.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:29.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:29.740 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:29.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:29.835 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:13:29.841 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:29.890 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.3 to 1 (10000) 2024-03-21T13:13:29.890 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:29.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:29.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:30.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:30.025 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:30.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:30.126 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:30.161 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:30.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:30.194 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:29 smithi204 ceph-mon[33153]: pgmap v699: 148 pgs: 148 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 719 B/s rd, 48 MiB/s wr, 762 op/s; 30 B/s, 0 objects/s recovering 2024-03-21T13:13:30.194 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:29 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1386666345' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:13:30.194 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:29 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:13:30.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:29 smithi163 ceph-mon[29621]: pgmap v699: 148 pgs: 148 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 719 B/s rd, 48 MiB/s wr, 762 op/s; 30 B/s, 0 objects/s recovering 2024-03-21T13:13:30.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:29 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1386666345' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:13:30.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:29 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:13:30.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:29 smithi163 ceph-mon[40675]: pgmap v699: 148 pgs: 148 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 719 B/s rd, 48 MiB/s wr, 762 op/s; 30 B/s, 0 objects/s recovering 2024-03-21T13:13:30.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:29 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1386666345' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:13:30.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:29 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:13:30.264 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:30.301 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:30.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:30.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:30.393 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 58 16 96016 96000 103.437 69.5 0.0101311 0.00966315 2024-03-21T13:13:30.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:30.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:30.442 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:30.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:30.543 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:30.581 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:30.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:30.684 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:30.715 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:30.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:30.768 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:13:30.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:30.853 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:30.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:30.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:30.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:31.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:31.006 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:31.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:31.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:30 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:13:31.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:30 smithi163 ceph-mon[29621]: osdmap e351: 8 total, 8 up, 7 in 2024-03-21T13:13:31.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:30 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:13:31.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:30 smithi163 ceph-mon[40675]: osdmap e351: 8 total, 8 up, 7 in 2024-03-21T13:13:31.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:31.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:31.146 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:31.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:31.247 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:31.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:30 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:13:31.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:30 smithi204 ceph-mon[33153]: osdmap e351: 8 total, 8 up, 7 in 2024-03-21T13:13:31.276 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:31.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:31.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:31.378 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:31.393 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 59 16 97196 97180 102.933 73.75 0.0158218 0.0097102 2024-03-21T13:13:31.412 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:31.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:31.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:31.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:31.560 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:31.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:31.662 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:31.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:31.832 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:31.867 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:31.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:31.969 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:32.002 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:32.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:32.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:32.104 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:32.139 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:32.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:32.241 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:32.243 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-21T13:13:32.322 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:32.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:32.394 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:13:32.394406+0000 min lat: 0.00117124 max lat: 0.239747 avg lat: 0.00975645 2024-03-21T13:13:32.394 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:13:32.394 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 60 16 98375 98359 102.446 73.6875 0.0181779 0.00975645 2024-03-21T13:13:32.417 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:13:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:13:32.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:32.463 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:32.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:32.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:32 smithi163 ceph-mon[29621]: pgmap v701: 148 pgs: 148 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.7 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 36 B/s, 0 objects/s recovering 2024-03-21T13:13:32.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:32 smithi163 ceph-mon[40675]: pgmap v701: 148 pgs: 148 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.7 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 36 B/s, 0 objects/s recovering 2024-03-21T13:13:32.565 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:32.591 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:32.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:32.693 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:32.732 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:32.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:32.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:32 smithi204 ceph-mon[33153]: pgmap v701: 148 pgs: 148 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.7 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 36 B/s, 0 objects/s recovering 2024-03-21T13:13:32.834 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:32.872 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:32.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:32.887 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:13:32.887 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:13:32.973 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:33.015 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:33.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:33.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:33.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:33.125 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:33.142 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:33.148 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:13:33.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:33.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:33.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:33.187 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:13:33.235 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:13:33.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:33.274 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:13:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:33.294 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:33.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:33.321 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:13:33.361 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:13:33.394 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 61 16 99467 99451 101.885 68.25 0.0133915 0.00981073 2024-03-21T13:13:33.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:33.403 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:13:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:33.436 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:33.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:33.440 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:13:33.449 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:13:33.449 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:13:33.449 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:13:33.449 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:13:33.449 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:13:33.449 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:13:33.449 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:13:33.449 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:13:33.537 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:33.582 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:33.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:33.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:33.682 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:33 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:33.682 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:33 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/676671788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:33.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:33 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:33.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:33 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/676671788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:33.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:33.732 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:33.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:33.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:33.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:33 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:33.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:33 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/676671788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:33.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-21T13:13:33.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:33.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:33.919 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:33.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:34.020 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:34.083 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:34.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:34.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:34.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:34.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:34.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:34.331 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:13:34.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:34.383 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:34.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:34.394 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 62 16 100601 100585 101.385 70.875 0.030589 0.00985817 2024-03-21T13:13:34.484 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:34.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:34 smithi204 ceph-mon[33153]: pgmap v702: 148 pgs: 148 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 789 B/s rd, 56 MiB/s wr, 903 op/s 2024-03-21T13:13:34.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:34.521 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:34.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:34.622 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:34.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:34.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:34.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:34 smithi163 ceph-mon[29621]: pgmap v702: 148 pgs: 148 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 789 B/s rd, 56 MiB/s wr, 903 op/s 2024-03-21T13:13:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:34 smithi163 ceph-mon[40675]: pgmap v702: 148 pgs: 148 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 789 B/s rd, 56 MiB/s wr, 903 op/s 2024-03-21T13:13:34.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:34.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:34.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:34.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:34.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:34.954 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:34.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:34.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:35.056 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:35.094 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:35.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:35.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:35.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:35.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:35.235 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:35.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:35.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:35.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:35.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:35.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:35.394 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 63 16 101882 101866 101.046 80.0625 0.00926619 0.00989201 2024-03-21T13:13:35.474 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:35.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:35.514 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:35.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:35.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:35.654 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:35.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:35.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:35.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:35.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:13:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:13:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:13:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:35.800 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:35.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:35.901 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:35.933 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:35.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:35.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:36.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:36.069 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:36.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:36.171 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:36.208 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:36.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:36.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:36.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:36 smithi204 ceph-mon[33153]: pgmap v703: 148 pgs: 148 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 823 op/s 2024-03-21T13:13:36.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:36 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:36.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:36 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:36.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:36.351 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:36.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:36.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:36.394 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 64 16 103074 103058 100.631 74.5 0.0104628 0.00993231 2024-03-21T13:13:36.453 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:36.487 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:36.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:36.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:36 smithi163 ceph-mon[29621]: pgmap v703: 148 pgs: 148 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 823 op/s 2024-03-21T13:13:36.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:36 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:36.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:36 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:36.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:36 smithi163 ceph-mon[40675]: pgmap v703: 148 pgs: 148 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 823 op/s 2024-03-21T13:13:36.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:36 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:36.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:36 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:36.532 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:13:36.532 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-03-21T13:13:36.532 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-03-21T13:13:36.532 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-03-21T13:13:36.533 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:13:36.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:36.635 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:36.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:36.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:36.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:36.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:36.779 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:36.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:36.880 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:36.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:36.920 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:36.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:37.021 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:37.059 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:37.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:37.161 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:37.203 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:37.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:37.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:37 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:13:37.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:37 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:13:37.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:37.359 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:37.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:37.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:37.394 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 65 16 104152 104136 100.12 67.375 0.0187305 0.00998309 2024-03-21T13:13:37.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:37.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:37 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:13:37.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:37.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:37.583 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-21T13:13:37.608 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:37.643 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:37.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:37.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:37.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:37.801 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:37.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:37.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:37.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:37.953 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:37.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:38.055 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:38.090 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:38.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:38.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:38.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:38.238 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:38.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:38.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:38.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:38 smithi163 ceph-mon[29621]: pgmap v704: 148 pgs: 148 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-21T13:13:38.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:38 smithi163 ceph-mon[40675]: pgmap v704: 148 pgs: 148 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-21T13:13:38.339 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:38.370 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:38.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:38.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:38.394 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 66 15 105250 105235 99.6435 68.6875 0.00967576 0.0100312 2024-03-21T13:13:38.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:38.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:38 smithi204 ceph-mon[33153]: pgmap v704: 148 pgs: 148 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-21T13:13:38.513 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:38.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:38.616 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:38.646 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:38.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:38.661 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:13:38.687 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:13:38.749 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:38.782 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:38.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:38.884 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:38.926 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:38.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:39.028 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:39.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:39.078 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:39.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:39.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:39.203 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:39.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:39.305 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:39.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:39.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:39.394 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 67 16 106302 106286 99.1366 65.6875 0.0091784 0.0100822 2024-03-21T13:13:39.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:39.466 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:39.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:39.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:39.565 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:39.568 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:39.631 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:39.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:39.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:39.733 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:39.776 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:39.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:39.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:39.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:39.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:40.012 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:40.043 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:40.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:40.145 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.4 to 1 (10000) 2024-03-21T13:13:40.185 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:13:40.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:40.206 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:40.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:40.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:40.328 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:40 smithi204 ceph-mon[33153]: pgmap v705: 148 pgs: 148 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-21T13:13:40.328 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:40 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/102589779' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:13:40.329 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:40 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:13:40.329 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:40 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1607731575' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:13:40.329 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:40 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:13:40.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:40.351 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:40.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:40.395 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 68 16 107414 107398 98.7007 69.5 0.0099289 0.0101271 2024-03-21T13:13:40.413 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:40 smithi163 ceph-mon[29621]: pgmap v705: 148 pgs: 148 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-21T13:13:40.413 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:40 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/102589779' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:13:40.413 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:40 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:13:40.413 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:40 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1607731575' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:13:40.413 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:40 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:13:40.413 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:40 smithi163 ceph-mon[40675]: pgmap v705: 148 pgs: 148 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-21T13:13:40.414 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:40 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/102589779' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:13:40.414 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:40 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:13:40.414 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:40 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1607731575' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:13:40.414 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:40 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:13:40.453 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:40.494 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:40.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:40.595 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:40.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:40.633 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:40.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:40.734 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:40.770 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:40.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:40.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:40.872 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:40.911 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:40.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:41.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:41.064 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:41.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:41.165 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:41.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:41.219 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:41.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:41.321 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:41.357 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:41.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:41.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:41.394 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 69 15 108612 108597 98.3562 74.9375 0.00711269 0.0101629 2024-03-21T13:13:41.450 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:41 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:13:41.450 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:41 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:13:41.450 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:41 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:13:41.450 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:41 smithi204 ceph-mon[33153]: osdmap e352: 8 total, 8 up, 7 in 2024-03-21T13:13:41.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:41.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:41 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:13:41.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:41 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:13:41.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:41 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:13:41.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:41 smithi163 ceph-mon[29621]: osdmap e352: 8 total, 8 up, 7 in 2024-03-21T13:13:41.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:41 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:13:41.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:41 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:13:41.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:41 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:13:41.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:41 smithi163 ceph-mon[40675]: osdmap e352: 8 total, 8 up, 7 in 2024-03-21T13:13:41.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:41.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:41.628 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:41.666 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:41.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:41.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:41.768 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:41.796 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:13:41.797 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-03-21T13:13:41.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-03-21T13:13:41.797 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:13:41.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.797 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:36863 10.6 10:6c204189:::benchmark_data_smithi163_154138_object36862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-03-21T13:13:41.797 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:12:47.016389+0000", 2024-03-21T13:13:41.797 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.768304931000003, 2024-03-21T13:13:41.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.068458837999999994, 2024-03-21T13:13:41.797 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.797 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.797 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.798 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.798 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.798 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 36863 2024-03-21T13:13:41.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.798 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.016389+0000", 2024-03-21T13:13:41.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.016389+0000", 2024-03-21T13:13:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.016390+0000", 2024-03-21T13:13:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9899999999999999e-07 2024-03-21T13:13:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.016458+0000", 2024-03-21T13:13:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8800999999999994e-05 2024-03-21T13:13:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.016472+0000", 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3427e-05 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.016480+0000", 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9570000000000002e-06 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.016531+0000", 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1514999999999997e-05 2024-03-21T13:13:41.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.016644+0000", 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112409 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.016769+0000", 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012490600000000001 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.019119+0000", 2024-03-21T13:13:41.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023499319999999999 2024-03-21T13:13:41.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.019314+0000", 2024-03-21T13:13:41.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000195038 2024-03-21T13:13:41.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.084769+0000", 2024-03-21T13:13:41.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.065455046000000003 2024-03-21T13:13:41.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.084806+0000", 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7772999999999999e-05 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:47.084848+0000", 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1235000000000001e-05 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:38922 10.9 10:9c56557c:::benchmark_data_smithi163_154138_object38921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:12:48.224138+0000", 2024-03-21T13:13:41.803 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.560555960999999, 2024-03-21T13:13:41.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.11783644, 2024-03-21T13:13:41.804 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.804 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.804 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.804 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.804 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.804 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 38922 2024-03-21T13:13:41.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.804 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.224138+0000", 2024-03-21T13:13:41.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.224138+0000", 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.224138+0000", 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2099999999999996e-07 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.224165+0000", 2024-03-21T13:13:41.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6443000000000001e-05 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.224171+0000", 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4119999999999998e-06 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.224177+0000", 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3900000000000001e-06 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.225792+0000", 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001615708 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.225910+0000", 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117549 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.226032+0000", 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000122504 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.227703+0000", 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001670749 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.230180+0000", 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024764399999999999 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.341895+0000", 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.11171542600000001 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.341938+0000", 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3356000000000003e-05 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:48.341974+0000", 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5842e-05 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.809 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:48771 10.9 10:9aeefb9c:::benchmark_data_smithi163_154138_object48770:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-03-21T13:13:41.810 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:12:53.864709+0000", 2024-03-21T13:13:41.810 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.919984941000003, 2024-03-21T13:13:41.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.083986034000000001, 2024-03-21T13:13:41.810 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.810 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.810 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.810 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.810 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.810 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 48771 2024-03-21T13:13:41.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.810 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.864709+0000", 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.864709+0000", 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.864714+0000", 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8439999999999997e-06 2024-03-21T13:13:41.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.864763+0000", 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9085999999999997e-05 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.864779+0000", 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5837000000000001e-05 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.864788+0000", 2024-03-21T13:13:41.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0599999999999997e-06 2024-03-21T13:13:41.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.813 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.864854+0000", 2024-03-21T13:13:41.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6068000000000005e-05 2024-03-21T13:13:41.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.813 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.864984+0000", 2024-03-21T13:13:41.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000130159 2024-03-21T13:13:41.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-03-21T13:13:41.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.865283+0000", 2024-03-21T13:13:41.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000299337 2024-03-21T13:13:41.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.871494+0000", 2024-03-21T13:13:41.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062105870000000001 2024-03-21T13:13:41.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.873421+0000", 2024-03-21T13:13:41.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001927032 2024-03-21T13:13:41.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.948627+0000", 2024-03-21T13:13:41.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.075206057000000007 2024-03-21T13:13:41.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.948665+0000", 2024-03-21T13:13:41.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8578999999999998e-05 2024-03-21T13:13:41.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.948695+0000", 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9388000000000001e-05 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:48773 10.9 10:9f738e9e:::benchmark_data_smithi163_154138_object48772:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:12:53.865594+0000", 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.919099676000002, 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.085256049, 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.816 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 48773 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.865594+0000", 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.865594+0000", 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.865595+0000", 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1629999999999999e-06 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.865654+0000", 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8399000000000001e-05 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.865665+0000", 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1796e-05 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.865673+0000", 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7829999999999999e-06 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.869297+0000", 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003624206 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.869423+0000", 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012535700000000001 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.869555+0000", 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013273000000000001 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.871632+0000", 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020766169999999998 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.875490+0000", 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038577199999999998 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.950786+0000", 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.075296554000000002 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.950823+0000", 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6712999999999999e-05 2024-03-21T13:13:41.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.950850+0000", 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7010999999999999e-05 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:48789 10.9 10:9b95dc92:::benchmark_data_smithi163_154138_object48788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:12:53.876464+0000", 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.908229499999997, 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.074528019000000001, 2024-03-21T13:13:41.822 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 48789 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.876464+0000", 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.876464+0000", 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.876465+0000", 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6499999999999998e-07 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.876541+0000", 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6340000000000004e-05 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.876555+0000", 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4035e-05 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.876563+0000", 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4379999999999998e-06 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.876621+0000", 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8e-05 2024-03-21T13:13:41.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.876733+0000", 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112119 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.876860+0000", 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012732100000000001 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.878866+0000", 2024-03-21T13:13:41.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020060640000000001 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.880769+0000", 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001902536 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.950899+0000", 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.070130089000000007 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.950956+0000", 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6718999999999997e-05 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:53.950992+0000", 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6492999999999998e-05 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:52288 10.6 10:69d4bc9a:::benchmark_data_smithi163_154138_object52287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:12:56.038290+0000", 2024-03-21T13:13:41.828 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 45.746403291999997, 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.082740734999999996, 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 52288 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.038290+0000", 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.038290+0000", 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.038292+0000", 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.223e-06 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.038325+0000", 2024-03-21T13:13:41.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3707000000000001e-05 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.038340+0000", 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4449e-05 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.038349+0000", 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1370000000000008e-06 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.038407+0000", 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8396000000000001e-05 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.038520+0000", 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011221 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.038662+0000", 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014270400000000001 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.041498+0000", 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028354080000000002 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.043433+0000", 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019351570000000001 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.120863+0000", 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.077430070000000004 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.120975+0000", 2024-03-21T13:13:41.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011254799999999999 2024-03-21T13:13:41.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:12:56.121031+0000", 2024-03-21T13:13:41.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5726000000000001e-05 2024-03-21T13:13:41.834 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.834 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.834 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.834 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:73113 10.6 10:659c9eff:::benchmark_data_smithi163_154138_object73112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:10.585980+0000", 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 31.198713595000001, 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.080570055000000002, 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 73113 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.585980+0000", 2024-03-21T13:13:41.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.585980+0000", 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.585987+0000", 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5930000000000002e-06 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.586024+0000", 2024-03-21T13:13:41.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7137e-05 2024-03-21T13:13:41.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.586034+0000", 2024-03-21T13:13:41.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0552e-05 2024-03-21T13:13:41.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.586042+0000", 2024-03-21T13:13:41.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9109999999999993e-06 2024-03-21T13:13:41.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.586856+0000", 2024-03-21T13:13:41.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00081334300000000001 2024-03-21T13:13:41.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.586989+0000", 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013344999999999999 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.587119+0000", 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012988400000000001 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.592643+0000", 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0055237410000000004 2024-03-21T13:13:41.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.592820+0000", 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017777799999999999 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.666290+0000", 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.073469534000000003 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.666460+0000", 2024-03-21T13:13:41.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017013400000000001 2024-03-21T13:13:41.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.666550+0000", 2024-03-21T13:13:41.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9998000000000001e-05 2024-03-21T13:13:41.841 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.841 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.841 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.841 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:73115 10.6 10:67b3854d:::benchmark_data_smithi163_154138_object73114:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-21T13:13:41.841 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:10.587412+0000", 2024-03-21T13:13:41.841 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 31.197281879999998, 2024-03-21T13:13:41.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.079344484000000007, 2024-03-21T13:13:41.842 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.842 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.842 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.842 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.842 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.842 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 73115 2024-03-21T13:13:41.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.842 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.587412+0000", 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.587412+0000", 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.587413+0000", 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.105e-06 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.587508+0000", 2024-03-21T13:13:41.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5187999999999997e-05 2024-03-21T13:13:41.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.587517+0000", 2024-03-21T13:13:41.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3659999999999998e-06 2024-03-21T13:13:41.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.587524+0000", 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8569999999999997e-06 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.589839+0000", 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023147850000000002 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.589971+0000", 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000131435 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.590167+0000", 2024-03-21T13:13:41.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000196126 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.592879+0000", 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027121760000000002 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.592972+0000", 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3010999999999994e-05 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.666633+0000", 2024-03-21T13:13:41.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.073661253999999995 2024-03-21T13:13:41.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.666716+0000", 2024-03-21T13:13:41.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2526000000000002e-05 2024-03-21T13:13:41.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.666756+0000", 2024-03-21T13:13:41.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0655000000000003e-05 2024-03-21T13:13:41.848 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.848 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.848 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.848 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:73121 10.6 10:682938d8:::benchmark_data_smithi163_154138_object73120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-21T13:13:41.848 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:10.594977+0000", 2024-03-21T13:13:41.848 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 31.189716397000002, 2024-03-21T13:13:41.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.080808611000000002, 2024-03-21T13:13:41.848 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 73121 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.594977+0000", 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.594977+0000", 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.594978+0000", 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7899999999999999e-07 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.595029+0000", 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1122999999999998e-05 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.595036+0000", 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1169999999999998e-06 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.595042+0000", 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9109999999999998e-06 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.596506+0000", 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014638209999999999 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.596631+0000", 2024-03-21T13:13:41.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012529800000000001 2024-03-21T13:13:41.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:13:41.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.596760+0000", 2024-03-21T13:13:41.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000128832 2024-03-21T13:13:41.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.666924+0000", 2024-03-21T13:13:41.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.070164159000000004 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.675419+0000", 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084945139999999999 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.675645+0000", 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022689100000000001 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.675743+0000", 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7607000000000002e-05 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.675786+0000", 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2858999999999997e-05 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:73124 10.6 10:6f4b1540:::benchmark_data_smithi163_154138_object73123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:10.595057+0000", 2024-03-21T13:13:41.854 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 31.189636242999999, 2024-03-21T13:13:41.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.081154232000000007, 2024-03-21T13:13:41.855 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.855 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.855 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.855 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.855 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.855 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 73124 2024-03-21T13:13:41.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.855 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.595057+0000", 2024-03-21T13:13:41.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.595057+0000", 2024-03-21T13:13:41.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.595058+0000", 2024-03-21T13:13:41.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3700000000000001e-07 2024-03-21T13:13:41.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.595084+0000", 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.605e-05 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.595090+0000", 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0290000000000002e-06 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.595095+0000", 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8339999999999998e-06 2024-03-21T13:13:41.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.663033+0000", 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.067938304000000005 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.663157+0000", 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000123678 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.663286+0000", 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012947600000000001 2024-03-21T13:13:41.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.667129+0000", 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038427829999999998 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.675424+0000", 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0082948799999999993 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.675828+0000", 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00040373299999999998 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.676043+0000", 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021545000000000001 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.676212+0000", 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016857800000000001 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.25481.0:73113 10.6 e343/330 ondisk, result = 0)", 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:10.598696+0000", 2024-03-21T13:13:41.860 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 31.185997492999999, 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.067863638000000004, 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.598696+0000", 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.598696+0000", 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.598697+0000", 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6300000000000005e-07 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.598702+0000", 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1290000000000004e-06 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.660428+0000", 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.061725543000000001 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.660438+0000", 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0648999999999999e-05 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.666246+0000", 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058078840000000001 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.666271+0000", 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4453000000000001e-05 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:10.666560+0000", 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028931699999999998 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:83624 10.6 10:6dcf6583:::benchmark_data_smithi163_154138_object83623:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:19.314517+0000", 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.470176413000001, 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.082736546999999994, 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.864 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.865 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.865 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.865 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 83624 2024-03-21T13:13:41.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.865 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.314517+0000", 2024-03-21T13:13:41.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:41.867 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:41.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:41.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.314517+0000", 2024-03-21T13:13:41.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.314519+0000", 2024-03-21T13:13:41.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.358e-06 2024-03-21T13:13:41.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.314550+0000", 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1069999999999999e-05 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.314560+0000", 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0327e-05 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.314568+0000", 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5839999999999997e-06 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.316542+0000", 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019740809999999999 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.316657+0000", 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114813 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.316782+0000", 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012591499999999999 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.320734+0000", 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039513839999999996 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.325359+0000", 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0046254829999999997 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.397123+0000", 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.071763338999999995 2024-03-21T13:13:41.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.397210+0000", 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7818000000000005e-05 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:19.397254+0000", 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3374999999999997e-05 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.872 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:86736 10.9 10:9d6d2d55:::benchmark_data_smithi163_154138_object86735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:22.273238+0000", 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 19.511455552000001, 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.073142612999999995, 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 86736 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.273238+0000", 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.273238+0000", 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.273240+0000", 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5579999999999999e-06 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.273298+0000", 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7827e-05 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.273309+0000", 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1861e-05 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.273317+0000", 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6229999999999997e-06 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.273366+0000", 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9079000000000002e-05 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.273456+0000", 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9919000000000004e-05 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.273546+0000", 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9975999999999994e-05 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.278679+0000", 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051329840000000002 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.278722+0000", 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3312999999999997e-05 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346301+0000", 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.067578312000000001 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346350+0000", 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9381e-05 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346381+0000", 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.078e-05 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.878 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:86737 10.9 10:933a2130:::benchmark_data_smithi163_154138_object86736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:22.274987+0000", 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 19.509706305999998, 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.071503387000000002, 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 86737 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.274987+0000", 2024-03-21T13:13:41.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.274987+0000", 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.274988+0000", 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3099999999999997e-07 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.275031+0000", 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3124999999999998e-05 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.275040+0000", 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1190000000000003e-06 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.275046+0000", 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.755e-06 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.275589+0000", 2024-03-21T13:13:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00054264199999999997 2024-03-21T13:13:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.275664+0000", 2024-03-21T13:13:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5012000000000006e-05 2024-03-21T13:13:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-03-21T13:13:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.275763+0000", 2024-03-21T13:13:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9327999999999995e-05 2024-03-21T13:13:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.278700+0000", 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029370030000000001 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.279238+0000", 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053779499999999998 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346422+0000", 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.067184025999999994 2024-03-21T13:13:41.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346457+0000", 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4579999999999998e-05 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346491+0000", 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4270999999999998e-05 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.884 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:86738 10.9 10:9e7f466a:::benchmark_data_smithi163_154138_object86737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:22.275247+0000", 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 19.509446429, 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.071341525000000003, 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 86738 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.275247+0000", 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.275247+0000", 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.275248+0000", 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0800000000000005e-07 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.275307+0000", 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8964e-05 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.275312+0000", 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2340000000000002e-06 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.275316+0000", 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.805e-06 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.277291+0000", 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019753940000000001 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.277406+0000", 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011523000000000001 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.277491+0000", 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4645e-05 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.278835+0000", 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013435949999999999 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.279430+0000", 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059526999999999996 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346524+0000", 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.067094218999999997 2024-03-21T13:13:41.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346555+0000", 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1043000000000001e-05 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346589+0000", 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3618000000000002e-05 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:86743 10.6 10:68e217fd:::benchmark_data_smithi163_154138_object86742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:22.346633+0000", 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 19.438060590999999, 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.147842855, 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 86743 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346633+0000", 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346633+0000", 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346634+0000", 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5600000000000004e-07 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346691+0000", 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7169000000000001e-05 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346699+0000", 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.418e-06 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346704+0000", 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3290000000000001e-06 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346754+0000", 2024-03-21T13:13:41.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0000999999999998e-05 2024-03-21T13:13:41.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346922+0000", 2024-03-21T13:13:41.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016808900000000001 2024-03-21T13:13:41.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:13:41.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.347068+0000", 2024-03-21T13:13:41.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014599200000000001 2024-03-21T13:13:41.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.494282+0000", 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.14721404299999999 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.494294+0000", 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.237e-05 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.494367+0000", 2024-03-21T13:13:41.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2541999999999998e-05 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.494423+0000", 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5776999999999998e-05 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.494476+0000", 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3168999999999999e-05 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:86744 10.6 10:61f5e644:::benchmark_data_smithi163_154138_object86743:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:22.346749+0000", 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 19.437944741999999, 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.149484533, 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 86744 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346749+0000", 2024-03-21T13:13:41.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346749+0000", 2024-03-21T13:13:41.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346750+0000", 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.99e-07 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346832+0000", 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2144999999999994e-05 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346839+0000", 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2219999999999996e-06 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.346845+0000", 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0719999999999996e-06 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.490731+0000", 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.14388594299999999 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.490904+0000", 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017266800000000001 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.491462+0000", 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00055826599999999997 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.494496+0000", 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030341970000000002 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.494696+0000", 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020032200000000001 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.496137+0000", 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014403739999999999 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.496191+0000", 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4014999999999997e-05 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.496233+0000", 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2710000000000003e-05 2024-03-21T13:13:41.902 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.25481.0:86743 10.6 e348/330 ondisk, result = 0)", 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:22.349607+0000", 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 19.435086792, 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.144688341, 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.349607+0000", 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.349607+0000", 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.349607+0000", 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.349611+0000", 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1709999999999998e-06 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.490685+0000", 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.141074117 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.490712+0000", 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7310000000000001e-05 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.494268+0000", 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035560650000000002 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.494275+0000", 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1760000000000004e-06 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.494295+0000", 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9911e-05 2024-03-21T13:13:41.906 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25481.0:86766 10.3 e348/330 10:c1200345:::benchmark_data_smithi163_154138_object86765:head v 348'5357, mlcod=348'5355)", 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:22.374546+0000", 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 19.410147598999998, 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.11967480899999999, 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.374546+0000", 2024-03-21T13:13:41.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.374546+0000", 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.374547+0000", 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.395e-06 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.374611+0000", 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3076000000000004e-05 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.490726+0000", 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.116115616 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.490737+0000", 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0331e-05 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.490794+0000", 2024-03-21T13:13:41.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7219999999999998e-05 2024-03-21T13:13:41.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.490866+0000", 2024-03-21T13:13:41.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2084999999999998e-05 2024-03-21T13:13:41.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.494176+0000", 2024-03-21T13:13:41.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033105959999999999 2024-03-21T13:13:41.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.494221+0000", 2024-03-21T13:13:41.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4490000000000003e-05 2024-03-21T13:13:41.911 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.911 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.911 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.911 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:108718 10.6 10:6c1f1d86:::benchmark_data_smithi163_154138_object108717:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-21T13:13:41.911 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:41.528129+0000", 2024-03-21T13:13:41.911 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.25656486499999998, 2024-03-21T13:13:41.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.068569033000000001, 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 108718 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.528129+0000", 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.528129+0000", 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.528134+0000", 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2889999999999997e-06 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.528175+0000", 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0871999999999998e-05 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:41.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.528191+0000", 2024-03-21T13:13:41.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.605e-05 2024-03-21T13:13:41.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:41.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.528201+0000", 2024-03-21T13:13:41.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0162e-05 2024-03-21T13:13:41.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:41.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.528332+0000", 2024-03-21T13:13:41.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000130686 2024-03-21T13:13:41.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.528495+0000", 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016336399999999999 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.528644+0000", 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014872900000000001 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.535090+0000", 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064456299999999999 2024-03-21T13:13:41.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:41.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.535105+0000", 2024-03-21T13:13:41.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5809999999999999e-05 2024-03-21T13:13:41.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:41.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.596566+0000", 2024-03-21T13:13:41.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.061460331999999999 2024-03-21T13:13:41.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:41.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.596644+0000", 2024-03-21T13:13:41.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8407999999999999e-05 2024-03-21T13:13:41.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:41.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:41.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:41.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.596698+0000", 2024-03-21T13:13:41.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3701e-05 2024-03-21T13:13:41.917 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.917 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.917 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.917 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:41.917 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:41.917 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:13:41.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:42.015 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:42.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:42.110 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-21T13:13:42.116 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:42.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:42.155 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:42.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:42.256 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:42.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:42.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:42.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:42.395 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 70 16 109643 109627 97.8694 64.375 0.0174594 0.0102127 2024-03-21T13:13:42.401 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:42.423 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:13:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:13:42.433 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:42.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:42.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:42.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:42.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:42.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:42.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:42.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:42.718 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:42.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:42.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:42 smithi204 ceph-mon[33153]: pgmap v707: 148 pgs: 148 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-21T13:13:42.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:42 smithi204 ceph-mon[33153]: osdmap e353: 8 total, 8 up, 7 in 2024-03-21T13:13:42.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:42 smithi163 ceph-mon[29621]: pgmap v707: 148 pgs: 148 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-21T13:13:42.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:42 smithi163 ceph-mon[29621]: osdmap e353: 8 total, 8 up, 7 in 2024-03-21T13:13:42.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:42 smithi163 ceph-mon[40675]: pgmap v707: 148 pgs: 148 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-21T13:13:42.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:42 smithi163 ceph-mon[40675]: osdmap e353: 8 total, 8 up, 7 in 2024-03-21T13:13:42.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:42.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:42.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:42.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:42.959 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:43.007 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:43.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:43.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:43.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:43.146 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:43.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:43.248 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:43.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:43.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:43.395 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 71 16 110766 110750 97.4795 70.1875 0.0241668 0.0102535 2024-03-21T13:13:43.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-21T13:13:43.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:43.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:43.456 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:43.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:43.557 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:43.592 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:43.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:43.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:43.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:43.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:43.758 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:43.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:43.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:43.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:43.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:43.860 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:43.886 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:13:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:43.898 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:43.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:43.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:44.046 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:44.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:44.125 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:44.147 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:44.218 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:44.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:44.319 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:44.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:44.359 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:44.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:44.395 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 72 16 111764 111748 96.9918 62.375 0.0246774 0.0103053 2024-03-21T13:13:44.460 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:44.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:44 smithi204 ceph-mon[33153]: pgmap v709: 148 pgs: 148 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.34k op/s 2024-03-21T13:13:44.489 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:44.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:44.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:44.630 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:44.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:44.697 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:44 smithi163 ceph-mon[29621]: pgmap v709: 148 pgs: 148 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.34k op/s 2024-03-21T13:13:44.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:44 smithi163 ceph-mon[40675]: pgmap v709: 148 pgs: 148 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.34k op/s 2024-03-21T13:13:44.733 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:44.769 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:44.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:44.827 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:13:44.871 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:44.936 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:13:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:44.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:44.980 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:13:45.025 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:13:45.044 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:45.062 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:13:45.075 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:45.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:45.102 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:13:45.136 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:13:45.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:45.181 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:13:45.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:45.216 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:45.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:45.230 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:13:45.248 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:13:45.249 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:13:45.249 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:13:45.249 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:13:45.249 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:13:45.249 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:13:45.249 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:13:45.249 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:13:45.317 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:45.354 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:45.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:45.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:45.396 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 73 16 112794 112778 96.545 64.375 0.0100884 0.0103529 2024-03-21T13:13:45.457 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:45.490 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:45.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:45.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:45 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:13:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:45 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:13:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:45 smithi163 ceph-mon[29621]: osdmap e354: 8 total, 8 up, 7 in 2024-03-21T13:13:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:45 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2847138705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:45 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:13:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:45 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:13:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:45 smithi163 ceph-mon[40675]: osdmap e354: 8 total, 8 up, 7 in 2024-03-21T13:13:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:45 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2847138705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:45.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:45.626 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:45.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:45.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:45.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:45 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:13:45.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:45 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:13:45.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:45 smithi204 ceph-mon[33153]: osdmap e354: 8 total, 8 up, 7 in 2024-03-21T13:13:45.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:45 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2847138705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:45.728 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:45.774 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:45.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:45.875 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:45.900 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:45.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:46.001 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:46.013 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:13:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:13:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:13:46.037 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:46.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:46.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:46.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:46.175 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:46.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:46.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:46.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:46.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:46.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:46.396 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 74 16 113953 113937 96.2191 72.4375 0.00917422 0.0103889 2024-03-21T13:13:46.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:46.466 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:46.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:46.568 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:46.602 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:46.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:46.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:46.703 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:46.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:46 smithi163 ceph-mon[29621]: pgmap v711: 148 pgs: 148 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T13:13:46.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:46 smithi163 ceph-mon[29621]: osdmap e355: 8 total, 8 up, 7 in 2024-03-21T13:13:46.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:46 smithi163 ceph-mon[40675]: pgmap v711: 148 pgs: 148 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T13:13:46.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:46 smithi163 ceph-mon[40675]: osdmap e355: 8 total, 8 up, 7 in 2024-03-21T13:13:46.740 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:46.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:46.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:46 smithi204 ceph-mon[33153]: pgmap v711: 148 pgs: 148 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T13:13:46.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:46 smithi204 ceph-mon[33153]: osdmap e355: 8 total, 8 up, 7 in 2024-03-21T13:13:46.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:46.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:46.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:46.982 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:47.016 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:47.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:47.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:47.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:47.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:47.157 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:47.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:47.258 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:47.286 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:47.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:47.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:47.388 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:47.396 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 75 16 115111 115095 95.9011 72.375 0.0151723 0.0104229 2024-03-21T13:13:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:47.423 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:47.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:47.524 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:47.561 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:47.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:47.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:47.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:47.702 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:47.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:47.804 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:47.834 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:47.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:47.935 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:47.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:47.984 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:47.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:48.026 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:13:48.026 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:13:48.085 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:48.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:48.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:48.224 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:48.265 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:48.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:48.366 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:48.396 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 76 16 116229 116213 95.5586 69.875 0.0190542 0.0104601 2024-03-21T13:13:48.404 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:48.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:48.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:48.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:48 smithi163 ceph-mon[29621]: pgmap v713: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 954 op/s 2024-03-21T13:13:48.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:48 smithi163 ceph-mon[40675]: pgmap v713: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 954 op/s 2024-03-21T13:13:48.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:48.547 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:48.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:48.648 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:48.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:48.688 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:48.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:48.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:48 smithi204 ceph-mon[33153]: pgmap v713: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 954 op/s 2024-03-21T13:13:48.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:48.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:48.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:48.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:48.968 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:48.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:49.012 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:49.070 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:49.117 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:49.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:49.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:49.218 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:49.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:49.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:49.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:49.396 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 77 16 117352 117336 95.229 70.1875 0.0210585 0.0104963 2024-03-21T13:13:49.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:49.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:49.491 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.6 to 1 (10000) 2024-03-21T13:13:49.505 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:49.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-21T13:13:49.544 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:49.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:49.646 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:49.659 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:13:49.681 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:49.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:49.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:49 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/519614072' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:13:49.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:49 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/519614072' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:13:49.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:49 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/519614072' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:13:49.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:49.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:49.848 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:49.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:49.949 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:49.979 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:49.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:50.081 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:50.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:50.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:50.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:50.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:50.264 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:50.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:50.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:50.396 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 78 16 118445 118429 94.8839 68.3125 0.0114882 0.0105345 2024-03-21T13:13:50.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:50.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:50.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:50.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:50.548 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:50.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:50.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:50.613 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:50 smithi163 ceph-mon[29621]: pgmap v714: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 684 B/s rd, 45 MiB/s wr, 720 op/s 2024-03-21T13:13:50.614 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:50 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/519614072' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:13:50.614 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:50 smithi163 ceph-mon[29621]: osdmap e356: 8 total, 8 up, 7 in 2024-03-21T13:13:50.614 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:50.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:50 smithi163 ceph-mon[40675]: pgmap v714: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 684 B/s rd, 45 MiB/s wr, 720 op/s 2024-03-21T13:13:50.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:50 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/519614072' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:13:50.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:50 smithi163 ceph-mon[40675]: osdmap e356: 8 total, 8 up, 7 in 2024-03-21T13:13:50.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:50.641 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:50 smithi204 ceph-mon[33153]: pgmap v714: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 684 B/s rd, 45 MiB/s wr, 720 op/s 2024-03-21T13:13:50.641 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:50 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/519614072' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:13:50.641 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:50 smithi204 ceph-mon[33153]: osdmap e356: 8 total, 8 up, 7 in 2024-03-21T13:13:50.641 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:50.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:50.688 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:50.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:50.789 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:50.823 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:50.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:50.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:50.925 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:50.941 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:13:50.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:50.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:50.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:51.076 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:51.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:51.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:51.217 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:51.249 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:51.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:51.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:51.396 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 79 16 119572 119556 94.5744 70.4375 0.0109234 0.0105687 2024-03-21T13:13:51.398 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:51.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:51.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:51.476 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-21T13:13:51.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:51.541 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:51.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:51.642 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:51.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:51.688 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:51.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:51.789 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:51.820 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:51.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:51.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:51.922 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:51.957 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:51.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:51.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:52.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:52.097 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:52.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:52.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:52.234 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:52.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:52.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:51 smithi204 ceph-mon[33153]: pgmap v716: 148 pgs: 148 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.5 KiB/s rd, 103 MiB/s wr, 1.66k op/s 2024-03-21T13:13:52.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:52.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:52.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:51 smithi204 ceph-mon[33153]: osdmap e357: 8 total, 8 up, 7 in 2024-03-21T13:13:52.253 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:13:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:13:52.259 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:51 smithi163 ceph-mon[29621]: pgmap v716: 148 pgs: 148 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.5 KiB/s rd, 103 MiB/s wr, 1.66k op/s 2024-03-21T13:13:52.259 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:52.259 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:52.259 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:51 smithi163 ceph-mon[29621]: osdmap e357: 8 total, 8 up, 7 in 2024-03-21T13:13:52.259 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:51 smithi163 ceph-mon[40675]: pgmap v716: 148 pgs: 148 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.5 KiB/s rd, 103 MiB/s wr, 1.66k op/s 2024-03-21T13:13:52.259 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:52.259 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:52.259 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:51 smithi163 ceph-mon[40675]: osdmap e357: 8 total, 8 up, 7 in 2024-03-21T13:13:52.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:52.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:52.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:52.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:52.396 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:13:52.397044+0000 min lat: 0.00117124 max lat: 0.239747 avg lat: 0.0106023 2024-03-21T13:13:52.396 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:13:52.396 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 80 16 120711 120695 94.282 71.1875 0.0148692 0.0106023 2024-03-21T13:13:52.489 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:52.524 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:52.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:52.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:52.626 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:52.684 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:52.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:52.786 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:52.826 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:52.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:52.928 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:52.969 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:52.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:52.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:53.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:53.107 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:53.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:53.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:53.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:52 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:13:53.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:52 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:13:53.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:52 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:53.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:52 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:53.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:52 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:53.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:53.222 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:53.243 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:53.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:53.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:53.255 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:13:53.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:52 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:13:53.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:52 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:13:53.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:52 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:53.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:52 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:53.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:52 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:53.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:52 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:13:53.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:52 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:13:53.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:52 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:53.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:52 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:53.263 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:52 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:53.345 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:53.367 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:13:53.367 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:13:53.382 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:53.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:53.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:53.396 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 81 16 121832 121816 93.9829 70.0625 0.00867223 0.0106358 2024-03-21T13:13:53.484 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:53.524 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:53.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:53.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:53.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:53.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:53.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:53.768 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:53.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:53.802 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:53.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:53.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:53.948 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:53.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:54.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:54.089 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:54.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:54.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:54.191 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:54.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:53 smithi163 ceph-mon[29621]: pgmap v718: 148 pgs: 148 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 789 B/s rd, 54 MiB/s wr, 870 op/s 2024-03-21T13:13:54.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:53 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:13:54.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:53 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:13:54.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:53 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:13:54.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:53 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:13:54.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:53 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:54.229 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:53 smithi163 ceph-mon[40675]: pgmap v718: 148 pgs: 148 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 789 B/s rd, 54 MiB/s wr, 870 op/s 2024-03-21T13:13:54.229 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:53 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:13:54.230 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:53 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:13:54.230 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:53 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:13:54.230 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:53 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:13:54.230 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:53 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:54.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:54.232 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:54.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:54.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:53 smithi204 ceph-mon[33153]: pgmap v718: 148 pgs: 148 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 789 B/s rd, 54 MiB/s wr, 870 op/s 2024-03-21T13:13:54.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:53 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:13:54.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:53 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:13:54.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:53 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:13:54.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:53 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:13:54.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:53 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:13:54.332 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:54.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:54.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:54.397 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 82 16 123049 123033 93.7641 76.0625 0.0140826 0.0106607 2024-03-21T13:13:54.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:54.520 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:54.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:54.621 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:54.658 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:54.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:54.760 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:13:54.785 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:54.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:13:54.887 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:54.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:54.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:55.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:55.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:55.063 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:55.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:55.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:54 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:13:55.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:54 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:13:55.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:54 smithi163 ceph-mon[29621]: osdmap e358: 8 total, 8 up, 7 in 2024-03-21T13:13:55.145 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:54 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:13:55.145 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:54 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:13:55.145 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:54 smithi163 ceph-mon[40675]: osdmap e358: 8 total, 8 up, 7 in 2024-03-21T13:13:55.163 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:55.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-21T13:13:55.204 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:55.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:55.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:55.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:54 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:13:55.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:54 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:13:55.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:54 smithi204 ceph-mon[33153]: osdmap e358: 8 total, 8 up, 7 in 2024-03-21T13:13:55.306 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:55.340 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:55.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:55.397 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 83 16 124201 124185 93.5018 72 0.016767 0.0106904 2024-03-21T13:13:55.442 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:55.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:55.491 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:55.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:55.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:55.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:55.645 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:13:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:55.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:55.747 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:55.762 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:13:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:13:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:13:55.789 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:55.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:55.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:55.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:55.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:55.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:55.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:56.027 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:56.067 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:56.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:56.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:56.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:56 smithi163 ceph-mon[29621]: pgmap v720: 148 pgs: 148 active+clean; 7.3 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:13:56.127 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:56 smithi163 ceph-mon[40675]: pgmap v720: 148 pgs: 148 active+clean; 7.3 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:13:56.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:13:56.202 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:56.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:13:56.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:56 smithi204 ceph-mon[33153]: pgmap v720: 148 pgs: 148 active+clean; 7.3 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T13:13:56.304 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:56.331 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:13:56.343 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:56.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:56.372 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:13:56.397 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 84 16 125376 125360 93.2629 73.4375 0.0125831 0.010718 2024-03-21T13:13:56.407 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:13:56.445 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:56.447 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:13:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:56.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:56.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:56.484 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:13:56.526 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:13:56.567 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:13:56.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:56.603 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:13:56.604 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:13:56.604 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:13:56.605 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:13:56.605 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:13:56.605 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:13:56.605 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:13:56.605 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:13:56.605 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:13:56.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:56.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:56.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:56.712 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:56.749 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:56.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:56.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:56.850 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:56.888 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:56.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:56.990 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:57.015 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:57.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:57.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:57.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:57.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:57 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:13:57.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:57 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:13:57.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:57 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2580130252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:57.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:57 smithi163 ceph-mon[29621]: osdmap e359: 8 total, 8 up, 7 in 2024-03-21T13:13:57.145 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:57 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:13:57.145 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:57 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:13:57.145 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:57 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2580130252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:57.145 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:57 smithi163 ceph-mon[40675]: osdmap e359: 8 total, 8 up, 7 in 2024-03-21T13:13:57.158 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:57.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:57.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:57.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:57 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:13:57.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:57 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:13:57.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:57 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2580130252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:13:57.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:57 smithi204 ceph-mon[33153]: osdmap e359: 8 total, 8 up, 7 in 2024-03-21T13:13:57.260 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:57.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:57.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:57.397 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 85 16 126474 126458 92.973 68.625 0.00939583 0.0107519 2024-03-21T13:13:57.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:57.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:57.453 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:57.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:57.554 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:57.596 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:57.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:57.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:57.740 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:57.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:57.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:57.845 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:57.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:57.894 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:57.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:57.995 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:58.046 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:58.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:58.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:58.178 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:58.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:58.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:58.280 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:58.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:58.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:58.397 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 86 16 127528 127512 92.6578 65.875 0.00734425 0.0107881 2024-03-21T13:13:58.417 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:58.463 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:58.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:58.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:58.479 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:13:58.479 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:58 smithi163 ceph-mon[29621]: pgmap v722: 148 pgs: 148 active+clean; 7.6 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-21T13:13:58.479 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:58 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/287131646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:13:58.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:58 smithi163 ceph-mon[40675]: pgmap v722: 148 pgs: 148 active+clean; 7.6 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-21T13:13:58.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:58 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/287131646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:13:58.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:58.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:58.603 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:58.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:13:58.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:58.744 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:58.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:58 smithi204 ceph-mon[33153]: pgmap v722: 148 pgs: 148 active+clean; 7.6 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-21T13:13:58.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:58 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/287131646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:13:58.846 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:58.879 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:58.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:58.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:58.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:13:59.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:59.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:59.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:13:59.136 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:13:59.169 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:59.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:13:59.271 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:59.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:59.305 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:59.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:59.397 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 87 16 128631 128615 92.3851 68.9375 0.029228 0.010819 2024-03-21T13:13:59.406 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:13:59.437 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:59.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:13:59.486 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.7 to 1 (10000) 2024-03-21T13:13:59.539 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:13:59.575 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:13:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:59.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:13:59.677 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:13:59.717 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:59.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:13:59.750 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:59 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:13:59.750 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:59 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/287131646' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:13:59.750 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:59 smithi204 ceph-mon[33153]: osdmap e360: 8 total, 8 up, 7 in 2024-03-21T13:13:59.750 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:59 smithi204 ceph-mon[33153]: 10.f scrub starts 2024-03-21T13:13:59.750 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:13:59 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3491039614' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:13:59.751 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:13:59.751 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-03-21T13:13:59.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-03-21T13:13:59.751 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:13:59.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.751 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:180928 8.3 8:c9037379:::benchmark_data_smithi163_107079_object45055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-21T13:13:59.751 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:08.363985+0000", 2024-03-21T13:13:59.751 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 411.38213874399997, 2024-03-21T13:13:59.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.993946526, 2024-03-21T13:13:59.751 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 180928 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.363985+0000", 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.363985+0000", 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.363986+0000", 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6360000000000001e-06 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.363992+0000", 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3809999999999998e-06 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.364005+0000", 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3112000000000001e-05 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.364015+0000", 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6989999999999998e-06 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.364094+0000", 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9091999999999994e-05 2024-03-21T13:13:59.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.364112+0000", 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7886000000000001e-05 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.353413+0000", 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98930138700000003 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.353519+0000", 2024-03-21T13:13:59.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010623100000000001 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,4", 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.353610+0000", 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0628e-05 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.357695+0000", 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040855850000000001 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.357709+0000", 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3341e-05 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.357824+0000", 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011544000000000001 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.357889+0000", 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5016000000000002e-05 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.757 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.357931+0000", 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2092000000000003e-05 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:180936 8.3 8:c3210a5c:::benchmark_data_smithi163_107079_object45063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:08.375804+0000", 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 411.37031953799999, 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98556327399999999, 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.758 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 180936 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375804+0000", 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375804+0000", 2024-03-21T13:13:59.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375805+0000", 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0920000000000001e-06 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375810+0000", 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2870000000000004e-06 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375820+0000", 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0121e-05 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375826+0000", 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5999999999999997e-06 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375897+0000", 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1359000000000006e-05 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375907+0000", 2024-03-21T13:13:59.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2210000000000003e-06 2024-03-21T13:13:59.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.355959+0000", 2024-03-21T13:13:59.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98005200599999998 2024-03-21T13:13:59.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.356079+0000", 2024-03-21T13:13:59.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119867 2024-03-21T13:13:59.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,4", 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.356175+0000", 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6142000000000002e-05 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361160+0000", 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0049857160000000003 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361217+0000", 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6706000000000002e-05 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361261+0000", 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4069000000000001e-05 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361319+0000", 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7938e-05 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361367+0000", 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8149999999999998e-05 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.764 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:180962 8.3 8:cd1c3043:::benchmark_data_smithi163_107079_object45089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:08.401153+0000", 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 411.34497061399998, 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96533337100000005, 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 180962 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.765 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401153+0000", 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401153+0000", 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401155+0000", 2024-03-21T13:13:59.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6440000000000001e-06 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401160+0000", 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9969999999999998e-06 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401171+0000", 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1049999999999999e-05 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401178+0000", 2024-03-21T13:13:59.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6459999999999993e-06 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401234+0000", 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5554999999999999e-05 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401243+0000", 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7679999999999997e-06 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.357990+0000", 2024-03-21T13:13:59.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95674721299999999 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.358108+0000", 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118019 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,4", 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.358207+0000", 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8783000000000006e-05 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361381+0000", 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003174035 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361428+0000", 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7685000000000002e-05 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.366384+0000", 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0049555010000000002 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.771 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.366438+0000", 2024-03-21T13:13:59.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4215000000000002e-05 2024-03-21T13:13:59.771 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.771 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.771 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.366486+0000", 2024-03-21T13:13:59.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8260000000000002e-05 2024-03-21T13:13:59.771 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.771 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.771 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.771 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.771 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:52326 9.0 9:038ef788:::benchmark_data_smithi163_131650_object52325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e246)", 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:17.468332+0000", 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 282.27779176500002, 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99509188100000001, 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 52326 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.468332+0000", 2024-03-21T13:13:59.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.468332+0000", 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.468335+0000", 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8090000000000001e-06 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.468380+0000", 2024-03-21T13:13:59.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5525000000000002e-05 2024-03-21T13:13:59.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.468399+0000", 2024-03-21T13:13:59.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8855000000000002e-05 2024-03-21T13:13:59.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.468410+0000", 2024-03-21T13:13:59.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1066999999999999e-05 2024-03-21T13:13:59.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.551585+0000", 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.083175313000000001 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.551597+0000", 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1311999999999999e-05 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:18.460573+0000", 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.90897664600000005 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:18.461383+0000", 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00080961300000000004 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:18.463342+0000", 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019586120000000002 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:18.463424+0000", 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2128999999999999e-05 2024-03-21T13:13:59.776 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.777 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.777 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.777 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:54275 9.18 9.cf8139d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-03-21T13:13:59.777 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:19.460531+0000", 2024-03-21T13:13:59.777 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 280.28559257299997, 2024-03-21T13:13:59.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97871142700000002, 2024-03-21T13:13:59.777 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.777 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:13:59.777 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.777 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:13:59.778 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:59 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:13:59.778 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:59 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/287131646' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:13:59.778 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:59 smithi163 ceph-mon[40675]: osdmap e360: 8 total, 8 up, 7 in 2024-03-21T13:13:59.778 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:59 smithi163 ceph-mon[40675]: 10.f scrub starts 2024-03-21T13:13:59.778 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:13:59 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3491039614' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:13:59.779 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:59 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:13:59.779 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:59 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/287131646' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:13:59.779 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:59 smithi163 ceph-mon[29621]: osdmap e360: 8 total, 8 up, 7 in 2024-03-21T13:13:59.779 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:59 smithi163 ceph-mon[29621]: 10.f scrub starts 2024-03-21T13:13:59.779 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:13:59 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3491039614' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:13:59.779 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:13:59.779 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 54275 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460531+0000", 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460531+0000", 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460532+0000", 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.144e-06 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460554+0000", 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1467999999999998e-05 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460563+0000", 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.55e-06 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460570+0000", 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5030000000000002e-06 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460608+0000", 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8875999999999998e-05 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460613+0000", 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6789999999999996e-06 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:20.439232+0000", 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97861839500000003 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:20.439242+0000", 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0812000000000001e-05 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100381 9.b 9:dd44d674:::benchmark_data_smithi163_131650_object100380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.561457+0000", 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.18466696799999, 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1224653619999998, 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100381 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.561457+0000", 2024-03-21T13:13:59.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.561457+0000", 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.561458+0000", 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.502e-06 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.561495+0000", 2024-03-21T13:13:59.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7379000000000002e-05 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.561512+0000", 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6322000000000001e-05 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.561520+0000", 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5469999999999997e-06 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.562792+0000", 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001272198 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.562944+0000", 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000151242 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.563101+0000", 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015759199999999999 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567303+0000", 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042017759999999999 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567324+0000", 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0729000000000001e-05 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.586023+0000", 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018698887000000001 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.586037+0000", 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4079e-05 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574162+0000", 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98812536900000003 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574175+0000", 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3006e-05 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.683847+0000", 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1096722409999999 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.683922+0000", 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4493000000000007e-05 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100386 9.b 9:d957d1b3:::benchmark_data_smithi163_131650_object100385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.567504+0000", 2024-03-21T13:13:59.790 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.17861933899999, 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.129416671, 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100386 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567504+0000", 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567504+0000", 2024-03-21T13:13:59.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567506+0000", 2024-03-21T13:13:59.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.345e-06 2024-03-21T13:13:59.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567541+0000", 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5472999999999997e-05 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567555+0000", 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3511e-05 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567563+0000", 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3669999999999997e-06 2024-03-21T13:13:59.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567640+0000", 2024-03-21T13:13:59.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7366000000000004e-05 2024-03-21T13:13:59.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567793+0000", 2024-03-21T13:13:59.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015228 2024-03-21T13:13:59.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:13:59.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.568487+0000", 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00069475500000000003 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571382+0000", 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028947220000000002 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.588432+0000", 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017050108000000001 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.588443+0000", 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1352000000000001e-05 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574213+0000", 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98576913200000005 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574223+0000", 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0275000000000001e-05 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.684130+0000", 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.10990674 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.684673+0000", 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00054377400000000002 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.696860+0000", 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012186591 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.696921+0000", 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.088e-05 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100389 9.0 9:063bd6a6:::benchmark_data_smithi163_131650_object100388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.571614+0000", 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.174509481, 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.256985604, 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.798 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100389 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571614+0000", 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571614+0000", 2024-03-21T13:13:59.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571616+0000", 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.748e-06 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571650+0000", 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4409000000000002e-05 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571664+0000", 2024-03-21T13:13:59.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.36e-05 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571673+0000", 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3770000000000002e-06 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.592173+0000", 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020499746999999999 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.592183+0000", 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8800000000000003e-06 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.571390+0000", 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.979206727 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.571404+0000", 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4791e-05 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.803242+0000", 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2318381460000001 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.803399+0000", 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000156587 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.803796+0000", 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00039736600000000002 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.818306+0000", 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014509762000000001 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.818452+0000", 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014624500000000001 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.824862+0000", 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064098009999999997 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.824923+0000", 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0291000000000003e-05 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.828600+0000", 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036771270000000001 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.805 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100403 9.b 9:d27bacb7:::benchmark_data_smithi163_131650_object100402:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:13:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.597070+0000", 2024-03-21T13:13:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.14905328699999, 2024-03-21T13:13:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1000044849999999, 2024-03-21T13:13:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:13:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:13:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100403 2024-03-21T13:13:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597070+0000", 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597070+0000", 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597072+0000", 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.325e-06 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597104+0000", 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2159000000000002e-05 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597113+0000", 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0599999999999997e-06 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597119+0000", 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7150000000000001e-06 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597232+0000", 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112543 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597242+0000", 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0071000000000001e-05 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574254+0000", 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.977012141 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574263+0000", 2024-03-21T13:13:59.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1130000000000007e-06 2024-03-21T13:13:59.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.684729+0000", 2024-03-21T13:13:59.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.110465461 2024-03-21T13:13:59.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.684879+0000", 2024-03-21T13:13:59.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014982799999999999 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,3", 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.684982+0000", 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103797 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.689118+0000", 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041350960000000004 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.691334+0000", 2024-03-21T13:13:59.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022162530000000001 2024-03-21T13:13:59.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.691364+0000", 2024-03-21T13:13:59.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9961999999999999e-05 2024-03-21T13:13:59.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.696956+0000", 2024-03-21T13:13:59.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0055922439999999997 2024-03-21T13:13:59.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697002+0000", 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6020000000000003e-05 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697075+0000", 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2696999999999999e-05 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100405 9.0 9:03b1f289:::benchmark_data_smithi163_131650_object100404:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:13:59.813 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.602198+0000", 2024-03-21T13:13:59.814 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.143925317, 2024-03-21T13:13:59.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2290228340000002, 2024-03-21T13:13:59.814 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.814 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.814 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.814 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:13:59.814 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:13:59.814 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100405 2024-03-21T13:13:59.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.814 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602198+0000", 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602198+0000", 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602200+0000", 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5039999999999999e-06 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602231+0000", 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1597000000000003e-05 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602248+0000", 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6399e-05 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602257+0000", 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6549999999999999e-06 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602323+0000", 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5296999999999995e-05 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602334+0000", 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1402e-05 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.573102+0000", 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97076796700000001 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.573115+0000", 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2679e-05 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.813300+0000", 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2401856250000001 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.813442+0000", 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014174399999999999 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-03-21T13:13:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.813831+0000", 2024-03-21T13:13:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038913100000000003 2024-03-21T13:13:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.818610+0000", 2024-03-21T13:13:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047784689999999996 2024-03-21T13:13:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.821637+0000", 2024-03-21T13:13:59.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:13:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030269530000000002 2024-03-21T13:13:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.828668+0000", 2024-03-21T13:13:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0070315559999999996 2024-03-21T13:13:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.828723+0000", 2024-03-21T13:13:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5105999999999997e-05 2024-03-21T13:13:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.831221+0000", 2024-03-21T13:13:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00249775 2024-03-21T13:13:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100413 9.b 9:de0331ce:::benchmark_data_smithi163_131650_object100412:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:13:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.608307+0000", 2024-03-21T13:13:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.13781657199999, 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0889179439999999, 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100413 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608307+0000", 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608307+0000", 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608308+0000", 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.322e-06 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608360+0000", 2024-03-21T13:13:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1860999999999997e-05 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608369+0000", 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6170000000000007e-06 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608376+0000", 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5440000000000001e-06 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608442+0000", 2024-03-21T13:13:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5980000000000002e-05 2024-03-21T13:13:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608452+0000", 2024-03-21T13:13:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0205e-05 2024-03-21T13:13:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574295+0000", 2024-03-21T13:13:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96584290500000003 2024-03-21T13:13:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574304+0000", 2024-03-21T13:13:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5399999999999996e-06 2024-03-21T13:13:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.686498+0000", 2024-03-21T13:13:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1121937829999999 2024-03-21T13:13:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.686598+0000", 2024-03-21T13:13:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000100296 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,3", 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.686706+0000", 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107565 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.691297+0000", 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0045910040000000001 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.695175+0000", 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003878102 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.695383+0000", 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000208657 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697114+0000", 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001730678 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697150+0000", 2024-03-21T13:13:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5581999999999999e-05 2024-03-21T13:13:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697225+0000", 2024-03-21T13:13:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5302999999999994e-05 2024-03-21T13:13:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100418 9.b 9:dd6b4148:::benchmark_data_smithi163_131650_object100417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:13:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.615444+0000", 2024-03-21T13:13:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 237.13067998099999, 2024-03-21T13:13:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0820358419999998, 2024-03-21T13:13:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100418 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615444+0000", 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615444+0000", 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615445+0000", 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5400000000000001e-06 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615499+0000", 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3872000000000002e-05 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615511+0000", 2024-03-21T13:13:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2054e-05 2024-03-21T13:13:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615520+0000", 2024-03-21T13:13:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6950000000000006e-06 2024-03-21T13:13:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615579+0000", 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9378e-05 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615586+0000", 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5250000000000002e-06 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574333+0000", 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95874714800000005 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574341+0000", 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.49e-06 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.687885+0000", 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.113543653 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.687993+0000", 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108359 2024-03-21T13:13:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,3", 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.688151+0000", 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000157947 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.691372+0000", 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032209890000000001 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.695216+0000", 2024-03-21T13:13:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038435320000000002 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.695415+0000", 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019962300000000001 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697343+0000", 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001927443 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697390+0000", 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7639000000000002e-05 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697479+0000", 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8955000000000005e-05 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:86745 10.7 10:e893f8c5:::benchmark_data_smithi163_154138_object86744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:22.351225+0000", 2024-03-21T13:13:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 37.394898759, 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15515072599999999, 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 86745 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.351225+0000", 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.351225+0000", 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.351225+0000", 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4499999999999997e-07 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.351281+0000", 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5439999999999998e-05 2024-03-21T13:13:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.351287+0000", 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2990000000000002e-06 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.351293+0000", 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8760000000000002e-06 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.361981+0000", 2024-03-21T13:13:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010688543999999999 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.362131+0000", 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014936600000000001 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,5", 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.362312+0000", 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018144499999999999 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.369070+0000", 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067576429999999998 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.371990+0000", 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029197260000000001 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.506288+0000", 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13429824900000001 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.506337+0000", 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9629999999999997e-05 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.506375+0000", 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8062999999999998e-05 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:108710 10.f 10:fd86497b:::benchmark_data_smithi163_154138_object108709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:41.524816+0000", 2024-03-21T13:13:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 18.221307936999999, 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.098077211999999997, 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 108710 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.524816+0000", 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.524816+0000", 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.524816+0000", 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0200000000000002e-07 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.524892+0000", 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5870000000000004e-05 2024-03-21T13:13:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.524901+0000", 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.098e-06 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.524908+0000", 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7610000000000002e-06 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.527947+0000", 2024-03-21T13:13:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030387840000000001 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.528094+0000", 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014754899999999999 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.528245+0000", 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000151139 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.532186+0000", 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003940687 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.533054+0000", 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00086801000000000001 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.622783+0000", 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.089729069999999994 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.622836+0000", 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3084000000000002e-05 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.622893+0000", 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6558000000000003e-05 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:108713 10.7 10:e2f3458e:::benchmark_data_smithi163_154138_object108712:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-21T13:13:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:41.524920+0000", 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 18.221203943999999, 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.098107776999999993, 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 108713 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.524920+0000", 2024-03-21T13:13:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.524920+0000", 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.524920+0000", 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.75e-07 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.524975+0000", 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4993e-05 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.524982+0000", 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3670000000000002e-06 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.524987+0000", 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5890000000000002e-06 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.527927+0000", 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029395939999999998 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.528022+0000", 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4897999999999998e-05 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,5", 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.528109+0000", 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7522000000000007e-05 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.532178+0000", 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040693229999999997 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.537376+0000", 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051976920000000003 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.622947+0000", 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.085570701999999998 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.622992+0000", 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5281000000000002e-05 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.623027+0000", 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5240999999999997e-05 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:108722 10.7 10:e726ac97:::benchmark_data_smithi163_154138_object108721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-21T13:13:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:41.532338+0000", 2024-03-21T13:13:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 18.213785668, 2024-03-21T13:13:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.090907376999999998, 2024-03-21T13:13:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 108722 2024-03-21T13:13:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.532338+0000", 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.532338+0000", 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.532339+0000", 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6300000000000004e-07 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.532370+0000", 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1018e-05 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.532382+0000", 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2808e-05 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.532389+0000", 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.973e-06 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.532471+0000", 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1840000000000002e-05 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.532597+0000", 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125649 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,5", 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.532739+0000", 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000141967 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.535588+0000", 2024-03-21T13:13:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028489769999999999 2024-03-21T13:13:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.538300+0000", 2024-03-21T13:13:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027118110000000002 2024-03-21T13:13:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.623172+0000", 2024-03-21T13:13:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.084871975000000002 2024-03-21T13:13:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.623213+0000", 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1619000000000003e-05 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:41.623245+0000", 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1977000000000002e-05 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:126000 10.b 10:d899005e:::benchmark_data_smithi163_154138_object125999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:56.943207+0000", 2024-03-21T13:13:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 2.8029169839999999, 2024-03-21T13:13:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.093997488000000004, 2024-03-21T13:13:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 126000 2024-03-21T13:13:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.943207+0000", 2024-03-21T13:13:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.943207+0000", 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.943208+0000", 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2270000000000001e-06 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.943238+0000", 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9751e-05 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.943249+0000", 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1309e-05 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.943257+0000", 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0530000000000007e-06 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.943388+0000", 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013066299999999999 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.943530+0000", 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014289000000000001 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.943689+0000", 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015866900000000001 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.947209+0000", 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003519861 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.947236+0000", 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6604000000000001e-05 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.037102+0000", 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.089866633000000001 2024-03-21T13:13:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.037157+0000", 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5052000000000001e-05 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.037204+0000", 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6776000000000001e-05 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:126002 10.a 10:5e54b1e9:::benchmark_data_smithi163_154138_object126001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:56.945793+0000", 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 2.8003302850000003, 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.094174913999999998, 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 126002 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.945793+0000", 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.945793+0000", 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.945795+0000", 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3090000000000001e-06 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.945826+0000", 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1674999999999997e-05 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.945837+0000", 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0502999999999999e-05 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.945845+0000", 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2179999999999999e-06 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.945991+0000", 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014615099999999999 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.946132+0000", 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014126399999999999 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,5", 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.946272+0000", 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014001899999999999 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.950251+0000", 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039789630000000003 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.950523+0000", 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027171500000000001 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.039883+0000", 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.089360239999999994 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.039929+0000", 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6181999999999999e-05 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.039968+0000", 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8674999999999998e-05 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:126015 10.a 10:5469ab62:::benchmark_data_smithi163_154138_object126014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:56.952827+0000", 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 2.7932961230000002, 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.096980888000000001, 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 126015 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.952827+0000", 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.952827+0000", 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.952829+0000", 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.249e-06 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.952859+0000", 2024-03-21T13:13:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0754999999999999e-05 2024-03-21T13:13:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.952876+0000", 2024-03-21T13:13:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6184e-05 2024-03-21T13:13:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.952883+0000", 2024-03-21T13:13:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8050000000000007e-06 2024-03-21T13:13:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.952939+0000", 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5418999999999999e-05 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.953078+0000", 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013897000000000001 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,5", 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.037111+0000", 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.084033344999999995 2024-03-21T13:13:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.046782+0000", 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096704449999999997 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.049588+0000", 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028066300000000001 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.049668+0000", 2024-03-21T13:13:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9424000000000004e-05 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.049767+0000", 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9395000000000006e-05 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.049808+0000", 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1267000000000003e-05 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:126016 10.a 10:57d5bc12:::benchmark_data_smithi163_154138_object126015:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:56.953621+0000", 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 2.7925024600000001, 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.109168348, 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 126016 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:13:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.953621+0000", 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.953621+0000", 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.953622+0000", 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5700000000000001e-07 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:56.953674+0000", 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1700999999999998e-05 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.037022+0000", 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.083348699999999998 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.037037+0000", 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4312e-05 2024-03-21T13:13:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.039998+0000", 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029614419999999999 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.040129+0000", 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000131051 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,5", 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.040281+0000", 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000152113 2024-03-21T13:13:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.049630+0000", 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0093491909999999998 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.055903+0000", 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006272494 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.062689+0000", 2024-03-21T13:13:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006786561 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.062742+0000", 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2788999999999999e-05 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:57.062789+0000", 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7136999999999999e-05 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:13:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:13:59.889 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:13:59.899 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:13:59.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:13:59.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:00.001 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:00.034 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:00.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:00.137 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:00.177 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:00.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:00.278 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:00.296 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-21T13:14:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:00.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:00.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:00.397 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 88 16 129687 129671 92.0852 66 0.00982153 0.0108552 2024-03-21T13:14:00.417 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:00.452 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:00.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:00 smithi163 ceph-mon[29621]: pgmap v724: 148 pgs: 148 active+clean; 7.6 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 959 op/s 2024-03-21T13:14:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:00 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3491039614' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:14:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:00 smithi163 ceph-mon[29621]: osdmap e361: 8 total, 8 up, 7 in 2024-03-21T13:14:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:00 smithi163 ceph-mon[29621]: 4.6 scrub starts 2024-03-21T13:14:00.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:00 smithi163 ceph-mon[29621]: 4.6 scrub ok 2024-03-21T13:14:00.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:00 smithi163 ceph-mon[40675]: pgmap v724: 148 pgs: 148 active+clean; 7.6 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 959 op/s 2024-03-21T13:14:00.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:00 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3491039614' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:14:00.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:00 smithi163 ceph-mon[40675]: osdmap e361: 8 total, 8 up, 7 in 2024-03-21T13:14:00.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:00 smithi163 ceph-mon[40675]: 4.6 scrub starts 2024-03-21T13:14:00.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:00 smithi163 ceph-mon[40675]: 4.6 scrub ok 2024-03-21T13:14:00.555 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:00.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:00.606 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:00.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:00.675 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:00.707 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:00.753 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:00 smithi204 ceph-mon[33153]: pgmap v724: 148 pgs: 148 active+clean; 7.6 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 959 op/s 2024-03-21T13:14:00.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:00 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3491039614' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:14:00.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:00 smithi204 ceph-mon[33153]: osdmap e361: 8 total, 8 up, 7 in 2024-03-21T13:14:00.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:00 smithi204 ceph-mon[33153]: 4.6 scrub starts 2024-03-21T13:14:00.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:00 smithi204 ceph-mon[33153]: 4.6 scrub ok 2024-03-21T13:14:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:00.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:00.812 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:14:00.856 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:00.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:00.886 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:00.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:00.988 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:01.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:01.028 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:01.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:01.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:01.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:01.158 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:01.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:01.259 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:01.293 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:01.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:01.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:01.398 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 89 16 130712 130696 91.7703 64.0625 0.0114037 0.0108921 2024-03-21T13:14:01.431 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:01.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:01.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:01.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:01.554 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:14:01.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-21T13:14:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:01.592 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:01.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:01.693 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:01.746 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:01.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:01.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:01 smithi204 ceph-mon[33153]: 4.f scrub starts 2024-03-21T13:14:01.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:01 smithi204 ceph-mon[33153]: 4.f scrub ok 2024-03-21T13:14:01.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:01 smithi204 ceph-mon[33153]: 4.8 scrub starts 2024-03-21T13:14:01.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:01 smithi204 ceph-mon[33153]: 4.8 scrub ok 2024-03-21T13:14:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:01 smithi163 ceph-mon[29621]: 4.f scrub starts 2024-03-21T13:14:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:01 smithi163 ceph-mon[29621]: 4.f scrub ok 2024-03-21T13:14:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:01 smithi163 ceph-mon[29621]: 4.8 scrub starts 2024-03-21T13:14:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:01 smithi163 ceph-mon[29621]: 4.8 scrub ok 2024-03-21T13:14:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:01 smithi163 ceph-mon[40675]: 4.f scrub starts 2024-03-21T13:14:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:01 smithi163 ceph-mon[40675]: 4.f scrub ok 2024-03-21T13:14:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:01 smithi163 ceph-mon[40675]: 4.8 scrub starts 2024-03-21T13:14:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:01 smithi163 ceph-mon[40675]: 4.8 scrub ok 2024-03-21T13:14:01.847 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:01.885 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:01.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:01.987 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:02.032 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:14:02.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:02.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:02.136 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:02.195 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:02.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:02.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:02.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:02.339 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:02.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:02.397 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Total time run: 90.0053 2024-03-21T13:14:02.398 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Total writes made: 131796 2024-03-21T13:14:02.398 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Write size: 65536 2024-03-21T13:14:02.398 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Object size: 65536 2024-03-21T13:14:02.398 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Bandwidth (MB/sec): 91.5196 2024-03-21T13:14:02.398 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev Bandwidth: 34.1244 2024-03-21T13:14:02.398 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max bandwidth (MB/sec): 213.938 2024-03-21T13:14:02.398 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min bandwidth (MB/sec): 61.5 2024-03-21T13:14:02.398 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Average IOPS: 1464 2024-03-21T13:14:02.398 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev IOPS: 545.99 2024-03-21T13:14:02.398 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max IOPS: 3423 2024-03-21T13:14:02.398 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min IOPS: 984 2024-03-21T13:14:02.399 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Average Latency(s): 0.010923 2024-03-21T13:14:02.431 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev Latency(s): 0.00755684 2024-03-21T13:14:02.432 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max latency(s): 0.239747 2024-03-21T13:14:02.432 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min latency(s): 0.00117124 2024-03-21T13:14:02.432 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:14:02 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:14:02.433 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Cleaning up (deleting benchmark objects) 2024-03-21T13:14:02.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:02.479 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:02.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:02.580 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:02.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:02.615 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:02.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:02.705 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:02 smithi204 ceph-mon[33153]: pgmap v726: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.9 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.7 KiB/s rd, 106 MiB/s wr, 1.71k op/s 2024-03-21T13:14:02.705 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:02 smithi204 ceph-mon[33153]: 6.0 scrub starts 2024-03-21T13:14:02.705 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:02 smithi204 ceph-mon[33153]: 6.0 scrub ok 2024-03-21T13:14:02.705 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:02 smithi204 ceph-mon[33153]: 1.0 scrub starts 2024-03-21T13:14:02.705 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:02 smithi204 ceph-mon[33153]: 1.0 scrub ok 2024-03-21T13:14:02.705 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:02 smithi204 ceph-mon[33153]: 3.3 scrub starts 2024-03-21T13:14:02.706 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:02 smithi204 ceph-mon[33153]: 3.3 scrub ok 2024-03-21T13:14:02.706 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:02 smithi204 ceph-mon[33153]: 10.f scrub ok 2024-03-21T13:14:02.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:02.731 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:02.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:02.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:02.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[29621]: pgmap v726: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.9 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.7 KiB/s rd, 106 MiB/s wr, 1.71k op/s 2024-03-21T13:14:02.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[29621]: 6.0 scrub starts 2024-03-21T13:14:02.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[29621]: 6.0 scrub ok 2024-03-21T13:14:02.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[29621]: 1.0 scrub starts 2024-03-21T13:14:02.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[29621]: 1.0 scrub ok 2024-03-21T13:14:02.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[29621]: 3.3 scrub starts 2024-03-21T13:14:02.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[29621]: 3.3 scrub ok 2024-03-21T13:14:02.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[29621]: 10.f scrub ok 2024-03-21T13:14:02.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[40675]: pgmap v726: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.9 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.7 KiB/s rd, 106 MiB/s wr, 1.71k op/s 2024-03-21T13:14:02.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[40675]: 6.0 scrub starts 2024-03-21T13:14:02.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[40675]: 6.0 scrub ok 2024-03-21T13:14:02.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[40675]: 1.0 scrub starts 2024-03-21T13:14:02.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[40675]: 1.0 scrub ok 2024-03-21T13:14:02.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[40675]: 3.3 scrub starts 2024-03-21T13:14:02.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[40675]: 3.3 scrub ok 2024-03-21T13:14:02.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:02 smithi163 ceph-mon[40675]: 10.f scrub ok 2024-03-21T13:14:02.853 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:02.859 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:02.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:02.911 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:02.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:03.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:03.049 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:03.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:03.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:03.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:03.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:03.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:03.282 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:03.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:03.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:03.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:03.418 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:03.453 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:03.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:03.511 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.2 to 1 (10000) 2024-03-21T13:14:03.555 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:03.591 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:03.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:03.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:03 smithi204 ceph-mon[33153]: 3.19 scrub starts 2024-03-21T13:14:03.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:03.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:03 smithi204 ceph-mon[33153]: 3.19 scrub ok 2024-03-21T13:14:03.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:03 smithi204 ceph-mon[33153]: 5.9 scrub starts 2024-03-21T13:14:03.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:03 smithi204 ceph-mon[33153]: 5.9 scrub ok 2024-03-21T13:14:03.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:03 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1960572290' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:14:03.693 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:03.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:03.739 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:03.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:03.840 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:03.865 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:03.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:03.967 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:04.008 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:04.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:03 smithi163 ceph-mon[29621]: 3.19 scrub starts 2024-03-21T13:14:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:03 smithi163 ceph-mon[29621]: 3.19 scrub ok 2024-03-21T13:14:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:03 smithi163 ceph-mon[29621]: 5.9 scrub starts 2024-03-21T13:14:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:03 smithi163 ceph-mon[29621]: 5.9 scrub ok 2024-03-21T13:14:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:03 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1960572290' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:14:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:03 smithi163 ceph-mon[40675]: 3.19 scrub starts 2024-03-21T13:14:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:03 smithi163 ceph-mon[40675]: 3.19 scrub ok 2024-03-21T13:14:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:03 smithi163 ceph-mon[40675]: 5.9 scrub starts 2024-03-21T13:14:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:03 smithi163 ceph-mon[40675]: 5.9 scrub ok 2024-03-21T13:14:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:03 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1960572290' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:14:04.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:04.145 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:04.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:04.248 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:04.282 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:04.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:04.383 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:04.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:04.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:04.520 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:04.554 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:04.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:04.586 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:14:04.586 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:14:04.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.587 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:134524 10.c 10:3877297c:::benchmark_data_smithi163_154138_object2725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-03-21T13:14:04.587 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:14:04.573935+0000", 2024-03-21T13:14:04.587 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.010595831999999999, 2024-03-21T13:14:04.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010632364, 2024-03-21T13:14:04.587 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:14:04.587 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:14:04.587 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:14:04.587 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:14:04.587 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:14:04.587 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 134524 2024-03-21T13:14:04.587 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.587 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:14:04.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:14:04.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.573935+0000", 2024-03-21T13:14:04.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:14:04.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:14:04.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.573935+0000", 2024-03-21T13:14:04.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:14:04.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:14:04.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.573936+0000", 2024-03-21T13:14:04.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5599999999999995e-07 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.573939+0000", 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8789999999999999e-06 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.573946+0000", 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8140000000000003e-06 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:14:04.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.573952+0000", 2024-03-21T13:14:04.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9680000000000003e-06 2024-03-21T13:14:04.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:14:04.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.577337+0000", 2024-03-21T13:14:04.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033853300000000002 2024-03-21T13:14:04.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:14:04.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.577463+0000", 2024-03-21T13:14:04.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012574 2024-03-21T13:14:04.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:14:04.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.577585+0000", 2024-03-21T13:14:04.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012203500000000001 2024-03-21T13:14:04.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:04.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.582148+0000", 2024-03-21T13:14:04.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004563147 2024-03-21T13:14:04.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:14:04.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.582465+0000", 2024-03-21T13:14:04.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031723800000000001 2024-03-21T13:14:04.592 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:14:04.592 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:14:04.592 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:14:04.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.592 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25481.0:134525 10.9 e362/330 10:9fb82f67:::benchmark_data_smithi163_154138_object2726:head v 362'8378, mlcod=362'8374)", 2024-03-21T13:14:04.592 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:14:04.574695+0000", 2024-03-21T13:14:04.592 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0098364009999999998, 2024-03-21T13:14:04.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0099767599999999994, 2024-03-21T13:14:04.592 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:14:04.592 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:14:04.592 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.574695+0000", 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.574695+0000", 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.574695+0000", 2024-03-21T13:14:04.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7000000000000004e-07 2024-03-21T13:14:04.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:14:04.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.574701+0000", 2024-03-21T13:14:04.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7270000000000004e-06 2024-03-21T13:14:04.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:14:04.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.574726+0000", 2024-03-21T13:14:04.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4491e-05 2024-03-21T13:14:04.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.574732+0000", 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7259999999999996e-06 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.575200+0000", 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000467808 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.575254+0000", 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3783000000000003e-05 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:14:04.595 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25481.0:134527 10.8 e362/324 10:105c5f9f:::benchmark_data_smithi163_154138_object2728:head v 362'8385, mlcod=362'8382)", 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:14:04.578148+0000", 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0063830349999999996, 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0065898700000000003, 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.578148+0000", 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:14:04.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.578148+0000", 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.578149+0000", 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7700000000000004e-07 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.578155+0000", 2024-03-21T13:14:04.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2929999999999997e-06 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.578171+0000", 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6266999999999999e-05 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.578179+0000", 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.396e-06 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:14:04.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.580141+0000", 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019621629999999998 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.580186+0000", 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4888999999999997e-05 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25481.0:134530 10.8 e362/324 10:1eafb372:::benchmark_data_smithi163_154138_object2731:head v 362'8386, mlcod=362'8382)", 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:14:04.581543+0000", 2024-03-21T13:14:04.599 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0029881629999999998, 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00326006, 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.581543+0000", 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.581543+0000", 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:14:04.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.581544+0000", 2024-03-21T13:14:04.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1999999999999999e-07 2024-03-21T13:14:04.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:14:04.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.581550+0000", 2024-03-21T13:14:04.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3940000000000001e-06 2024-03-21T13:14:04.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:14:04.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.581559+0000", 2024-03-21T13:14:04.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2299999999999997e-06 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.581566+0000", 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.472e-06 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.583382+0000", 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00181584 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:14:04.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.583426+0000", 2024-03-21T13:14:04.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3949999999999998e-05 2024-03-21T13:14:04.603 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:14:04.603 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:14:04.603 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:14:04.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.603 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:134535 10.1 10:808e6ff6:::benchmark_data_smithi163_154138_object2736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-03-21T13:14:04.603 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:14:04.582293+0000", 2024-03-21T13:14:04.603 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.002238412, 2024-03-21T13:14:04.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025870699999999999, 2024-03-21T13:14:04.603 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:14:04.603 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:14:04.603 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 134535 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.582293+0000", 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.582293+0000", 2024-03-21T13:14:04.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.582294+0000", 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5510000000000001e-06 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.582300+0000", 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5450000000000003e-06 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:14:04.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.582315+0000", 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4810999999999999e-05 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.582323+0000", 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8739999999999995e-06 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.582376+0000", 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2984e-05 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.582533+0000", 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015770000000000001 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.582644+0000", 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110999 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:134536 10.c 10:330080fd:::benchmark_data_smithi163_154138_object2737:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:14:04.584147+0000", 2024-03-21T13:14:04.607 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.00038386699999999998, 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00081378699999999995, 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 134536 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.584147+0000", 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:14:04.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:14:04.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.584147+0000", 2024-03-21T13:14:04.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:14:04.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:14:04.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.584149+0000", 2024-03-21T13:14:04.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2330000000000001e-06 2024-03-21T13:14:04.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:14:04.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.584154+0000", 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1880000000000001e-06 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.584167+0000", 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3067999999999999e-05 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.584175+0000", 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9410000000000007e-06 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.584225+0000", 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9789000000000001e-05 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.584342+0000", 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011754799999999999 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:04.584569+0000", 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000226438 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:14:04.611 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:14:04.612 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:14:04.612 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:14:04.612 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-03-21T13:14:04.612 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 6 2024-03-21T13:14:04.612 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:14:04.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:04.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:04.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:04.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:04.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:04.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:04.831 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:04.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:04.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[29621]: pgmap v727: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.9 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 789 B/s rd, 53 MiB/s wr, 852 op/s 2024-03-21T13:14:04.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[29621]: 3.9 scrub starts 2024-03-21T13:14:04.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[29621]: 3.9 scrub ok 2024-03-21T13:14:04.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[29621]: 6.16 scrub starts 2024-03-21T13:14:04.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1960572290' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:14:04.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[29621]: osdmap e362: 8 total, 8 up, 7 in 2024-03-21T13:14:04.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[29621]: 6.16 scrub ok 2024-03-21T13:14:04.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[29621]: 10.3 scrub starts 2024-03-21T13:14:04.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[40675]: pgmap v727: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.9 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 789 B/s rd, 53 MiB/s wr, 852 op/s 2024-03-21T13:14:04.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[40675]: 3.9 scrub starts 2024-03-21T13:14:04.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[40675]: 3.9 scrub ok 2024-03-21T13:14:04.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[40675]: 6.16 scrub starts 2024-03-21T13:14:04.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1960572290' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:14:04.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[40675]: osdmap e362: 8 total, 8 up, 7 in 2024-03-21T13:14:04.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[40675]: 6.16 scrub ok 2024-03-21T13:14:04.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:04 smithi163 ceph-mon[40675]: 10.3 scrub starts 2024-03-21T13:14:04.886 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:04 smithi204 ceph-mon[33153]: pgmap v727: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.9 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 789 B/s rd, 53 MiB/s wr, 852 op/s 2024-03-21T13:14:04.886 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:04 smithi204 ceph-mon[33153]: 3.9 scrub starts 2024-03-21T13:14:04.886 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:04 smithi204 ceph-mon[33153]: 3.9 scrub ok 2024-03-21T13:14:04.886 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:04 smithi204 ceph-mon[33153]: 6.16 scrub starts 2024-03-21T13:14:04.886 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:04 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1960572290' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:14:04.887 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:04 smithi204 ceph-mon[33153]: osdmap e362: 8 total, 8 up, 7 in 2024-03-21T13:14:04.887 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:04 smithi204 ceph-mon[33153]: 6.16 scrub ok 2024-03-21T13:14:04.887 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:04 smithi204 ceph-mon[33153]: 10.3 scrub starts 2024-03-21T13:14:04.932 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:04.976 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:04.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:04.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:05.078 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:05.116 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:05.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:05.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:05.256 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:05.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:05.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:05.396 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:05.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:05.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:05.490 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-21T13:14:05.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:05.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:05.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:05.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:05.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:05 smithi163 ceph-mon[29621]: 4.1d scrub starts 2024-03-21T13:14:05.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:05 smithi163 ceph-mon[29621]: 4.1d scrub ok 2024-03-21T13:14:05.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:05.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:05 smithi163 ceph-mon[40675]: 4.1d scrub starts 2024-03-21T13:14:05.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:05 smithi163 ceph-mon[40675]: 4.1d scrub ok 2024-03-21T13:14:05.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:05.718 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:14:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:14:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:14:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:05.722 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:05.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:05.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:05 smithi204 ceph-mon[33153]: 4.1d scrub starts 2024-03-21T13:14:05.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:05 smithi204 ceph-mon[33153]: 4.1d scrub ok 2024-03-21T13:14:05.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:05.824 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:05.859 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:05.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:05.925 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:14:05.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:06.009 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:06.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:06.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:06.012 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-21T13:14:06.111 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:06.144 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:06.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:06.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:06.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:06.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:06.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:06.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:06.400 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:06.439 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:06.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:06.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:06.541 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:06.572 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:06.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:06.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:06.715 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:06.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:06 smithi204 ceph-mon[33153]: pgmap v729: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.9 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 794 B/s rd, 57 MiB/s wr, 970 op/s 2024-03-21T13:14:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:06 smithi204 ceph-mon[33153]: 10.2 scrub starts 2024-03-21T13:14:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:06 smithi204 ceph-mon[33153]: 6.19 scrub starts 2024-03-21T13:14:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:06 smithi204 ceph-mon[33153]: 6.19 scrub ok 2024-03-21T13:14:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:06 smithi204 ceph-mon[33153]: 5.18 scrub starts 2024-03-21T13:14:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:06 smithi204 ceph-mon[33153]: 5.18 scrub ok 2024-03-21T13:14:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:06 smithi204 ceph-mon[33153]: 10.3 scrub ok 2024-03-21T13:14:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[29621]: pgmap v729: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.9 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 794 B/s rd, 57 MiB/s wr, 970 op/s 2024-03-21T13:14:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[29621]: 10.2 scrub starts 2024-03-21T13:14:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[29621]: 6.19 scrub starts 2024-03-21T13:14:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[29621]: 6.19 scrub ok 2024-03-21T13:14:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[29621]: 5.18 scrub starts 2024-03-21T13:14:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[29621]: 5.18 scrub ok 2024-03-21T13:14:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[29621]: 10.3 scrub ok 2024-03-21T13:14:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[40675]: pgmap v729: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.9 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 794 B/s rd, 57 MiB/s wr, 970 op/s 2024-03-21T13:14:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[40675]: 10.2 scrub starts 2024-03-21T13:14:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[40675]: 6.19 scrub starts 2024-03-21T13:14:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[40675]: 6.19 scrub ok 2024-03-21T13:14:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[40675]: 5.18 scrub starts 2024-03-21T13:14:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[40675]: 5.18 scrub ok 2024-03-21T13:14:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:06 smithi163 ceph-mon[40675]: 10.3 scrub ok 2024-03-21T13:14:06.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:06.850 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:06.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:06.952 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:06.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:06.990 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:06.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:07.091 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:07.128 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:07.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:07.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:07.277 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:07.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:07.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:07.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:07.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:07.448 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:07.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:07.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:07.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:07.659 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:07.693 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:07.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:07.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:07 smithi163 ceph-mon[29621]: 6.10 scrub starts 2024-03-21T13:14:07.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:07 smithi163 ceph-mon[29621]: 6.10 scrub ok 2024-03-21T13:14:07.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:07 smithi163 ceph-mon[29621]: 6.5 scrub starts 2024-03-21T13:14:07.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:07 smithi163 ceph-mon[29621]: 6.5 scrub ok 2024-03-21T13:14:07.727 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:07 smithi163 ceph-mon[40675]: 6.10 scrub starts 2024-03-21T13:14:07.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:07 smithi163 ceph-mon[40675]: 6.10 scrub ok 2024-03-21T13:14:07.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:07 smithi163 ceph-mon[40675]: 6.5 scrub starts 2024-03-21T13:14:07.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:07 smithi163 ceph-mon[40675]: 6.5 scrub ok 2024-03-21T13:14:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:07 smithi204 ceph-mon[33153]: 6.10 scrub starts 2024-03-21T13:14:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:07 smithi204 ceph-mon[33153]: 6.10 scrub ok 2024-03-21T13:14:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:07 smithi204 ceph-mon[33153]: 6.5 scrub starts 2024-03-21T13:14:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:07 smithi204 ceph-mon[33153]: 6.5 scrub ok 2024-03-21T13:14:07.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:07.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:07.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:07.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:07.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:07.972 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:07.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:07.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:08.074 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:08.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:08.103 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:08.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:08.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:08.237 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:08.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:08.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:08.339 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:08.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:08.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:08.475 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:08.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:08.519 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:08.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:08.608 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:08 smithi163 ceph-mon[29621]: pgmap v730: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 1.42k op/s 2024-03-21T13:14:08.608 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:08 smithi163 ceph-mon[29621]: 5.7 scrub starts 2024-03-21T13:14:08.608 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:08 smithi163 ceph-mon[29621]: 5.7 scrub ok 2024-03-21T13:14:08.608 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:08 smithi163 ceph-mon[29621]: 6.18 scrub starts 2024-03-21T13:14:08.608 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:08 smithi163 ceph-mon[29621]: 6.18 scrub ok 2024-03-21T13:14:08.608 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:08 smithi163 ceph-mon[29621]: 10.2 scrub ok 2024-03-21T13:14:08.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:08 smithi163 ceph-mon[40675]: pgmap v730: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 1.42k op/s 2024-03-21T13:14:08.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:08 smithi163 ceph-mon[40675]: 5.7 scrub starts 2024-03-21T13:14:08.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:08 smithi163 ceph-mon[40675]: 5.7 scrub ok 2024-03-21T13:14:08.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:08 smithi163 ceph-mon[40675]: 6.18 scrub starts 2024-03-21T13:14:08.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:08 smithi163 ceph-mon[40675]: 6.18 scrub ok 2024-03-21T13:14:08.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:08 smithi163 ceph-mon[40675]: 10.2 scrub ok 2024-03-21T13:14:08.620 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:08.658 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:08.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:08.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:08 smithi204 ceph-mon[33153]: pgmap v730: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 1.42k op/s 2024-03-21T13:14:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:08 smithi204 ceph-mon[33153]: 5.7 scrub starts 2024-03-21T13:14:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:08 smithi204 ceph-mon[33153]: 5.7 scrub ok 2024-03-21T13:14:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:08 smithi204 ceph-mon[33153]: 6.18 scrub starts 2024-03-21T13:14:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:08 smithi204 ceph-mon[33153]: 6.18 scrub ok 2024-03-21T13:14:08.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:08 smithi204 ceph-mon[33153]: 10.2 scrub ok 2024-03-21T13:14:08.755 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:14:08.755 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:14:08.755 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:14:08.755 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:14:08.758 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:14:08.760 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:08.798 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:08.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:08.813 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:14:08.865 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:14:08.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:08.907 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:14:08.915 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:14:08.915 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:14:08.915 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:14:08.916 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:14:08.916 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:14:08.916 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:14:08.916 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:14:08.916 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:14:08.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:08.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:09.046 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:09.088 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:09.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:09.189 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:09.227 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:09.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:09.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:09.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:09.366 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:09.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:09.468 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:09.503 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:09.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:09.557 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:14:09.560 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:14:09.560 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:14:09.604 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:09.639 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:09.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:09.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:09.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:09.776 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:09.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:09.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:09.879 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:09.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:09.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:09.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[29621]: 4.b scrub starts 2024-03-21T13:14:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[29621]: 4.b scrub ok 2024-03-21T13:14:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1824161373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[29621]: pgmap v731: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 1.21k op/s 2024-03-21T13:14:09.945 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/136359001' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:14:09.945 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:14:09.945 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[29621]: 4.17 scrub starts 2024-03-21T13:14:09.945 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[29621]: 4.17 scrub ok 2024-03-21T13:14:09.945 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[40675]: 4.b scrub starts 2024-03-21T13:14:09.945 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[40675]: 4.b scrub ok 2024-03-21T13:14:09.945 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1824161373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:09.945 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[40675]: pgmap v731: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 1.21k op/s 2024-03-21T13:14:09.945 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/136359001' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:14:09.945 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:14:09.945 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[40675]: 4.17 scrub starts 2024-03-21T13:14:09.945 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:09 smithi163 ceph-mon[40675]: 4.17 scrub ok 2024-03-21T13:14:09.964 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:09 smithi204 ceph-mon[33153]: 4.b scrub starts 2024-03-21T13:14:09.965 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:09 smithi204 ceph-mon[33153]: 4.b scrub ok 2024-03-21T13:14:09.965 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:09 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1824161373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:09.965 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:09 smithi204 ceph-mon[33153]: pgmap v731: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 1.21k op/s 2024-03-21T13:14:09.965 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:09 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/136359001' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:14:09.965 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:09 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:14:09.965 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:09 smithi204 ceph-mon[33153]: 4.17 scrub starts 2024-03-21T13:14:09.965 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:09 smithi204 ceph-mon[33153]: 4.17 scrub ok 2024-03-21T13:14:10.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:10.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:10.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:10.162 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:10.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:10.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:10.299 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:10.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-21T13:14:10.338 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:10.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:10.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:10.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:10.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:10.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:10.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:10.593 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:10.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:10.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:10.743 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:14:10.761 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:10.792 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:10.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:10.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:10.898 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:10 smithi204 ceph-mon[33153]: 5.c scrub starts 2024-03-21T13:14:10.899 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:10 smithi204 ceph-mon[33153]: 5.c scrub ok 2024-03-21T13:14:10.899 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:10 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:14:10.899 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:10 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:14:10.899 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:10 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:14:10.899 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:10 smithi204 ceph-mon[33153]: osdmap e363: 8 total, 8 up, 7 in 2024-03-21T13:14:10.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:10.930 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:10.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:11.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:10 smithi163 ceph-mon[29621]: 5.c scrub starts 2024-03-21T13:14:11.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:10 smithi163 ceph-mon[29621]: 5.c scrub ok 2024-03-21T13:14:11.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:10 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:14:11.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:10 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:14:11.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:10 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:14:11.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:10 smithi163 ceph-mon[29621]: osdmap e363: 8 total, 8 up, 7 in 2024-03-21T13:14:11.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:10 smithi163 ceph-mon[40675]: 5.c scrub starts 2024-03-21T13:14:11.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:10 smithi163 ceph-mon[40675]: 5.c scrub ok 2024-03-21T13:14:11.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:10 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:14:11.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:10 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:14:11.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:10 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:14:11.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:10 smithi163 ceph-mon[40675]: osdmap e363: 8 total, 8 up, 7 in 2024-03-21T13:14:11.030 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:11.065 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:11.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:11.167 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:11.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:11.205 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:11.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:11.305 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:11.347 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:11.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:11.448 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:11.480 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:11.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:11.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:11.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:11.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:11.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:11.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:11.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:11.760 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:11.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[29621]: 6.1e scrub starts 2024-03-21T13:14:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[29621]: 6.1e scrub ok 2024-03-21T13:14:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[29621]: 3.e scrub starts 2024-03-21T13:14:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[29621]: 3.e scrub ok 2024-03-21T13:14:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[29621]: pgmap v733: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 1.4 KiB/s rd, 18 MiB/s wr, 1.42k op/s 2024-03-21T13:14:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[29621]: 3.1e deep-scrub starts 2024-03-21T13:14:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[29621]: 3.1e deep-scrub ok 2024-03-21T13:14:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[40675]: 6.1e scrub starts 2024-03-21T13:14:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[40675]: 6.1e scrub ok 2024-03-21T13:14:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[40675]: 3.e scrub starts 2024-03-21T13:14:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[40675]: 3.e scrub ok 2024-03-21T13:14:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[40675]: pgmap v733: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 1.4 KiB/s rd, 18 MiB/s wr, 1.42k op/s 2024-03-21T13:14:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[40675]: 3.1e deep-scrub starts 2024-03-21T13:14:11.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:11 smithi163 ceph-mon[40675]: 3.1e deep-scrub ok 2024-03-21T13:14:11.861 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:11.898 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:11.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:11.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:12.000 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:11 smithi204 ceph-mon[33153]: 6.1e scrub starts 2024-03-21T13:14:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:11 smithi204 ceph-mon[33153]: 6.1e scrub ok 2024-03-21T13:14:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:11 smithi204 ceph-mon[33153]: 3.e scrub starts 2024-03-21T13:14:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:11 smithi204 ceph-mon[33153]: 3.e scrub ok 2024-03-21T13:14:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:11 smithi204 ceph-mon[33153]: pgmap v733: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 1.4 KiB/s rd, 18 MiB/s wr, 1.42k op/s 2024-03-21T13:14:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:11 smithi204 ceph-mon[33153]: 3.1e deep-scrub starts 2024-03-21T13:14:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:11 smithi204 ceph-mon[33153]: 3.1e deep-scrub ok 2024-03-21T13:14:12.048 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:12.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:12.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:12.192 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:12.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:12.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:12.294 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:12.331 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:14:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:14:12.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:14:12.335 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:12.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:12.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:12.437 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:12.472 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:12.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:12.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:12.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:12.615 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:12.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:12.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:12.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:12 smithi204 ceph-mon[33153]: 4.e scrub starts 2024-03-21T13:14:12.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:12 smithi204 ceph-mon[33153]: 4.e scrub ok 2024-03-21T13:14:12.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:12 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:12.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:12.754 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:12.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:12.819 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:12 smithi163 ceph-mon[29621]: 4.e scrub starts 2024-03-21T13:14:12.819 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:12 smithi163 ceph-mon[29621]: 4.e scrub ok 2024-03-21T13:14:12.819 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:12 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:12 smithi163 ceph-mon[40675]: 4.e scrub starts 2024-03-21T13:14:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:12 smithi163 ceph-mon[40675]: 4.e scrub ok 2024-03-21T13:14:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:12 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:12.854 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:12.901 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:12.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:12.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:12.964 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:13.003 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:13.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:13.053 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:13.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:13.154 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:13.193 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:13.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:13.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:13.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:13.329 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:14:13.329 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:14:13.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:13.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:13.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:13.433 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:14:13.438 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:13.513 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:13.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:13.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:13.583 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.5 to 1 (10000) 2024-03-21T13:14:13.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:13.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:13.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:13.744 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:13.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[40675]: 4.1 scrub starts 2024-03-21T13:14:13.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[40675]: 4.1 scrub ok 2024-03-21T13:14:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[40675]: 3.11 scrub starts 2024-03-21T13:14:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[40675]: 3.11 scrub ok 2024-03-21T13:14:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[40675]: pgmap v734: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 1.25k op/s 2024-03-21T13:14:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2032730689' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:14:13.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:14:13.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[29621]: 4.1 scrub starts 2024-03-21T13:14:13.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[29621]: 4.1 scrub ok 2024-03-21T13:14:13.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[29621]: 3.11 scrub starts 2024-03-21T13:14:13.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[29621]: 3.11 scrub ok 2024-03-21T13:14:13.763 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[29621]: pgmap v734: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 1.25k op/s 2024-03-21T13:14:13.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2032730689' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:14:13.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:13 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:14:13.782 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:13.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:13.884 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:13.920 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:13.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:13.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:13 smithi204 ceph-mon[33153]: 4.1 scrub starts 2024-03-21T13:14:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:13 smithi204 ceph-mon[33153]: 4.1 scrub ok 2024-03-21T13:14:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:13 smithi204 ceph-mon[33153]: 3.11 scrub starts 2024-03-21T13:14:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:13 smithi204 ceph-mon[33153]: 3.11 scrub ok 2024-03-21T13:14:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:13 smithi204 ceph-mon[33153]: pgmap v734: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 1.25k op/s 2024-03-21T13:14:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:13 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2032730689' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:14:14.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:13 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:14:14.022 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:14.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:14.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:14.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:14.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:14.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:14.201 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:14.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:14.228 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-21T13:14:14.302 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:14.334 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:14.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:14.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:14.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:14.469 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:14.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:14.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:14.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:14.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:14.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:14.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:14.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:14.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:14.848 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:14.881 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:14.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:14.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:14.983 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:14 smithi204 ceph-mon[33153]: 10.8 scrub starts 2024-03-21T13:14:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:14 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:14:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:14 smithi204 ceph-mon[33153]: osdmap e364: 8 total, 8 up, 7 in 2024-03-21T13:14:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:14 smithi204 ceph-mon[33153]: 6.6 scrub starts 2024-03-21T13:14:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:14 smithi204 ceph-mon[33153]: 6.6 scrub ok 2024-03-21T13:14:15.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:14 smithi163 ceph-mon[29621]: 10.8 scrub starts 2024-03-21T13:14:15.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:14 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:14:15.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:14 smithi163 ceph-mon[29621]: osdmap e364: 8 total, 8 up, 7 in 2024-03-21T13:14:15.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:14 smithi163 ceph-mon[29621]: 6.6 scrub starts 2024-03-21T13:14:15.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:14 smithi163 ceph-mon[29621]: 6.6 scrub ok 2024-03-21T13:14:15.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:14 smithi163 ceph-mon[40675]: 10.8 scrub starts 2024-03-21T13:14:15.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:14 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:14:15.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:14 smithi163 ceph-mon[40675]: osdmap e364: 8 total, 8 up, 7 in 2024-03-21T13:14:15.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:14 smithi163 ceph-mon[40675]: 6.6 scrub starts 2024-03-21T13:14:15.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:14 smithi163 ceph-mon[40675]: 6.6 scrub ok 2024-03-21T13:14:15.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:15.023 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:15.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:15.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:15.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:15.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:15.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:15.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:15.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:15.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:15.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:15.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:15.444 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:15.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:15.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:15.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:15.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:15.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:15.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:15.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:15.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:15.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:15.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:15.964 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:15 smithi204 ceph-mon[33153]: 5.1d scrub starts 2024-03-21T13:14:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:15 smithi204 ceph-mon[33153]: 5.1d scrub ok 2024-03-21T13:14:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:15 smithi204 ceph-mon[33153]: 3.10 scrub starts 2024-03-21T13:14:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:15 smithi204 ceph-mon[33153]: 3.10 scrub ok 2024-03-21T13:14:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:15 smithi204 ceph-mon[33153]: pgmap v736: 148 pgs: 148 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 833 op/s 2024-03-21T13:14:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:15 smithi204 ceph-mon[33153]: 4.16 scrub starts 2024-03-21T13:14:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:15 smithi204 ceph-mon[33153]: 4.16 scrub ok 2024-03-21T13:14:16.008 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:16.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:16.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:16.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[29621]: 5.1d scrub starts 2024-03-21T13:14:16.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[29621]: 5.1d scrub ok 2024-03-21T13:14:16.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[29621]: 3.10 scrub starts 2024-03-21T13:14:16.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[29621]: 3.10 scrub ok 2024-03-21T13:14:16.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[29621]: pgmap v736: 148 pgs: 148 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 833 op/s 2024-03-21T13:14:16.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[29621]: 4.16 scrub starts 2024-03-21T13:14:16.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[29621]: 4.16 scrub ok 2024-03-21T13:14:16.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[40675]: 5.1d scrub starts 2024-03-21T13:14:16.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[40675]: 5.1d scrub ok 2024-03-21T13:14:16.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[40675]: 3.10 scrub starts 2024-03-21T13:14:16.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[40675]: 3.10 scrub ok 2024-03-21T13:14:16.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[40675]: pgmap v736: 148 pgs: 148 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 833 op/s 2024-03-21T13:14:16.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[40675]: 4.16 scrub starts 2024-03-21T13:14:16.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-mon[40675]: 4.16 scrub ok 2024-03-21T13:14:16.012 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:14:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:14:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:14:16.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:16.148 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:16.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:16.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:16.250 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:16.286 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:16.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:16.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:16.422 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:16.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:16.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:16.561 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:16.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:16.663 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:16.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:16.692 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:16.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:16.793 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:16.831 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:16.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:16.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:16.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:16.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:16.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:17.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:16 smithi204 ceph-mon[33153]: 3.c scrub starts 2024-03-21T13:14:17.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:16 smithi204 ceph-mon[33153]: 3.c scrub ok 2024-03-21T13:14:17.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:16 smithi204 ceph-mon[33153]: 10.8 scrub ok 2024-03-21T13:14:17.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:16 smithi204 ceph-mon[33153]: 6.17 scrub starts 2024-03-21T13:14:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:16 smithi163 ceph-mon[29621]: 3.c scrub starts 2024-03-21T13:14:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:16 smithi163 ceph-mon[29621]: 3.c scrub ok 2024-03-21T13:14:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:16 smithi163 ceph-mon[29621]: 10.8 scrub ok 2024-03-21T13:14:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:16 smithi163 ceph-mon[29621]: 6.17 scrub starts 2024-03-21T13:14:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:16 smithi163 ceph-mon[40675]: 3.c scrub starts 2024-03-21T13:14:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:16 smithi163 ceph-mon[40675]: 3.c scrub ok 2024-03-21T13:14:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:16 smithi163 ceph-mon[40675]: 10.8 scrub ok 2024-03-21T13:14:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:16 smithi163 ceph-mon[40675]: 6.17 scrub starts 2024-03-21T13:14:17.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:17.108 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:17.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:17.210 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:17.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:17.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:17.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:17.351 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:17.393 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:17.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:17.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:17.495 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:17.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:17.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:17.641 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:17.678 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:17.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:17.779 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:17.827 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:17.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:17.923 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:17 smithi163 ceph-mon[29621]: 4.3 scrub starts 2024-03-21T13:14:17.923 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:17 smithi163 ceph-mon[29621]: 4.3 scrub ok 2024-03-21T13:14:17.923 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:17 smithi163 ceph-mon[29621]: 6.17 scrub ok 2024-03-21T13:14:17.924 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:17 smithi163 ceph-mon[29621]: pgmap v737: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-21T13:14:17.924 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:17 smithi163 ceph-mon[29621]: 3.8 scrub starts 2024-03-21T13:14:17.924 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:17 smithi163 ceph-mon[29621]: 3.8 scrub ok 2024-03-21T13:14:17.924 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:17 smithi163 ceph-mon[40675]: 4.3 scrub starts 2024-03-21T13:14:17.924 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:17 smithi163 ceph-mon[40675]: 4.3 scrub ok 2024-03-21T13:14:17.924 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:17 smithi163 ceph-mon[40675]: 6.17 scrub ok 2024-03-21T13:14:17.924 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:17 smithi163 ceph-mon[40675]: pgmap v737: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-21T13:14:17.924 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:17 smithi163 ceph-mon[40675]: 3.8 scrub starts 2024-03-21T13:14:17.924 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:17 smithi163 ceph-mon[40675]: 3.8 scrub ok 2024-03-21T13:14:17.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:17.962 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:17.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:17 smithi204 ceph-mon[33153]: 4.3 scrub starts 2024-03-21T13:14:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:17 smithi204 ceph-mon[33153]: 4.3 scrub ok 2024-03-21T13:14:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:17 smithi204 ceph-mon[33153]: 6.17 scrub ok 2024-03-21T13:14:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:17 smithi204 ceph-mon[33153]: pgmap v737: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-21T13:14:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:17 smithi204 ceph-mon[33153]: 3.8 scrub starts 2024-03-21T13:14:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:17 smithi204 ceph-mon[33153]: 3.8 scrub ok 2024-03-21T13:14:18.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:18.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:18.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:18.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:18.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:18.239 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:18.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:18.340 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:18.372 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-21T13:14:18.372 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:14:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:18.387 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:18.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:18.489 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:18.528 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:18.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:18.630 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:18.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:18.671 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:18.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:18.773 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:18.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:18.817 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:18.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:18.871 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:18 smithi163 ceph-mon[29621]: 10.e scrub starts 2024-03-21T13:14:18.871 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:18 smithi163 ceph-mon[29621]: 3.7 scrub starts 2024-03-21T13:14:18.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:18 smithi163 ceph-mon[29621]: 3.7 scrub ok 2024-03-21T13:14:18.872 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:18 smithi163 ceph-mon[40675]: 10.e scrub starts 2024-03-21T13:14:18.872 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:18 smithi163 ceph-mon[40675]: 3.7 scrub starts 2024-03-21T13:14:18.872 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:18 smithi163 ceph-mon[40675]: 3.7 scrub ok 2024-03-21T13:14:18.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:18.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:18.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:18.993 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:18.993 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:19.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:18 smithi204 ceph-mon[33153]: 10.e scrub starts 2024-03-21T13:14:19.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:18 smithi204 ceph-mon[33153]: 3.7 scrub starts 2024-03-21T13:14:19.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:18 smithi204 ceph-mon[33153]: 3.7 scrub ok 2024-03-21T13:14:19.057 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:19.093 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:14:19.114 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:19.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:19.132 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:14:19.166 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:14:19.207 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:14:19.217 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:19.253 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:14:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:19.261 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:19.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:19.269 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:19.292 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:14:19.329 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:14:19.362 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:19.367 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:14:19.410 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:19.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:19.512 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:19.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:19.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:19.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:19.696 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:19.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:19.713 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:19 smithi163 ceph-mon[29621]: pgmap v738: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-21T13:14:19.713 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:19 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/233091986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:19.713 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:19 smithi163 ceph-mon[40675]: pgmap v738: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-21T13:14:19.713 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:19 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/233091986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:19.798 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:19.834 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:19.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:19.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:19.936 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:19.968 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:19.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:19.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:20.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:19 smithi204 ceph-mon[33153]: pgmap v738: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-21T13:14:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:19 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/233091986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:20.070 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:20.108 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:20.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:20.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:20.211 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:20.255 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:20.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:20.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:20.357 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:20.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:20.397 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:20.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:20.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:20.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:20.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:20.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:20.636 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:20.671 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:20.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:20.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:20.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:20.812 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:20.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:20.913 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:20.948 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:20.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:21.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:21.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:21.088 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:21.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:21.189 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:21.221 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:21.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:21.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[29621]: 3.d scrub starts 2024-03-21T13:14:21.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[29621]: 3.d scrub ok 2024-03-21T13:14:21.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[29621]: 10.e scrub ok 2024-03-21T13:14:21.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:21.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:21.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[29621]: 3.18 scrub starts 2024-03-21T13:14:21.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[29621]: 3.18 scrub ok 2024-03-21T13:14:21.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[29621]: 6.12 scrub starts 2024-03-21T13:14:21.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[29621]: 6.12 scrub ok 2024-03-21T13:14:21.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[40675]: 3.d scrub starts 2024-03-21T13:14:21.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[40675]: 3.d scrub ok 2024-03-21T13:14:21.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[40675]: 10.e scrub ok 2024-03-21T13:14:21.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:21.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:21.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[40675]: 3.18 scrub starts 2024-03-21T13:14:21.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[40675]: 3.18 scrub ok 2024-03-21T13:14:21.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[40675]: 6.12 scrub starts 2024-03-21T13:14:21.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:21 smithi163 ceph-mon[40675]: 6.12 scrub ok 2024-03-21T13:14:21.323 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:21.362 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:21.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:21.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:21.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:21.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:21.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:21 smithi204 ceph-mon[33153]: 3.d scrub starts 2024-03-21T13:14:21.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:21 smithi204 ceph-mon[33153]: 3.d scrub ok 2024-03-21T13:14:21.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:21 smithi204 ceph-mon[33153]: 10.e scrub ok 2024-03-21T13:14:21.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:21.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:21.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:21 smithi204 ceph-mon[33153]: 3.18 scrub starts 2024-03-21T13:14:21.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:21 smithi204 ceph-mon[33153]: 3.18 scrub ok 2024-03-21T13:14:21.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:21 smithi204 ceph-mon[33153]: 6.12 scrub starts 2024-03-21T13:14:21.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:21 smithi204 ceph-mon[33153]: 6.12 scrub ok 2024-03-21T13:14:21.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:21.622 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:21.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:21.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:21.724 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:21.759 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:21.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:21.860 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:21.895 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:21.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:21.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:22.036 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:22.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:22.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:22.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:22.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:22.277 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:22.317 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:22.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:22.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:22.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:22.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:22.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:22.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:22 smithi204 ceph-mon[33153]: 10.d scrub starts 2024-03-21T13:14:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:22 smithi204 ceph-mon[33153]: pgmap v739: 148 pgs: 148 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-21T13:14:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:22 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:14:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:22 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:14:22.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:14:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:14:22.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:22 smithi163 ceph-mon[29621]: 10.d scrub starts 2024-03-21T13:14:22.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:22 smithi163 ceph-mon[29621]: pgmap v739: 148 pgs: 148 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-21T13:14:22.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:22 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:14:22.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:22 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:14:22.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:22 smithi163 ceph-mon[40675]: 10.d scrub starts 2024-03-21T13:14:22.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:22 smithi163 ceph-mon[40675]: pgmap v739: 148 pgs: 148 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-21T13:14:22.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:22 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:14:22.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:22 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:14:22.562 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:22.598 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:22.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:22.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:22.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:22.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:22.733 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:22.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:22.834 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:22.869 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:22.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:22.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:23.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:23.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:23.107 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:23.143 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:23.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:23.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:23.231 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:14:23.244 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:23.279 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:23.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:23.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:23.381 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:23.420 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:23.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:23.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:23 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:23.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:23 smithi204 ceph-mon[33153]: 4.18 scrub starts 2024-03-21T13:14:23.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:23 smithi204 ceph-mon[33153]: 4.18 scrub ok 2024-03-21T13:14:23.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:23 smithi204 ceph-mon[33153]: 6.e scrub starts 2024-03-21T13:14:23.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:23 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1991871803' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:14:23.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:23 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:14:23.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:23 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:23.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:23 smithi163 ceph-mon[29621]: 4.18 scrub starts 2024-03-21T13:14:23.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:23 smithi163 ceph-mon[29621]: 4.18 scrub ok 2024-03-21T13:14:23.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:23 smithi163 ceph-mon[29621]: 6.e scrub starts 2024-03-21T13:14:23.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:23 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1991871803' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:14:23.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:23 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:14:23.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:23 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:23.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:23 smithi163 ceph-mon[40675]: 4.18 scrub starts 2024-03-21T13:14:23.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:23 smithi163 ceph-mon[40675]: 4.18 scrub ok 2024-03-21T13:14:23.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:23 smithi163 ceph-mon[40675]: 6.e scrub starts 2024-03-21T13:14:23.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:23 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1991871803' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:14:23.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:23 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:14:23.521 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:23.560 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:23.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:23.661 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:23.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:23.698 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:23.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:23.799 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:23.840 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:23.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:23.920 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:14:23.942 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:24.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:24.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:24.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:24.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:24.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:24.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:24.276 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:24.320 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:24.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:24.421 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:24.454 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:24.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:24.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:24 smithi204 ceph-mon[33153]: 6.e scrub ok 2024-03-21T13:14:24.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:24 smithi204 ceph-mon[33153]: 10.d scrub ok 2024-03-21T13:14:24.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:24 smithi204 ceph-mon[33153]: pgmap v740: 148 pgs: 148 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-21T13:14:24.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:24 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:14:24.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:24 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:14:24.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:24 smithi204 ceph-mon[33153]: osdmap e365: 8 total, 8 up, 7 in 2024-03-21T13:14:24.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:24 smithi163 ceph-mon[29621]: 6.e scrub ok 2024-03-21T13:14:24.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:24 smithi163 ceph-mon[29621]: 10.d scrub ok 2024-03-21T13:14:24.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:24 smithi163 ceph-mon[29621]: pgmap v740: 148 pgs: 148 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-21T13:14:24.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:24 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:14:24.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:24 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:14:24.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:24 smithi163 ceph-mon[29621]: osdmap e365: 8 total, 8 up, 7 in 2024-03-21T13:14:24.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:24 smithi163 ceph-mon[40675]: 6.e scrub ok 2024-03-21T13:14:24.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:24 smithi163 ceph-mon[40675]: 10.d scrub ok 2024-03-21T13:14:24.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:24 smithi163 ceph-mon[40675]: pgmap v740: 148 pgs: 148 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-21T13:14:24.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:24 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:14:24.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:24 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:14:24.513 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:24 smithi163 ceph-mon[40675]: osdmap e365: 8 total, 8 up, 7 in 2024-03-21T13:14:24.556 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:24.590 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:24.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:24.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:24.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:24.732 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:24.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:24.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:24.870 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:24.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:24.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:24.972 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:25.008 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:25.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:25.109 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:25.145 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:25.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:25.247 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:25.285 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:25.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:25.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:25.427 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:25.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:25.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:25.562 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:25.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:25.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:25.665 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:25.702 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:25.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:25.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:25.735 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:14:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:14:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:14:25.803 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.25481.0:160555 10.5 e366/324 10:a6f4fd81:::benchmark_data_smithi163_154138_object28756:head v 366'9987, mlcod=366'9985)", 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:14:25.817073+0000", 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.001810456, 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001843393, 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:25.817073+0000", 2024-03-21T13:14:25.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:25.817073+0000", 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:25.817074+0000", 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3e-07 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:25.817079+0000", 2024-03-21T13:14:25.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1499999999999998e-06 2024-03-21T13:14:25.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:25.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:25.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:25.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:25.817091+0000", 2024-03-21T13:14:25.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2048000000000001e-05 2024-03-21T13:14:25.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:25.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:25.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:25.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:25.817099+0000", 2024-03-21T13:14:25.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5020000000000003e-06 2024-03-21T13:14:25.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:25.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:25.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:25.817150+0000", 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1017999999999998e-05 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:25.817169+0000", 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9471999999999999e-05 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 1 2024-03-21T13:14:25.823 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:14:25.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:25.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:25.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:25.940 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:25.981 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:25.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:25.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:26.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:26.118 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:26.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:26.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:26.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:26.259 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:26.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:26.297 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:26.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:26.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:26.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:26.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:26.472 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:14:26.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:26 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:14:26.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:26 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:14:26.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:26 smithi163 ceph-mon[29621]: pgmap v742: 148 pgs: 148 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-21T13:14:26.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:26 smithi163 ceph-mon[29621]: osdmap e366: 8 total, 8 up, 7 in 2024-03-21T13:14:26.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:26 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:14:26.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:26 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:14:26.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:26 smithi163 ceph-mon[40675]: pgmap v742: 148 pgs: 148 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-21T13:14:26.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:26 smithi163 ceph-mon[40675]: osdmap e366: 8 total, 8 up, 7 in 2024-03-21T13:14:26.484 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:14:26.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:26.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:26 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:14:26.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:26 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:14:26.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:26 smithi204 ceph-mon[33153]: pgmap v742: 148 pgs: 148 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-21T13:14:26.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:26 smithi204 ceph-mon[33153]: osdmap e366: 8 total, 8 up, 7 in 2024-03-21T13:14:26.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:26.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:26.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:26.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:26.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:26.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:26.776 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:26.809 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:26.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:26.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:26.916 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-21T13:14:26.945 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:26.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:26.966 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:14:27.048 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:27.124 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:27.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:27.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:27.264 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:27.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:27.365 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:27.412 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:27.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:27.514 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:27.550 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:27.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:27.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:27.652 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:27.689 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:27.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:27 smithi204 ceph-mon[33153]: from='client.15696 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:14:27.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:27 smithi163 ceph-mon[29621]: from='client.15696 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:14:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:27 smithi163 ceph-mon[40675]: from='client.15696 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:14:27.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:27.826 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:27.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:27.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:27.928 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:27.963 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:27.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:28.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:28.090 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:28.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:28.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:28.231 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:28.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:28.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:28.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:28.364 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:28.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:28.465 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:28.500 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:28.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:28.601 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:28.636 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:28.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:28.738 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:28.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:28 smithi163 ceph-mon[29621]: pgmap v744: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-21T13:14:28.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:28 smithi163 ceph-mon[40675]: pgmap v744: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-21T13:14:28.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:28 smithi204 ceph-mon[33153]: pgmap v744: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-21T13:14:28.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:28.788 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:28.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:28.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:28.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:28.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:29.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:29.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:29.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:29.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:29.200 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:29.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:29.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:29.302 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:29.339 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:29.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:29.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:29.478 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:29.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:29.579 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:29.593 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":743,"stamp":"2024-03-21T13:14:25.228920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7139236288,"num_objects":109126,"num_object_clones":0,"num_object_copies":327378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109126,"num_whiteouts":0,"num_read":2272,"num_read_kb":2118,"num_write":155567,"num_write_kb":8437406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":1779887,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80340,"ondisk_log_size":80340,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":26060860,"kb_used_data":20970580,"kb_used_omap":10,"kb_used_meta":5089717,"kb_avail":630155204,"statfs":{"total":671965249536,"available":645278928896,"internally_reserved":0,"allocated":21473873920,"data_stored":21470559833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":5211870315},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-476381184,"num_objects":-7269,"num_object_clones":0,"num_object_copies":-21807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7269,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7269,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.314611"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":79,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.228198+0000","last_change":"2024-03-21T13:12:29.526257+0000","last_active":"2024-03-21T13:14:23.228198+0000","last_peered":"2024-03-21T13:14:23.228198+0000","last_clean":"2024-03-21T13:14:23.228198+0000","last_became_active":"2024-03-21T13:12:29.525755+0000","last_became_peered":"2024-03-21T13:12:29.525755+0000","last_unstale":"2024-03-21T13:14:23.228198+0000","last_undegraded":"2024-03-21T13:14:23.228198+0000","last_fullsized":"2024-03-21T13:14:23.228198+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:11.717992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:09:11.717992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":87,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.227206+0000","last_change":"2024-03-21T13:13:14.412543+0000","last_active":"2024-03-21T13:14:23.227206+0000","last_peered":"2024-03-21T13:14:23.227206+0000","last_clean":"2024-03-21T13:14:23.227206+0000","last_became_active":"2024-03-21T13:12:29.510271+0000","last_became_peered":"2024-03-21T13:12:29.510271+0000","last_unstale":"2024-03-21T13:14:23.227206+0000","last_undegraded":"2024-03-21T13:14:23.227206+0000","last_fullsized":"2024-03-21T13:14:23.227206+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:14.412145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:14.412145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:30.918992+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":87,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.227501+0000","last_change":"2024-03-21T13:13:11.426511+0000","last_active":"2024-03-21T13:14:23.227501+0000","last_peered":"2024-03-21T13:14:23.227501+0000","last_clean":"2024-03-21T13:14:23.227501+0000","last_became_active":"2024-03-21T13:12:29.517794+0000","last_became_peered":"2024-03-21T13:12:29.517794+0000","last_unstale":"2024-03-21T13:14:23.227501+0000","last_undegraded":"2024-03-21T13:14:23.227501+0000","last_fullsized":"2024-03-21T13:14:23.227501+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:11.426239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:13:11.426239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:28.919433+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"234'19","reported_seq":731,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.223936+0000","last_change":"2024-03-21T13:14:22.200097+0000","last_active":"2024-03-21T13:14:23.223936+0000","last_peered":"2024-03-21T13:14:23.223936+0000","last_clean":"2024-03-21T13:14:23.223936+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:14:23.223936+0000","last_undegraded":"2024-03-21T13:14:23.223936+0000","last_fullsized":"2024-03-21T13:14:23.223936+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:14:22.199823+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:14:22.199823+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:45.142829+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":628,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.588602+0000","last_change":"2024-03-21T13:11:32.995208+0000","last_active":"2024-03-21T13:14:13.588602+0000","last_peered":"2024-03-21T13:14:13.588602+0000","last_clean":"2024-03-21T13:14:13.588602+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:14:13.588602+0000","last_undegraded":"2024-03-21T13:14:13.588602+0000","last_fullsized":"2024-03-21T13:14:13.588602+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:32.995084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:11:32.995084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":587,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.580368+0000","last_change":"2024-03-21T13:14:05.582597+0000","last_active":"2024-03-21T13:14:13.580368+0000","last_peered":"2024-03-21T13:14:13.580368+0000","last_clean":"2024-03-21T13:14:13.580368+0000","last_became_active":"2024-03-21T13:10:03.591543+0000","last_became_peered":"2024-03-21T13:10:03.591543+0000","last_unstale":"2024-03-21T13:14:13.580368+0000","last_undegraded":"2024-03-21T13:14:13.580368+0000","last_fullsized":"2024-03-21T13:14:13.580368+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:05.582473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:14:05.582473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:10.999808+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":668,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589648+0000","last_change":"2024-03-21T13:14:11.126777+0000","last_active":"2024-03-21T13:14:13.589648+0000","last_peered":"2024-03-21T13:14:13.589648+0000","last_clean":"2024-03-21T13:14:13.589648+0000","last_became_active":"2024-03-21T13:10:03.610318+0000","last_became_peered":"2024-03-21T13:10:03.610318+0000","last_unstale":"2024-03-21T13:14:13.589648+0000","last_undegraded":"2024-03-21T13:14:13.589648+0000","last_fullsized":"2024-03-21T13:14:13.589648+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_clean_scrub_stamp":"2024-03-21T13:14:11.126531+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:22.684839+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":160,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.591988+0000","last_change":"2024-03-21T13:11:28.976696+0000","last_active":"2024-03-21T13:14:13.591988+0000","last_peered":"2024-03-21T13:14:13.591988+0000","last_clean":"2024-03-21T13:14:13.591988+0000","last_became_active":"2024-03-21T13:11:21.702209+0000","last_became_peered":"2024-03-21T13:11:21.702209+0000","last_unstale":"2024-03-21T13:14:13.591988+0000","last_undegraded":"2024-03-21T13:14:13.591988+0000","last_fullsized":"2024-03-21T13:14:13.591988+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:11:28.976545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:28.976545+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":547,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.591518+0000","last_change":"2024-03-21T13:14:07.576452+0000","last_active":"2024-03-21T13:14:13.591518+0000","last_peered":"2024-03-21T13:14:13.591518+0000","last_clean":"2024-03-21T13:14:13.591518+0000","last_became_active":"2024-03-21T13:10:03.597963+0000","last_became_peered":"2024-03-21T13:10:03.597963+0000","last_unstale":"2024-03-21T13:14:13.591518+0000","last_undegraded":"2024-03-21T13:14:13.591518+0000","last_fullsized":"2024-03-21T13:14:13.591518+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:07.576369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:07.576369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:23.341119+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":79,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.230530+0000","last_change":"2024-03-21T13:12:29.519152+0000","last_active":"2024-03-21T13:14:23.230530+0000","last_peered":"2024-03-21T13:14:23.230530+0000","last_clean":"2024-03-21T13:14:23.230530+0000","last_became_active":"2024-03-21T13:12:29.518784+0000","last_became_peered":"2024-03-21T13:12:29.518784+0000","last_unstale":"2024-03-21T13:14:23.230530+0000","last_undegraded":"2024-03-21T13:14:23.230530+0000","last_fullsized":"2024-03-21T13:14:23.230530+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:20.342189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:11:20.342189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":79,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.226882+0000","last_change":"2024-03-21T13:12:29.526491+0000","last_active":"2024-03-21T13:14:23.226882+0000","last_peered":"2024-03-21T13:14:23.226882+0000","last_clean":"2024-03-21T13:14:23.226882+0000","last_became_active":"2024-03-21T13:12:29.526129+0000","last_became_peered":"2024-03-21T13:12:29.526129+0000","last_unstale":"2024-03-21T13:14:23.226882+0000","last_undegraded":"2024-03-21T13:14:23.226882+0000","last_fullsized":"2024-03-21T13:14:23.226882+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:33.515220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:33.515220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":639,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.590400+0000","last_change":"2024-03-21T13:13:13.181558+0000","last_active":"2024-03-21T13:14:13.590400+0000","last_peered":"2024-03-21T13:14:13.590400+0000","last_clean":"2024-03-21T13:14:13.590400+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:14:13.590400+0000","last_undegraded":"2024-03-21T13:14:13.590400+0000","last_fullsized":"2024-03-21T13:14:13.590400+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:13:13.181385+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:13:13.181385+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:41.227273+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":87,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.228456+0000","last_change":"2024-03-21T13:14:05.545713+0000","last_active":"2024-03-21T13:14:23.228456+0000","last_peered":"2024-03-21T13:14:23.228456+0000","last_clean":"2024-03-21T13:14:23.228456+0000","last_became_active":"2024-03-21T13:12:29.508520+0000","last_became_peered":"2024-03-21T13:12:29.508520+0000","last_unstale":"2024-03-21T13:14:23.228456+0000","last_undegraded":"2024-03-21T13:14:23.228456+0000","last_fullsized":"2024-03-21T13:14:23.228456+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:05.545425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:05.545425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:21.804231+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":652,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589922+0000","last_change":"2024-03-21T13:13:03.359188+0000","last_active":"2024-03-21T13:14:13.589922+0000","last_peered":"2024-03-21T13:14:13.589922+0000","last_clean":"2024-03-21T13:14:13.589922+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:14:13.589922+0000","last_undegraded":"2024-03-21T13:14:13.589922+0000","last_fullsized":"2024-03-21T13:14:13.589922+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:03.359078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:03.359078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":79,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.226206+0000","last_change":"2024-03-21T13:12:29.524840+0000","last_active":"2024-03-21T13:14:23.226206+0000","last_peered":"2024-03-21T13:14:23.226206+0000","last_clean":"2024-03-21T13:14:23.226206+0000","last_became_active":"2024-03-21T13:12:29.524473+0000","last_became_peered":"2024-03-21T13:12:29.524473+0000","last_unstale":"2024-03-21T13:14:23.226206+0000","last_undegraded":"2024-03-21T13:14:23.226206+0000","last_fullsized":"2024-03-21T13:14:23.226206+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:25.792349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:25.792349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"234'9","reported_seq":659,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.588293+0000","last_change":"2024-03-21T13:10:03.627839+0000","last_active":"2024-03-21T13:14:13.588293+0000","last_peered":"2024-03-21T13:14:13.588293+0000","last_clean":"2024-03-21T13:14:13.588293+0000","last_became_active":"2024-03-21T13:10:03.595745+0000","last_became_peered":"2024-03-21T13:10:03.595745+0000","last_unstale":"2024-03-21T13:14:13.588293+0000","last_undegraded":"2024-03-21T13:14:13.588293+0000","last_fullsized":"2024-03-21T13:14:13.588293+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'9","last_scrub_stamp":"2024-03-21T13:10:00.872509+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:10:00.872509+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":683,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.224262+0000","last_change":"2024-03-21T13:14:10.384752+0000","last_active":"2024-03-21T13:14:23.224262+0000","last_peered":"2024-03-21T13:14:23.224262+0000","last_clean":"2024-03-21T13:14:23.224262+0000","last_became_active":"2024-03-21T13:12:29.520189+0000","last_became_peered":"2024-03-21T13:12:29.520189+0000","last_unstale":"2024-03-21T13:14:23.224262+0000","last_undegraded":"2024-03-21T13:14:23.224262+0000","last_fullsized":"2024-03-21T13:14:23.224262+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:10.384430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:23.865785+0000","last_clean_scrub_stamp":"2024-03-21T13:14:10.384430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:27.709391+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":570,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.223767+0000","last_change":"2024-03-21T13:14:14.230013+0000","last_active":"2024-03-21T13:14:23.223767+0000","last_peered":"2024-03-21T13:14:23.223767+0000","last_clean":"2024-03-21T13:14:23.223767+0000","last_became_active":"2024-03-21T13:12:29.520241+0000","last_became_peered":"2024-03-21T13:12:29.520241+0000","last_unstale":"2024-03-21T13:14:23.223767+0000","last_undegraded":"2024-03-21T13:14:23.223767+0000","last_fullsized":"2024-03-21T13:14:23.223767+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:14.224068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:14:14.224068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:39.198450+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":539,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.590809+0000","last_change":"2024-03-21T13:12:33.276793+0000","last_active":"2024-03-21T13:14:13.590809+0000","last_peered":"2024-03-21T13:14:13.590809+0000","last_clean":"2024-03-21T13:14:13.590809+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:14:13.590809+0000","last_undegraded":"2024-03-21T13:14:13.590809+0000","last_fullsized":"2024-03-21T13:14:13.590809+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:33.276672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:12:33.276672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":220,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.627256+0000","last_change":"2024-03-21T13:12:15.899688+0000","last_active":"2024-03-21T13:14:13.627256+0000","last_peered":"2024-03-21T13:14:13.627256+0000","last_clean":"2024-03-21T13:14:13.627256+0000","last_became_active":"2024-03-21T13:10:03.604616+0000","last_became_peered":"2024-03-21T13:10:03.604616+0000","last_unstale":"2024-03-21T13:14:13.627256+0000","last_undegraded":"2024-03-21T13:14:13.627256+0000","last_fullsized":"2024-03-21T13:14:13.627256+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:12:15.899551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:12:15.899551+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":653,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.590283+0000","last_change":"2024-03-21T13:13:21.480427+0000","last_active":"2024-03-21T13:14:13.590283+0000","last_peered":"2024-03-21T13:14:13.590283+0000","last_clean":"2024-03-21T13:14:13.590283+0000","last_became_active":"2024-03-21T13:12:29.502808+0000","last_became_peered":"2024-03-21T13:12:29.502808+0000","last_unstale":"2024-03-21T13:14:13.590283+0000","last_undegraded":"2024-03-21T13:14:13.590283+0000","last_fullsized":"2024-03-21T13:14:13.590283+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:21.480296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:13:21.480296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.326257+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":565,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589369+0000","last_change":"2024-03-21T13:10:03.613920+0000","last_active":"2024-03-21T13:14:13.589369+0000","last_peered":"2024-03-21T13:14:13.589369+0000","last_clean":"2024-03-21T13:14:13.589369+0000","last_became_active":"2024-03-21T13:10:03.612409+0000","last_became_peered":"2024-03-21T13:10:03.612409+0000","last_unstale":"2024-03-21T13:14:13.589369+0000","last_undegraded":"2024-03-21T13:14:13.589369+0000","last_fullsized":"2024-03-21T13:14:13.589369+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:56.868411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:56.868411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":692,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.224979+0000","last_change":"2024-03-21T13:13:07.954508+0000","last_active":"2024-03-21T13:14:23.224979+0000","last_peered":"2024-03-21T13:14:23.224979+0000","last_clean":"2024-03-21T13:14:23.224979+0000","last_became_active":"2024-03-21T13:10:03.586563+0000","last_became_peered":"2024-03-21T13:10:03.586563+0000","last_unstale":"2024-03-21T13:14:23.224979+0000","last_undegraded":"2024-03-21T13:14:23.224979+0000","last_fullsized":"2024-03-21T13:14:23.224979+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:07.954268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:20.956778+0000","last_clean_scrub_stamp":"2024-03-21T13:13:07.954268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:36.523160+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":704,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589754+0000","last_change":"2024-03-21T13:14:04.151045+0000","last_active":"2024-03-21T13:14:13.589754+0000","last_peered":"2024-03-21T13:14:13.589754+0000","last_clean":"2024-03-21T13:14:13.589754+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:14:13.589754+0000","last_undegraded":"2024-03-21T13:14:13.589754+0000","last_fullsized":"2024-03-21T13:14:13.589754+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-21T13:14:04.150913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:04.150913+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:09.479392+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":57,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589561+0000","last_change":"2024-03-21T13:13:06.433430+0000","last_active":"2024-03-21T13:14:13.589561+0000","last_peered":"2024-03-21T13:14:13.589561+0000","last_clean":"2024-03-21T13:14:13.589561+0000","last_became_active":"2024-03-21T13:13:06.420220+0000","last_became_peered":"2024-03-21T13:13:06.420220+0000","last_unstale":"2024-03-21T13:14:13.589561+0000","last_undegraded":"2024-03-21T13:14:13.589561+0000","last_fullsized":"2024-03-21T13:14:13.589561+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:12:19.913538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:12:19.913538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":841,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.579937+0000","last_change":"2024-03-21T13:12:25.267410+0000","last_active":"2024-03-21T13:14:13.579937+0000","last_peered":"2024-03-21T13:14:13.579937+0000","last_clean":"2024-03-21T13:14:13.579937+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:14:13.579937+0000","last_undegraded":"2024-03-21T13:14:13.579937+0000","last_fullsized":"2024-03-21T13:14:13.579937+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:12:25.267265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:12:25.267265+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":547,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.583146+0000","last_change":"2024-03-21T13:14:02.109905+0000","last_active":"2024-03-21T13:14:13.583146+0000","last_peered":"2024-03-21T13:14:13.583146+0000","last_clean":"2024-03-21T13:14:13.583146+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:14:13.583146+0000","last_undegraded":"2024-03-21T13:14:13.583146+0000","last_fullsized":"2024-03-21T13:14:13.583146+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:14:02.109739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:02.109739+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:12.931461+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":589,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.591385+0000","last_change":"2024-03-21T13:11:16.033191+0000","last_active":"2024-03-21T13:14:13.591385+0000","last_peered":"2024-03-21T13:14:13.591385+0000","last_clean":"2024-03-21T13:14:13.591385+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:14:13.591385+0000","last_undegraded":"2024-03-21T13:14:13.591385+0000","last_fullsized":"2024-03-21T13:14:13.591385+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:11:16.033036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:16.033036+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":79,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.226800+0000","last_change":"2024-03-21T13:12:29.527843+0000","last_active":"2024-03-21T13:14:23.226800+0000","last_peered":"2024-03-21T13:14:23.226800+0000","last_clean":"2024-03-21T13:14:23.226800+0000","last_became_active":"2024-03-21T13:12:29.527129+0000","last_became_peered":"2024-03-21T13:12:29.527129+0000","last_unstale":"2024-03-21T13:14:23.226800+0000","last_undegraded":"2024-03-21T13:14:23.226800+0000","last_fullsized":"2024-03-21T13:14:23.226800+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:12.924305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:12:12.924305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":552,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.591235+0000","last_change":"2024-03-21T13:11:13.977186+0000","last_active":"2024-03-21T13:14:13.591235+0000","last_peered":"2024-03-21T13:14:13.591235+0000","last_clean":"2024-03-21T13:14:13.591235+0000","last_became_active":"2024-03-21T13:10:03.599880+0000","last_became_peered":"2024-03-21T13:10:03.599880+0000","last_unstale":"2024-03-21T13:14:13.591235+0000","last_undegraded":"2024-03-21T13:14:13.591235+0000","last_fullsized":"2024-03-21T13:14:13.591235+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:13.977069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:13.977069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":666,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:20.212307+0000","last_change":"2024-03-21T13:14:20.212339+0000","last_active":"2024-03-21T13:14:20.212307+0000","last_peered":"2024-03-21T13:14:20.212307+0000","last_clean":"2024-03-21T13:14:20.212307+0000","last_became_active":"2024-03-21T13:10:03.610489+0000","last_became_peered":"2024-03-21T13:10:03.610489+0000","last_unstale":"2024-03-21T13:14:20.212307+0000","last_undegraded":"2024-03-21T13:14:20.212307+0000","last_fullsized":"2024-03-21T13:14:20.212307+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:20.212222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:20.212222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:23.956162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"331'24","reported_seq":735,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.579991+0000","last_change":"2024-03-21T13:12:30.276844+0000","last_active":"2024-03-21T13:14:13.579991+0000","last_peered":"2024-03-21T13:14:13.579991+0000","last_clean":"2024-03-21T13:14:13.579991+0000","last_became_active":"2024-03-21T13:12:29.522466+0000","last_became_peered":"2024-03-21T13:12:29.522466+0000","last_unstale":"2024-03-21T13:14:13.579991+0000","last_undegraded":"2024-03-21T13:14:13.579991+0000","last_fullsized":"2024-03-21T13:14:13.579991+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'24","last_scrub_stamp":"2024-03-21T13:12:30.276748+0000","last_deep_scrub":"331'24","last_deep_scrub_stamp":"2024-03-21T13:12:30.276748+0000","last_clean_scrub_stamp":"2024-03-21T13:12:30.276748+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":487,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.626888+0000","last_change":"2024-03-21T13:13:21.828465+0000","last_active":"2024-03-21T13:14:13.626888+0000","last_peered":"2024-03-21T13:14:13.626888+0000","last_clean":"2024-03-21T13:14:13.626888+0000","last_became_active":"2024-03-21T13:13:21.827921+0000","last_became_peered":"2024-03-21T13:13:21.827921+0000","last_unstale":"2024-03-21T13:14:13.626888+0000","last_undegraded":"2024-03-21T13:14:13.626888+0000","last_fullsized":"2024-03-21T13:14:13.626888+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:21.183347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:12:21.183347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":648,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.579140+0000","last_change":"2024-03-21T13:13:12.942908+0000","last_active":"2024-03-21T13:14:13.579140+0000","last_peered":"2024-03-21T13:14:13.579140+0000","last_clean":"2024-03-21T13:14:13.579140+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:14:13.579140+0000","last_undegraded":"2024-03-21T13:14:13.579140+0000","last_fullsized":"2024-03-21T13:14:13.579140+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:12.942785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:12.942785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:24.031345+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":79,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.227909+0000","last_change":"2024-03-21T13:12:29.518880+0000","last_active":"2024-03-21T13:14:23.227909+0000","last_peered":"2024-03-21T13:14:23.227909+0000","last_clean":"2024-03-21T13:14:23.227909+0000","last_became_active":"2024-03-21T13:12:29.517486+0000","last_became_peered":"2024-03-21T13:12:29.517486+0000","last_unstale":"2024-03-21T13:14:23.227909+0000","last_undegraded":"2024-03-21T13:14:23.227909+0000","last_fullsized":"2024-03-21T13:14:23.227909+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:22.955737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:22.955737+0000","last_clean_scrub_stamp":"2024-03-21T13:12:22.955737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":660,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.582974+0000","last_change":"2024-03-21T13:13:26.091403+0000","last_active":"2024-03-21T13:14:13.582974+0000","last_peered":"2024-03-21T13:14:13.582974+0000","last_clean":"2024-03-21T13:14:13.582974+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:14:13.582974+0000","last_undegraded":"2024-03-21T13:14:13.582974+0000","last_fullsized":"2024-03-21T13:14:13.582974+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-21T13:13:26.091169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:26.091169+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:50.023578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":628,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589348+0000","last_change":"2024-03-21T13:11:33.943308+0000","last_active":"2024-03-21T13:14:13.589348+0000","last_peered":"2024-03-21T13:14:13.589348+0000","last_clean":"2024-03-21T13:14:13.589348+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:14:13.589348+0000","last_undegraded":"2024-03-21T13:14:13.589348+0000","last_fullsized":"2024-03-21T13:14:13.589348+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:33.943015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:33.943015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":294,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.580185+0000","last_change":"2024-03-21T13:13:05.952941+0000","last_active":"2024-03-21T13:14:13.580185+0000","last_peered":"2024-03-21T13:14:13.580185+0000","last_clean":"2024-03-21T13:14:13.580185+0000","last_became_active":"2024-03-21T13:08:20.955293+0000","last_became_peered":"2024-03-21T13:08:20.955293+0000","last_unstale":"2024-03-21T13:14:13.580185+0000","last_undegraded":"2024-03-21T13:14:13.580185+0000","last_fullsized":"2024-03-21T13:14:13.580185+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:05.952808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:05.952808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:18.970924+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":87,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.228641+0000","last_change":"2024-03-21T13:13:15.451096+0000","last_active":"2024-03-21T13:14:23.228641+0000","last_peered":"2024-03-21T13:14:23.228641+0000","last_clean":"2024-03-21T13:14:23.228641+0000","last_became_active":"2024-03-21T13:12:29.507706+0000","last_became_peered":"2024-03-21T13:12:29.507706+0000","last_unstale":"2024-03-21T13:14:23.228641+0000","last_undegraded":"2024-03-21T13:14:23.228641+0000","last_fullsized":"2024-03-21T13:14:23.228641+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:15.447666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:13:15.447666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.842740+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":382,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.583200+0000","last_change":"2024-03-21T13:13:09.092652+0000","last_active":"2024-03-21T13:14:13.583200+0000","last_peered":"2024-03-21T13:14:13.583200+0000","last_clean":"2024-03-21T13:14:13.583200+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:14:13.583200+0000","last_undegraded":"2024-03-21T13:14:13.583200+0000","last_fullsized":"2024-03-21T13:14:13.583200+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:13:09.092038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:09.092038+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:20.695441+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":87,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.225901+0000","last_change":"2024-03-21T13:14:06.489352+0000","last_active":"2024-03-21T13:14:23.225901+0000","last_peered":"2024-03-21T13:14:23.225901+0000","last_clean":"2024-03-21T13:14:23.225901+0000","last_became_active":"2024-03-21T13:12:29.518334+0000","last_became_peered":"2024-03-21T13:12:29.518334+0000","last_unstale":"2024-03-21T13:14:23.225901+0000","last_undegraded":"2024-03-21T13:14:23.225901+0000","last_fullsized":"2024-03-21T13:14:23.225901+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:06.489212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:06.489212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:20.036491+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":650,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.588516+0000","last_change":"2024-03-21T13:13:08.066898+0000","last_active":"2024-03-21T13:14:13.588516+0000","last_peered":"2024-03-21T13:14:13.588516+0000","last_clean":"2024-03-21T13:14:13.588516+0000","last_became_active":"2024-03-21T13:12:29.502203+0000","last_became_peered":"2024-03-21T13:12:29.502203+0000","last_unstale":"2024-03-21T13:14:13.588516+0000","last_undegraded":"2024-03-21T13:14:13.588516+0000","last_fullsized":"2024-03-21T13:14:13.588516+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:08.066673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:08.066673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":646,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.590355+0000","last_change":"2024-03-21T13:13:19.383175+0000","last_active":"2024-03-21T13:14:13.590355+0000","last_peered":"2024-03-21T13:14:13.590355+0000","last_clean":"2024-03-21T13:14:13.590355+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:14:13.590355+0000","last_undegraded":"2024-03-21T13:14:13.590355+0000","last_fullsized":"2024-03-21T13:14:13.590355+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:19.383090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:19.383090+0000","last_clean_scrub_stamp":"2024-03-21T13:13:19.383090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:43.999038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":653,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.580553+0000","last_change":"2024-03-21T13:13:22.789241+0000","last_active":"2024-03-21T13:14:13.580553+0000","last_peered":"2024-03-21T13:14:13.580553+0000","last_clean":"2024-03-21T13:14:13.580553+0000","last_became_active":"2024-03-21T13:10:03.591857+0000","last_became_peered":"2024-03-21T13:10:03.591857+0000","last_unstale":"2024-03-21T13:14:13.580553+0000","last_undegraded":"2024-03-21T13:14:13.580553+0000","last_fullsized":"2024-03-21T13:14:13.580553+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:13:22.789110+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:13:22.789110+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:24.232663+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":473,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589664+0000","last_change":"2024-03-21T13:11:34.090139+0000","last_active":"2024-03-21T13:14:13.589664+0000","last_peered":"2024-03-21T13:14:13.589664+0000","last_clean":"2024-03-21T13:14:13.589664+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:14:13.589664+0000","last_undegraded":"2024-03-21T13:14:13.589664+0000","last_fullsized":"2024-03-21T13:14:13.589664+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:34.087671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:34.087671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":87,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.228956+0000","last_change":"2024-03-21T13:13:22.501168+0000","last_active":"2024-03-21T13:14:23.228956+0000","last_peered":"2024-03-21T13:14:23.228956+0000","last_clean":"2024-03-21T13:14:23.228956+0000","last_became_active":"2024-03-21T13:12:29.510100+0000","last_became_peered":"2024-03-21T13:12:29.510100+0000","last_unstale":"2024-03-21T13:14:23.228956+0000","last_undegraded":"2024-03-21T13:14:23.228956+0000","last_fullsized":"2024-03-21T13:14:23.228956+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:22.501055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:22.501055+0000","last_clean_scrub_stamp":"2024-03-21T13:13:22.501055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:44.976431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":359,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.590250+0000","last_change":"2024-03-21T13:13:20.420469+0000","last_active":"2024-03-21T13:14:13.590250+0000","last_peered":"2024-03-21T13:14:13.590250+0000","last_clean":"2024-03-21T13:14:13.590250+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:14:13.590250+0000","last_undegraded":"2024-03-21T13:14:13.590250+0000","last_fullsized":"2024-03-21T13:14:13.590250+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:20.420313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:13:20.420313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:30.906361+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":96,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.230745+0000","last_change":"2024-03-21T13:12:29.601629+0000","last_active":"2024-03-21T13:14:23.230745+0000","last_peered":"2024-03-21T13:14:23.230745+0000","last_clean":"2024-03-21T13:14:23.230745+0000","last_became_active":"2024-03-21T13:12:29.521758+0000","last_became_peered":"2024-03-21T13:12:29.521758+0000","last_unstale":"2024-03-21T13:14:23.230745+0000","last_undegraded":"2024-03-21T13:14:23.230745+0000","last_fullsized":"2024-03-21T13:14:23.230745+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:11:29.734236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:29.734236+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":547,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.591004+0000","last_change":"2024-03-21T13:14:03.518501+0000","last_active":"2024-03-21T13:14:13.591004+0000","last_peered":"2024-03-21T13:14:13.591004+0000","last_clean":"2024-03-21T13:14:13.591004+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:14:13.591004+0000","last_undegraded":"2024-03-21T13:14:13.591004+0000","last_fullsized":"2024-03-21T13:14:13.591004+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:03.518372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:03.518372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:13.362750+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"331'11","reported_seq":115,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.226968+0000","last_change":"2024-03-21T13:13:23.466272+0000","last_active":"2024-03-21T13:14:23.226968+0000","last_peered":"2024-03-21T13:14:23.226968+0000","last_clean":"2024-03-21T13:14:23.226968+0000","last_became_active":"2024-03-21T13:12:29.521385+0000","last_became_peered":"2024-03-21T13:12:29.521385+0000","last_unstale":"2024-03-21T13:14:23.226968+0000","last_undegraded":"2024-03-21T13:14:23.226968+0000","last_fullsized":"2024-03-21T13:14:23.226968+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'11","last_scrub_stamp":"2024-03-21T13:13:23.466140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:23.466140+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:44.052256+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":658,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589627+0000","last_change":"2024-03-21T13:12:18.171392+0000","last_active":"2024-03-21T13:14:13.589627+0000","last_peered":"2024-03-21T13:14:13.589627+0000","last_clean":"2024-03-21T13:14:13.589627+0000","last_became_active":"2024-03-21T13:10:03.599265+0000","last_became_peered":"2024-03-21T13:10:03.599265+0000","last_unstale":"2024-03-21T13:14:13.589627+0000","last_undegraded":"2024-03-21T13:14:13.589627+0000","last_fullsized":"2024-03-21T13:14:13.589627+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.171299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.171299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'20","reported_seq":720,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589749+0000","last_change":"2024-03-21T13:13:04.129254+0000","last_active":"2024-03-21T13:14:13.589749+0000","last_peered":"2024-03-21T13:14:13.589749+0000","last_clean":"2024-03-21T13:14:13.589749+0000","last_became_active":"2024-03-21T13:10:03.613119+0000","last_became_peered":"2024-03-21T13:10:03.613119+0000","last_unstale":"2024-03-21T13:14:13.589749+0000","last_undegraded":"2024-03-21T13:14:13.589749+0000","last_fullsized":"2024-03-21T13:14:13.589749+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:13:04.128932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:04.128932+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":218,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:16.555610+0000","last_change":"2024-03-21T13:14:16.555652+0000","last_active":"2024-03-21T13:14:16.555610+0000","last_peered":"2024-03-21T13:14:16.555610+0000","last_clean":"2024-03-21T13:14:16.555610+0000","last_became_active":"2024-03-21T13:12:29.507751+0000","last_became_peered":"2024-03-21T13:12:29.507751+0000","last_unstale":"2024-03-21T13:14:16.555610+0000","last_undegraded":"2024-03-21T13:14:16.555610+0000","last_fullsized":"2024-03-21T13:14:16.555610+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:16.555516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:14:16.555516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:26.167124+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"272'10","reported_seq":850,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589896+0000","last_change":"2024-03-21T13:13:25.102337+0000","last_active":"2024-03-21T13:14:13.589896+0000","last_peered":"2024-03-21T13:14:13.589896+0000","last_clean":"2024-03-21T13:14:13.589896+0000","last_became_active":"2024-03-21T13:10:03.594651+0000","last_became_peered":"2024-03-21T13:10:03.594651+0000","last_unstale":"2024-03-21T13:14:13.589896+0000","last_undegraded":"2024-03-21T13:14:13.589896+0000","last_fullsized":"2024-03-21T13:14:13.589896+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'10","last_scrub_stamp":"2024-03-21T13:13:25.102208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:25.102208+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:42.694340+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":547,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589764+0000","last_change":"2024-03-21T13:13:21.373908+0000","last_active":"2024-03-21T13:14:13.589764+0000","last_peered":"2024-03-21T13:14:13.589764+0000","last_clean":"2024-03-21T13:14:13.589764+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:14:13.589764+0000","last_undegraded":"2024-03-21T13:14:13.589764+0000","last_fullsized":"2024-03-21T13:14:13.589764+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:21.373667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:13:21.373667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:28.725558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"234'15","reported_seq":97,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.226565+0000","last_change":"2024-03-21T13:13:18.440575+0000","last_active":"2024-03-21T13:14:23.226565+0000","last_peered":"2024-03-21T13:14:23.226565+0000","last_clean":"2024-03-21T13:14:23.226565+0000","last_became_active":"2024-03-21T13:12:29.510247+0000","last_became_peered":"2024-03-21T13:12:29.510247+0000","last_unstale":"2024-03-21T13:14:23.226565+0000","last_undegraded":"2024-03-21T13:14:23.226565+0000","last_fullsized":"2024-03-21T13:14:23.226565+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:13:18.440466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:18.440466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:38.918558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":204,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.627067+0000","last_change":"2024-03-21T13:13:11.639568+0000","last_active":"2024-03-21T13:14:13.627067+0000","last_peered":"2024-03-21T13:14:13.627067+0000","last_clean":"2024-03-21T13:14:13.627067+0000","last_became_active":"2024-03-21T13:10:03.613841+0000","last_became_peered":"2024-03-21T13:10:03.613841+0000","last_unstale":"2024-03-21T13:14:13.627067+0000","last_undegraded":"2024-03-21T13:14:13.627067+0000","last_fullsized":"2024-03-21T13:14:13.627067+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:11.639450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:13:11.639450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:27.549004+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":638,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.590922+0000","last_change":"2024-03-21T13:11:26.697833+0000","last_active":"2024-03-21T13:14:13.590922+0000","last_peered":"2024-03-21T13:14:13.590922+0000","last_clean":"2024-03-21T13:14:13.590922+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:14:13.590922+0000","last_undegraded":"2024-03-21T13:14:13.590922+0000","last_fullsized":"2024-03-21T13:14:13.590922+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:26.697728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:26.697728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":660,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.591613+0000","last_change":"2024-03-21T13:14:12.561473+0000","last_active":"2024-03-21T13:14:13.591613+0000","last_peered":"2024-03-21T13:14:13.591613+0000","last_clean":"2024-03-21T13:14:13.591613+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:14:13.591613+0000","last_undegraded":"2024-03-21T13:14:13.591613+0000","last_fullsized":"2024-03-21T13:14:13.591613+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:14:12.561236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:12.561236+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:42.231028+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'15","reported_seq":583,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:15.193050+0000","last_change":"2024-03-21T13:14:15.193109+0000","last_active":"2024-03-21T13:14:15.193050+0000","last_peered":"2024-03-21T13:14:15.193050+0000","last_clean":"2024-03-21T13:14:15.193050+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:14:15.193050+0000","last_undegraded":"2024-03-21T13:14:15.193050+0000","last_fullsized":"2024-03-21T13:14:15.193050+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:14:15.192958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:15.192958+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:23.439586+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":634,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.590259+0000","last_change":"2024-03-21T13:13:11.462718+0000","last_active":"2024-03-21T13:14:13.590259+0000","last_peered":"2024-03-21T13:14:13.590259+0000","last_clean":"2024-03-21T13:14:13.590259+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:14:13.590259+0000","last_undegraded":"2024-03-21T13:14:13.590259+0000","last_fullsized":"2024-03-21T13:14:13.590259+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:11.462567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:13:11.462567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:30.945841+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":87,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.228552+0000","last_change":"2024-03-21T13:13:07.458319+0000","last_active":"2024-03-21T13:14:23.228552+0000","last_peered":"2024-03-21T13:14:23.228552+0000","last_clean":"2024-03-21T13:14:23.228552+0000","last_became_active":"2024-03-21T13:12:29.514113+0000","last_became_peered":"2024-03-21T13:12:29.514113+0000","last_unstale":"2024-03-21T13:14:23.228552+0000","last_undegraded":"2024-03-21T13:14:23.228552+0000","last_fullsized":"2024-03-21T13:14:23.228552+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:07.458199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:07.458199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":668,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:14.647029+0000","last_change":"2024-03-21T13:14:14.647062+0000","last_active":"2024-03-21T13:14:14.647029+0000","last_peered":"2024-03-21T13:14:14.647029+0000","last_clean":"2024-03-21T13:14:14.647029+0000","last_became_active":"2024-03-21T13:12:29.520286+0000","last_became_peered":"2024-03-21T13:12:29.520286+0000","last_unstale":"2024-03-21T13:14:14.647029+0000","last_undegraded":"2024-03-21T13:14:14.647029+0000","last_fullsized":"2024-03-21T13:14:14.647029+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:14.646933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:14.646933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:16.998551+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":704,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.590776+0000","last_change":"2024-03-21T13:14:09.141688+0000","last_active":"2024-03-21T13:14:13.590776+0000","last_peered":"2024-03-21T13:14:13.590776+0000","last_clean":"2024-03-21T13:14:13.590776+0000","last_became_active":"2024-03-21T13:10:03.610810+0000","last_became_peered":"2024-03-21T13:10:03.610810+0000","last_unstale":"2024-03-21T13:14:13.590776+0000","last_undegraded":"2024-03-21T13:14:13.590776+0000","last_fullsized":"2024-03-21T13:14:13.590776+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-21T13:14:09.141520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:09.141520+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:21.501338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"365'10037","reported_seq":10117,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:24.784807+0000","last_change":"2024-03-21T13:12:29.526267+0000","last_active":"2024-03-21T13:14:24.784807+0000","last_peered":"2024-03-21T13:14:24.784807+0000","last_clean":"2024-03-21T13:14:24.784807+0000","last_became_active":"2024-03-21T13:12:29.525906+0000","last_became_peered":"2024-03-21T13:12:29.525906+0000","last_unstale":"2024-03-21T13:14:24.784807+0000","last_undegraded":"2024-03-21T13:14:24.784807+0000","last_fullsized":"2024-03-21T13:14:24.784807+0000","mapping_epoch":330,"log_start":"343'4700","ondisk_log_start":"343'4700","created":324,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":5337,"log_dups_size":0,"ondisk_log_size":5337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429457408,"num_objects":6553,"num_object_clones":0,"num_object_copies":19659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":79,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.230584+0000","last_change":"2024-03-21T13:12:29.518161+0000","last_active":"2024-03-21T13:14:23.230584+0000","last_peered":"2024-03-21T13:14:23.230584+0000","last_clean":"2024-03-21T13:14:23.230584+0000","last_became_active":"2024-03-21T13:12:29.517413+0000","last_became_peered":"2024-03-21T13:12:29.517413+0000","last_unstale":"2024-03-21T13:14:23.230584+0000","last_undegraded":"2024-03-21T13:14:23.230584+0000","last_fullsized":"2024-03-21T13:14:23.230584+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:02.758017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:11:02.758017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":682,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.591487+0000","last_change":"2024-03-21T13:11:31.040564+0000","last_active":"2024-03-21T13:14:13.591487+0000","last_peered":"2024-03-21T13:14:13.591487+0000","last_clean":"2024-03-21T13:14:13.591487+0000","last_became_active":"2024-03-21T13:06:29.487439+0000","last_became_peered":"2024-03-21T13:06:29.487439+0000","last_unstale":"2024-03-21T13:14:13.591487+0000","last_undegraded":"2024-03-21T13:14:13.591487+0000","last_fullsized":"2024-03-21T13:14:13.591487+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:11:31.040414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:31.040414+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":103,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.225799+0000","last_change":"2024-03-21T13:14:00.557564+0000","last_active":"2024-03-21T13:14:23.225799+0000","last_peered":"2024-03-21T13:14:23.225799+0000","last_clean":"2024-03-21T13:14:23.225799+0000","last_became_active":"2024-03-21T13:12:29.513019+0000","last_became_peered":"2024-03-21T13:12:29.513019+0000","last_unstale":"2024-03-21T13:14:23.225799+0000","last_undegraded":"2024-03-21T13:14:23.225799+0000","last_fullsized":"2024-03-21T13:14:23.225799+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:14:00.557372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:00.557372+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:03.309924+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":840,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.592131+0000","last_change":"2024-03-21T13:14:02.524497+0000","last_active":"2024-03-21T13:14:13.592131+0000","last_peered":"2024-03-21T13:14:13.592131+0000","last_clean":"2024-03-21T13:14:13.592131+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:14:13.592131+0000","last_undegraded":"2024-03-21T13:14:13.592131+0000","last_fullsized":"2024-03-21T13:14:13.592131+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:14:02.524111+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:14:02.524111+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:09.116528+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"364'9716","reported_seq":9812,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:20.248204+0000","last_change":"2024-03-21T13:12:29.522488+0000","last_active":"2024-03-21T13:14:20.248204+0000","last_peered":"2024-03-21T13:14:20.248204+0000","last_clean":"2024-03-21T13:14:20.248204+0000","last_became_active":"2024-03-21T13:12:29.521010+0000","last_became_peered":"2024-03-21T13:12:29.521010+0000","last_unstale":"2024-03-21T13:14:20.248204+0000","last_undegraded":"2024-03-21T13:14:20.248204+0000","last_fullsized":"2024-03-21T13:14:20.248204+0000","mapping_epoch":330,"log_start":"343'4900","ondisk_log_start":"343'4900","created":324,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4816,"log_dups_size":0,"ondisk_log_size":4816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456916992,"num_objects":6972,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9716,"num_write_kb":534016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":634,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.588688+0000","last_change":"2024-03-21T13:13:15.340376+0000","last_active":"2024-03-21T13:14:13.588688+0000","last_peered":"2024-03-21T13:14:13.588688+0000","last_clean":"2024-03-21T13:14:13.588688+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:14:13.588688+0000","last_undegraded":"2024-03-21T13:14:13.588688+0000","last_fullsized":"2024-03-21T13:14:13.588688+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:15.340248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:13:15.340248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:26.266564+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":660,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.591147+0000","last_change":"2024-03-21T13:14:10.542906+0000","last_active":"2024-03-21T13:14:13.591147+0000","last_peered":"2024-03-21T13:14:13.591147+0000","last_clean":"2024-03-21T13:14:13.591147+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:14:13.591147+0000","last_undegraded":"2024-03-21T13:14:13.591147+0000","last_fullsized":"2024-03-21T13:14:13.591147+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:10.542755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:10.542755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:34.356866+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"346'23","reported_seq":55,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.626516+0000","last_change":"2024-03-21T13:13:21.852476+0000","last_active":"2024-03-21T13:14:13.626516+0000","last_peered":"2024-03-21T13:14:13.626516+0000","last_clean":"2024-03-21T13:14:13.626516+0000","last_became_active":"2024-03-21T13:13:21.819111+0000","last_became_peered":"2024-03-21T13:13:21.819111+0000","last_unstale":"2024-03-21T13:14:13.626516+0000","last_undegraded":"2024-03-21T13:14:13.626516+0000","last_fullsized":"2024-03-21T13:14:13.626516+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'23","last_scrub_stamp":"2024-03-21T13:13:20.232807+0000","last_deep_scrub":"346'23","last_deep_scrub_stamp":"2024-03-21T13:13:20.232807+0000","last_clean_scrub_stamp":"2024-03-21T13:13:20.232807+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:31.174789+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":210,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.630988+0000","last_change":"2024-03-21T13:13:12.686137+0000","last_active":"2024-03-21T13:14:13.630988+0000","last_peered":"2024-03-21T13:14:13.630988+0000","last_clean":"2024-03-21T13:14:13.630988+0000","last_became_active":"2024-03-21T13:12:29.499921+0000","last_became_peered":"2024-03-21T13:12:29.499921+0000","last_unstale":"2024-03-21T13:14:13.630988+0000","last_undegraded":"2024-03-21T13:14:13.630988+0000","last_fullsized":"2024-03-21T13:14:13.630988+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:12.685999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:12.685999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:24.618313+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"364'9674","reported_seq":9764,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:21.056893+0000","last_change":"2024-03-21T13:12:19.520750+0000","last_active":"2024-03-21T13:14:21.056893+0000","last_peered":"2024-03-21T13:14:21.056893+0000","last_clean":"2024-03-21T13:14:21.056893+0000","last_became_active":"2024-03-21T13:12:19.519902+0000","last_became_peered":"2024-03-21T13:12:19.519902+0000","last_unstale":"2024-03-21T13:14:21.056893+0000","last_undegraded":"2024-03-21T13:14:21.056893+0000","last_fullsized":"2024-03-21T13:14:21.056893+0000","mapping_epoch":324,"log_start":"345'5100","ondisk_log_start":"345'5100","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446431232,"num_objects":6812,"num_object_clones":0,"num_object_copies":20436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9674,"num_write_kb":527552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":655,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.591570+0000","last_change":"2024-03-21T13:13:23.379786+0000","last_active":"2024-03-21T13:14:13.591570+0000","last_peered":"2024-03-21T13:14:13.591570+0000","last_clean":"2024-03-21T13:14:13.591570+0000","last_became_active":"2024-03-21T13:10:03.598366+0000","last_became_peered":"2024-03-21T13:10:03.598366+0000","last_unstale":"2024-03-21T13:14:13.591570+0000","last_undegraded":"2024-03-21T13:14:13.591570+0000","last_fullsized":"2024-03-21T13:14:13.591570+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:23.379475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:13:23.379475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:39.026216+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":218,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.631880+0000","last_change":"2024-03-21T13:13:21.644266+0000","last_active":"2024-03-21T13:14:13.631880+0000","last_peered":"2024-03-21T13:14:13.631880+0000","last_clean":"2024-03-21T13:14:13.631880+0000","last_became_active":"2024-03-21T13:12:29.500199+0000","last_became_peered":"2024-03-21T13:12:29.500199+0000","last_unstale":"2024-03-21T13:14:13.631880+0000","last_undegraded":"2024-03-21T13:14:13.631880+0000","last_fullsized":"2024-03-21T13:14:13.631880+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:21.644151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:21.644151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:24.056338+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":666,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:19.700412+0000","last_change":"2024-03-21T13:14:19.700448+0000","last_active":"2024-03-21T13:14:19.700412+0000","last_peered":"2024-03-21T13:14:19.700412+0000","last_clean":"2024-03-21T13:14:19.700412+0000","last_became_active":"2024-03-21T13:12:29.508984+0000","last_became_peered":"2024-03-21T13:12:29.508984+0000","last_unstale":"2024-03-21T13:14:19.700412+0000","last_undegraded":"2024-03-21T13:14:19.700412+0000","last_fullsized":"2024-03-21T13:14:19.700412+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:19.700320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:19.700320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:30.698005+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'35","reported_seq":786,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.580076+0000","last_change":"2024-03-21T13:13:24.732800+0000","last_active":"2024-03-21T13:14:13.580076+0000","last_peered":"2024-03-21T13:14:13.580076+0000","last_clean":"2024-03-21T13:14:13.580076+0000","last_became_active":"2024-03-21T13:12:29.523359+0000","last_became_peered":"2024-03-21T13:12:29.523359+0000","last_unstale":"2024-03-21T13:14:13.580076+0000","last_undegraded":"2024-03-21T13:14:13.580076+0000","last_fullsized":"2024-03-21T13:14:13.580076+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:13:24.732562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:24.732562+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:50.994968+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"364'9567","reported_seq":9722,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:20.268962+0000","last_change":"2024-03-21T13:13:27.478302+0000","last_active":"2024-03-21T13:14:20.268962+0000","last_peered":"2024-03-21T13:14:20.268962+0000","last_clean":"2024-03-21T13:14:20.268962+0000","last_became_active":"2024-03-21T13:12:19.519440+0000","last_became_peered":"2024-03-21T13:12:19.519440+0000","last_unstale":"2024-03-21T13:14:20.268962+0000","last_undegraded":"2024-03-21T13:14:20.268962+0000","last_fullsized":"2024-03-21T13:14:20.268962+0000","mapping_epoch":324,"log_start":"341'4000","ondisk_log_start":"341'4000","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'5750","last_scrub_stamp":"2024-03-21T13:13:27.478094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:13:27.478094+0000","objects_scrubbed":5677,"log_size":5567,"log_dups_size":0,"ondisk_log_size":5567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:45.751471+0000","scrub_duration":1868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445579264,"num_objects":6799,"num_object_clones":0,"num_object_copies":20397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9567,"num_write_kb":523712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":551,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589811+0000","last_change":"2024-03-21T13:11:38.166199+0000","last_active":"2024-03-21T13:14:13.589811+0000","last_peered":"2024-03-21T13:14:13.589811+0000","last_clean":"2024-03-21T13:14:13.589811+0000","last_became_active":"2024-03-21T13:10:03.606655+0000","last_became_peered":"2024-03-21T13:10:03.606655+0000","last_unstale":"2024-03-21T13:14:13.589811+0000","last_undegraded":"2024-03-21T13:14:13.589811+0000","last_fullsized":"2024-03-21T13:14:13.589811+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:38.166077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:38.166077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":204,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.626991+0000","last_change":"2024-03-21T13:13:06.624315+0000","last_active":"2024-03-21T13:14:13.626991+0000","last_peered":"2024-03-21T13:14:13.626991+0000","last_clean":"2024-03-21T13:14:13.626991+0000","last_became_active":"2024-03-21T13:10:03.603646+0000","last_became_peered":"2024-03-21T13:10:03.603646+0000","last_unstale":"2024-03-21T13:14:13.626991+0000","last_undegraded":"2024-03-21T13:14:13.626991+0000","last_fullsized":"2024-03-21T13:14:13.626991+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:06.624233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:06.624233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":87,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.226145+0000","last_change":"2024-03-21T13:14:15.386244+0000","last_active":"2024-03-21T13:14:23.226145+0000","last_peered":"2024-03-21T13:14:23.226145+0000","last_clean":"2024-03-21T13:14:23.226145+0000","last_became_active":"2024-03-21T13:12:29.512741+0000","last_became_peered":"2024-03-21T13:12:29.512741+0000","last_unstale":"2024-03-21T13:14:23.226145+0000","last_undegraded":"2024-03-21T13:14:23.226145+0000","last_fullsized":"2024-03-21T13:14:23.226145+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:15.386132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:15.386132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:21.391493+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"234'19","reported_seq":638,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.224231+0000","last_change":"2024-03-21T13:14:08.259829+0000","last_active":"2024-03-21T13:14:23.224231+0000","last_peered":"2024-03-21T13:14:23.224231+0000","last_clean":"2024-03-21T13:14:23.224231+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:14:23.224231+0000","last_undegraded":"2024-03-21T13:14:23.224231+0000","last_fullsized":"2024-03-21T13:14:23.224231+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:14:08.259691+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:14:08.259691+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:25.539719+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.a","version":"364'9455","reported_seq":9551,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:20.247355+0000","last_change":"2024-03-21T13:12:29.521470+0000","last_active":"2024-03-21T13:14:20.247355+0000","last_peered":"2024-03-21T13:14:20.247355+0000","last_clean":"2024-03-21T13:14:20.247355+0000","last_became_active":"2024-03-21T13:12:29.520693+0000","last_became_peered":"2024-03-21T13:12:29.520693+0000","last_unstale":"2024-03-21T13:14:20.247355+0000","last_undegraded":"2024-03-21T13:14:20.247355+0000","last_fullsized":"2024-03-21T13:14:20.247355+0000","mapping_epoch":330,"log_start":"343'4600","ondisk_log_start":"343'4600","created":324,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452788224,"num_objects":6909,"num_object_clones":0,"num_object_copies":20727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9455,"num_write_kb":523648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":629,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:14.199529+0000","last_change":"2024-03-21T13:14:14.199570+0000","last_active":"2024-03-21T13:14:14.199529+0000","last_peered":"2024-03-21T13:14:14.199529+0000","last_clean":"2024-03-21T13:14:14.199529+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:14:14.199529+0000","last_undegraded":"2024-03-21T13:14:14.199529+0000","last_fullsized":"2024-03-21T13:14:14.199529+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:14.199450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:14:14.199450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:34.853946+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":83,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.228819+0000","last_change":"2024-03-21T13:12:29.554139+0000","last_active":"2024-03-21T13:14:23.228819+0000","last_peered":"2024-03-21T13:14:23.228819+0000","last_clean":"2024-03-21T13:14:23.228819+0000","last_became_active":"2024-03-21T13:12:29.521275+0000","last_became_peered":"2024-03-21T13:12:29.521275+0000","last_unstale":"2024-03-21T13:14:23.228819+0000","last_undegraded":"2024-03-21T13:14:23.228819+0000","last_fullsized":"2024-03-21T13:14:23.228819+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:12:16.224897+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:12:16.224897+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":359,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.579264+0000","last_change":"2024-03-21T13:14:01.668714+0000","last_active":"2024-03-21T13:14:13.579264+0000","last_peered":"2024-03-21T13:14:13.579264+0000","last_clean":"2024-03-21T13:14:13.579264+0000","last_became_active":"2024-03-21T13:06:45.410017+0000","last_became_peered":"2024-03-21T13:06:45.410017+0000","last_unstale":"2024-03-21T13:14:13.579264+0000","last_undegraded":"2024-03-21T13:14:13.579264+0000","last_fullsized":"2024-03-21T13:14:13.579264+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:01.668582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:01.668582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:02.323590+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"272'43","reported_seq":287,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:15.271337+0000","last_change":"2024-03-21T13:13:20.682786+0000","last_active":"2024-03-21T13:14:15.271337+0000","last_peered":"2024-03-21T13:14:15.271337+0000","last_clean":"2024-03-21T13:14:15.271337+0000","last_became_active":"2024-03-21T13:10:03.603889+0000","last_became_peered":"2024-03-21T13:10:03.603889+0000","last_unstale":"2024-03-21T13:14:15.271337+0000","last_undegraded":"2024-03-21T13:14:15.271337+0000","last_fullsized":"2024-03-21T13:14:15.271337+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'43","last_scrub_stamp":"2024-03-21T13:13:20.682623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:20.682623+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:43.173396+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":363,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.579798+0000","last_change":"2024-03-21T13:13:15.969147+0000","last_active":"2024-03-21T13:14:13.579798+0000","last_peered":"2024-03-21T13:14:13.579798+0000","last_clean":"2024-03-21T13:14:13.579798+0000","last_became_active":"2024-03-21T13:06:45.406734+0000","last_became_peered":"2024-03-21T13:06:45.406734+0000","last_unstale":"2024-03-21T13:14:13.579798+0000","last_undegraded":"2024-03-21T13:14:13.579798+0000","last_fullsized":"2024-03-21T13:14:13.579798+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:15.968926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:15.968926+0000","last_clean_scrub_stamp":"2024-03-21T13:13:15.968926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:21.926255+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"365'9915","reported_seq":9994,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:24.811638+0000","last_change":"2024-03-21T13:12:29.509025+0000","last_active":"2024-03-21T13:14:24.811638+0000","last_peered":"2024-03-21T13:14:24.811638+0000","last_clean":"2024-03-21T13:14:24.811638+0000","last_became_active":"2024-03-21T13:12:29.508658+0000","last_became_peered":"2024-03-21T13:12:29.508658+0000","last_unstale":"2024-03-21T13:14:24.811638+0000","last_undegraded":"2024-03-21T13:14:24.811638+0000","last_fullsized":"2024-03-21T13:14:24.811638+0000","mapping_epoch":330,"log_start":"343'4600","ondisk_log_start":"343'4600","created":324,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":5315,"log_dups_size":0,"ondisk_log_size":5315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426180608,"num_objects":6503,"num_object_clones":0,"num_object_copies":19509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9915,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":643,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.590709+0000","last_change":"2024-03-21T13:14:06.644063+0000","last_active":"2024-03-21T13:14:13.590709+0000","last_peered":"2024-03-21T13:14:13.590709+0000","last_clean":"2024-03-21T13:14:13.590709+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:14:13.590709+0000","last_undegraded":"2024-03-21T13:14:13.590709+0000","last_fullsized":"2024-03-21T13:14:13.590709+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:06.643911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:06.643911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:29.513060+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":205,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.631112+0000","last_change":"2024-03-21T13:13:09.639250+0000","last_active":"2024-03-21T13:14:13.631112+0000","last_peered":"2024-03-21T13:14:13.631112+0000","last_clean":"2024-03-21T13:14:13.631112+0000","last_became_active":"2024-03-21T13:10:03.604303+0000","last_became_peered":"2024-03-21T13:10:03.604303+0000","last_unstale":"2024-03-21T13:14:13.631112+0000","last_undegraded":"2024-03-21T13:14:13.631112+0000","last_fullsized":"2024-03-21T13:14:13.631112+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:09.636696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:13:09.636696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:36.946207+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":205,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.627353+0000","last_change":"2024-03-21T13:12:27.851986+0000","last_active":"2024-03-21T13:14:13.627353+0000","last_peered":"2024-03-21T13:14:13.627353+0000","last_clean":"2024-03-21T13:14:13.627353+0000","last_became_active":"2024-03-21T13:10:03.604427+0000","last_became_peered":"2024-03-21T13:10:03.604427+0000","last_unstale":"2024-03-21T13:14:13.627353+0000","last_undegraded":"2024-03-21T13:14:13.627353+0000","last_fullsized":"2024-03-21T13:14:13.627353+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:27.851875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:12:27.851875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"243'20","reported_seq":91,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.227996+0000","last_change":"2024-03-21T13:12:29.544839+0000","last_active":"2024-03-21T13:14:23.227996+0000","last_peered":"2024-03-21T13:14:23.227996+0000","last_clean":"2024-03-21T13:14:23.227996+0000","last_became_active":"2024-03-21T13:12:29.518048+0000","last_became_peered":"2024-03-21T13:12:29.518048+0000","last_unstale":"2024-03-21T13:14:23.227996+0000","last_undegraded":"2024-03-21T13:14:23.227996+0000","last_fullsized":"2024-03-21T13:14:23.227996+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'20","last_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_deep_scrub":"243'20","last_deep_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.683382+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":210,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.630261+0000","last_change":"2024-03-21T13:13:19.683549+0000","last_active":"2024-03-21T13:14:13.630261+0000","last_peered":"2024-03-21T13:14:13.630261+0000","last_clean":"2024-03-21T13:14:13.630261+0000","last_became_active":"2024-03-21T13:12:29.506751+0000","last_became_peered":"2024-03-21T13:12:29.506751+0000","last_unstale":"2024-03-21T13:14:13.630261+0000","last_undegraded":"2024-03-21T13:14:13.630261+0000","last_fullsized":"2024-03-21T13:14:13.630261+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:19.683292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:19.683292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:42.374427+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"364'9530","reported_seq":9620,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:20.224268+0000","last_change":"2024-03-21T13:12:19.517136+0000","last_active":"2024-03-21T13:14:20.224268+0000","last_peered":"2024-03-21T13:14:20.224268+0000","last_clean":"2024-03-21T13:14:20.224268+0000","last_became_active":"2024-03-21T13:12:19.516747+0000","last_became_peered":"2024-03-21T13:12:19.516747+0000","last_unstale":"2024-03-21T13:14:20.224268+0000","last_undegraded":"2024-03-21T13:14:20.224268+0000","last_fullsized":"2024-03-21T13:14:20.224268+0000","mapping_epoch":324,"log_start":"343'4700","ondisk_log_start":"343'4700","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4830,"log_dups_size":0,"ondisk_log_size":4830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450756608,"num_objects":6878,"num_object_clones":0,"num_object_copies":20634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9530,"num_write_kb":525056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":79,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.227815+0000","last_change":"2024-03-21T13:12:29.529126+0000","last_active":"2024-03-21T13:14:23.227815+0000","last_peered":"2024-03-21T13:14:23.227815+0000","last_clean":"2024-03-21T13:14:23.227815+0000","last_became_active":"2024-03-21T13:12:29.527083+0000","last_became_peered":"2024-03-21T13:12:29.527083+0000","last_unstale":"2024-03-21T13:14:23.227815+0000","last_undegraded":"2024-03-21T13:14:23.227815+0000","last_fullsized":"2024-03-21T13:14:23.227815+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:57.956080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:09:57.956080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":905,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.591829+0000","last_change":"2024-03-21T13:14:01.491401+0000","last_active":"2024-03-21T13:14:13.591829+0000","last_peered":"2024-03-21T13:14:13.591829+0000","last_clean":"2024-03-21T13:14:13.591829+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:14:13.591829+0000","last_undegraded":"2024-03-21T13:14:13.591829+0000","last_fullsized":"2024-03-21T13:14:13.591829+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:14:01.491237+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:14:01.491237+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:09.775211+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":150,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.626551+0000","last_change":"2024-03-21T13:13:02.645250+0000","last_active":"2024-03-21T13:14:13.626551+0000","last_peered":"2024-03-21T13:14:13.626551+0000","last_clean":"2024-03-21T13:14:13.626551+0000","last_became_active":"2024-03-21T13:12:29.499989+0000","last_became_peered":"2024-03-21T13:12:29.499989+0000","last_unstale":"2024-03-21T13:14:13.626551+0000","last_undegraded":"2024-03-21T13:14:13.626551+0000","last_fullsized":"2024-03-21T13:14:13.626551+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:02.645139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:13:02.645139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:25.580098+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"234'36","reported_seq":882,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.579960+0000","last_change":"2024-03-21T13:12:26.260721+0000","last_active":"2024-03-21T13:14:13.579960+0000","last_peered":"2024-03-21T13:14:13.579960+0000","last_clean":"2024-03-21T13:14:13.579960+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:14:13.579960+0000","last_undegraded":"2024-03-21T13:14:13.579960+0000","last_fullsized":"2024-03-21T13:14:13.579960+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'36","last_scrub_stamp":"2024-03-21T13:12:26.260559+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:12:26.260559+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":663,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.590396+0000","last_change":"2024-03-21T13:12:29.508145+0000","last_active":"2024-03-21T13:14:13.590396+0000","last_peered":"2024-03-21T13:14:13.590396+0000","last_clean":"2024-03-21T13:14:13.590396+0000","last_became_active":"2024-03-21T13:12:29.507753+0000","last_became_peered":"2024-03-21T13:12:29.507753+0000","last_unstale":"2024-03-21T13:14:13.590396+0000","last_undegraded":"2024-03-21T13:14:13.590396+0000","last_fullsized":"2024-03-21T13:14:13.590396+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:11.001239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:11:11.001239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"364'9586","reported_seq":9758,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:20.226628+0000","last_change":"2024-03-21T13:14:15.730969+0000","last_active":"2024-03-21T13:14:20.226628+0000","last_peered":"2024-03-21T13:14:20.226628+0000","last_clean":"2024-03-21T13:14:20.226628+0000","last_became_active":"2024-03-21T13:12:19.518131+0000","last_became_peered":"2024-03-21T13:12:19.518131+0000","last_unstale":"2024-03-21T13:14:20.226628+0000","last_undegraded":"2024-03-21T13:14:20.226628+0000","last_fullsized":"2024-03-21T13:14:20.226628+0000","mapping_epoch":324,"log_start":"341'4000","ondisk_log_start":"341'4000","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'9234","last_scrub_stamp":"2024-03-21T13:14:15.730731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:14:15.730731+0000","objects_scrubbed":7276,"log_size":5586,"log_dups_size":0,"ondisk_log_size":5586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:16.733134+0000","scrub_duration":2341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448790528,"num_objects":6848,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9586,"num_write_kb":525888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":79,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.226285+0000","last_change":"2024-03-21T13:12:29.515061+0000","last_active":"2024-03-21T13:14:23.226285+0000","last_peered":"2024-03-21T13:14:23.226285+0000","last_clean":"2024-03-21T13:14:23.226285+0000","last_became_active":"2024-03-21T13:12:29.514542+0000","last_became_peered":"2024-03-21T13:12:29.514542+0000","last_unstale":"2024-03-21T13:14:23.226285+0000","last_undegraded":"2024-03-21T13:14:23.226285+0000","last_fullsized":"2024-03-21T13:14:23.226285+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:36.890785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:36.890785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":566,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.223697+0000","last_change":"2024-03-21T13:13:24.068810+0000","last_active":"2024-03-21T13:14:23.223697+0000","last_peered":"2024-03-21T13:14:23.223697+0000","last_clean":"2024-03-21T13:14:23.223697+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:14:23.223697+0000","last_undegraded":"2024-03-21T13:14:23.223697+0000","last_fullsized":"2024-03-21T13:14:23.223697+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:24.068699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:13:24.068699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:33.301925+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":1601,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:18.895210+0000","last_change":"2024-03-21T13:13:24.417605+0000","last_active":"2024-03-21T13:14:18.895210+0000","last_peered":"2024-03-21T13:14:18.895210+0000","last_clean":"2024-03-21T13:14:18.895210+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:14:18.895210+0000","last_undegraded":"2024-03-21T13:14:18.895210+0000","last_fullsized":"2024-03-21T13:14:18.895210+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:13:24.417338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:13:24.417338+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:47.228515+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":932,"num_read_kb":927,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":780,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589103+0000","last_change":"2024-03-21T13:13:59.662999+0000","last_active":"2024-03-21T13:14:13.589103+0000","last_peered":"2024-03-21T13:14:13.589103+0000","last_clean":"2024-03-21T13:14:13.589103+0000","last_became_active":"2024-03-21T13:10:03.606672+0000","last_became_peered":"2024-03-21T13:10:03.606672+0000","last_unstale":"2024-03-21T13:14:13.589103+0000","last_undegraded":"2024-03-21T13:14:13.589103+0000","last_fullsized":"2024-03-21T13:14:13.589103+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-21T13:13:59.662850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:59.662850+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:05.428216+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":87,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.227734+0000","last_change":"2024-03-21T13:14:07.406567+0000","last_active":"2024-03-21T13:14:23.227734+0000","last_peered":"2024-03-21T13:14:23.227734+0000","last_clean":"2024-03-21T13:14:23.227734+0000","last_became_active":"2024-03-21T13:12:29.526100+0000","last_became_peered":"2024-03-21T13:12:29.526100+0000","last_unstale":"2024-03-21T13:14:23.227734+0000","last_undegraded":"2024-03-21T13:14:23.227734+0000","last_fullsized":"2024-03-21T13:14:23.227734+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:07.406434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:14:07.406434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:32.235053+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"364'9697","reported_seq":9788,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:20.249647+0000","last_change":"2024-03-21T13:12:19.518955+0000","last_active":"2024-03-21T13:14:20.249647+0000","last_peered":"2024-03-21T13:14:20.249647+0000","last_clean":"2024-03-21T13:14:20.249647+0000","last_became_active":"2024-03-21T13:12:19.518554+0000","last_became_peered":"2024-03-21T13:12:19.518554+0000","last_unstale":"2024-03-21T13:14:20.249647+0000","last_undegraded":"2024-03-21T13:14:20.249647+0000","last_fullsized":"2024-03-21T13:14:20.249647+0000","mapping_epoch":324,"log_start":"344'4900","ondisk_log_start":"344'4900","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4797,"log_dups_size":0,"ondisk_log_size":4797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452198424,"num_objects":6901,"num_object_clones":0,"num_object_copies":20703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6901,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9697,"num_write_kb":531073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":545,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.579127+0000","last_change":"2024-03-21T13:13:15.013884+0000","last_active":"2024-03-21T13:14:13.579127+0000","last_peered":"2024-03-21T13:14:13.579127+0000","last_clean":"2024-03-21T13:14:13.579127+0000","last_became_active":"2024-03-21T13:10:03.591088+0000","last_became_peered":"2024-03-21T13:10:03.591088+0000","last_unstale":"2024-03-21T13:14:13.579127+0000","last_undegraded":"2024-03-21T13:14:13.579127+0000","last_fullsized":"2024-03-21T13:14:13.579127+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:15.013717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:13:15.013717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:22.891741+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":211,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.627208+0000","last_change":"2024-03-21T13:13:08.629175+0000","last_active":"2024-03-21T13:14:13.627208+0000","last_peered":"2024-03-21T13:14:13.627208+0000","last_clean":"2024-03-21T13:14:13.627208+0000","last_became_active":"2024-03-21T13:12:29.507783+0000","last_became_peered":"2024-03-21T13:12:29.507783+0000","last_unstale":"2024-03-21T13:14:13.627208+0000","last_undegraded":"2024-03-21T13:14:13.627208+0000","last_fullsized":"2024-03-21T13:14:13.627208+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:08.628946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:13:08.628946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":105,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.230901+0000","last_change":"2024-03-21T13:14:16.366110+0000","last_active":"2024-03-21T13:14:23.230901+0000","last_peered":"2024-03-21T13:14:23.230901+0000","last_clean":"2024-03-21T13:14:23.230901+0000","last_became_active":"2024-03-21T13:12:29.518800+0000","last_became_peered":"2024-03-21T13:12:29.518800+0000","last_unstale":"2024-03-21T13:14:23.230901+0000","last_undegraded":"2024-03-21T13:14:23.230901+0000","last_fullsized":"2024-03-21T13:14:23.230901+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-21T13:14:16.365267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:16.365267+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:27.792685+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":673,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.580677+0000","last_change":"2024-03-21T13:13:21.836944+0000","last_active":"2024-03-21T13:14:13.580677+0000","last_peered":"2024-03-21T13:14:13.580677+0000","last_clean":"2024-03-21T13:14:13.580677+0000","last_became_active":"2024-03-21T13:12:29.520657+0000","last_became_peered":"2024-03-21T13:12:29.520657+0000","last_unstale":"2024-03-21T13:14:13.580677+0000","last_undegraded":"2024-03-21T13:14:13.580677+0000","last_fullsized":"2024-03-21T13:14:13.580677+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:21.836810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:21.836810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.069624+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"364'9510","reported_seq":9600,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:20.288425+0000","last_change":"2024-03-21T13:12:19.519139+0000","last_active":"2024-03-21T13:14:20.288425+0000","last_peered":"2024-03-21T13:14:20.288425+0000","last_clean":"2024-03-21T13:14:20.288425+0000","last_became_active":"2024-03-21T13:12:19.518797+0000","last_became_peered":"2024-03-21T13:12:19.518797+0000","last_unstale":"2024-03-21T13:14:20.288425+0000","last_undegraded":"2024-03-21T13:14:20.288425+0000","last_fullsized":"2024-03-21T13:14:20.288425+0000","mapping_epoch":324,"log_start":"344'4900","ondisk_log_start":"344'4900","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4610,"log_dups_size":0,"ondisk_log_size":4610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443940864,"num_objects":6774,"num_object_clones":0,"num_object_copies":20322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":521088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":552,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.582857+0000","last_change":"2024-03-21T13:14:01.163827+0000","last_active":"2024-03-21T13:14:13.582857+0000","last_peered":"2024-03-21T13:14:13.582857+0000","last_clean":"2024-03-21T13:14:13.582857+0000","last_became_active":"2024-03-21T13:10:03.595454+0000","last_became_peered":"2024-03-21T13:10:03.595454+0000","last_unstale":"2024-03-21T13:14:13.582857+0000","last_undegraded":"2024-03-21T13:14:13.582857+0000","last_fullsized":"2024-03-21T13:14:13.582857+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:01.163722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:01.163722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:15.245067+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":79,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.226646+0000","last_change":"2024-03-21T13:12:29.514972+0000","last_active":"2024-03-21T13:14:23.226646+0000","last_peered":"2024-03-21T13:14:23.226646+0000","last_clean":"2024-03-21T13:14:23.226646+0000","last_became_active":"2024-03-21T13:12:29.512291+0000","last_became_peered":"2024-03-21T13:12:29.512291+0000","last_unstale":"2024-03-21T13:14:23.226646+0000","last_undegraded":"2024-03-21T13:14:23.226646+0000","last_fullsized":"2024-03-21T13:14:23.226646+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:28.030369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:28.030369+0000","last_clean_scrub_stamp":"2024-03-21T13:12:28.030369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"234'25","reported_seq":91,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.230262+0000","last_change":"2024-03-21T13:12:29.567830+0000","last_active":"2024-03-21T13:14:23.230262+0000","last_peered":"2024-03-21T13:14:23.230262+0000","last_clean":"2024-03-21T13:14:23.230262+0000","last_became_active":"2024-03-21T13:12:29.526114+0000","last_became_peered":"2024-03-21T13:12:29.526114+0000","last_unstale":"2024-03-21T13:14:23.230262+0000","last_undegraded":"2024-03-21T13:14:23.230262+0000","last_fullsized":"2024-03-21T13:14:23.230262+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-21T13:11:37.749901+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:11:37.749901+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"331'13","reported_seq":747,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:17.962654+0000","last_change":"2024-03-21T13:13:09.998270+0000","last_active":"2024-03-21T13:14:17.962654+0000","last_peered":"2024-03-21T13:14:17.962654+0000","last_clean":"2024-03-21T13:14:17.962654+0000","last_became_active":"2024-03-21T13:12:29.521017+0000","last_became_peered":"2024-03-21T13:12:29.521017+0000","last_unstale":"2024-03-21T13:14:17.962654+0000","last_undegraded":"2024-03-21T13:14:17.962654+0000","last_fullsized":"2024-03-21T13:14:17.962654+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:13:09.997984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:09.997984+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:34.382798+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.f","version":"364'9517","reported_seq":9696,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:20.255193+0000","last_change":"2024-03-21T13:14:01.880864+0000","last_active":"2024-03-21T13:14:20.255193+0000","last_peered":"2024-03-21T13:14:20.255193+0000","last_clean":"2024-03-21T13:14:20.255193+0000","last_became_active":"2024-03-21T13:12:19.468502+0000","last_became_peered":"2024-03-21T13:12:19.468502+0000","last_unstale":"2024-03-21T13:14:20.255193+0000","last_undegraded":"2024-03-21T13:14:20.255193+0000","last_fullsized":"2024-03-21T13:14:20.255193+0000","mapping_epoch":324,"log_start":"343'4700","ondisk_log_start":"343'4700","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'8144","last_scrub_stamp":"2024-03-21T13:14:01.880514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:14:01.880514+0000","objects_scrubbed":8036,"log_size":4817,"log_dups_size":0,"ondisk_log_size":4817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:18.207182+0000","scrub_duration":3250,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449380352,"num_objects":6857,"num_object_clones":0,"num_object_copies":20571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9517,"num_write_kb":523968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":647,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.590927+0000","last_change":"2024-03-21T13:12:19.261227+0000","last_active":"2024-03-21T13:14:13.590927+0000","last_peered":"2024-03-21T13:14:13.590927+0000","last_clean":"2024-03-21T13:14:13.590927+0000","last_became_active":"2024-03-21T13:10:03.606557+0000","last_became_peered":"2024-03-21T13:10:03.606557+0000","last_unstale":"2024-03-21T13:14:13.590927+0000","last_undegraded":"2024-03-21T13:14:13.590927+0000","last_fullsized":"2024-03-21T13:14:13.590927+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:19.261040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:12:19.261040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":699,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589129+0000","last_change":"2024-03-21T13:13:09.602357+0000","last_active":"2024-03-21T13:14:13.589129+0000","last_peered":"2024-03-21T13:14:13.589129+0000","last_clean":"2024-03-21T13:14:13.589129+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:14:13.589129+0000","last_undegraded":"2024-03-21T13:14:13.589129+0000","last_fullsized":"2024-03-21T13:14:13.589129+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:13:09.602090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:13:09.602090+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:20.389303+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'35","reported_seq":103,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.227095+0000","last_change":"2024-03-21T13:14:12.399995+0000","last_active":"2024-03-21T13:14:23.227095+0000","last_peered":"2024-03-21T13:14:23.227095+0000","last_clean":"2024-03-21T13:14:23.227095+0000","last_became_active":"2024-03-21T13:12:29.525154+0000","last_became_peered":"2024-03-21T13:12:29.525154+0000","last_unstale":"2024-03-21T13:14:23.227095+0000","last_undegraded":"2024-03-21T13:14:23.227095+0000","last_fullsized":"2024-03-21T13:14:23.227095+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:14:12.399785+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:14:12.399785+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:22.967195+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"226'10","reported_seq":842,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.583289+0000","last_change":"2024-03-21T13:13:27.063138+0000","last_active":"2024-03-21T13:14:13.583289+0000","last_peered":"2024-03-21T13:14:13.583289+0000","last_clean":"2024-03-21T13:14:13.583289+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:14:13.583289+0000","last_undegraded":"2024-03-21T13:14:13.583289+0000","last_fullsized":"2024-03-21T13:14:13.583289+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T13:13:27.062991+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:13:27.062991+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:43.639228+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"364'9821","reported_seq":9989,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:20.258649+0000","last_change":"2024-03-21T13:14:20.020388+0000","last_active":"2024-03-21T13:14:20.258649+0000","last_peered":"2024-03-21T13:14:20.258649+0000","last_clean":"2024-03-21T13:14:20.258649+0000","last_became_active":"2024-03-21T13:12:19.468361+0000","last_became_peered":"2024-03-21T13:12:19.468361+0000","last_unstale":"2024-03-21T13:14:20.258649+0000","last_undegraded":"2024-03-21T13:14:20.258649+0000","last_fullsized":"2024-03-21T13:14:20.258649+0000","mapping_epoch":324,"log_start":"343'5000","ondisk_log_start":"343'5000","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'9793","last_scrub_stamp":"2024-03-21T13:14:20.020240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:14:20.020240+0000","objects_scrubbed":7101,"log_size":4821,"log_dups_size":0,"ondisk_log_size":4821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:31.881440+0000","scrub_duration":2344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457637888,"num_objects":6983,"num_object_clones":0,"num_object_copies":20949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9821,"num_write_kb":537728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":566,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.590512+0000","last_change":"2024-03-21T13:12:28.983993+0000","last_active":"2024-03-21T13:14:13.590512+0000","last_peered":"2024-03-21T13:14:13.590512+0000","last_clean":"2024-03-21T13:14:13.590512+0000","last_became_active":"2024-03-21T13:10:03.610471+0000","last_became_peered":"2024-03-21T13:10:03.610471+0000","last_unstale":"2024-03-21T13:14:13.590512+0000","last_undegraded":"2024-03-21T13:14:13.590512+0000","last_fullsized":"2024-03-21T13:14:13.590512+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:28.983915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:12:28.983915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":649,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:18.185888+0000","last_change":"2024-03-21T13:14:18.185914+0000","last_active":"2024-03-21T13:14:18.185888+0000","last_peered":"2024-03-21T13:14:18.185888+0000","last_clean":"2024-03-21T13:14:18.185888+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:14:18.185888+0000","last_undegraded":"2024-03-21T13:14:18.185888+0000","last_fullsized":"2024-03-21T13:14:18.185888+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:18.185782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:14:18.185782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:30.895236+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"234'30","reported_seq":687,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.583189+0000","last_change":"2024-03-21T13:12:26.004699+0000","last_active":"2024-03-21T13:14:13.583189+0000","last_peered":"2024-03-21T13:14:13.583189+0000","last_clean":"2024-03-21T13:14:13.583189+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:14:13.583189+0000","last_undegraded":"2024-03-21T13:14:13.583189+0000","last_fullsized":"2024-03-21T13:14:13.583189+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'30","last_scrub_stamp":"2024-03-21T13:12:26.004582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:12:26.004582+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":651,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.223436+0000","last_change":"2024-03-21T13:13:09.984533+0000","last_active":"2024-03-21T13:14:23.223436+0000","last_peered":"2024-03-21T13:14:23.223436+0000","last_clean":"2024-03-21T13:14:23.223436+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:14:23.223436+0000","last_undegraded":"2024-03-21T13:14:23.223436+0000","last_fullsized":"2024-03-21T13:14:23.223436+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:09.984411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:09.984411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:37.909935+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"364'9673","reported_seq":9763,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:20.255805+0000","last_change":"2024-03-21T13:12:19.518000+0000","last_active":"2024-03-21T13:14:20.255805+0000","last_peered":"2024-03-21T13:14:20.255805+0000","last_clean":"2024-03-21T13:14:20.255805+0000","last_became_active":"2024-03-21T13:12:19.517486+0000","last_became_peered":"2024-03-21T13:12:19.517486+0000","last_unstale":"2024-03-21T13:14:20.255805+0000","last_undegraded":"2024-03-21T13:14:20.255805+0000","last_fullsized":"2024-03-21T13:14:20.255805+0000","mapping_epoch":324,"log_start":"345'5100","ondisk_log_start":"345'5100","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4573,"log_dups_size":0,"ondisk_log_size":4573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453312512,"num_objects":6917,"num_object_clones":0,"num_object_copies":20751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9673,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":79,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.226722+0000","last_change":"2024-03-21T13:12:29.521775+0000","last_active":"2024-03-21T13:14:23.226722+0000","last_peered":"2024-03-21T13:14:23.226722+0000","last_clean":"2024-03-21T13:14:23.226722+0000","last_became_active":"2024-03-21T13:12:29.518722+0000","last_became_peered":"2024-03-21T13:12:29.518722+0000","last_unstale":"2024-03-21T13:14:23.226722+0000","last_undegraded":"2024-03-21T13:14:23.226722+0000","last_fullsized":"2024-03-21T13:14:23.226722+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:25.878314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:12:25.878314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"331'13","reported_seq":107,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.228735+0000","last_change":"2024-03-21T13:12:29.557416+0000","last_active":"2024-03-21T13:14:23.228735+0000","last_peered":"2024-03-21T13:14:23.228735+0000","last_clean":"2024-03-21T13:14:23.228735+0000","last_became_active":"2024-03-21T13:12:29.525058+0000","last_became_peered":"2024-03-21T13:12:29.525058+0000","last_unstale":"2024-03-21T13:14:23.228735+0000","last_undegraded":"2024-03-21T13:14:23.228735+0000","last_fullsized":"2024-03-21T13:14:23.228735+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'12","last_scrub_stamp":"2024-03-21T13:11:18.400270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:18.400270+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":655,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:17.224171+0000","last_change":"2024-03-21T13:14:17.224206+0000","last_active":"2024-03-21T13:14:17.224171+0000","last_peered":"2024-03-21T13:14:17.224171+0000","last_clean":"2024-03-21T13:14:17.224171+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:14:17.224171+0000","last_undegraded":"2024-03-21T13:14:17.224171+0000","last_fullsized":"2024-03-21T13:14:17.224171+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:17.224086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:17.224086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:20.410698+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"234'34","reported_seq":610,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.590677+0000","last_change":"2024-03-21T13:14:00.092878+0000","last_active":"2024-03-21T13:14:13.590677+0000","last_peered":"2024-03-21T13:14:13.590677+0000","last_clean":"2024-03-21T13:14:13.590677+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:14:13.590677+0000","last_undegraded":"2024-03-21T13:14:13.590677+0000","last_fullsized":"2024-03-21T13:14:13.590677+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'34","last_scrub_stamp":"2024-03-21T13:14:00.092594+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:14:00.092594+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:03.480207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"364'9604","reported_seq":9692,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:20.250628+0000","last_change":"2024-03-21T13:12:19.467189+0000","last_active":"2024-03-21T13:14:20.250628+0000","last_peered":"2024-03-21T13:14:20.250628+0000","last_clean":"2024-03-21T13:14:20.250628+0000","last_became_active":"2024-03-21T13:12:19.466848+0000","last_became_peered":"2024-03-21T13:12:19.466848+0000","last_unstale":"2024-03-21T13:14:20.250628+0000","last_undegraded":"2024-03-21T13:14:20.250628+0000","last_fullsized":"2024-03-21T13:14:20.250628+0000","mapping_epoch":324,"log_start":"343'4800","ondisk_log_start":"343'4800","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4804,"log_dups_size":0,"ondisk_log_size":4804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454688768,"num_objects":6938,"num_object_clones":0,"num_object_copies":20814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9604,"num_write_kb":529344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":649,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589268+0000","last_change":"2024-03-21T13:12:33.035682+0000","last_active":"2024-03-21T13:14:13.589268+0000","last_peered":"2024-03-21T13:14:13.589268+0000","last_clean":"2024-03-21T13:14:13.589268+0000","last_became_active":"2024-03-21T13:12:29.502816+0000","last_became_peered":"2024-03-21T13:12:29.502816+0000","last_unstale":"2024-03-21T13:14:13.589268+0000","last_undegraded":"2024-03-21T13:14:13.589268+0000","last_fullsized":"2024-03-21T13:14:13.589268+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:12:33.035545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:12:33.035545+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":79,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.227649+0000","last_change":"2024-03-21T13:12:29.524403+0000","last_active":"2024-03-21T13:14:23.227649+0000","last_peered":"2024-03-21T13:14:23.227649+0000","last_clean":"2024-03-21T13:14:23.227649+0000","last_became_active":"2024-03-21T13:12:29.524087+0000","last_became_peered":"2024-03-21T13:12:29.524087+0000","last_unstale":"2024-03-21T13:14:23.227649+0000","last_undegraded":"2024-03-21T13:14:23.227649+0000","last_fullsized":"2024-03-21T13:14:23.227649+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:27.742792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:27.742792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":687,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.223546+0000","last_change":"2024-03-21T13:14:03.182206+0000","last_active":"2024-03-21T13:14:23.223546+0000","last_peered":"2024-03-21T13:14:23.223546+0000","last_clean":"2024-03-21T13:14:23.223546+0000","last_became_active":"2024-03-21T13:10:03.586593+0000","last_became_peered":"2024-03-21T13:10:03.586593+0000","last_unstale":"2024-03-21T13:14:23.223546+0000","last_undegraded":"2024-03-21T13:14:23.223546+0000","last_fullsized":"2024-03-21T13:14:23.223546+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:03.182101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:03.182101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:22.301082+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":409,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.579788+0000","last_change":"2024-03-21T13:14:11.615096+0000","last_active":"2024-03-21T13:14:13.579788+0000","last_peered":"2024-03-21T13:14:13.579788+0000","last_clean":"2024-03-21T13:14:13.579788+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:14:13.579788+0000","last_undegraded":"2024-03-21T13:14:13.579788+0000","last_fullsized":"2024-03-21T13:14:13.579788+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:14:11.611538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:11.611538+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:38.005827+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"364'9424","reported_seq":9608,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:20.256916+0000","last_change":"2024-03-21T13:14:05.715845+0000","last_active":"2024-03-21T13:14:20.256916+0000","last_peered":"2024-03-21T13:14:20.256916+0000","last_clean":"2024-03-21T13:14:20.256916+0000","last_became_active":"2024-03-21T13:12:29.500228+0000","last_became_peered":"2024-03-21T13:12:29.500228+0000","last_unstale":"2024-03-21T13:14:20.256916+0000","last_undegraded":"2024-03-21T13:14:20.256916+0000","last_fullsized":"2024-03-21T13:14:20.256916+0000","mapping_epoch":330,"log_start":"340'3800","ondisk_log_start":"340'3800","created":324,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'8369","last_scrub_stamp":"2024-03-21T13:14:05.715580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:14:05.715580+0000","objects_scrubbed":7944,"log_size":5624,"log_dups_size":0,"ondisk_log_size":5624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:24.792407+0000","scrub_duration":2147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446955520,"num_objects":6820,"num_object_clones":0,"num_object_copies":20460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9424,"num_write_kb":519808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":212,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.639542+0000","last_change":"2024-03-21T13:13:24.855383+0000","last_active":"2024-03-21T13:14:13.639542+0000","last_peered":"2024-03-21T13:14:13.639542+0000","last_clean":"2024-03-21T13:14:13.639542+0000","last_became_active":"2024-03-21T13:10:03.602752+0000","last_became_peered":"2024-03-21T13:10:03.602752+0000","last_unstale":"2024-03-21T13:14:13.639542+0000","last_undegraded":"2024-03-21T13:14:13.639542+0000","last_fullsized":"2024-03-21T13:14:13.639542+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:24.855268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:13:24.855268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:50.759371+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":559,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:23.226180+0000","last_change":"2024-03-21T13:14:09.276862+0000","last_active":"2024-03-21T13:14:23.226180+0000","last_peered":"2024-03-21T13:14:23.226180+0000","last_clean":"2024-03-21T13:14:23.226180+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:14:23.226180+0000","last_undegraded":"2024-03-21T13:14:23.226180+0000","last_fullsized":"2024-03-21T13:14:23.226180+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:09.276760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:14:09.276760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:17.691979+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":663,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.580394+0000","last_change":"2024-03-21T13:12:31.226728+0000","last_active":"2024-03-21T13:14:13.580394+0000","last_peered":"2024-03-21T13:14:13.580394+0000","last_clean":"2024-03-21T13:14:13.580394+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:14:13.580394+0000","last_undegraded":"2024-03-21T13:14:13.580394+0000","last_fullsized":"2024-03-21T13:14:13.580394+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:31.226652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:12:31.226652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"234'31","reported_seq":222,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.627280+0000","last_change":"2024-03-21T13:13:15.705112+0000","last_active":"2024-03-21T13:14:13.627280+0000","last_peered":"2024-03-21T13:14:13.627280+0000","last_clean":"2024-03-21T13:14:13.627280+0000","last_became_active":"2024-03-21T13:10:03.591832+0000","last_became_peered":"2024-03-21T13:10:03.591832+0000","last_unstale":"2024-03-21T13:14:13.627280+0000","last_undegraded":"2024-03-21T13:14:13.627280+0000","last_fullsized":"2024-03-21T13:14:13.627280+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'31","last_scrub_stamp":"2024-03-21T13:13:15.704929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:15.704929+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.307944+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"365'9940","reported_seq":10120,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-21T13:14:25.167548+0000","last_change":"2024-03-21T13:14:07.672937+0000","last_active":"2024-03-21T13:14:25.167548+0000","last_peered":"2024-03-21T13:14:25.167548+0000","last_clean":"2024-03-21T13:14:25.167548+0000","last_became_active":"2024-03-21T13:12:19.518390+0000","last_became_peered":"2024-03-21T13:12:19.518390+0000","last_unstale":"2024-03-21T13:14:25.167548+0000","last_undegraded":"2024-03-21T13:14:25.167548+0000","last_fullsized":"2024-03-21T13:14:25.167548+0000","mapping_epoch":324,"log_start":"348'5500","ondisk_log_start":"348'5500","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'8572","last_scrub_stamp":"2024-03-21T13:14:07.671427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:14:07.671427+0000","objects_scrubbed":7915,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:34.495574+0000","scrub_duration":2473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423624704,"num_objects":6464,"num_object_clones":0,"num_object_copies":19392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9940,"num_write_kb":524928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":210,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.630120+0000","last_change":"2024-03-21T13:12:29.499704+0000","last_active":"2024-03-21T13:14:13.630120+0000","last_peered":"2024-03-21T13:14:13.630120+0000","last_clean":"2024-03-21T13:14:13.630120+0000","last_became_active":"2024-03-21T13:12:29.499442+0000","last_became_peered":"2024-03-21T13:12:29.499442+0000","last_unstale":"2024-03-21T13:14:13.630120+0000","last_undegraded":"2024-03-21T13:14:13.630120+0000","last_fullsized":"2024-03-21T13:14:13.630120+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:20.890918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:12:20.890918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"331'13","reported_seq":271,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.627222+0000","last_change":"2024-03-21T13:12:29.518453+0000","last_active":"2024-03-21T13:14:13.627222+0000","last_peered":"2024-03-21T13:14:13.627222+0000","last_clean":"2024-03-21T13:14:13.627222+0000","last_became_active":"2024-03-21T13:12:29.507925+0000","last_became_peered":"2024-03-21T13:12:29.507925+0000","last_unstale":"2024-03-21T13:14:13.627222+0000","last_undegraded":"2024-03-21T13:14:13.627222+0000","last_fullsized":"2024-03-21T13:14:13.627222+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'12","last_scrub_stamp":"2024-03-21T13:12:18.891042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.891042+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":640,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.588423+0000","last_change":"2024-03-21T13:13:12.141409+0000","last_active":"2024-03-21T13:14:13.588423+0000","last_peered":"2024-03-21T13:14:13.588423+0000","last_clean":"2024-03-21T13:14:13.588423+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:14:13.588423+0000","last_undegraded":"2024-03-21T13:14:13.588423+0000","last_fullsized":"2024-03-21T13:14:13.588423+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:12.141284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:13:12.141284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:36.256401+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"234'20","reported_seq":602,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-21T13:14:13.589458+0000","last_change":"2024-03-21T13:12:17.040702+0000","last_active":"2024-03-21T13:14:13.589458+0000","last_peered":"2024-03-21T13:14:13.589458+0000","last_clean":"2024-03-21T13:14:13.589458+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:14:13.589458+0000","last_undegraded":"2024-03-21T13:14:13.589458+0000","last_fullsized":"2024-03-21T13:14:13.589458+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:12:17.040608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:12:17.040608+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":7138639896,"num_objects":108928,"num_object_clones":0,"num_object_copies":326784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108928,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154666,"num_write_kb":8434945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21260873728,"data_stored":21260861512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79366,"ondisk_log_size":79366,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":932,"num_read_kb":927,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1103,"num_read_kb":860,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":6766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":740,"ondisk_log_size":740,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168769,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3008716,"kb_used_data":2215364,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":90736436,"statfs":{"total":95995035648,"available":92914110464,"internally_reserved":0,"allocated":2268532736,"data_stored":2268022563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":812382666},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463111,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3970320,"kb_used_data":3122760,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":89774832,"statfs":{"total":95995035648,"available":91929427968,"internally_reserved":0,"allocated":3197706240,"data_stored":3197191704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":867826118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575072,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738597,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5178824,"kb_used_data":4425664,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":88566328,"statfs":{"total":95995035648,"available":90691919872,"internally_reserved":0,"allocated":4531879936,"data_stored":4531372531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411547,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3388320,"kb_used_data":2667160,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":90356832,"statfs":{"total":95995035648,"available":92525395968,"internally_reserved":0,"allocated":2731171840,"data_stored":2730704455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182614,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3554180,"kb_used_data":2638652,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":90190972,"statfs":{"total":95995035648,"available":92355555328,"internally_reserved":0,"allocated":2701979648,"data_stored":2701476247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":937490887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019089,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4686720,"kb_used_data":3793216,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":89058432,"statfs":{"total":95995035648,"available":91195834368,"internally_reserved":0,"allocated":3884253184,"data_stored":3883841859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":914749893},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724748,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2273780,"kb_used_data":2107764,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91471372,"statfs":{"total":95995035648,"available":93666684928,"internally_reserved":0,"allocated":2158350336,"data_stored":2157950474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4488564736,"data_stored":4488560664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2699493376,"data_stored":2699493376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2671247360,"data_stored":2671247360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3834515456,"data_stored":3834511384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2142109696,"data_stored":2142109696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3173781504,"data_stored":3173777432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2251161600,"data_stored":2251161600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:14:29.600 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:14:29.609 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:29.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:29.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:29.750 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:29.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:29.851 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:29.896 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:29.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:29.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:29.971 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:29.998 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:30.055 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:30.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:30.157 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:30.196 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:30.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:30.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:30.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:30.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:30.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:30.394 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:30.438 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:30.491 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:30.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:30.503 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:14:30.540 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:14:30.541 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:30 smithi163 ceph-mon[29621]: pgmap v745: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 843 op/s 2024-03-21T13:14:30.541 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:30 smithi163 ceph-mon[40675]: pgmap v745: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 843 op/s 2024-03-21T13:14:30.582 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:14:30.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:30.618 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:14:30.625 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:30.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:30.657 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:14:30.697 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:14:30.727 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:30.743 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:14:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:30 smithi204 ceph-mon[33153]: pgmap v745: 148 pgs: 148 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 843 op/s 2024-03-21T13:14:30.762 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:30.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:30.780 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:14:30.783 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:14:30.783 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:14:30.784 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:14:30.784 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:14:30.784 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:14:30.784 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:14:30.784 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:14:30.784 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:14:30.864 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:30.902 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:30.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:31.003 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:31.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:31.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:31.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:31.189 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:31.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:31.291 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:31.334 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:31.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:31.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:31.436 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:31.475 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:31.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:31.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:31.618 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:31.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:31.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:31.720 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:31.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:31 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3185350371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:31.757 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:31.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:31.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:31.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:31 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3185350371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:31.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:31 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3185350371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:31.860 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:31.893 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:31.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:31.996 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:32.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:32.028 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:32.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:32.129 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:32.161 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:32.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:32.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:32.263 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:32.297 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:32.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:32.398 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:32.433 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:32.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:32.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:32.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:14:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:14:32.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:32.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:32.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:32.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:32.713 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:32.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:32.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:32 smithi204 ceph-mon[33153]: pgmap v746: 148 pgs: 148 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-21T13:14:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:32 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:14:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:32 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:14:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:32 smithi204 ceph-mon[33153]: osdmap e367: 8 total, 8 up, 7 in 2024-03-21T13:14:32.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:32 smithi163 ceph-mon[29621]: pgmap v746: 148 pgs: 148 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-21T13:14:32.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:32 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:14:32.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:32 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:14:32.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:32 smithi163 ceph-mon[29621]: osdmap e367: 8 total, 8 up, 7 in 2024-03-21T13:14:32.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:32 smithi163 ceph-mon[40675]: pgmap v746: 148 pgs: 148 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-21T13:14:32.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:32 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:14:32.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:32 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:14:32.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:32 smithi163 ceph-mon[40675]: osdmap e367: 8 total, 8 up, 7 in 2024-03-21T13:14:32.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:32.853 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:32.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:32.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:32.991 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:32.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:32.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:33.093 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:33.170 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:33.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:33.271 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:33.307 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:33.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:33.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:33.446 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:33.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:33.547 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:33.580 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:33.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:33.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:33.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:33.706 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:33.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:33.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:33 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:33 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:33.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:33 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:33.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:33.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:33.847 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:33.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:33.948 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:33.981 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:33.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:33.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:34.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:34.119 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:34.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:34.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:34.155 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:34.222 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:34.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:34.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:34.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:34.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:34.393 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:34.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:34.495 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:34.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:34.559 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:34.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:34.660 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:34.693 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:34.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:34.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:34.754 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:34 smithi204 ceph-mon[33153]: pgmap v748: 148 pgs: 148 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-03-21T13:14:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:34 smithi163 ceph-mon[29621]: pgmap v748: 148 pgs: 148 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-03-21T13:14:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:34 smithi163 ceph-mon[40675]: pgmap v748: 148 pgs: 148 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-03-21T13:14:34.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:34.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:34.832 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:34.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:34.933 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:34.971 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:34.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:35.029 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:14:35.073 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:35.106 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:14:35.106 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:14:35.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:35.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:35.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:35.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:35.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:35.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:35.350 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:35.387 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:35.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:35.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:35.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:35.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:35.521 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:35.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:35.623 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:35.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:35.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:14:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:14:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:14:35.764 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:35.805 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:35.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:35.907 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:35.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:35.940 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:35.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:36.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:36.072 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:36.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:36.174 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:36.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:36.218 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:36.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:36.319 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:36.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:36.366 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:36.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:36.467 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:36.499 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:14:36.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:36 smithi163 ceph-mon[29621]: pgmap v749: 148 pgs: 148 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-21T13:14:36.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:36 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:36.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:36 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:36.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:36 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/641166440' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:14:36.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:36 smithi163 ceph-mon[40675]: pgmap v749: 148 pgs: 148 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-21T13:14:36.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:36 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:36.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:36 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:36.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:36 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/641166440' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:14:36.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:36 smithi204 ceph-mon[33153]: pgmap v749: 148 pgs: 148 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-21T13:14:36.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:36 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:36.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:36 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:36.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:36 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/641166440' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:14:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:36.511 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:36.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:36.612 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:36.654 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:36.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:36.755 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:36.788 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:36.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:36.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:36.926 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:36.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:37.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:37.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:37.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:37.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:37.199 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:37.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:37.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:37.300 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:37.323 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:37.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:37.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:37.424 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:37.461 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:37.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:37.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:37.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:37.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:37.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:37.740 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:37.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:37.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:37 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:14:37.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:37 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/641166440' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:14:37.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:37 smithi204 ceph-mon[33153]: osdmap e368: 8 total, 8 up, 7 in 2024-03-21T13:14:37.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:37 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:14:37.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:37 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/641166440' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:14:37.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:37 smithi163 ceph-mon[40675]: osdmap e368: 8 total, 8 up, 7 in 2024-03-21T13:14:37.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:37 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:14:37.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:37 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/641166440' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:14:37.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:37 smithi163 ceph-mon[29621]: osdmap e368: 8 total, 8 up, 7 in 2024-03-21T13:14:37.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:37.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:37.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:37.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:37.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:38.008 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:38.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:38.109 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:38.147 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:38.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:38.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:38.249 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:38.289 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:38.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:38.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:38.390 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:38.414 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:38.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:38.516 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:38.550 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:38.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:38.651 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:38.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:38.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:38.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:38.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:38 smithi204 ceph-mon[33153]: pgmap v751: 148 pgs: 148 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-03-21T13:14:38.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:38 smithi163 ceph-mon[29621]: pgmap v751: 148 pgs: 148 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-03-21T13:14:38.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:38 smithi163 ceph-mon[40675]: pgmap v751: 148 pgs: 148 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-03-21T13:14:38.792 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:38.825 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:38.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:38.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:38.927 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:38.962 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:38.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:38.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:39.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:39.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:39.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:39.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:39.127 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-21T13:14:39.201 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:39.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:39.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:39.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:39.335 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:39.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:39.374 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:39.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:39.475 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:39.514 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:39.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:39.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:39.616 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:39.646 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:39.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:39.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:39.781 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:39.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:39.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:39.882 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:39.918 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:39.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:40.019 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:40.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:40.050 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:40.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:40.131 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:40.151 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:40.200 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:40.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:40.301 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:40.333 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:40.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:40.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:40.465 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:40.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:40.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:40.601 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:40.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:40.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:40.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:40 smithi163 ceph-mon[29621]: pgmap v752: 148 pgs: 148 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 881 op/s 2024-03-21T13:14:40.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:40 smithi163 ceph-mon[40675]: pgmap v752: 148 pgs: 148 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 881 op/s 2024-03-21T13:14:40.703 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:40.737 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:40.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:40.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:40 smithi204 ceph-mon[33153]: pgmap v752: 148 pgs: 148 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 881 op/s 2024-03-21T13:14:40.840 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:40.877 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:40.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:40.979 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:41.017 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:41.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:41.109 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:14:41.118 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:14:41.119 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:41.154 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:41.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:41.256 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:41.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:41.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:41.394 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:41.426 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:41.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:41.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:41.528 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:41.558 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:41.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:41.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:41.681 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:41.697 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:41.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:41.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:41 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:14:41.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:41 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:14:41.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:41 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:14:41.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:41 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:14:41.798 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:41.834 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:41.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:41.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:41.936 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:41.969 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:41.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:41.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:41 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:14:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:41 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:14:42.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:42.106 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:42.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:42.116 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:14:42.207 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:42.242 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:42.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:42.343 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:42.372 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":753,"stamp":"2024-03-21T13:14:40.921580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5540813248,"num_objects":84736,"num_object_clones":0,"num_object_copies":254208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84736,"num_whiteouts":0,"num_read":2292,"num_read_kb":2138,"num_write":179957,"num_write_kb":8437406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":1779887,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80330,"ondisk_log_size":80330,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":21673064,"kb_used_data":16488496,"kb_used_omap":10,"kb_used_meta":5184117,"kb_avail":634543000,"statfs":{"total":671965249536,"available":649772032000,"internally_reserved":0,"allocated":16884219904,"data_stored":16881417345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":5308535915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-837287936,"num_objects":-12776,"num_object_clones":0,"num_object_copies":-38328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12776,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12776,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.459943"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":85,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.502980+0000","last_change":"2024-03-21T13:12:29.526257+0000","last_active":"2024-03-21T13:14:36.502980+0000","last_peered":"2024-03-21T13:14:36.502980+0000","last_clean":"2024-03-21T13:14:36.502980+0000","last_became_active":"2024-03-21T13:12:29.525755+0000","last_became_peered":"2024-03-21T13:12:29.525755+0000","last_unstale":"2024-03-21T13:14:36.502980+0000","last_undegraded":"2024-03-21T13:14:36.502980+0000","last_fullsized":"2024-03-21T13:14:36.502980+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:11.717992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:09:11.717992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127074+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":93,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.502149+0000","last_change":"2024-03-21T13:13:14.412543+0000","last_active":"2024-03-21T13:14:36.502149+0000","last_peered":"2024-03-21T13:14:36.502149+0000","last_clean":"2024-03-21T13:14:36.502149+0000","last_became_active":"2024-03-21T13:12:29.510271+0000","last_became_peered":"2024-03-21T13:12:29.510271+0000","last_unstale":"2024-03-21T13:14:36.502149+0000","last_undegraded":"2024-03-21T13:14:36.502149+0000","last_fullsized":"2024-03-21T13:14:36.502149+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:14.412145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:14.412145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127633+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":94,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.505943+0000","last_change":"2024-03-21T13:13:11.426511+0000","last_active":"2024-03-21T13:14:36.505943+0000","last_peered":"2024-03-21T13:14:36.505943+0000","last_clean":"2024-03-21T13:14:36.505943+0000","last_became_active":"2024-03-21T13:12:29.517794+0000","last_became_peered":"2024-03-21T13:12:29.517794+0000","last_unstale":"2024-03-21T13:14:36.505943+0000","last_undegraded":"2024-03-21T13:14:36.505943+0000","last_fullsized":"2024-03-21T13:14:36.505943+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:11.426239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:13:11.426239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127608+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"234'19","reported_seq":737,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.492478+0000","last_change":"2024-03-21T13:14:22.200097+0000","last_active":"2024-03-21T13:14:36.492478+0000","last_peered":"2024-03-21T13:14:36.492478+0000","last_clean":"2024-03-21T13:14:36.492478+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:14:36.492478+0000","last_undegraded":"2024-03-21T13:14:36.492478+0000","last_fullsized":"2024-03-21T13:14:36.492478+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:14:22.199823+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-03-21T13:01:37.692932+0000","last_clean_scrub_stamp":"2024-03-21T13:14:22.199823+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:45.142829+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":634,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.474799+0000","last_change":"2024-03-21T13:11:32.995208+0000","last_active":"2024-03-21T13:14:31.474799+0000","last_peered":"2024-03-21T13:14:31.474799+0000","last_clean":"2024-03-21T13:14:31.474799+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:14:31.474799+0000","last_undegraded":"2024-03-21T13:14:31.474799+0000","last_fullsized":"2024-03-21T13:14:31.474799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:32.995084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:11:32.995084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.154964+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":595,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496981+0000","last_change":"2024-03-21T13:14:05.582597+0000","last_active":"2024-03-21T13:14:36.496981+0000","last_peered":"2024-03-21T13:14:36.496981+0000","last_clean":"2024-03-21T13:14:36.496981+0000","last_became_active":"2024-03-21T13:10:03.591543+0000","last_became_peered":"2024-03-21T13:10:03.591543+0000","last_unstale":"2024-03-21T13:14:36.496981+0000","last_undegraded":"2024-03-21T13:14:36.496981+0000","last_fullsized":"2024-03-21T13:14:36.496981+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:05.582473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:14:05.582473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:10.999808+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":674,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.475157+0000","last_change":"2024-03-21T13:14:11.126777+0000","last_active":"2024-03-21T13:14:31.475157+0000","last_peered":"2024-03-21T13:14:31.475157+0000","last_clean":"2024-03-21T13:14:31.475157+0000","last_became_active":"2024-03-21T13:10:03.610318+0000","last_became_peered":"2024-03-21T13:10:03.610318+0000","last_unstale":"2024-03-21T13:14:31.475157+0000","last_undegraded":"2024-03-21T13:14:31.475157+0000","last_fullsized":"2024-03-21T13:14:31.475157+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_clean_scrub_stamp":"2024-03-21T13:14:11.126531+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:22.684839+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":168,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496040+0000","last_change":"2024-03-21T13:11:28.976696+0000","last_active":"2024-03-21T13:14:36.496040+0000","last_peered":"2024-03-21T13:14:36.496040+0000","last_clean":"2024-03-21T13:14:36.496040+0000","last_became_active":"2024-03-21T13:11:21.702209+0000","last_became_peered":"2024-03-21T13:11:21.702209+0000","last_unstale":"2024-03-21T13:14:36.496040+0000","last_undegraded":"2024-03-21T13:14:36.496040+0000","last_fullsized":"2024-03-21T13:14:36.496040+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:11:28.976545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:28.976545+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":555,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495034+0000","last_change":"2024-03-21T13:14:07.576452+0000","last_active":"2024-03-21T13:14:36.495034+0000","last_peered":"2024-03-21T13:14:36.495034+0000","last_clean":"2024-03-21T13:14:36.495034+0000","last_became_active":"2024-03-21T13:10:03.597963+0000","last_became_peered":"2024-03-21T13:10:03.597963+0000","last_unstale":"2024-03-21T13:14:36.495034+0000","last_undegraded":"2024-03-21T13:14:36.495034+0000","last_fullsized":"2024-03-21T13:14:36.495034+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:07.576369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:07.576369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:23.341119+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":85,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.503129+0000","last_change":"2024-03-21T13:12:29.519152+0000","last_active":"2024-03-21T13:14:36.503129+0000","last_peered":"2024-03-21T13:14:36.503129+0000","last_clean":"2024-03-21T13:14:36.503129+0000","last_became_active":"2024-03-21T13:12:29.518784+0000","last_became_peered":"2024-03-21T13:12:29.518784+0000","last_unstale":"2024-03-21T13:14:36.503129+0000","last_undegraded":"2024-03-21T13:14:36.503129+0000","last_fullsized":"2024-03-21T13:14:36.503129+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:20.342189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:11:20.342189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127174+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":85,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.504326+0000","last_change":"2024-03-21T13:12:29.526491+0000","last_active":"2024-03-21T13:14:36.504326+0000","last_peered":"2024-03-21T13:14:36.504326+0000","last_clean":"2024-03-21T13:14:36.504326+0000","last_became_active":"2024-03-21T13:12:29.526129+0000","last_became_peered":"2024-03-21T13:12:29.526129+0000","last_unstale":"2024-03-21T13:14:36.504326+0000","last_undegraded":"2024-03-21T13:14:36.504326+0000","last_fullsized":"2024-03-21T13:14:36.504326+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:33.515220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:33.515220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":645,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.475589+0000","last_change":"2024-03-21T13:13:13.181558+0000","last_active":"2024-03-21T13:14:31.475589+0000","last_peered":"2024-03-21T13:14:31.475589+0000","last_clean":"2024-03-21T13:14:31.475589+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:14:31.475589+0000","last_undegraded":"2024-03-21T13:14:31.475589+0000","last_fullsized":"2024-03-21T13:14:31.475589+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:13:13.181385+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:13:13.181385+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:41.227273+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":93,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.502831+0000","last_change":"2024-03-21T13:14:05.545713+0000","last_active":"2024-03-21T13:14:36.502831+0000","last_peered":"2024-03-21T13:14:36.502831+0000","last_clean":"2024-03-21T13:14:36.502831+0000","last_became_active":"2024-03-21T13:12:29.508520+0000","last_became_peered":"2024-03-21T13:12:29.508520+0000","last_unstale":"2024-03-21T13:14:36.502831+0000","last_undegraded":"2024-03-21T13:14:36.502831+0000","last_fullsized":"2024-03-21T13:14:36.502831+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:05.545425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:05.545425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:21.804231+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":660,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495691+0000","last_change":"2024-03-21T13:13:03.359188+0000","last_active":"2024-03-21T13:14:36.495691+0000","last_peered":"2024-03-21T13:14:36.495691+0000","last_clean":"2024-03-21T13:14:36.495691+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:14:36.495691+0000","last_undegraded":"2024-03-21T13:14:36.495691+0000","last_fullsized":"2024-03-21T13:14:36.495691+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:03.359078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:03.359078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":85,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.503529+0000","last_change":"2024-03-21T13:12:29.524840+0000","last_active":"2024-03-21T13:14:36.503529+0000","last_peered":"2024-03-21T13:14:36.503529+0000","last_clean":"2024-03-21T13:14:36.503529+0000","last_became_active":"2024-03-21T13:12:29.524473+0000","last_became_peered":"2024-03-21T13:12:29.524473+0000","last_unstale":"2024-03-21T13:14:36.503529+0000","last_undegraded":"2024-03-21T13:14:36.503529+0000","last_fullsized":"2024-03-21T13:14:36.503529+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:25.792349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:25.792349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"234'9","reported_seq":665,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.474505+0000","last_change":"2024-03-21T13:10:03.627839+0000","last_active":"2024-03-21T13:14:31.474505+0000","last_peered":"2024-03-21T13:14:31.474505+0000","last_clean":"2024-03-21T13:14:31.474505+0000","last_became_active":"2024-03-21T13:10:03.595745+0000","last_became_peered":"2024-03-21T13:10:03.595745+0000","last_unstale":"2024-03-21T13:14:31.474505+0000","last_undegraded":"2024-03-21T13:14:31.474505+0000","last_fullsized":"2024-03-21T13:14:31.474505+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'9","last_scrub_stamp":"2024-03-21T13:10:00.872509+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:10:00.872509+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.154841+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":689,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496999+0000","last_change":"2024-03-21T13:14:10.384752+0000","last_active":"2024-03-21T13:14:36.496999+0000","last_peered":"2024-03-21T13:14:36.496999+0000","last_clean":"2024-03-21T13:14:36.496999+0000","last_became_active":"2024-03-21T13:12:29.520189+0000","last_became_peered":"2024-03-21T13:12:29.520189+0000","last_unstale":"2024-03-21T13:14:36.496999+0000","last_undegraded":"2024-03-21T13:14:36.496999+0000","last_fullsized":"2024-03-21T13:14:36.496999+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:10.384430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:23.865785+0000","last_clean_scrub_stamp":"2024-03-21T13:14:10.384430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:27.709391+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":576,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495209+0000","last_change":"2024-03-21T13:14:14.230013+0000","last_active":"2024-03-21T13:14:36.495209+0000","last_peered":"2024-03-21T13:14:36.495209+0000","last_clean":"2024-03-21T13:14:36.495209+0000","last_became_active":"2024-03-21T13:12:29.520241+0000","last_became_peered":"2024-03-21T13:12:29.520241+0000","last_unstale":"2024-03-21T13:14:36.495209+0000","last_undegraded":"2024-03-21T13:14:36.495209+0000","last_fullsized":"2024-03-21T13:14:36.495209+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:14.224068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:14:14.224068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:39.198450+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":547,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496271+0000","last_change":"2024-03-21T13:12:33.276793+0000","last_active":"2024-03-21T13:14:36.496271+0000","last_peered":"2024-03-21T13:14:36.496271+0000","last_clean":"2024-03-21T13:14:36.496271+0000","last_became_active":"2024-03-21T13:01:48.209845+0000","last_became_peered":"2024-03-21T13:01:48.209845+0000","last_unstale":"2024-03-21T13:14:36.496271+0000","last_undegraded":"2024-03-21T13:14:36.496271+0000","last_fullsized":"2024-03-21T13:14:36.496271+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:33.276672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:12:33.276672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":228,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496189+0000","last_change":"2024-03-21T13:12:15.899688+0000","last_active":"2024-03-21T13:14:36.496189+0000","last_peered":"2024-03-21T13:14:36.496189+0000","last_clean":"2024-03-21T13:14:36.496189+0000","last_became_active":"2024-03-21T13:10:03.604616+0000","last_became_peered":"2024-03-21T13:10:03.604616+0000","last_unstale":"2024-03-21T13:14:36.496189+0000","last_undegraded":"2024-03-21T13:14:36.496189+0000","last_fullsized":"2024-03-21T13:14:36.496189+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:12:15.899551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:12:15.899551+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":659,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.476271+0000","last_change":"2024-03-21T13:13:21.480427+0000","last_active":"2024-03-21T13:14:31.476271+0000","last_peered":"2024-03-21T13:14:31.476271+0000","last_clean":"2024-03-21T13:14:31.476271+0000","last_became_active":"2024-03-21T13:12:29.502808+0000","last_became_peered":"2024-03-21T13:12:29.502808+0000","last_unstale":"2024-03-21T13:14:31.476271+0000","last_undegraded":"2024-03-21T13:14:31.476271+0000","last_fullsized":"2024-03-21T13:14:31.476271+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:21.480296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:13:21.480296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.326257+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":571,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.479648+0000","last_change":"2024-03-21T13:10:03.613920+0000","last_active":"2024-03-21T13:14:31.479648+0000","last_peered":"2024-03-21T13:14:31.479648+0000","last_clean":"2024-03-21T13:14:31.479648+0000","last_became_active":"2024-03-21T13:10:03.612409+0000","last_became_peered":"2024-03-21T13:10:03.612409+0000","last_unstale":"2024-03-21T13:14:31.479648+0000","last_undegraded":"2024-03-21T13:14:31.479648+0000","last_fullsized":"2024-03-21T13:14:31.479648+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:56.868411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:09:56.868411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.154878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":698,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.494131+0000","last_change":"2024-03-21T13:13:07.954508+0000","last_active":"2024-03-21T13:14:36.494131+0000","last_peered":"2024-03-21T13:14:36.494131+0000","last_clean":"2024-03-21T13:14:36.494131+0000","last_became_active":"2024-03-21T13:10:03.586563+0000","last_became_peered":"2024-03-21T13:10:03.586563+0000","last_unstale":"2024-03-21T13:14:36.494131+0000","last_undegraded":"2024-03-21T13:14:36.494131+0000","last_fullsized":"2024-03-21T13:14:36.494131+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:07.954268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:20.956778+0000","last_clean_scrub_stamp":"2024-03-21T13:13:07.954268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:36.523160+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":710,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.476447+0000","last_change":"2024-03-21T13:14:04.151045+0000","last_active":"2024-03-21T13:14:31.476447+0000","last_peered":"2024-03-21T13:14:31.476447+0000","last_clean":"2024-03-21T13:14:31.476447+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:14:31.476447+0000","last_undegraded":"2024-03-21T13:14:31.476447+0000","last_fullsized":"2024-03-21T13:14:31.476447+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-21T13:14:04.150913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:04.150913+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:09.479392+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":65,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495551+0000","last_change":"2024-03-21T13:13:06.433430+0000","last_active":"2024-03-21T13:14:36.495551+0000","last_peered":"2024-03-21T13:14:36.495551+0000","last_clean":"2024-03-21T13:14:36.495551+0000","last_became_active":"2024-03-21T13:13:06.420220+0000","last_became_peered":"2024-03-21T13:13:06.420220+0000","last_unstale":"2024-03-21T13:14:36.495551+0000","last_undegraded":"2024-03-21T13:14:36.495551+0000","last_fullsized":"2024-03-21T13:14:36.495551+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:12:19.913538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:12:19.913538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":853,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.499043+0000","last_change":"2024-03-21T13:12:25.267410+0000","last_active":"2024-03-21T13:14:36.499043+0000","last_peered":"2024-03-21T13:14:36.499043+0000","last_clean":"2024-03-21T13:14:36.499043+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:14:36.499043+0000","last_undegraded":"2024-03-21T13:14:36.499043+0000","last_fullsized":"2024-03-21T13:14:36.499043+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:12:25.267265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:12:25.267265+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":553,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.470891+0000","last_change":"2024-03-21T13:14:02.109905+0000","last_active":"2024-03-21T13:14:31.470891+0000","last_peered":"2024-03-21T13:14:31.470891+0000","last_clean":"2024-03-21T13:14:31.470891+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:14:31.470891+0000","last_undegraded":"2024-03-21T13:14:31.470891+0000","last_fullsized":"2024-03-21T13:14:31.470891+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:14:02.109739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:02.109739+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:12.931461+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":597,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496453+0000","last_change":"2024-03-21T13:11:16.033191+0000","last_active":"2024-03-21T13:14:36.496453+0000","last_peered":"2024-03-21T13:14:36.496453+0000","last_clean":"2024-03-21T13:14:36.496453+0000","last_became_active":"2024-03-21T13:01:48.214769+0000","last_became_peered":"2024-03-21T13:01:48.214769+0000","last_unstale":"2024-03-21T13:14:36.496453+0000","last_undegraded":"2024-03-21T13:14:36.496453+0000","last_fullsized":"2024-03-21T13:14:36.496453+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:11:16.033036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:16.033036+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":85,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.504240+0000","last_change":"2024-03-21T13:12:29.527843+0000","last_active":"2024-03-21T13:14:36.504240+0000","last_peered":"2024-03-21T13:14:36.504240+0000","last_clean":"2024-03-21T13:14:36.504240+0000","last_became_active":"2024-03-21T13:12:29.527129+0000","last_became_peered":"2024-03-21T13:12:29.527129+0000","last_unstale":"2024-03-21T13:14:36.504240+0000","last_undegraded":"2024-03-21T13:14:36.504240+0000","last_fullsized":"2024-03-21T13:14:36.504240+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:12.924305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:12:12.924305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127381+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":560,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496332+0000","last_change":"2024-03-21T13:11:13.977186+0000","last_active":"2024-03-21T13:14:36.496332+0000","last_peered":"2024-03-21T13:14:36.496332+0000","last_clean":"2024-03-21T13:14:36.496332+0000","last_became_active":"2024-03-21T13:10:03.599880+0000","last_became_peered":"2024-03-21T13:10:03.599880+0000","last_unstale":"2024-03-21T13:14:36.496332+0000","last_undegraded":"2024-03-21T13:14:36.496332+0000","last_fullsized":"2024-03-21T13:14:36.496332+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:13.977069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:13.977069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":672,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.470082+0000","last_change":"2024-03-21T13:14:20.212339+0000","last_active":"2024-03-21T13:14:31.470082+0000","last_peered":"2024-03-21T13:14:31.470082+0000","last_clean":"2024-03-21T13:14:31.470082+0000","last_became_active":"2024-03-21T13:10:03.610489+0000","last_became_peered":"2024-03-21T13:10:03.610489+0000","last_unstale":"2024-03-21T13:14:31.470082+0000","last_undegraded":"2024-03-21T13:14:31.470082+0000","last_fullsized":"2024-03-21T13:14:31.470082+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:20.212222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:20.212222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:23.956162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"331'24","reported_seq":743,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.499092+0000","last_change":"2024-03-21T13:12:30.276844+0000","last_active":"2024-03-21T13:14:36.499092+0000","last_peered":"2024-03-21T13:14:36.499092+0000","last_clean":"2024-03-21T13:14:36.499092+0000","last_became_active":"2024-03-21T13:12:29.522466+0000","last_became_peered":"2024-03-21T13:12:29.522466+0000","last_unstale":"2024-03-21T13:14:36.499092+0000","last_undegraded":"2024-03-21T13:14:36.499092+0000","last_fullsized":"2024-03-21T13:14:36.499092+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'24","last_scrub_stamp":"2024-03-21T13:12:30.276748+0000","last_deep_scrub":"331'24","last_deep_scrub_stamp":"2024-03-21T13:12:30.276748+0000","last_clean_scrub_stamp":"2024-03-21T13:12:30.276748+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:37.805373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":503,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496450+0000","last_change":"2024-03-21T13:14:20.369440+0000","last_active":"2024-03-21T13:14:36.496450+0000","last_peered":"2024-03-21T13:14:36.496450+0000","last_clean":"2024-03-21T13:14:36.496450+0000","last_became_active":"2024-03-21T13:13:21.827921+0000","last_became_peered":"2024-03-21T13:13:21.827921+0000","last_unstale":"2024-03-21T13:14:36.496450+0000","last_undegraded":"2024-03-21T13:14:36.496450+0000","last_fullsized":"2024-03-21T13:14:36.496450+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:20.369324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:14:20.369324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:37.770360+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":656,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.497657+0000","last_change":"2024-03-21T13:13:12.942908+0000","last_active":"2024-03-21T13:14:36.497657+0000","last_peered":"2024-03-21T13:14:36.497657+0000","last_clean":"2024-03-21T13:14:36.497657+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:14:36.497657+0000","last_undegraded":"2024-03-21T13:14:36.497657+0000","last_fullsized":"2024-03-21T13:14:36.497657+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:12.942785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:12.942785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:37.807362+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":85,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.506384+0000","last_change":"2024-03-21T13:12:29.518880+0000","last_active":"2024-03-21T13:14:36.506384+0000","last_peered":"2024-03-21T13:14:36.506384+0000","last_clean":"2024-03-21T13:14:36.506384+0000","last_became_active":"2024-03-21T13:12:29.517486+0000","last_became_peered":"2024-03-21T13:12:29.517486+0000","last_unstale":"2024-03-21T13:14:36.506384+0000","last_undegraded":"2024-03-21T13:14:36.506384+0000","last_fullsized":"2024-03-21T13:14:36.506384+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:22.955737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:22.955737+0000","last_clean_scrub_stamp":"2024-03-21T13:12:22.955737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":666,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.470176+0000","last_change":"2024-03-21T13:13:26.091403+0000","last_active":"2024-03-21T13:14:31.470176+0000","last_peered":"2024-03-21T13:14:31.470176+0000","last_clean":"2024-03-21T13:14:31.470176+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:14:31.470176+0000","last_undegraded":"2024-03-21T13:14:31.470176+0000","last_fullsized":"2024-03-21T13:14:31.470176+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-21T13:13:26.091169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:26.091169+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:50.023578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":634,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.474990+0000","last_change":"2024-03-21T13:11:33.943308+0000","last_active":"2024-03-21T13:14:31.474990+0000","last_peered":"2024-03-21T13:14:31.474990+0000","last_clean":"2024-03-21T13:14:31.474990+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:14:31.474990+0000","last_undegraded":"2024-03-21T13:14:31.474990+0000","last_fullsized":"2024-03-21T13:14:31.474990+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:33.943015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:33.943015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.154936+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":302,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496883+0000","last_change":"2024-03-21T13:13:05.952941+0000","last_active":"2024-03-21T13:14:36.496883+0000","last_peered":"2024-03-21T13:14:36.496883+0000","last_clean":"2024-03-21T13:14:36.496883+0000","last_became_active":"2024-03-21T13:08:20.955293+0000","last_became_peered":"2024-03-21T13:08:20.955293+0000","last_unstale":"2024-03-21T13:14:36.496883+0000","last_undegraded":"2024-03-21T13:14:36.496883+0000","last_fullsized":"2024-03-21T13:14:36.496883+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:05.952808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:05.952808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:37.807286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":93,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.504468+0000","last_change":"2024-03-21T13:13:15.451096+0000","last_active":"2024-03-21T13:14:36.504468+0000","last_peered":"2024-03-21T13:14:36.504468+0000","last_clean":"2024-03-21T13:14:36.504468+0000","last_became_active":"2024-03-21T13:12:29.507706+0000","last_became_peered":"2024-03-21T13:12:29.507706+0000","last_unstale":"2024-03-21T13:14:36.504468+0000","last_undegraded":"2024-03-21T13:14:36.504468+0000","last_fullsized":"2024-03-21T13:14:36.504468+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:15.447666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:13:15.447666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.842740+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":388,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.471048+0000","last_change":"2024-03-21T13:13:09.092652+0000","last_active":"2024-03-21T13:14:31.471048+0000","last_peered":"2024-03-21T13:14:31.471048+0000","last_clean":"2024-03-21T13:14:31.471048+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:14:31.471048+0000","last_undegraded":"2024-03-21T13:14:31.471048+0000","last_fullsized":"2024-03-21T13:14:31.471048+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:13:09.092038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:09.092038+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.155660+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":93,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.502169+0000","last_change":"2024-03-21T13:14:06.489352+0000","last_active":"2024-03-21T13:14:36.502169+0000","last_peered":"2024-03-21T13:14:36.502169+0000","last_clean":"2024-03-21T13:14:36.502169+0000","last_became_active":"2024-03-21T13:12:29.518334+0000","last_became_peered":"2024-03-21T13:12:29.518334+0000","last_unstale":"2024-03-21T13:14:36.502169+0000","last_undegraded":"2024-03-21T13:14:36.502169+0000","last_fullsized":"2024-03-21T13:14:36.502169+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:06.489212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:06.489212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:20.036491+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":656,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.474710+0000","last_change":"2024-03-21T13:13:08.066898+0000","last_active":"2024-03-21T13:14:31.474710+0000","last_peered":"2024-03-21T13:14:31.474710+0000","last_clean":"2024-03-21T13:14:31.474710+0000","last_became_active":"2024-03-21T13:12:29.502203+0000","last_became_peered":"2024-03-21T13:12:29.502203+0000","last_unstale":"2024-03-21T13:14:31.474710+0000","last_undegraded":"2024-03-21T13:14:31.474710+0000","last_fullsized":"2024-03-21T13:14:31.474710+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:08.066673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:08.066673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.155632+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":654,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495845+0000","last_change":"2024-03-21T13:13:19.383175+0000","last_active":"2024-03-21T13:14:36.495845+0000","last_peered":"2024-03-21T13:14:36.495845+0000","last_clean":"2024-03-21T13:14:36.495845+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:14:36.495845+0000","last_undegraded":"2024-03-21T13:14:36.495845+0000","last_fullsized":"2024-03-21T13:14:36.495845+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:19.383090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:19.383090+0000","last_clean_scrub_stamp":"2024-03-21T13:13:19.383090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:43.999038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":661,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.498044+0000","last_change":"2024-03-21T13:13:22.789241+0000","last_active":"2024-03-21T13:14:36.498044+0000","last_peered":"2024-03-21T13:14:36.498044+0000","last_clean":"2024-03-21T13:14:36.498044+0000","last_became_active":"2024-03-21T13:10:03.591857+0000","last_became_peered":"2024-03-21T13:10:03.591857+0000","last_unstale":"2024-03-21T13:14:36.498044+0000","last_undegraded":"2024-03-21T13:14:36.498044+0000","last_fullsized":"2024-03-21T13:14:36.498044+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:13:22.789110+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:13:22.789110+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:37.807393+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":481,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495210+0000","last_change":"2024-03-21T13:11:34.090139+0000","last_active":"2024-03-21T13:14:36.495210+0000","last_peered":"2024-03-21T13:14:36.495210+0000","last_clean":"2024-03-21T13:14:36.495210+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:14:36.495210+0000","last_undegraded":"2024-03-21T13:14:36.495210+0000","last_fullsized":"2024-03-21T13:14:36.495210+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:34.087671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:34.087671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":93,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.503606+0000","last_change":"2024-03-21T13:13:22.501168+0000","last_active":"2024-03-21T13:14:36.503606+0000","last_peered":"2024-03-21T13:14:36.503606+0000","last_clean":"2024-03-21T13:14:36.503606+0000","last_became_active":"2024-03-21T13:12:29.510100+0000","last_became_peered":"2024-03-21T13:12:29.510100+0000","last_unstale":"2024-03-21T13:14:36.503606+0000","last_undegraded":"2024-03-21T13:14:36.503606+0000","last_fullsized":"2024-03-21T13:14:36.503606+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:22.501055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:22.501055+0000","last_clean_scrub_stamp":"2024-03-21T13:13:22.501055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:44.976431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":367,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.499729+0000","last_change":"2024-03-21T13:13:20.420469+0000","last_active":"2024-03-21T13:14:36.499729+0000","last_peered":"2024-03-21T13:14:36.499729+0000","last_clean":"2024-03-21T13:14:36.499729+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:14:36.499729+0000","last_undegraded":"2024-03-21T13:14:36.499729+0000","last_fullsized":"2024-03-21T13:14:36.499729+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:20.420313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:13:20.420313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":102,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.503197+0000","last_change":"2024-03-21T13:12:29.601629+0000","last_active":"2024-03-21T13:14:36.503197+0000","last_peered":"2024-03-21T13:14:36.503197+0000","last_clean":"2024-03-21T13:14:36.503197+0000","last_became_active":"2024-03-21T13:12:29.521758+0000","last_became_peered":"2024-03-21T13:12:29.521758+0000","last_unstale":"2024-03-21T13:14:36.503197+0000","last_undegraded":"2024-03-21T13:14:36.503197+0000","last_fullsized":"2024-03-21T13:14:36.503197+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:11:29.734236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:11:29.734236+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127352+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":555,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496120+0000","last_change":"2024-03-21T13:14:03.518501+0000","last_active":"2024-03-21T13:14:36.496120+0000","last_peered":"2024-03-21T13:14:36.496120+0000","last_clean":"2024-03-21T13:14:36.496120+0000","last_became_active":"2024-03-21T13:01:48.214341+0000","last_became_peered":"2024-03-21T13:01:48.214341+0000","last_unstale":"2024-03-21T13:14:36.496120+0000","last_undegraded":"2024-03-21T13:14:36.496120+0000","last_fullsized":"2024-03-21T13:14:36.496120+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:03.518372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:03.518372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:13.362750+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"331'11","reported_seq":124,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.504462+0000","last_change":"2024-03-21T13:13:23.466272+0000","last_active":"2024-03-21T13:14:36.504462+0000","last_peered":"2024-03-21T13:14:36.504462+0000","last_clean":"2024-03-21T13:14:36.504462+0000","last_became_active":"2024-03-21T13:12:29.521385+0000","last_became_peered":"2024-03-21T13:12:29.521385+0000","last_unstale":"2024-03-21T13:14:36.504462+0000","last_undegraded":"2024-03-21T13:14:36.504462+0000","last_fullsized":"2024-03-21T13:14:36.504462+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'11","last_scrub_stamp":"2024-03-21T13:13:23.466140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:23.466140+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:44.052256+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":666,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495531+0000","last_change":"2024-03-21T13:12:18.171392+0000","last_active":"2024-03-21T13:14:36.495531+0000","last_peered":"2024-03-21T13:14:36.495531+0000","last_clean":"2024-03-21T13:14:36.495531+0000","last_became_active":"2024-03-21T13:10:03.599265+0000","last_became_peered":"2024-03-21T13:10:03.599265+0000","last_unstale":"2024-03-21T13:14:36.495531+0000","last_undegraded":"2024-03-21T13:14:36.495531+0000","last_fullsized":"2024-03-21T13:14:36.495531+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.171299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.171299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'20","reported_seq":726,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.479942+0000","last_change":"2024-03-21T13:13:04.129254+0000","last_active":"2024-03-21T13:14:31.479942+0000","last_peered":"2024-03-21T13:14:31.479942+0000","last_clean":"2024-03-21T13:14:31.479942+0000","last_became_active":"2024-03-21T13:10:03.613119+0000","last_became_peered":"2024-03-21T13:10:03.613119+0000","last_unstale":"2024-03-21T13:14:31.479942+0000","last_undegraded":"2024-03-21T13:14:31.479942+0000","last_fullsized":"2024-03-21T13:14:31.479942+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:13:04.128932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:04.128932+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.155608+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":226,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.497152+0000","last_change":"2024-03-21T13:14:16.555652+0000","last_active":"2024-03-21T13:14:36.497152+0000","last_peered":"2024-03-21T13:14:36.497152+0000","last_clean":"2024-03-21T13:14:36.497152+0000","last_became_active":"2024-03-21T13:12:29.507751+0000","last_became_peered":"2024-03-21T13:12:29.507751+0000","last_unstale":"2024-03-21T13:14:36.497152+0000","last_undegraded":"2024-03-21T13:14:36.497152+0000","last_fullsized":"2024-03-21T13:14:36.497152+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:16.555516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:36.699469+0000","last_clean_scrub_stamp":"2024-03-21T13:14:16.555516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:26.167124+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"272'10","reported_seq":859,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.480086+0000","last_change":"2024-03-21T13:13:25.102337+0000","last_active":"2024-03-21T13:14:31.480086+0000","last_peered":"2024-03-21T13:14:31.480086+0000","last_clean":"2024-03-21T13:14:31.480086+0000","last_became_active":"2024-03-21T13:10:03.594651+0000","last_became_peered":"2024-03-21T13:10:03.594651+0000","last_unstale":"2024-03-21T13:14:31.480086+0000","last_undegraded":"2024-03-21T13:14:31.480086+0000","last_fullsized":"2024-03-21T13:14:31.480086+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'10","last_scrub_stamp":"2024-03-21T13:13:25.102208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:25.102208+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:42.694340+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":555,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495525+0000","last_change":"2024-03-21T13:13:21.373908+0000","last_active":"2024-03-21T13:14:36.495525+0000","last_peered":"2024-03-21T13:14:36.495525+0000","last_clean":"2024-03-21T13:14:36.495525+0000","last_became_active":"2024-03-21T13:01:48.212477+0000","last_became_peered":"2024-03-21T13:01:48.212477+0000","last_unstale":"2024-03-21T13:14:36.495525+0000","last_undegraded":"2024-03-21T13:14:36.495525+0000","last_fullsized":"2024-03-21T13:14:36.495525+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:21.373667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:13:21.373667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"234'15","reported_seq":103,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.503963+0000","last_change":"2024-03-21T13:13:18.440575+0000","last_active":"2024-03-21T13:14:36.503963+0000","last_peered":"2024-03-21T13:14:36.503963+0000","last_clean":"2024-03-21T13:14:36.503963+0000","last_became_active":"2024-03-21T13:12:29.510247+0000","last_became_peered":"2024-03-21T13:12:29.510247+0000","last_unstale":"2024-03-21T13:14:36.503963+0000","last_undegraded":"2024-03-21T13:14:36.503963+0000","last_fullsized":"2024-03-21T13:14:36.503963+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:13:18.440466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:18.440466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:38.918558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":212,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495998+0000","last_change":"2024-03-21T13:13:11.639568+0000","last_active":"2024-03-21T13:14:36.495998+0000","last_peered":"2024-03-21T13:14:36.495998+0000","last_clean":"2024-03-21T13:14:36.495998+0000","last_became_active":"2024-03-21T13:10:03.613841+0000","last_became_peered":"2024-03-21T13:10:03.613841+0000","last_unstale":"2024-03-21T13:14:36.495998+0000","last_undegraded":"2024-03-21T13:14:36.495998+0000","last_fullsized":"2024-03-21T13:14:36.495998+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:11.639450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:13:11.639450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:39.513137+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":644,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.476754+0000","last_change":"2024-03-21T13:11:26.697833+0000","last_active":"2024-03-21T13:14:31.476754+0000","last_peered":"2024-03-21T13:14:31.476754+0000","last_clean":"2024-03-21T13:14:31.476754+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:14:31.476754+0000","last_undegraded":"2024-03-21T13:14:31.476754+0000","last_fullsized":"2024-03-21T13:14:31.476754+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:26.697728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:26.697728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.154907+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":668,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495661+0000","last_change":"2024-03-21T13:14:12.561473+0000","last_active":"2024-03-21T13:14:36.495661+0000","last_peered":"2024-03-21T13:14:36.495661+0000","last_clean":"2024-03-21T13:14:36.495661+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:14:36.495661+0000","last_undegraded":"2024-03-21T13:14:36.495661+0000","last_fullsized":"2024-03-21T13:14:36.495661+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:14:12.561236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:12.561236+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:42.231028+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'15","reported_seq":589,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.475050+0000","last_change":"2024-03-21T13:14:15.193109+0000","last_active":"2024-03-21T13:14:31.475050+0000","last_peered":"2024-03-21T13:14:31.475050+0000","last_clean":"2024-03-21T13:14:31.475050+0000","last_became_active":"2024-03-21T13:01:48.210554+0000","last_became_peered":"2024-03-21T13:01:48.210554+0000","last_unstale":"2024-03-21T13:14:31.475050+0000","last_undegraded":"2024-03-21T13:14:31.475050+0000","last_fullsized":"2024-03-21T13:14:31.475050+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:14:15.192958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:15.192958+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:23.439586+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":642,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495770+0000","last_change":"2024-03-21T13:13:11.462718+0000","last_active":"2024-03-21T13:14:36.495770+0000","last_peered":"2024-03-21T13:14:36.495770+0000","last_clean":"2024-03-21T13:14:36.495770+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:14:36.495770+0000","last_undegraded":"2024-03-21T13:14:36.495770+0000","last_fullsized":"2024-03-21T13:14:36.495770+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:11.462567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:13:11.462567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":93,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.504390+0000","last_change":"2024-03-21T13:13:07.458319+0000","last_active":"2024-03-21T13:14:36.504390+0000","last_peered":"2024-03-21T13:14:36.504390+0000","last_clean":"2024-03-21T13:14:36.504390+0000","last_became_active":"2024-03-21T13:12:29.514113+0000","last_became_peered":"2024-03-21T13:12:29.514113+0000","last_unstale":"2024-03-21T13:14:36.504390+0000","last_undegraded":"2024-03-21T13:14:36.504390+0000","last_fullsized":"2024-03-21T13:14:36.504390+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:07.458199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:07.458199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127583+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":676,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496926+0000","last_change":"2024-03-21T13:14:14.647062+0000","last_active":"2024-03-21T13:14:36.496926+0000","last_peered":"2024-03-21T13:14:36.496926+0000","last_clean":"2024-03-21T13:14:36.496926+0000","last_became_active":"2024-03-21T13:12:29.520286+0000","last_became_peered":"2024-03-21T13:12:29.520286+0000","last_unstale":"2024-03-21T13:14:36.496926+0000","last_undegraded":"2024-03-21T13:14:36.496926+0000","last_fullsized":"2024-03-21T13:14:36.496926+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:14.646933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:14.646933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:16.998551+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":710,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.476624+0000","last_change":"2024-03-21T13:14:09.141688+0000","last_active":"2024-03-21T13:14:31.476624+0000","last_peered":"2024-03-21T13:14:31.476624+0000","last_clean":"2024-03-21T13:14:31.476624+0000","last_became_active":"2024-03-21T13:10:03.610810+0000","last_became_peered":"2024-03-21T13:10:03.610810+0000","last_unstale":"2024-03-21T13:14:31.476624+0000","last_undegraded":"2024-03-21T13:14:31.476624+0000","last_fullsized":"2024-03-21T13:14:31.476624+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-21T13:14:09.141520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:09.141520+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:21.501338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"368'11295","reported_seq":11381,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:39.824422+0000","last_change":"2024-03-21T13:12:29.526267+0000","last_active":"2024-03-21T13:14:39.824422+0000","last_peered":"2024-03-21T13:14:39.824422+0000","last_clean":"2024-03-21T13:14:39.824422+0000","last_became_active":"2024-03-21T13:12:29.525906+0000","last_became_peered":"2024-03-21T13:12:29.525906+0000","last_unstale":"2024-03-21T13:14:39.824422+0000","last_undegraded":"2024-03-21T13:14:39.824422+0000","last_fullsized":"2024-03-21T13:14:39.824422+0000","mapping_epoch":330,"log_start":"351'6000","ondisk_log_start":"351'6000","created":324,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":5295,"log_dups_size":0,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347013120,"num_objects":5295,"num_object_clones":0,"num_object_copies":15885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":86,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.503840+0000","last_change":"2024-03-21T13:12:29.518161+0000","last_active":"2024-03-21T13:14:36.503840+0000","last_peered":"2024-03-21T13:14:36.503840+0000","last_clean":"2024-03-21T13:14:36.503840+0000","last_became_active":"2024-03-21T13:12:29.517413+0000","last_became_peered":"2024-03-21T13:12:29.517413+0000","last_unstale":"2024-03-21T13:14:36.503840+0000","last_undegraded":"2024-03-21T13:14:36.503840+0000","last_fullsized":"2024-03-21T13:14:36.503840+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:02.758017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:11:02.758017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127124+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":691,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495672+0000","last_change":"2024-03-21T13:11:31.040564+0000","last_active":"2024-03-21T13:14:36.495672+0000","last_peered":"2024-03-21T13:14:36.495672+0000","last_clean":"2024-03-21T13:14:36.495672+0000","last_became_active":"2024-03-21T13:06:29.487439+0000","last_became_peered":"2024-03-21T13:06:29.487439+0000","last_unstale":"2024-03-21T13:14:36.495672+0000","last_undegraded":"2024-03-21T13:14:36.495672+0000","last_fullsized":"2024-03-21T13:14:36.495672+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:11:31.040414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:11:31.040414+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":110,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.502224+0000","last_change":"2024-03-21T13:14:00.557564+0000","last_active":"2024-03-21T13:14:36.502224+0000","last_peered":"2024-03-21T13:14:36.502224+0000","last_clean":"2024-03-21T13:14:36.502224+0000","last_became_active":"2024-03-21T13:12:29.513019+0000","last_became_peered":"2024-03-21T13:12:29.513019+0000","last_unstale":"2024-03-21T13:14:36.502224+0000","last_undegraded":"2024-03-21T13:14:36.502224+0000","last_fullsized":"2024-03-21T13:14:36.502224+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:14:00.557372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:00.557372+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:03.309924+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":852,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496611+0000","last_change":"2024-03-21T13:14:02.524497+0000","last_active":"2024-03-21T13:14:36.496611+0000","last_peered":"2024-03-21T13:14:36.496611+0000","last_clean":"2024-03-21T13:14:36.496611+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:14:36.496611+0000","last_undegraded":"2024-03-21T13:14:36.496611+0000","last_fullsized":"2024-03-21T13:14:36.496611+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:14:02.524111+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:14:02.524111+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:09.116528+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"368'11358","reported_seq":11463,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:40.273157+0000","last_change":"2024-03-21T13:12:29.522488+0000","last_active":"2024-03-21T13:14:40.273157+0000","last_peered":"2024-03-21T13:14:40.273157+0000","last_clean":"2024-03-21T13:14:40.273157+0000","last_became_active":"2024-03-21T13:12:29.521010+0000","last_became_peered":"2024-03-21T13:12:29.521010+0000","last_unstale":"2024-03-21T13:14:40.273157+0000","last_undegraded":"2024-03-21T13:14:40.273157+0000","last_fullsized":"2024-03-21T13:14:40.273157+0000","mapping_epoch":330,"log_start":"351'6500","ondisk_log_start":"351'6500","created":324,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349306880,"num_objects":5330,"num_object_clones":0,"num_object_copies":15990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11358,"num_write_kb":534016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":641,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.474903+0000","last_change":"2024-03-21T13:13:15.340376+0000","last_active":"2024-03-21T13:14:31.474903+0000","last_peered":"2024-03-21T13:14:31.474903+0000","last_clean":"2024-03-21T13:14:31.474903+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:14:31.474903+0000","last_undegraded":"2024-03-21T13:14:31.474903+0000","last_fullsized":"2024-03-21T13:14:31.474903+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:15.340248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:13:15.340248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.155699+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":669,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496251+0000","last_change":"2024-03-21T13:14:10.542906+0000","last_active":"2024-03-21T13:14:36.496251+0000","last_peered":"2024-03-21T13:14:36.496251+0000","last_clean":"2024-03-21T13:14:36.496251+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:14:36.496251+0000","last_undegraded":"2024-03-21T13:14:36.496251+0000","last_fullsized":"2024-03-21T13:14:36.496251+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:10.542755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:10.542755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:34.356866+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"346'23","reported_seq":63,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495313+0000","last_change":"2024-03-21T13:13:21.852476+0000","last_active":"2024-03-21T13:14:36.495313+0000","last_peered":"2024-03-21T13:14:36.495313+0000","last_clean":"2024-03-21T13:14:36.495313+0000","last_became_active":"2024-03-21T13:13:21.819111+0000","last_became_peered":"2024-03-21T13:13:21.819111+0000","last_unstale":"2024-03-21T13:14:36.495313+0000","last_undegraded":"2024-03-21T13:14:36.495313+0000","last_fullsized":"2024-03-21T13:14:36.495313+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'23","last_scrub_stamp":"2024-03-21T13:13:20.232807+0000","last_deep_scrub":"346'23","last_deep_scrub_stamp":"2024-03-21T13:13:20.232807+0000","last_clean_scrub_stamp":"2024-03-21T13:13:20.232807+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":219,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495690+0000","last_change":"2024-03-21T13:13:12.686137+0000","last_active":"2024-03-21T13:14:36.495690+0000","last_peered":"2024-03-21T13:14:36.495690+0000","last_clean":"2024-03-21T13:14:36.495690+0000","last_became_active":"2024-03-21T13:12:29.499921+0000","last_became_peered":"2024-03-21T13:12:29.499921+0000","last_unstale":"2024-03-21T13:14:36.495690+0000","last_undegraded":"2024-03-21T13:14:36.495690+0000","last_fullsized":"2024-03-21T13:14:36.495690+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:12.685999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:12.685999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"367'10914","reported_seq":11011,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.056665+0000","last_change":"2024-03-21T13:12:19.520750+0000","last_active":"2024-03-21T13:14:36.056665+0000","last_peered":"2024-03-21T13:14:36.056665+0000","last_clean":"2024-03-21T13:14:36.056665+0000","last_became_active":"2024-03-21T13:12:19.519902+0000","last_became_peered":"2024-03-21T13:12:19.519902+0000","last_unstale":"2024-03-21T13:14:36.056665+0000","last_undegraded":"2024-03-21T13:14:36.056665+0000","last_fullsized":"2024-03-21T13:14:36.056665+0000","mapping_epoch":324,"log_start":"351'6300","ondisk_log_start":"351'6300","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4614,"log_dups_size":0,"ondisk_log_size":4614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365166592,"num_objects":5572,"num_object_clones":0,"num_object_copies":16716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10914,"num_write_kb":527552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":663,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495104+0000","last_change":"2024-03-21T13:13:23.379786+0000","last_active":"2024-03-21T13:14:36.495104+0000","last_peered":"2024-03-21T13:14:36.495104+0000","last_clean":"2024-03-21T13:14:36.495104+0000","last_became_active":"2024-03-21T13:10:03.598366+0000","last_became_peered":"2024-03-21T13:10:03.598366+0000","last_unstale":"2024-03-21T13:14:36.495104+0000","last_undegraded":"2024-03-21T13:14:36.495104+0000","last_fullsized":"2024-03-21T13:14:36.495104+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:23.379475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:13:23.379475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:39.026216+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":227,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496223+0000","last_change":"2024-03-21T13:13:21.644266+0000","last_active":"2024-03-21T13:14:36.496223+0000","last_peered":"2024-03-21T13:14:36.496223+0000","last_clean":"2024-03-21T13:14:36.496223+0000","last_became_active":"2024-03-21T13:12:29.500199+0000","last_became_peered":"2024-03-21T13:12:29.500199+0000","last_unstale":"2024-03-21T13:14:36.496223+0000","last_undegraded":"2024-03-21T13:14:36.496223+0000","last_fullsized":"2024-03-21T13:14:36.496223+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:21.644151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:21.644151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":675,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496005+0000","last_change":"2024-03-21T13:14:19.700448+0000","last_active":"2024-03-21T13:14:36.496005+0000","last_peered":"2024-03-21T13:14:36.496005+0000","last_clean":"2024-03-21T13:14:36.496005+0000","last_became_active":"2024-03-21T13:12:29.508984+0000","last_became_peered":"2024-03-21T13:12:29.508984+0000","last_unstale":"2024-03-21T13:14:36.496005+0000","last_undegraded":"2024-03-21T13:14:36.496005+0000","last_fullsized":"2024-03-21T13:14:36.496005+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:19.700320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:19.700320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:30.698005+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'35","reported_seq":794,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.497727+0000","last_change":"2024-03-21T13:13:24.732800+0000","last_active":"2024-03-21T13:14:36.497727+0000","last_peered":"2024-03-21T13:14:36.497727+0000","last_clean":"2024-03-21T13:14:36.497727+0000","last_became_active":"2024-03-21T13:12:29.523359+0000","last_became_peered":"2024-03-21T13:12:29.523359+0000","last_unstale":"2024-03-21T13:14:36.497727+0000","last_undegraded":"2024-03-21T13:14:36.497727+0000","last_fullsized":"2024-03-21T13:14:36.497727+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:13:24.732562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:24.732562+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:50.994968+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"368'11162","reported_seq":11326,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:40.277954+0000","last_change":"2024-03-21T13:13:27.478302+0000","last_active":"2024-03-21T13:14:40.277954+0000","last_peered":"2024-03-21T13:14:40.277954+0000","last_clean":"2024-03-21T13:14:40.277954+0000","last_became_active":"2024-03-21T13:12:19.519440+0000","last_became_peered":"2024-03-21T13:12:19.519440+0000","last_unstale":"2024-03-21T13:14:40.277954+0000","last_undegraded":"2024-03-21T13:14:40.277954+0000","last_fullsized":"2024-03-21T13:14:40.277954+0000","mapping_epoch":324,"log_start":"348'5600","ondisk_log_start":"348'5600","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'5750","last_scrub_stamp":"2024-03-21T13:13:27.478094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:13:27.478094+0000","objects_scrubbed":5677,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:45.751471+0000","scrub_duration":1868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341049344,"num_objects":5204,"num_object_clones":0,"num_object_copies":15612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11162,"num_write_kb":523712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":559,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495361+0000","last_change":"2024-03-21T13:11:38.166199+0000","last_active":"2024-03-21T13:14:36.495361+0000","last_peered":"2024-03-21T13:14:36.495361+0000","last_clean":"2024-03-21T13:14:36.495361+0000","last_became_active":"2024-03-21T13:10:03.606655+0000","last_became_peered":"2024-03-21T13:10:03.606655+0000","last_unstale":"2024-03-21T13:14:36.495361+0000","last_undegraded":"2024-03-21T13:14:36.495361+0000","last_fullsized":"2024-03-21T13:14:36.495361+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:38.166077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:38.166077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":213,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496539+0000","last_change":"2024-03-21T13:13:06.624315+0000","last_active":"2024-03-21T13:14:36.496539+0000","last_peered":"2024-03-21T13:14:36.496539+0000","last_clean":"2024-03-21T13:14:36.496539+0000","last_became_active":"2024-03-21T13:10:03.603646+0000","last_became_peered":"2024-03-21T13:10:03.603646+0000","last_unstale":"2024-03-21T13:14:36.496539+0000","last_undegraded":"2024-03-21T13:14:36.496539+0000","last_fullsized":"2024-03-21T13:14:36.496539+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:06.624233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:06.624233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":93,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.503440+0000","last_change":"2024-03-21T13:14:15.386244+0000","last_active":"2024-03-21T13:14:36.503440+0000","last_peered":"2024-03-21T13:14:36.503440+0000","last_clean":"2024-03-21T13:14:36.503440+0000","last_became_active":"2024-03-21T13:12:29.512741+0000","last_became_peered":"2024-03-21T13:12:29.512741+0000","last_unstale":"2024-03-21T13:14:36.503440+0000","last_undegraded":"2024-03-21T13:14:36.503440+0000","last_fullsized":"2024-03-21T13:14:36.503440+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:15.386132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:15.386132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:21.391493+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"234'19","reported_seq":644,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.494000+0000","last_change":"2024-03-21T13:14:08.259829+0000","last_active":"2024-03-21T13:14:36.494000+0000","last_peered":"2024-03-21T13:14:36.494000+0000","last_clean":"2024-03-21T13:14:36.494000+0000","last_became_active":"2024-03-21T13:02:14.333996+0000","last_became_peered":"2024-03-21T13:02:14.333996+0000","last_unstale":"2024-03-21T13:14:36.494000+0000","last_undegraded":"2024-03-21T13:14:36.494000+0000","last_fullsized":"2024-03-21T13:14:36.494000+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:14:08.259691+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:14:08.259691+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:25.539719+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.a","version":"368'11091","reported_seq":11196,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:40.292406+0000","last_change":"2024-03-21T13:12:29.521470+0000","last_active":"2024-03-21T13:14:40.292406+0000","last_peered":"2024-03-21T13:14:40.292406+0000","last_clean":"2024-03-21T13:14:40.292406+0000","last_became_active":"2024-03-21T13:12:29.520693+0000","last_became_peered":"2024-03-21T13:12:29.520693+0000","last_unstale":"2024-03-21T13:14:40.292406+0000","last_undegraded":"2024-03-21T13:14:40.292406+0000","last_fullsized":"2024-03-21T13:14:40.292406+0000","mapping_epoch":330,"log_start":"351'6300","ondisk_log_start":"351'6300","created":324,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4791,"log_dups_size":0,"ondisk_log_size":4791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345571328,"num_objects":5273,"num_object_clones":0,"num_object_copies":15819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11091,"num_write_kb":523648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":635,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.475374+0000","last_change":"2024-03-21T13:14:14.199570+0000","last_active":"2024-03-21T13:14:31.475374+0000","last_peered":"2024-03-21T13:14:31.475374+0000","last_clean":"2024-03-21T13:14:31.475374+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:14:31.475374+0000","last_undegraded":"2024-03-21T13:14:31.475374+0000","last_fullsized":"2024-03-21T13:14:31.475374+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:14.199450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:14:14.199450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:34.853946+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":90,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.503512+0000","last_change":"2024-03-21T13:12:29.554139+0000","last_active":"2024-03-21T13:14:36.503512+0000","last_peered":"2024-03-21T13:14:36.503512+0000","last_clean":"2024-03-21T13:14:36.503512+0000","last_became_active":"2024-03-21T13:12:29.521275+0000","last_became_peered":"2024-03-21T13:12:29.521275+0000","last_unstale":"2024-03-21T13:14:36.503512+0000","last_undegraded":"2024-03-21T13:14:36.503512+0000","last_fullsized":"2024-03-21T13:14:36.503512+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:12:16.224897+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:12:16.224897+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":367,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.497194+0000","last_change":"2024-03-21T13:14:01.668714+0000","last_active":"2024-03-21T13:14:36.497194+0000","last_peered":"2024-03-21T13:14:36.497194+0000","last_clean":"2024-03-21T13:14:36.497194+0000","last_became_active":"2024-03-21T13:06:45.410017+0000","last_became_peered":"2024-03-21T13:06:45.410017+0000","last_unstale":"2024-03-21T13:14:36.497194+0000","last_undegraded":"2024-03-21T13:14:36.497194+0000","last_fullsized":"2024-03-21T13:14:36.497194+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:01.668582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:01.668582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:02.323590+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"272'43","reported_seq":299,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495898+0000","last_change":"2024-03-21T13:13:20.682786+0000","last_active":"2024-03-21T13:14:36.495898+0000","last_peered":"2024-03-21T13:14:36.495898+0000","last_clean":"2024-03-21T13:14:36.495898+0000","last_became_active":"2024-03-21T13:10:03.603889+0000","last_became_peered":"2024-03-21T13:10:03.603889+0000","last_unstale":"2024-03-21T13:14:36.495898+0000","last_undegraded":"2024-03-21T13:14:36.495898+0000","last_fullsized":"2024-03-21T13:14:36.495898+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'43","last_scrub_stamp":"2024-03-21T13:13:20.682623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:20.682623+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:43.173396+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":371,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.498557+0000","last_change":"2024-03-21T13:13:15.969147+0000","last_active":"2024-03-21T13:14:36.498557+0000","last_peered":"2024-03-21T13:14:36.498557+0000","last_clean":"2024-03-21T13:14:36.498557+0000","last_became_active":"2024-03-21T13:06:45.406734+0000","last_became_peered":"2024-03-21T13:06:45.406734+0000","last_unstale":"2024-03-21T13:14:36.498557+0000","last_undegraded":"2024-03-21T13:14:36.498557+0000","last_fullsized":"2024-03-21T13:14:36.498557+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:15.968926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:15.968926+0000","last_clean_scrub_stamp":"2024-03-21T13:13:15.968926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:37.807311+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"368'11116","reported_seq":11203,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:39.821892+0000","last_change":"2024-03-21T13:12:29.509025+0000","last_active":"2024-03-21T13:14:39.821892+0000","last_peered":"2024-03-21T13:14:39.821892+0000","last_clean":"2024-03-21T13:14:39.821892+0000","last_became_active":"2024-03-21T13:12:29.508658+0000","last_became_peered":"2024-03-21T13:12:29.508658+0000","last_unstale":"2024-03-21T13:14:39.821892+0000","last_undegraded":"2024-03-21T13:14:39.821892+0000","last_fullsized":"2024-03-21T13:14:39.821892+0000","mapping_epoch":330,"log_start":"350'5800","ondisk_log_start":"350'5800","created":324,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347471872,"num_objects":5302,"num_object_clones":0,"num_object_copies":15906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11116,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":651,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496155+0000","last_change":"2024-03-21T13:14:06.644063+0000","last_active":"2024-03-21T13:14:36.496155+0000","last_peered":"2024-03-21T13:14:36.496155+0000","last_clean":"2024-03-21T13:14:36.496155+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:14:36.496155+0000","last_undegraded":"2024-03-21T13:14:36.496155+0000","last_fullsized":"2024-03-21T13:14:36.496155+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:06.643911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:06.643911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:29.513060+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":213,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495789+0000","last_change":"2024-03-21T13:13:09.639250+0000","last_active":"2024-03-21T13:14:36.495789+0000","last_peered":"2024-03-21T13:14:36.495789+0000","last_clean":"2024-03-21T13:14:36.495789+0000","last_became_active":"2024-03-21T13:10:03.604303+0000","last_became_peered":"2024-03-21T13:10:03.604303+0000","last_unstale":"2024-03-21T13:14:36.495789+0000","last_undegraded":"2024-03-21T13:14:36.495789+0000","last_fullsized":"2024-03-21T13:14:36.495789+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:09.636696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:13:09.636696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:36.946207+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":213,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495898+0000","last_change":"2024-03-21T13:12:27.851986+0000","last_active":"2024-03-21T13:14:36.495898+0000","last_peered":"2024-03-21T13:14:36.495898+0000","last_clean":"2024-03-21T13:14:36.495898+0000","last_became_active":"2024-03-21T13:10:03.604427+0000","last_became_peered":"2024-03-21T13:10:03.604427+0000","last_unstale":"2024-03-21T13:14:36.495898+0000","last_undegraded":"2024-03-21T13:14:36.495898+0000","last_fullsized":"2024-03-21T13:14:36.495898+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:27.851875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:12:27.851875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"243'20","reported_seq":98,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.506528+0000","last_change":"2024-03-21T13:12:29.544839+0000","last_active":"2024-03-21T13:14:36.506528+0000","last_peered":"2024-03-21T13:14:36.506528+0000","last_clean":"2024-03-21T13:14:36.506528+0000","last_became_active":"2024-03-21T13:12:29.518048+0000","last_became_peered":"2024-03-21T13:12:29.518048+0000","last_unstale":"2024-03-21T13:14:36.506528+0000","last_undegraded":"2024-03-21T13:14:36.506528+0000","last_fullsized":"2024-03-21T13:14:36.506528+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'20","last_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_deep_scrub":"243'20","last_deep_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_clean_scrub_stamp":"2024-03-21T13:09:10.683382+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127045+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":219,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.497579+0000","last_change":"2024-03-21T13:13:19.683549+0000","last_active":"2024-03-21T13:14:36.497579+0000","last_peered":"2024-03-21T13:14:36.497579+0000","last_clean":"2024-03-21T13:14:36.497579+0000","last_became_active":"2024-03-21T13:12:29.506751+0000","last_became_peered":"2024-03-21T13:12:29.506751+0000","last_unstale":"2024-03-21T13:14:36.497579+0000","last_undegraded":"2024-03-21T13:14:36.497579+0000","last_fullsized":"2024-03-21T13:14:36.497579+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:19.683292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:19.683292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:42.374427+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"368'11172","reported_seq":11271,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:40.289223+0000","last_change":"2024-03-21T13:12:19.517136+0000","last_active":"2024-03-21T13:14:40.289223+0000","last_peered":"2024-03-21T13:14:40.289223+0000","last_clean":"2024-03-21T13:14:40.289223+0000","last_became_active":"2024-03-21T13:12:19.516747+0000","last_became_peered":"2024-03-21T13:12:19.516747+0000","last_unstale":"2024-03-21T13:14:40.289223+0000","last_undegraded":"2024-03-21T13:14:40.289223+0000","last_fullsized":"2024-03-21T13:14:40.289223+0000","mapping_epoch":324,"log_start":"351'6400","ondisk_log_start":"351'6400","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343146496,"num_objects":5236,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11172,"num_write_kb":525056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":86,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.506333+0000","last_change":"2024-03-21T13:12:29.529126+0000","last_active":"2024-03-21T13:14:36.506333+0000","last_peered":"2024-03-21T13:14:36.506333+0000","last_clean":"2024-03-21T13:14:36.506333+0000","last_became_active":"2024-03-21T13:12:29.527083+0000","last_became_peered":"2024-03-21T13:12:29.527083+0000","last_unstale":"2024-03-21T13:14:36.506333+0000","last_undegraded":"2024-03-21T13:14:36.506333+0000","last_fullsized":"2024-03-21T13:14:36.506333+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:57.956080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:09:57.956080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127100+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":914,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495323+0000","last_change":"2024-03-21T13:14:01.491401+0000","last_active":"2024-03-21T13:14:36.495323+0000","last_peered":"2024-03-21T13:14:36.495323+0000","last_clean":"2024-03-21T13:14:36.495323+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:14:36.495323+0000","last_undegraded":"2024-03-21T13:14:36.495323+0000","last_fullsized":"2024-03-21T13:14:36.495323+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:14:01.491237+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:14:01.491237+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:09.775211+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":158,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.495372+0000","last_change":"2024-03-21T13:13:02.645250+0000","last_active":"2024-03-21T13:14:36.495372+0000","last_peered":"2024-03-21T13:14:36.495372+0000","last_clean":"2024-03-21T13:14:36.495372+0000","last_became_active":"2024-03-21T13:12:29.499989+0000","last_became_peered":"2024-03-21T13:12:29.499989+0000","last_unstale":"2024-03-21T13:14:36.495372+0000","last_undegraded":"2024-03-21T13:14:36.495372+0000","last_fullsized":"2024-03-21T13:14:36.495372+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:02.645139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:13:02.645139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:39.513109+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"234'36","reported_seq":894,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.498710+0000","last_change":"2024-03-21T13:12:26.260721+0000","last_active":"2024-03-21T13:14:36.498710+0000","last_peered":"2024-03-21T13:14:36.498710+0000","last_clean":"2024-03-21T13:14:36.498710+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:14:36.498710+0000","last_undegraded":"2024-03-21T13:14:36.498710+0000","last_fullsized":"2024-03-21T13:14:36.498710+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'36","last_scrub_stamp":"2024-03-21T13:12:26.260559+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:12:26.260559+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":671,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.499892+0000","last_change":"2024-03-21T13:12:29.508145+0000","last_active":"2024-03-21T13:14:36.499892+0000","last_peered":"2024-03-21T13:14:36.499892+0000","last_clean":"2024-03-21T13:14:36.499892+0000","last_became_active":"2024-03-21T13:12:29.507753+0000","last_became_peered":"2024-03-21T13:12:29.507753+0000","last_unstale":"2024-03-21T13:14:36.499892+0000","last_undegraded":"2024-03-21T13:14:36.499892+0000","last_fullsized":"2024-03-21T13:14:36.499892+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:11.001239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:11:11.001239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"368'11204","reported_seq":11384,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:40.289864+0000","last_change":"2024-03-21T13:14:15.730969+0000","last_active":"2024-03-21T13:14:40.289864+0000","last_peered":"2024-03-21T13:14:40.289864+0000","last_clean":"2024-03-21T13:14:40.289864+0000","last_became_active":"2024-03-21T13:12:19.518131+0000","last_became_peered":"2024-03-21T13:12:19.518131+0000","last_unstale":"2024-03-21T13:14:40.289864+0000","last_undegraded":"2024-03-21T13:14:40.289864+0000","last_fullsized":"2024-03-21T13:14:40.289864+0000","mapping_epoch":324,"log_start":"348'5600","ondisk_log_start":"348'5600","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'9234","last_scrub_stamp":"2024-03-21T13:14:15.730731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:14:15.730731+0000","objects_scrubbed":7276,"log_size":5604,"log_dups_size":0,"ondisk_log_size":5604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:16.733134+0000","scrub_duration":2341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342753280,"num_objects":5230,"num_object_clones":0,"num_object_copies":15690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11204,"num_write_kb":525888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":86,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.503616+0000","last_change":"2024-03-21T13:12:29.515061+0000","last_active":"2024-03-21T13:14:36.503616+0000","last_peered":"2024-03-21T13:14:36.503616+0000","last_clean":"2024-03-21T13:14:36.503616+0000","last_became_active":"2024-03-21T13:12:29.514542+0000","last_became_peered":"2024-03-21T13:12:29.514542+0000","last_unstale":"2024-03-21T13:14:36.503616+0000","last_undegraded":"2024-03-21T13:14:36.503616+0000","last_fullsized":"2024-03-21T13:14:36.503616+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:36.890785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:11:36.890785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127292+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":572,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.493026+0000","last_change":"2024-03-21T13:13:24.068810+0000","last_active":"2024-03-21T13:14:36.493026+0000","last_peered":"2024-03-21T13:14:36.493026+0000","last_clean":"2024-03-21T13:14:36.493026+0000","last_became_active":"2024-03-21T13:01:48.211645+0000","last_became_peered":"2024-03-21T13:01:48.211645+0000","last_unstale":"2024-03-21T13:14:36.493026+0000","last_undegraded":"2024-03-21T13:14:36.493026+0000","last_fullsized":"2024-03-21T13:14:36.493026+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:24.068699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:13:24.068699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:39.230511+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":1630,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:39.955911+0000","last_change":"2024-03-21T13:13:24.417605+0000","last_active":"2024-03-21T13:14:39.955911+0000","last_peered":"2024-03-21T13:14:39.955911+0000","last_clean":"2024-03-21T13:14:39.955911+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:14:39.955911+0000","last_undegraded":"2024-03-21T13:14:39.955911+0000","last_fullsized":"2024-03-21T13:14:39.955911+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:13:24.417338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:13:24.417338+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:47.228515+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":952,"num_read_kb":947,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":788,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.502636+0000","last_change":"2024-03-21T13:13:59.662999+0000","last_active":"2024-03-21T13:14:36.502636+0000","last_peered":"2024-03-21T13:14:36.502636+0000","last_clean":"2024-03-21T13:14:36.502636+0000","last_became_active":"2024-03-21T13:10:03.606672+0000","last_became_peered":"2024-03-21T13:10:03.606672+0000","last_unstale":"2024-03-21T13:14:36.502636+0000","last_undegraded":"2024-03-21T13:14:36.502636+0000","last_fullsized":"2024-03-21T13:14:36.502636+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-21T13:13:59.662850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:59.662850+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:05.428216+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":94,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.506239+0000","last_change":"2024-03-21T13:14:07.406567+0000","last_active":"2024-03-21T13:14:36.506239+0000","last_peered":"2024-03-21T13:14:36.506239+0000","last_clean":"2024-03-21T13:14:36.506239+0000","last_became_active":"2024-03-21T13:12:29.526100+0000","last_became_peered":"2024-03-21T13:12:29.526100+0000","last_unstale":"2024-03-21T13:14:36.506239+0000","last_undegraded":"2024-03-21T13:14:36.506239+0000","last_fullsized":"2024-03-21T13:14:36.506239+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:07.406434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:14:07.406434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:32.235053+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"368'11247","reported_seq":11424,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:40.293340+0000","last_change":"2024-03-21T13:14:22.702712+0000","last_active":"2024-03-21T13:14:40.293340+0000","last_peered":"2024-03-21T13:14:40.293340+0000","last_clean":"2024-03-21T13:14:40.293340+0000","last_became_active":"2024-03-21T13:12:19.518554+0000","last_became_peered":"2024-03-21T13:12:19.518554+0000","last_unstale":"2024-03-21T13:14:40.293340+0000","last_undegraded":"2024-03-21T13:14:40.293340+0000","last_fullsized":"2024-03-21T13:14:40.293340+0000","mapping_epoch":324,"log_start":"351'6400","ondisk_log_start":"351'6400","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'9860","last_scrub_stamp":"2024-03-21T13:14:22.702574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:14:22.702574+0000","objects_scrubbed":6806,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:27.883071+0000","scrub_duration":2058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350617624,"num_objects":5351,"num_object_clones":0,"num_object_copies":16053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5351,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11247,"num_write_kb":531073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":553,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.497552+0000","last_change":"2024-03-21T13:13:15.013884+0000","last_active":"2024-03-21T13:14:36.497552+0000","last_peered":"2024-03-21T13:14:36.497552+0000","last_clean":"2024-03-21T13:14:36.497552+0000","last_became_active":"2024-03-21T13:10:03.591088+0000","last_became_peered":"2024-03-21T13:10:03.591088+0000","last_unstale":"2024-03-21T13:14:36.497552+0000","last_undegraded":"2024-03-21T13:14:36.497552+0000","last_fullsized":"2024-03-21T13:14:36.497552+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:15.013717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:13:15.013717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:37.807336+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":219,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496100+0000","last_change":"2024-03-21T13:13:08.629175+0000","last_active":"2024-03-21T13:14:36.496100+0000","last_peered":"2024-03-21T13:14:36.496100+0000","last_clean":"2024-03-21T13:14:36.496100+0000","last_became_active":"2024-03-21T13:12:29.507783+0000","last_became_peered":"2024-03-21T13:12:29.507783+0000","last_unstale":"2024-03-21T13:14:36.496100+0000","last_undegraded":"2024-03-21T13:14:36.496100+0000","last_fullsized":"2024-03-21T13:14:36.496100+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:08.628946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:13:08.628946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":112,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.503378+0000","last_change":"2024-03-21T13:14:16.366110+0000","last_active":"2024-03-21T13:14:36.503378+0000","last_peered":"2024-03-21T13:14:36.503378+0000","last_clean":"2024-03-21T13:14:36.503378+0000","last_became_active":"2024-03-21T13:12:29.518800+0000","last_became_peered":"2024-03-21T13:12:29.518800+0000","last_unstale":"2024-03-21T13:14:36.503378+0000","last_undegraded":"2024-03-21T13:14:36.503378+0000","last_fullsized":"2024-03-21T13:14:36.503378+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-21T13:14:16.365267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:16.365267+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:27.792685+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":681,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.497825+0000","last_change":"2024-03-21T13:13:21.836944+0000","last_active":"2024-03-21T13:14:36.497825+0000","last_peered":"2024-03-21T13:14:36.497825+0000","last_clean":"2024-03-21T13:14:36.497825+0000","last_became_active":"2024-03-21T13:12:29.520657+0000","last_became_peered":"2024-03-21T13:12:29.520657+0000","last_unstale":"2024-03-21T13:14:36.497825+0000","last_undegraded":"2024-03-21T13:14:36.497825+0000","last_fullsized":"2024-03-21T13:14:36.497825+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:21.836810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:21.836810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"368'11124","reported_seq":11223,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:40.293005+0000","last_change":"2024-03-21T13:12:19.519139+0000","last_active":"2024-03-21T13:14:40.293005+0000","last_peered":"2024-03-21T13:14:40.293005+0000","last_clean":"2024-03-21T13:14:40.293005+0000","last_became_active":"2024-03-21T13:12:19.518797+0000","last_became_peered":"2024-03-21T13:12:19.518797+0000","last_unstale":"2024-03-21T13:14:40.293005+0000","last_undegraded":"2024-03-21T13:14:40.293005+0000","last_fullsized":"2024-03-21T13:14:40.293005+0000","mapping_epoch":324,"log_start":"351'6500","ondisk_log_start":"351'6500","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4624,"log_dups_size":0,"ondisk_log_size":4624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:41.531559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338165760,"num_objects":5160,"num_object_clones":0,"num_object_copies":15480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11124,"num_write_kb":521088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":558,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.469970+0000","last_change":"2024-03-21T13:14:01.163827+0000","last_active":"2024-03-21T13:14:31.469970+0000","last_peered":"2024-03-21T13:14:31.469970+0000","last_clean":"2024-03-21T13:14:31.469970+0000","last_became_active":"2024-03-21T13:10:03.595454+0000","last_became_peered":"2024-03-21T13:10:03.595454+0000","last_unstale":"2024-03-21T13:14:31.469970+0000","last_undegraded":"2024-03-21T13:14:31.469970+0000","last_fullsized":"2024-03-21T13:14:31.469970+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:01.163722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:01.163722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:15.245067+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":86,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.504055+0000","last_change":"2024-03-21T13:12:29.514972+0000","last_active":"2024-03-21T13:14:36.504055+0000","last_peered":"2024-03-21T13:14:36.504055+0000","last_clean":"2024-03-21T13:14:36.504055+0000","last_became_active":"2024-03-21T13:12:29.512291+0000","last_became_peered":"2024-03-21T13:12:29.512291+0000","last_unstale":"2024-03-21T13:14:36.504055+0000","last_undegraded":"2024-03-21T13:14:36.504055+0000","last_fullsized":"2024-03-21T13:14:36.504055+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:28.030369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:28.030369+0000","last_clean_scrub_stamp":"2024-03-21T13:12:28.030369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127558+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"234'25","reported_seq":98,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.503699+0000","last_change":"2024-03-21T13:12:29.567830+0000","last_active":"2024-03-21T13:14:36.503699+0000","last_peered":"2024-03-21T13:14:36.503699+0000","last_clean":"2024-03-21T13:14:36.503699+0000","last_became_active":"2024-03-21T13:12:29.526114+0000","last_became_peered":"2024-03-21T13:12:29.526114+0000","last_unstale":"2024-03-21T13:14:36.503699+0000","last_undegraded":"2024-03-21T13:14:36.503699+0000","last_fullsized":"2024-03-21T13:14:36.503699+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-21T13:11:37.749901+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:11:37.749901+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127262+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"331'13","reported_seq":759,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.497431+0000","last_change":"2024-03-21T13:13:09.998270+0000","last_active":"2024-03-21T13:14:36.497431+0000","last_peered":"2024-03-21T13:14:36.497431+0000","last_clean":"2024-03-21T13:14:36.497431+0000","last_became_active":"2024-03-21T13:12:29.521017+0000","last_became_peered":"2024-03-21T13:12:29.521017+0000","last_unstale":"2024-03-21T13:14:36.497431+0000","last_undegraded":"2024-03-21T13:14:36.497431+0000","last_fullsized":"2024-03-21T13:14:36.497431+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:13:09.997984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:09.997984+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:39.733099+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.f","version":"368'11134","reported_seq":11321,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:40.273372+0000","last_change":"2024-03-21T13:14:01.880864+0000","last_active":"2024-03-21T13:14:40.273372+0000","last_peered":"2024-03-21T13:14:40.273372+0000","last_clean":"2024-03-21T13:14:40.273372+0000","last_became_active":"2024-03-21T13:12:19.468502+0000","last_became_peered":"2024-03-21T13:12:19.468502+0000","last_unstale":"2024-03-21T13:14:40.273372+0000","last_undegraded":"2024-03-21T13:14:40.273372+0000","last_fullsized":"2024-03-21T13:14:40.273372+0000","mapping_epoch":324,"log_start":"351'6300","ondisk_log_start":"351'6300","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'8144","last_scrub_stamp":"2024-03-21T13:14:01.880514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:14:01.880514+0000","objects_scrubbed":8036,"log_size":4834,"log_dups_size":0,"ondisk_log_size":4834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:18.207182+0000","scrub_duration":3250,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343408640,"num_objects":5240,"num_object_clones":0,"num_object_copies":15720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11134,"num_write_kb":523968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":655,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.496439+0000","last_change":"2024-03-21T13:12:19.261227+0000","last_active":"2024-03-21T13:14:36.496439+0000","last_peered":"2024-03-21T13:14:36.496439+0000","last_clean":"2024-03-21T13:14:36.496439+0000","last_became_active":"2024-03-21T13:10:03.606557+0000","last_became_peered":"2024-03-21T13:10:03.606557+0000","last_unstale":"2024-03-21T13:14:36.496439+0000","last_undegraded":"2024-03-21T13:14:36.496439+0000","last_fullsized":"2024-03-21T13:14:36.496439+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:19.261040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:12:19.261040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":715,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.502855+0000","last_change":"2024-03-21T13:14:21.647752+0000","last_active":"2024-03-21T13:14:36.502855+0000","last_peered":"2024-03-21T13:14:36.502855+0000","last_clean":"2024-03-21T13:14:36.502855+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:14:36.502855+0000","last_undegraded":"2024-03-21T13:14:36.502855+0000","last_fullsized":"2024-03-21T13:14:36.502855+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:14:21.647601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:21.647601+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:32.939731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'35","reported_seq":110,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.502046+0000","last_change":"2024-03-21T13:14:12.399995+0000","last_active":"2024-03-21T13:14:36.502046+0000","last_peered":"2024-03-21T13:14:36.502046+0000","last_clean":"2024-03-21T13:14:36.502046+0000","last_became_active":"2024-03-21T13:12:29.525154+0000","last_became_peered":"2024-03-21T13:12:29.525154+0000","last_unstale":"2024-03-21T13:14:36.502046+0000","last_undegraded":"2024-03-21T13:14:36.502046+0000","last_fullsized":"2024-03-21T13:14:36.502046+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:14:12.399785+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:14:12.399785+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:22.967195+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"226'10","reported_seq":851,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.470574+0000","last_change":"2024-03-21T13:13:27.063138+0000","last_active":"2024-03-21T13:14:31.470574+0000","last_peered":"2024-03-21T13:14:31.470574+0000","last_clean":"2024-03-21T13:14:31.470574+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:14:31.470574+0000","last_undegraded":"2024-03-21T13:14:31.470574+0000","last_fullsized":"2024-03-21T13:14:31.470574+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T13:13:27.062991+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:13:27.062991+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:43.639228+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"368'11541","reported_seq":11721,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:40.276808+0000","last_change":"2024-03-21T13:14:20.020388+0000","last_active":"2024-03-21T13:14:40.276808+0000","last_peered":"2024-03-21T13:14:40.276808+0000","last_clean":"2024-03-21T13:14:40.276808+0000","last_became_active":"2024-03-21T13:12:19.468361+0000","last_became_peered":"2024-03-21T13:12:19.468361+0000","last_unstale":"2024-03-21T13:14:40.276808+0000","last_undegraded":"2024-03-21T13:14:40.276808+0000","last_fullsized":"2024-03-21T13:14:40.276808+0000","mapping_epoch":324,"log_start":"351'6700","ondisk_log_start":"351'6700","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'9793","last_scrub_stamp":"2024-03-21T13:14:20.020240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:14:20.020240+0000","objects_scrubbed":7101,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:31.881440+0000","scrub_duration":2344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344915968,"num_objects":5263,"num_object_clones":0,"num_object_copies":15789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11541,"num_write_kb":537728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":572,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.475727+0000","last_change":"2024-03-21T13:12:28.983993+0000","last_active":"2024-03-21T13:14:31.475727+0000","last_peered":"2024-03-21T13:14:31.475727+0000","last_clean":"2024-03-21T13:14:31.475727+0000","last_became_active":"2024-03-21T13:10:03.610471+0000","last_became_peered":"2024-03-21T13:10:03.610471+0000","last_unstale":"2024-03-21T13:14:31.475727+0000","last_undegraded":"2024-03-21T13:14:31.475727+0000","last_fullsized":"2024-03-21T13:14:31.475727+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:28.983915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:12:28.983915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.155544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":655,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.476455+0000","last_change":"2024-03-21T13:14:18.185914+0000","last_active":"2024-03-21T13:14:31.476455+0000","last_peered":"2024-03-21T13:14:31.476455+0000","last_clean":"2024-03-21T13:14:31.476455+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:14:31.476455+0000","last_undegraded":"2024-03-21T13:14:31.476455+0000","last_fullsized":"2024-03-21T13:14:31.476455+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:18.185782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:14:18.185782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:30.895236+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"234'30","reported_seq":693,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.470423+0000","last_change":"2024-03-21T13:12:26.004699+0000","last_active":"2024-03-21T13:14:31.470423+0000","last_peered":"2024-03-21T13:14:31.470423+0000","last_clean":"2024-03-21T13:14:31.470423+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:14:31.470423+0000","last_undegraded":"2024-03-21T13:14:31.470423+0000","last_fullsized":"2024-03-21T13:14:31.470423+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'30","last_scrub_stamp":"2024-03-21T13:12:26.004582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:12:26.004582+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.155513+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":657,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.492396+0000","last_change":"2024-03-21T13:13:09.984533+0000","last_active":"2024-03-21T13:14:36.492396+0000","last_peered":"2024-03-21T13:14:36.492396+0000","last_clean":"2024-03-21T13:14:36.492396+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:14:36.492396+0000","last_undegraded":"2024-03-21T13:14:36.492396+0000","last_fullsized":"2024-03-21T13:14:36.492396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:09.984411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:09.984411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:37.909935+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"368'11268","reported_seq":11365,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:40.305004+0000","last_change":"2024-03-21T13:12:19.518000+0000","last_active":"2024-03-21T13:14:40.305004+0000","last_peered":"2024-03-21T13:14:40.305004+0000","last_clean":"2024-03-21T13:14:40.305004+0000","last_became_active":"2024-03-21T13:12:19.517486+0000","last_became_peered":"2024-03-21T13:12:19.517486+0000","last_unstale":"2024-03-21T13:14:40.305004+0000","last_undegraded":"2024-03-21T13:14:40.305004+0000","last_fullsized":"2024-03-21T13:14:40.305004+0000","mapping_epoch":324,"log_start":"351'6700","ondisk_log_start":"351'6700","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:41.531598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348782592,"num_objects":5322,"num_object_clones":0,"num_object_copies":15966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11268,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":85,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.504149+0000","last_change":"2024-03-21T13:12:29.521775+0000","last_active":"2024-03-21T13:14:36.504149+0000","last_peered":"2024-03-21T13:14:36.504149+0000","last_clean":"2024-03-21T13:14:36.504149+0000","last_became_active":"2024-03-21T13:12:29.518722+0000","last_became_peered":"2024-03-21T13:12:29.518722+0000","last_unstale":"2024-03-21T13:14:36.504149+0000","last_undegraded":"2024-03-21T13:14:36.504149+0000","last_fullsized":"2024-03-21T13:14:36.504149+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:25.878314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:12:25.878314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127504+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"331'13","reported_seq":116,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.504590+0000","last_change":"2024-03-21T13:12:29.557416+0000","last_active":"2024-03-21T13:14:36.504590+0000","last_peered":"2024-03-21T13:14:36.504590+0000","last_clean":"2024-03-21T13:14:36.504590+0000","last_became_active":"2024-03-21T13:12:29.525058+0000","last_became_peered":"2024-03-21T13:12:29.525058+0000","last_unstale":"2024-03-21T13:14:36.504590+0000","last_undegraded":"2024-03-21T13:14:36.504590+0000","last_fullsized":"2024-03-21T13:14:36.504590+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'12","last_scrub_stamp":"2024-03-21T13:11:18.400270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:18.400270+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":661,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.470333+0000","last_change":"2024-03-21T13:14:17.224206+0000","last_active":"2024-03-21T13:14:31.470333+0000","last_peered":"2024-03-21T13:14:31.470333+0000","last_clean":"2024-03-21T13:14:31.470333+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:14:31.470333+0000","last_undegraded":"2024-03-21T13:14:31.470333+0000","last_fullsized":"2024-03-21T13:14:31.470333+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:17.224086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:17.224086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:20.410698+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"234'34","reported_seq":616,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.476536+0000","last_change":"2024-03-21T13:14:00.092878+0000","last_active":"2024-03-21T13:14:31.476536+0000","last_peered":"2024-03-21T13:14:31.476536+0000","last_clean":"2024-03-21T13:14:31.476536+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:14:31.476536+0000","last_undegraded":"2024-03-21T13:14:31.476536+0000","last_fullsized":"2024-03-21T13:14:31.476536+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'34","last_scrub_stamp":"2024-03-21T13:14:00.092594+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:14:00.092594+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:03.480207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"368'11261","reported_seq":11360,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:40.196488+0000","last_change":"2024-03-21T13:12:19.467189+0000","last_active":"2024-03-21T13:14:40.196488+0000","last_peered":"2024-03-21T13:14:40.196488+0000","last_clean":"2024-03-21T13:14:40.196488+0000","last_became_active":"2024-03-21T13:12:19.466848+0000","last_became_peered":"2024-03-21T13:12:19.466848+0000","last_unstale":"2024-03-21T13:14:40.196488+0000","last_undegraded":"2024-03-21T13:14:40.196488+0000","last_fullsized":"2024-03-21T13:14:40.196488+0000","mapping_epoch":324,"log_start":"351'6400","ondisk_log_start":"351'6400","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.439666+0000","objects_scrubbed":0,"log_size":4861,"log_dups_size":0,"ondisk_log_size":4861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346095616,"num_objects":5281,"num_object_clones":0,"num_object_copies":15843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11261,"num_write_kb":529344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"71'1","reported_seq":656,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.479542+0000","last_change":"2024-03-21T13:12:33.035682+0000","last_active":"2024-03-21T13:14:31.479542+0000","last_peered":"2024-03-21T13:14:31.479542+0000","last_clean":"2024-03-21T13:14:31.479542+0000","last_became_active":"2024-03-21T13:12:29.502816+0000","last_became_peered":"2024-03-21T13:12:29.502816+0000","last_unstale":"2024-03-21T13:14:31.479542+0000","last_undegraded":"2024-03-21T13:14:31.479542+0000","last_fullsized":"2024-03-21T13:14:31.479542+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:12:33.035545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:12:33.035545+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.155575+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":85,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.506108+0000","last_change":"2024-03-21T13:12:29.524403+0000","last_active":"2024-03-21T13:14:36.506108+0000","last_peered":"2024-03-21T13:14:36.506108+0000","last_clean":"2024-03-21T13:14:36.506108+0000","last_became_active":"2024-03-21T13:12:29.524087+0000","last_became_peered":"2024-03-21T13:12:29.524087+0000","last_unstale":"2024-03-21T13:14:36.506108+0000","last_undegraded":"2024-03-21T13:14:36.506108+0000","last_fullsized":"2024-03-21T13:14:36.506108+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:27.742792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:11:27.742792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.127149+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":693,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.492884+0000","last_change":"2024-03-21T13:14:03.182206+0000","last_active":"2024-03-21T13:14:36.492884+0000","last_peered":"2024-03-21T13:14:36.492884+0000","last_clean":"2024-03-21T13:14:36.492884+0000","last_became_active":"2024-03-21T13:10:03.586593+0000","last_became_peered":"2024-03-21T13:10:03.586593+0000","last_unstale":"2024-03-21T13:14:36.492884+0000","last_undegraded":"2024-03-21T13:14:36.492884+0000","last_fullsized":"2024-03-21T13:14:36.492884+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:03.182101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:03.182101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:22.301082+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":417,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.499653+0000","last_change":"2024-03-21T13:14:11.615096+0000","last_active":"2024-03-21T13:14:36.499653+0000","last_peered":"2024-03-21T13:14:36.499653+0000","last_clean":"2024-03-21T13:14:36.499653+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:14:36.499653+0000","last_undegraded":"2024-03-21T13:14:36.499653+0000","last_fullsized":"2024-03-21T13:14:36.499653+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:14:11.611538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:11.611538+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:38.005827+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"368'11017","reported_seq":11209,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:40.287433+0000","last_change":"2024-03-21T13:14:05.715845+0000","last_active":"2024-03-21T13:14:40.287433+0000","last_peered":"2024-03-21T13:14:40.287433+0000","last_clean":"2024-03-21T13:14:40.287433+0000","last_became_active":"2024-03-21T13:12:29.500228+0000","last_became_peered":"2024-03-21T13:12:29.500228+0000","last_unstale":"2024-03-21T13:14:40.287433+0000","last_undegraded":"2024-03-21T13:14:40.287433+0000","last_fullsized":"2024-03-21T13:14:40.287433+0000","mapping_epoch":330,"log_start":"348'5400","ondisk_log_start":"348'5400","created":324,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'8369","last_scrub_stamp":"2024-03-21T13:14:05.715580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:14:05.715580+0000","objects_scrubbed":7944,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:24.792407+0000","scrub_duration":2147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342556672,"num_objects":5227,"num_object_clones":0,"num_object_copies":15681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11017,"num_write_kb":519808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":221,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.497702+0000","last_change":"2024-03-21T13:13:24.855383+0000","last_active":"2024-03-21T13:14:36.497702+0000","last_peered":"2024-03-21T13:14:36.497702+0000","last_clean":"2024-03-21T13:14:36.497702+0000","last_became_active":"2024-03-21T13:10:03.602752+0000","last_became_peered":"2024-03-21T13:10:03.602752+0000","last_unstale":"2024-03-21T13:14:36.497702+0000","last_undegraded":"2024-03-21T13:14:36.497702+0000","last_fullsized":"2024-03-21T13:14:36.497702+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:24.855268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:13:24.855268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:50.759371+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":565,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.494443+0000","last_change":"2024-03-21T13:14:09.276862+0000","last_active":"2024-03-21T13:14:36.494443+0000","last_peered":"2024-03-21T13:14:36.494443+0000","last_clean":"2024-03-21T13:14:36.494443+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:14:36.494443+0000","last_undegraded":"2024-03-21T13:14:36.494443+0000","last_fullsized":"2024-03-21T13:14:36.494443+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:09.276760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:14:09.276760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:17.691979+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":671,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.497922+0000","last_change":"2024-03-21T13:12:31.226728+0000","last_active":"2024-03-21T13:14:36.497922+0000","last_peered":"2024-03-21T13:14:36.497922+0000","last_clean":"2024-03-21T13:14:36.497922+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:14:36.497922+0000","last_undegraded":"2024-03-21T13:14:36.497922+0000","last_fullsized":"2024-03-21T13:14:36.497922+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:12:31.226652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:12:31.226652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:37.805338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"234'31","reported_seq":231,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.499085+0000","last_change":"2024-03-21T13:13:15.705112+0000","last_active":"2024-03-21T13:14:36.499085+0000","last_peered":"2024-03-21T13:14:36.499085+0000","last_clean":"2024-03-21T13:14:36.499085+0000","last_became_active":"2024-03-21T13:10:03.591832+0000","last_became_peered":"2024-03-21T13:10:03.591832+0000","last_unstale":"2024-03-21T13:14:36.499085+0000","last_undegraded":"2024-03-21T13:14:36.499085+0000","last_fullsized":"2024-03-21T13:14:36.499085+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'31","last_scrub_stamp":"2024-03-21T13:13:15.704929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:15.704929+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:40.307944+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"368'11152","reported_seq":11338,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:40.138044+0000","last_change":"2024-03-21T13:14:07.672937+0000","last_active":"2024-03-21T13:14:40.138044+0000","last_peered":"2024-03-21T13:14:40.138044+0000","last_clean":"2024-03-21T13:14:40.138044+0000","last_became_active":"2024-03-21T13:12:19.518390+0000","last_became_peered":"2024-03-21T13:12:19.518390+0000","last_unstale":"2024-03-21T13:14:40.138044+0000","last_undegraded":"2024-03-21T13:14:40.138044+0000","last_fullsized":"2024-03-21T13:14:40.138044+0000","mapping_epoch":324,"log_start":"353'6800","ondisk_log_start":"353'6800","created":324,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'8572","last_scrub_stamp":"2024-03-21T13:14:07.671427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:18.439666+0000","last_clean_scrub_stamp":"2024-03-21T13:14:07.671427+0000","objects_scrubbed":7915,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:34.495574+0000","scrub_duration":2473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344195072,"num_objects":5252,"num_object_clones":0,"num_object_copies":15756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11152,"num_write_kb":524928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":227,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.497495+0000","last_change":"2024-03-21T13:14:22.383118+0000","last_active":"2024-03-21T13:14:36.497495+0000","last_peered":"2024-03-21T13:14:36.497495+0000","last_clean":"2024-03-21T13:14:36.497495+0000","last_became_active":"2024-03-21T13:12:29.499442+0000","last_became_peered":"2024-03-21T13:12:29.499442+0000","last_unstale":"2024-03-21T13:14:36.497495+0000","last_undegraded":"2024-03-21T13:14:36.497495+0000","last_fullsized":"2024-03-21T13:14:36.497495+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:22.382971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:22.382971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:15:50.725987+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"331'13","reported_seq":284,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-21T13:14:36.499032+0000","last_change":"2024-03-21T13:12:29.518453+0000","last_active":"2024-03-21T13:14:36.499032+0000","last_peered":"2024-03-21T13:14:36.499032+0000","last_clean":"2024-03-21T13:14:36.499032+0000","last_became_active":"2024-03-21T13:12:29.507925+0000","last_became_peered":"2024-03-21T13:12:29.507925+0000","last_unstale":"2024-03-21T13:14:36.499032+0000","last_undegraded":"2024-03-21T13:14:36.499032+0000","last_fullsized":"2024-03-21T13:14:36.499032+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'12","last_scrub_stamp":"2024-03-21T13:12:18.891042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:12:18.891042+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:41.517227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":647,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.474618+0000","last_change":"2024-03-21T13:13:12.141409+0000","last_active":"2024-03-21T13:14:31.474618+0000","last_peered":"2024-03-21T13:14:31.474618+0000","last_clean":"2024-03-21T13:14:31.474618+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:14:31.474618+0000","last_undegraded":"2024-03-21T13:14:31.474618+0000","last_fullsized":"2024-03-21T13:14:31.474618+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:12.141284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:13:12.141284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:36.256401+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"234'20","reported_seq":608,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-21T13:14:31.479736+0000","last_change":"2024-03-21T13:12:17.040702+0000","last_active":"2024-03-21T13:14:31.479736+0000","last_peered":"2024-03-21T13:14:31.479736+0000","last_clean":"2024-03-21T13:14:31.479736+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:14:31.479736+0000","last_undegraded":"2024-03-21T13:14:31.479736+0000","last_fullsized":"2024-03-21T13:14:31.479736+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:12:17.040608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:12:17.040608+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:35.155480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5540216856,"num_objects":84538,"num_object_clones":0,"num_object_copies":253614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84538,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179056,"num_write_kb":8434945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16676564992,"data_stored":16676552776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79356,"ondisk_log_size":79356,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":952,"num_read_kb":947,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1103,"num_read_kb":860,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":6766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":740,"ondisk_log_size":740,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168773,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2491636,"kb_used_data":1702444,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":91253516,"statfs":{"total":95995035648,"available":93443600384,"internally_reserved":0,"allocated":1743302656,"data_stored":1742885363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":808122826},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463115,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3239672,"kb_used_data":2388016,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":90505480,"statfs":{"total":95995035648,"available":92677611520,"internally_reserved":0,"allocated":2445328384,"data_stored":2444906728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":872085958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575075,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738601,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4166576,"kb_used_data":3386984,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":89578576,"statfs":{"total":95995035648,"available":91728461824,"internally_reserved":0,"allocated":3468271616,"data_stored":3467857091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411551,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2779848,"kb_used_data":2036416,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":90965304,"statfs":{"total":95995035648,"available":93148471296,"internally_reserved":0,"allocated":2085289984,"data_stored":2084915479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182617,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3100896,"kb_used_data":2171352,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":90644256,"statfs":{"total":95995035648,"available":92819718144,"internally_reserved":0,"allocated":2223464448,"data_stored":2223059177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":951712199},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019092,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4021592,"kb_used_data":3083856,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":89723560,"statfs":{"total":95995035648,"available":91876925440,"internally_reserved":0,"allocated":3157868544,"data_stored":3157478158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724751,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1872844,"kb_used_data":1719428,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":91872308,"statfs":{"total":95995035648,"available":94077243392,"internally_reserved":0,"allocated":1760694272,"data_stored":1760315349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3430158336,"data_stored":3430154264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2059796480,"data_stored":2059796480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2195914752,"data_stored":2195914752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3107721216,"data_stored":3107717144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1739784192,"data_stored":1739784192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2422542336,"data_stored":2422538264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1720647680,"data_stored":1720647680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:14:42.378 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-21T13:14:42.379 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:14:42.379 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-03-21T13:14:42.379 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-21T13:14:42.379 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-21T13:14:42.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-03-21T13:14:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:42.381 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:42.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:42.459 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:14:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:14:42.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:42.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:42.541 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:42.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:42.642 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:42.670 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:42.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:42.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:42.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:42 smithi163 ceph-mon[40675]: pgmap v753: 148 pgs: 148 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-21T13:14:42.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:42 smithi163 ceph-mon[40675]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:14:42.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:42 smithi163 ceph-mon[40675]: osdmap e369: 8 total, 8 up, 7 in 2024-03-21T13:14:42.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:42 smithi163 ceph-mon[29621]: pgmap v753: 148 pgs: 148 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-21T13:14:42.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:42 smithi163 ceph-mon[29621]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:14:42.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:42 smithi163 ceph-mon[29621]: osdmap e369: 8 total, 8 up, 7 in 2024-03-21T13:14:42.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:42 smithi204 ceph-mon[33153]: pgmap v753: 148 pgs: 148 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-21T13:14:42.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:42 smithi204 ceph-mon[33153]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:14:42.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:42 smithi204 ceph-mon[33153]: osdmap e369: 8 total, 8 up, 7 in 2024-03-21T13:14:42.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:42.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:42.815 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:42.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:42.902 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:14:42.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:42.948 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:42.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:43.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:43.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:43.092 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:43.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:43.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:43.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:43.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:43.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:43.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:43.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:43.376 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:43.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:43.477 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:43.515 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:43.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:43.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:43.649 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:43.655 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:43.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:43.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:43.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:43.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:43.791 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:43.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:43.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:43.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:43.929 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:43.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:44.030 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:44.067 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:44.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:44.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:44.169 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:44.209 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:44.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:44.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:44.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:44.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:44.348 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:44.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:44.414 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:44.449 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:44.484 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:44.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:44.586 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:44.645 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:44.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:44.666 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:44 smithi163 ceph-mon[29621]: pgmap v755: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-21T13:14:44.666 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:44 smithi163 ceph-mon[40675]: pgmap v755: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-21T13:14:44.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:44.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:44 smithi204 ceph-mon[33153]: pgmap v755: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-21T13:14:44.790 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:44.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:44.892 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:44.929 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:44.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:45.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:45.057 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:14:45.069 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:45.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:45.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:45.093 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:14:45.134 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:14:45.169 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:14:45.171 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:45.213 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:14:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:45.214 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:45.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:45.248 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:14:45.285 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:14:45.315 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:45.327 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:14:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:14:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:14:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:14:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:14:45.333 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:14:45.333 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:14:45.333 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:14:45.333 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:45.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:45.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:45.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:45.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:45.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:45.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:45.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:45.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:45.734 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:45.758 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:14:45.758 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-21T13:14:45.759 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:14:45.770 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:45.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:45.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:45.813 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:45 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2868049637' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:45.813 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:14:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:14:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:14:45.813 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:14:45.813 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-03-21T13:14:45.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-03-21T13:14:45.814 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:14:45.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.814 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:197997 8.e 8:72fc6c89:::benchmark_data_smithi163_107079_object62124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.814 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.339543+0000", 2024-03-21T13:14:45.814 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.31306696299998, 2024-03-21T13:14:45.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98943099899999998, 2024-03-21T13:14:45.814 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.814 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.815 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.815 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 197997 2024-03-21T13:14:45.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.815 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339543+0000", 2024-03-21T13:14:45.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339543+0000", 2024-03-21T13:14:45.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339544+0000", 2024-03-21T13:14:45.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7400000000000002e-07 2024-03-21T13:14:45.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339548+0000", 2024-03-21T13:14:45.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0400000000000003e-06 2024-03-21T13:14:45.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339559+0000", 2024-03-21T13:14:45.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0631e-05 2024-03-21T13:14:45.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339566+0000", 2024-03-21T13:14:45.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.661e-06 2024-03-21T13:14:45.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339624+0000", 2024-03-21T13:14:45.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7519999999999998e-05 2024-03-21T13:14:45.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.339632+0000", 2024-03-21T13:14:45.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5690000000000005e-06 2024-03-21T13:14:45.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.325535+0000", 2024-03-21T13:14:45.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98590222599999999 2024-03-21T13:14:45.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.325675+0000", 2024-03-21T13:14:45.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014081 2024-03-21T13:14:45.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.325799+0000", 2024-03-21T13:14:45.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012380099999999999 2024-03-21T13:14:45.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.328858+0000", 2024-03-21T13:14:45.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003058659 2024-03-21T13:14:45.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.328892+0000", 2024-03-21T13:14:45.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4462999999999999e-05 2024-03-21T13:14:45.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.328903+0000", 2024-03-21T13:14:45.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0261000000000001e-05 2024-03-21T13:14:45.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.328933+0000", 2024-03-21T13:14:45.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9887e-05 2024-03-21T13:14:45.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.328974+0000", 2024-03-21T13:14:45.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1696999999999997e-05 2024-03-21T13:14:45.822 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.822 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.822 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.823 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198019 8.e 8:79bb99bc:::benchmark_data_smithi163_107079_object62146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.823 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.353797+0000", 2024-03-21T13:14:45.823 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.298812801, 2024-03-21T13:14:45.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.97779139900000001, 2024-03-21T13:14:45.823 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.823 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.823 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.823 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.823 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.823 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198019 2024-03-21T13:14:45.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.824 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353797+0000", 2024-03-21T13:14:45.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353797+0000", 2024-03-21T13:14:45.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353798+0000", 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9100000000000004e-07 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353801+0000", 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4060000000000001e-06 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353809+0000", 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8849999999999999e-06 2024-03-21T13:14:45.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353816+0000", 2024-03-21T13:14:45.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3969999999999999e-06 2024-03-21T13:14:45.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353870+0000", 2024-03-21T13:14:45.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4196000000000001e-05 2024-03-21T13:14:45.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.353876+0000", 2024-03-21T13:14:45.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.241e-06 2024-03-21T13:14:45.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.327830+0000", 2024-03-21T13:14:45.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.97395389600000004 2024-03-21T13:14:45.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.327900+0000", 2024-03-21T13:14:45.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9678000000000006e-05 2024-03-21T13:14:45.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.328004+0000", 2024-03-21T13:14:45.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104032 2024-03-21T13:14:45.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331359+0000", 2024-03-21T13:14:45.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033555260000000002 2024-03-21T13:14:45.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331410+0000", 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0231000000000001e-05 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331476+0000", 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6135000000000003e-05 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331534+0000", 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8661e-05 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331589+0000", 2024-03-21T13:14:45.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4524000000000002e-05 2024-03-21T13:14:45.830 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.830 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.830 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.830 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198149 8.e 8:7688df86:::benchmark_data_smithi163_107079_object62276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.830 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.432999+0000", 2024-03-21T13:14:45.830 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.21961105100002, 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.90199298900000002, 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198149 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.432999+0000", 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.432999+0000", 2024-03-21T13:14:45.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.433001+0000", 2024-03-21T13:14:45.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.395e-06 2024-03-21T13:14:45.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.433014+0000", 2024-03-21T13:14:45.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3903000000000001e-05 2024-03-21T13:14:45.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.433035+0000", 2024-03-21T13:14:45.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0395e-05 2024-03-21T13:14:45.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.433043+0000", 2024-03-21T13:14:45.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5750000000000001e-06 2024-03-21T13:14:45.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.433126+0000", 2024-03-21T13:14:45.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.297e-05 2024-03-21T13:14:45.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.433138+0000", 2024-03-21T13:14:45.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1861e-05 2024-03-21T13:14:45.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.329093+0000", 2024-03-21T13:14:45.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.89595484400000003 2024-03-21T13:14:45.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.329200+0000", 2024-03-21T13:14:45.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107267 2024-03-21T13:14:45.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.329265+0000", 2024-03-21T13:14:45.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4258999999999996e-05 2024-03-21T13:14:45.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331600+0000", 2024-03-21T13:14:45.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023357 2024-03-21T13:14:45.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.334786+0000", 2024-03-21T13:14:45.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031856190000000002 2024-03-21T13:14:45.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.334840+0000", 2024-03-21T13:14:45.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4367999999999999e-05 2024-03-21T13:14:45.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.334942+0000", 2024-03-21T13:14:45.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000101543 2024-03-21T13:14:45.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.334992+0000", 2024-03-21T13:14:45.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0290000000000001e-05 2024-03-21T13:14:45.838 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.838 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.838 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.838 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198166 8.e 8:70e11697:::benchmark_data_smithi163_107079_object62293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.839 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.448097+0000", 2024-03-21T13:14:45.839 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.20451306400003, 2024-03-21T13:14:45.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.89185160900000005, 2024-03-21T13:14:45.839 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.839 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.839 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.839 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.839 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.839 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198166 2024-03-21T13:14:45.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.839 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448097+0000", 2024-03-21T13:14:45.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448097+0000", 2024-03-21T13:14:45.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448099+0000", 2024-03-21T13:14:45.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.685e-06 2024-03-21T13:14:45.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448105+0000", 2024-03-21T13:14:45.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2999999999999998e-06 2024-03-21T13:14:45.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448123+0000", 2024-03-21T13:14:45.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8187000000000001e-05 2024-03-21T13:14:45.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448133+0000", 2024-03-21T13:14:45.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7790000000000007e-06 2024-03-21T13:14:45.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448218+0000", 2024-03-21T13:14:45.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5099000000000006e-05 2024-03-21T13:14:45.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.448229+0000", 2024-03-21T13:14:45.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0497e-05 2024-03-21T13:14:45.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.330366+0000", 2024-03-21T13:14:45.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.88213745099999996 2024-03-21T13:14:45.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.330435+0000", 2024-03-21T13:14:45.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8779e-05 2024-03-21T13:14:45.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.330499+0000", 2024-03-21T13:14:45.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4129999999999998e-05 2024-03-21T13:14:45.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.334736+0000", 2024-03-21T13:14:45.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0042372149999999999 2024-03-21T13:14:45.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.335060+0000", 2024-03-21T13:14:45.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032360700000000001 2024-03-21T13:14:45.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.339846+0000", 2024-03-21T13:14:45.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047860309999999996 2024-03-21T13:14:45.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.339902+0000", 2024-03-21T13:14:45.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.588e-05 2024-03-21T13:14:45.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.847 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:45 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2868049637' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:45.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.339949+0000", 2024-03-21T13:14:45.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6968999999999997e-05 2024-03-21T13:14:45.848 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.848 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.848 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.849 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198169 8.e 8:7f3d2014:::benchmark_data_smithi163_107079_object62296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.849 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.452482+0000", 2024-03-21T13:14:45.849 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.20012779299998, 2024-03-21T13:14:45.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.88774095099999994, 2024-03-21T13:14:45.849 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.849 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.849 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.849 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.849 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.849 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198169 2024-03-21T13:14:45.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.849 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452482+0000", 2024-03-21T13:14:45.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452482+0000", 2024-03-21T13:14:45.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452484+0000", 2024-03-21T13:14:45.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.486e-06 2024-03-21T13:14:45.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452491+0000", 2024-03-21T13:14:45.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8390000000000001e-06 2024-03-21T13:14:45.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452503+0000", 2024-03-21T13:14:45.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.253e-05 2024-03-21T13:14:45.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452512+0000", 2024-03-21T13:14:45.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9330000000000007e-06 2024-03-21T13:14:45.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452623+0000", 2024-03-21T13:14:45.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110587 2024-03-21T13:14:45.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.452632+0000", 2024-03-21T13:14:45.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7290000000000006e-06 2024-03-21T13:14:45.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331656+0000", 2024-03-21T13:14:45.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.879023997 2024-03-21T13:14:45.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331767+0000", 2024-03-21T13:14:45.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000111239 2024-03-21T13:14:45.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.331890+0000", 2024-03-21T13:14:45.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000123306 2024-03-21T13:14:45.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.334800+0000", 2024-03-21T13:14:45.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0029102360000000001 2024-03-21T13:14:45.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.335161+0000", 2024-03-21T13:14:45.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00036037599999999998 2024-03-21T13:14:45.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340133+0000", 2024-03-21T13:14:45.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049728039999999999 2024-03-21T13:14:45.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340184+0000", 2024-03-21T13:14:45.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0417999999999997e-05 2024-03-21T13:14:45.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340223+0000", 2024-03-21T13:14:45.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9471000000000002e-05 2024-03-21T13:14:45.857 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.857 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.857 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.857 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198177 8.e 8:704f885f:::benchmark_data_smithi163_107079_object62304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.857 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.456512+0000", 2024-03-21T13:14:45.857 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.19609847499999, 2024-03-21T13:14:45.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.883838499, 2024-03-21T13:14:45.857 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.857 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.858 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.858 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.858 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.858 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198177 2024-03-21T13:14:45.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.858 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456512+0000", 2024-03-21T13:14:45.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456512+0000", 2024-03-21T13:14:45.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456513+0000", 2024-03-21T13:14:45.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4750000000000001e-06 2024-03-21T13:14:45.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456520+0000", 2024-03-21T13:14:45.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7900000000000002e-06 2024-03-21T13:14:45.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456538+0000", 2024-03-21T13:14:45.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7883000000000001e-05 2024-03-21T13:14:45.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456548+0000", 2024-03-21T13:14:45.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8560000000000002e-06 2024-03-21T13:14:45.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456610+0000", 2024-03-21T13:14:45.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2644000000000005e-05 2024-03-21T13:14:45.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456617+0000", 2024-03-21T13:14:45.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9349999999999997e-06 2024-03-21T13:14:45.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.333229+0000", 2024-03-21T13:14:45.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.87661202800000004 2024-03-21T13:14:45.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.333343+0000", 2024-03-21T13:14:45.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114021 2024-03-21T13:14:45.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.333468+0000", 2024-03-21T13:14:45.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124461 2024-03-21T13:14:45.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.335178+0000", 2024-03-21T13:14:45.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0017099280000000001 2024-03-21T13:14:45.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.339788+0000", 2024-03-21T13:14:45.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0046099510000000002 2024-03-21T13:14:45.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340273+0000", 2024-03-21T13:14:45.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048556599999999999 2024-03-21T13:14:45.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340316+0000", 2024-03-21T13:14:45.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2725000000000002e-05 2024-03-21T13:14:45.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340350+0000", 2024-03-21T13:14:45.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4236000000000002e-05 2024-03-21T13:14:45.865 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.865 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.865 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.865 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198178 8.e 8:721bb05b:::benchmark_data_smithi163_107079_object62305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.866 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.456814+0000", 2024-03-21T13:14:45.866 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.19579582099999, 2024-03-21T13:14:45.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.88372909700000002, 2024-03-21T13:14:45.866 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.866 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.866 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.866 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.866 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.866 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198178 2024-03-21T13:14:45.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.866 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456814+0000", 2024-03-21T13:14:45.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456814+0000", 2024-03-21T13:14:45.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456815+0000", 2024-03-21T13:14:45.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-03-21T13:14:45.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456820+0000", 2024-03-21T13:14:45.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7890000000000002e-06 2024-03-21T13:14:45.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456832+0000", 2024-03-21T13:14:45.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1558e-05 2024-03-21T13:14:45.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456839+0000", 2024-03-21T13:14:45.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8380000000000002e-06 2024-03-21T13:14:45.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456893+0000", 2024-03-21T13:14:45.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3254999999999997e-05 2024-03-21T13:14:45.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.456903+0000", 2024-03-21T13:14:45.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9019999999999994e-06 2024-03-21T13:14:45.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.335208+0000", 2024-03-21T13:14:45.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.87830587000000004 2024-03-21T13:14:45.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.335859+0000", 2024-03-21T13:14:45.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00065099199999999998 2024-03-21T13:14:45.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.335970+0000", 2024-03-21T13:14:45.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110944 2024-03-21T13:14:45.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.339961+0000", 2024-03-21T13:14:45.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039907049999999998 2024-03-21T13:14:45.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340080+0000", 2024-03-21T13:14:45.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118522 2024-03-21T13:14:45.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340388+0000", 2024-03-21T13:14:45.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030865299999999999 2024-03-21T13:14:45.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340503+0000", 2024-03-21T13:14:45.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011429099999999999 2024-03-21T13:14:45.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340543+0000", 2024-03-21T13:14:45.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0973999999999998e-05 2024-03-21T13:14:45.873 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.873 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.873 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.873 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198182 8.e 8:7bcb4e00:::benchmark_data_smithi163_107079_object62309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.874 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.457774+0000", 2024-03-21T13:14:45.874 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.19483573299999, 2024-03-21T13:14:45.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.88472258800000003, 2024-03-21T13:14:45.874 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.874 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.874 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.874 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.874 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.874 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198182 2024-03-21T13:14:45.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.874 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457774+0000", 2024-03-21T13:14:45.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457774+0000", 2024-03-21T13:14:45.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457775+0000", 2024-03-21T13:14:45.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4300000000000001e-07 2024-03-21T13:14:45.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457781+0000", 2024-03-21T13:14:45.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2780000000000001e-06 2024-03-21T13:14:45.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457795+0000", 2024-03-21T13:14:45.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4754e-05 2024-03-21T13:14:45.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457804+0000", 2024-03-21T13:14:45.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3270000000000001e-06 2024-03-21T13:14:45.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457874+0000", 2024-03-21T13:14:45.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9811e-05 2024-03-21T13:14:45.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457881+0000", 2024-03-21T13:14:45.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2799999999999998e-06 2024-03-21T13:14:45.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.337893+0000", 2024-03-21T13:14:45.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.88001165699999995 2024-03-21T13:14:45.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.338013+0000", 2024-03-21T13:14:45.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120188 2024-03-21T13:14:45.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.338128+0000", 2024-03-21T13:14:45.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115423 2024-03-21T13:14:45.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340236+0000", 2024-03-21T13:14:45.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021079570000000001 2024-03-21T13:14:45.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342388+0000", 2024-03-21T13:14:45.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021521140000000001 2024-03-21T13:14:45.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342434+0000", 2024-03-21T13:14:45.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5337999999999999e-05 2024-03-21T13:14:45.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342469+0000", 2024-03-21T13:14:45.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5061999999999997e-05 2024-03-21T13:14:45.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342497+0000", 2024-03-21T13:14:45.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8456e-05 2024-03-21T13:14:45.881 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.881 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.881 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.882 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198184 8.e 8:7a1e0247:::benchmark_data_smithi163_107079_object62311:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.882 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.457973+0000", 2024-03-21T13:14:45.882 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.19463695500002, 2024-03-21T13:14:45.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.89018627299999997, 2024-03-21T13:14:45.882 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.882 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.882 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198184 2024-03-21T13:14:45.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.883 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457973+0000", 2024-03-21T13:14:45.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457973+0000", 2024-03-21T13:14:45.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457974+0000", 2024-03-21T13:14:45.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8899999999999998e-07 2024-03-21T13:14:45.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457979+0000", 2024-03-21T13:14:45.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2100000000000001e-06 2024-03-21T13:14:45.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457990+0000", 2024-03-21T13:14:45.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0969e-05 2024-03-21T13:14:45.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.457997+0000", 2024-03-21T13:14:45.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8329999999999996e-06 2024-03-21T13:14:45.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.458072+0000", 2024-03-21T13:14:45.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4911000000000002e-05 2024-03-21T13:14:45.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.458079+0000", 2024-03-21T13:14:45.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0140000000000001e-06 2024-03-21T13:14:45.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340576+0000", 2024-03-21T13:14:45.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.88249676899999996 2024-03-21T13:14:45.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340684+0000", 2024-03-21T13:14:45.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000108677 2024-03-21T13:14:45.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.340796+0000", 2024-03-21T13:14:45.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000111854 2024-03-21T13:14:45.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.347782+0000", 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0069861200000000002 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.347830+0000", 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7663000000000001e-05 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.347939+0000", 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010846899999999999 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348109+0000", 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017050400000000001 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348160+0000", 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0491000000000002e-05 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.889 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198188 8.e 8:71a99677:::benchmark_data_smithi163_107079_object62315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.890 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.462609+0000", 2024-03-21T13:14:45.890 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.19000125600002, 2024-03-21T13:14:45.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.89143663399999995, 2024-03-21T13:14:45.890 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.890 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.890 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.890 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.890 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.890 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198188 2024-03-21T13:14:45.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.890 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462609+0000", 2024-03-21T13:14:45.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462609+0000", 2024-03-21T13:14:45.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462611+0000", 2024-03-21T13:14:45.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6360000000000001e-06 2024-03-21T13:14:45.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462617+0000", 2024-03-21T13:14:45.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4749999999999998e-06 2024-03-21T13:14:45.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462630+0000", 2024-03-21T13:14:45.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2551999999999999e-05 2024-03-21T13:14:45.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462638+0000", 2024-03-21T13:14:45.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6030000000000005e-06 2024-03-21T13:14:45.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462696+0000", 2024-03-21T13:14:45.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7834999999999998e-05 2024-03-21T13:14:45.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.462706+0000", 2024-03-21T13:14:45.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6360000000000006e-06 2024-03-21T13:14:45.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342529+0000", 2024-03-21T13:14:45.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.87982343500000004 2024-03-21T13:14:45.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342627+0000", 2024-03-21T13:14:45.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7701999999999994e-05 2024-03-21T13:14:45.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.342753+0000", 2024-03-21T13:14:45.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012617499999999999 2024-03-21T13:14:45.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.347843+0000", 2024-03-21T13:14:45.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050896400000000003 2024-03-21T13:14:45.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348204+0000", 2024-03-21T13:14:45.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00036116300000000001 2024-03-21T13:14:45.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.353943+0000", 2024-03-21T13:14:45.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057395900000000001 2024-03-21T13:14:45.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.353996+0000", 2024-03-21T13:14:45.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2973000000000003e-05 2024-03-21T13:14:45.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354046+0000", 2024-03-21T13:14:45.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9218999999999998e-05 2024-03-21T13:14:45.897 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.897 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.897 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.897 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198243 8.e 8:700a35e6:::benchmark_data_smithi163_107079_object62370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.897 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.506276+0000", 2024-03-21T13:14:45.897 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.14633406799999, 2024-03-21T13:14:45.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.84810814400000001, 2024-03-21T13:14:45.897 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.898 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.898 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.898 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.898 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.898 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198243 2024-03-21T13:14:45.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.898 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506276+0000", 2024-03-21T13:14:45.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506276+0000", 2024-03-21T13:14:45.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506278+0000", 2024-03-21T13:14:45.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3659999999999999e-06 2024-03-21T13:14:45.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506284+0000", 2024-03-21T13:14:45.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0560000000000001e-06 2024-03-21T13:14:45.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506295+0000", 2024-03-21T13:14:45.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1175000000000001e-05 2024-03-21T13:14:45.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506312+0000", 2024-03-21T13:14:45.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.753e-05 2024-03-21T13:14:45.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506379+0000", 2024-03-21T13:14:45.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6618000000000005e-05 2024-03-21T13:14:45.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.506390+0000", 2024-03-21T13:14:45.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0638e-05 2024-03-21T13:14:45.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.344552+0000", 2024-03-21T13:14:45.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.83816236499999996 2024-03-21T13:14:45.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.344660+0000", 2024-03-21T13:14:45.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010765 2024-03-21T13:14:45.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.344726+0000", 2024-03-21T13:14:45.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6173000000000005e-05 2024-03-21T13:14:45.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.347849+0000", 2024-03-21T13:14:45.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031237109999999999 2024-03-21T13:14:45.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348241+0000", 2024-03-21T13:14:45.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039150099999999999 2024-03-21T13:14:45.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354126+0000", 2024-03-21T13:14:45.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0058847830000000002 2024-03-21T13:14:45.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354338+0000", 2024-03-21T13:14:45.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00021213800000000001 2024-03-21T13:14:45.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354384+0000", 2024-03-21T13:14:45.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6440000000000003e-05 2024-03-21T13:14:45.905 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.905 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.905 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.905 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198261 8.e 8:72991b00:::benchmark_data_smithi163_107079_object62388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.905 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.518817+0000", 2024-03-21T13:14:45.906 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.13379372200001, 2024-03-21T13:14:45.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.83579388399999999, 2024-03-21T13:14:45.906 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.906 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.906 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.906 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.906 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.906 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198261 2024-03-21T13:14:45.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.906 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.907 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518817+0000", 2024-03-21T13:14:45.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.907 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518817+0000", 2024-03-21T13:14:45.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.907 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518818+0000", 2024-03-21T13:14:45.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.776e-06 2024-03-21T13:14:45.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518824+0000", 2024-03-21T13:14:45.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8599999999999998e-06 2024-03-21T13:14:45.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518840+0000", 2024-03-21T13:14:45.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5987000000000001e-05 2024-03-21T13:14:45.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518849+0000", 2024-03-21T13:14:45.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4409999999999993e-06 2024-03-21T13:14:45.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518913+0000", 2024-03-21T13:14:45.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4414999999999999e-05 2024-03-21T13:14:45.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.518922+0000", 2024-03-21T13:14:45.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6500000000000002e-06 2024-03-21T13:14:45.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.346413+0000", 2024-03-21T13:14:45.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.82749177500000004 2024-03-21T13:14:45.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.346487+0000", 2024-03-21T13:14:45.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3609000000000006e-05 2024-03-21T13:14:45.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.346554+0000", 2024-03-21T13:14:45.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6489999999999995e-05 2024-03-21T13:14:45.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.353886+0000", 2024-03-21T13:14:45.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0073328710000000004 2024-03-21T13:14:45.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.353898+0000", 2024-03-21T13:14:45.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-03-21T13:14:45.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354507+0000", 2024-03-21T13:14:45.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00060872200000000002 2024-03-21T13:14:45.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354572+0000", 2024-03-21T13:14:45.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5137999999999999e-05 2024-03-21T13:14:45.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354610+0000", 2024-03-21T13:14:45.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8352000000000001e-05 2024-03-21T13:14:45.913 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.913 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.913 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.914 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198285 8.e 8:7c6267f4:::benchmark_data_smithi163_107079_object62412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.914 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.540290+0000", 2024-03-21T13:14:45.914 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.11231978799998, 2024-03-21T13:14:45.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.81888756500000004, 2024-03-21T13:14:45.914 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.914 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.914 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.914 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.914 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.914 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198285 2024-03-21T13:14:45.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.915 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540290+0000", 2024-03-21T13:14:45.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540290+0000", 2024-03-21T13:14:45.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540292+0000", 2024-03-21T13:14:45.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3570000000000001e-06 2024-03-21T13:14:45.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540297+0000", 2024-03-21T13:14:45.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2429999999999996e-06 2024-03-21T13:14:45.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540307+0000", 2024-03-21T13:14:45.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.026e-05 2024-03-21T13:14:45.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540315+0000", 2024-03-21T13:14:45.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5889999999999997e-06 2024-03-21T13:14:45.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540375+0000", 2024-03-21T13:14:45.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9960000000000002e-05 2024-03-21T13:14:45.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.540384+0000", 2024-03-21T13:14:45.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6509999999999996e-06 2024-03-21T13:14:45.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348268+0000", 2024-03-21T13:14:45.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.80788329000000003 2024-03-21T13:14:45.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348361+0000", 2024-03-21T13:14:45.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3122999999999996e-05 2024-03-21T13:14:45.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.348463+0000", 2024-03-21T13:14:45.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000102512 2024-03-21T13:14:45.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.353903+0000", 2024-03-21T13:14:45.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054392429999999999 2024-03-21T13:14:45.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.354086+0000", 2024-03-21T13:14:45.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018381300000000001 2024-03-21T13:14:45.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359076+0000", 2024-03-21T13:14:45.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049894859999999996 2024-03-21T13:14:45.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359131+0000", 2024-03-21T13:14:45.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5441e-05 2024-03-21T13:14:45.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359178+0000", 2024-03-21T13:14:45.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6597000000000001e-05 2024-03-21T13:14:45.922 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.922 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.922 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.922 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15078.0:198325 8.e 8:7a044c9a:::benchmark_data_smithi163_107079_object62452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-21T13:14:45.922 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:07:26.588828+0000", 2024-03-21T13:14:45.922 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 439.06378214, 2024-03-21T13:14:45.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.770531403, 2024-03-21T13:14:45.922 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.922 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.923 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.923 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15078", 2024-03-21T13:14:45.923 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:14:45.923 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 198325 2024-03-21T13:14:45.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.923 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588828+0000", 2024-03-21T13:14:45.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588828+0000", 2024-03-21T13:14:45.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588829+0000", 2024-03-21T13:14:45.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.26e-06 2024-03-21T13:14:45.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588834+0000", 2024-03-21T13:14:45.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4100000000000001e-06 2024-03-21T13:14:45.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588843+0000", 2024-03-21T13:14:45.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2439999999999999e-06 2024-03-21T13:14:45.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588849+0000", 2024-03-21T13:14:45.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8680000000000004e-06 2024-03-21T13:14:45.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588891+0000", 2024-03-21T13:14:45.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2385999999999997e-05 2024-03-21T13:14:45.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:26.588897+0000", 2024-03-21T13:14:45.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1940000000000003e-06 2024-03-21T13:14:45.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.350134+0000", 2024-03-21T13:14:45.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.76123635999999995 2024-03-21T13:14:45.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.350236+0000", 2024-03-21T13:14:45.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000102503 2024-03-21T13:14:45.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-03-21T13:14:45.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.350345+0000", 2024-03-21T13:14:45.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010907499999999999 2024-03-21T13:14:45.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.353907+0000", 2024-03-21T13:14:45.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035621030000000001 2024-03-21T13:14:45.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359012+0000", 2024-03-21T13:14:45.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0051040729999999998 2024-03-21T13:14:45.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359275+0000", 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000263341 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359323+0000", 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7743e-05 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:07:27.359359+0000", 2024-03-21T13:14:45.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6842999999999999e-05 2024-03-21T13:14:45.930 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.930 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.930 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.930 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:54244 9.12 9.7c67c952 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-03-21T13:14:45.930 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:09:19.445638+0000", 2024-03-21T13:14:45.930 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 326.20697271, 2024-03-21T13:14:45.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.99024263899999998, 2024-03-21T13:14:45.930 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.930 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:14:45.931 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.931 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:14:45.931 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:14:45.931 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 54244 2024-03-21T13:14:45.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.931 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445638+0000", 2024-03-21T13:14:45.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445638+0000", 2024-03-21T13:14:45.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445639+0000", 2024-03-21T13:14:45.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0890000000000001e-06 2024-03-21T13:14:45.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445668+0000", 2024-03-21T13:14:45.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9726e-05 2024-03-21T13:14:45.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445676+0000", 2024-03-21T13:14:45.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0660000000000004e-06 2024-03-21T13:14:45.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445683+0000", 2024-03-21T13:14:45.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3990000000000001e-06 2024-03-21T13:14:45.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445741+0000", 2024-03-21T13:14:45.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7972e-05 2024-03-21T13:14:45.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445751+0000", 2024-03-21T13:14:45.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0227999999999999e-05 2024-03-21T13:14:45.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:20.435868+0000", 2024-03-21T13:14:45.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.99011690600000002 2024-03-21T13:14:45.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:20.435880+0000", 2024-03-21T13:14:45.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2252999999999999e-05 2024-03-21T13:14:45.935 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.935 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.935 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.936 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:54255 9.12 9.40534e52 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-03-21T13:14:45.936 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:09:19.449680+0000", 2024-03-21T13:14:45.936 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 326.20293021100002, 2024-03-21T13:14:45.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98623570800000004, 2024-03-21T13:14:45.936 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.936 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:14:45.936 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.936 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:14:45.936 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:14:45.936 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 54255 2024-03-21T13:14:45.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.937 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449680+0000", 2024-03-21T13:14:45.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449680+0000", 2024-03-21T13:14:45.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449681+0000", 2024-03-21T13:14:45.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2559999999999999e-06 2024-03-21T13:14:45.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449704+0000", 2024-03-21T13:14:45.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2748e-05 2024-03-21T13:14:45.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449719+0000", 2024-03-21T13:14:45.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4584e-05 2024-03-21T13:14:45.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449726+0000", 2024-03-21T13:14:45.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6990000000000003e-06 2024-03-21T13:14:45.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449788+0000", 2024-03-21T13:14:45.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1667000000000004e-05 2024-03-21T13:14:45.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449797+0000", 2024-03-21T13:14:45.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.799e-06 2024-03-21T13:14:45.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:20.435907+0000", 2024-03-21T13:14:45.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98611042400000004 2024-03-21T13:14:45.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:20.435916+0000", 2024-03-21T13:14:45.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5310000000000001e-06 2024-03-21T13:14:45.941 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.941 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.941 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.942 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:100377 9.12 9:4e5aa4e2:::benchmark_data_smithi163_131650_object100376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-03-21T13:14:45.942 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:02.557198+0000", 2024-03-21T13:14:45.942 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 283.09541231100002, 2024-03-21T13:14:45.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1069594089999999, 2024-03-21T13:14:45.942 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.942 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:14:45.942 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.942 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:14:45.942 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:14:45.942 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100377 2024-03-21T13:14:45.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.943 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557198+0000", 2024-03-21T13:14:45.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557198+0000", 2024-03-21T13:14:45.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557199+0000", 2024-03-21T13:14:45.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.344e-06 2024-03-21T13:14:45.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557224+0000", 2024-03-21T13:14:45.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888e-05 2024-03-21T13:14:45.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557238+0000", 2024-03-21T13:14:45.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3759e-05 2024-03-21T13:14:45.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557245+0000", 2024-03-21T13:14:45.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3529999999999997e-06 2024-03-21T13:14:45.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557311+0000", 2024-03-21T13:14:45.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5513999999999996e-05 2024-03-21T13:14:45.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557434+0000", 2024-03-21T13:14:45.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012332599999999999 2024-03-21T13:14:45.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:14:45.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557755+0000", 2024-03-21T13:14:45.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000320718 2024-03-21T13:14:45.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.560432+0000", 2024-03-21T13:14:45.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026767940000000001 2024-03-21T13:14:45.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.561289+0000", 2024-03-21T13:14:45.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00085723799999999997 2024-03-21T13:14:45.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.573020+0000", 2024-03-21T13:14:45.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011730908999999999 2024-03-21T13:14:45.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.573043+0000", 2024-03-21T13:14:45.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3289999999999999e-05 2024-03-21T13:14:45.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584602+0000", 2024-03-21T13:14:45.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0115587029999999 2024-03-21T13:14:45.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584621+0000", 2024-03-21T13:14:45.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9125000000000001e-05 2024-03-21T13:14:45.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.664079+0000", 2024-03-21T13:14:45.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0794580840000001 2024-03-21T13:14:45.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.664157+0000", 2024-03-21T13:14:45.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8363999999999997e-05 2024-03-21T13:14:45.949 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.950 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.950 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.950 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:100402 9.12 9:4d69482a:::benchmark_data_smithi163_131650_object100401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:14:45.950 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:02.595353+0000", 2024-03-21T13:14:45.950 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 283.05725699999999, 2024-03-21T13:14:45.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0785798529999999, 2024-03-21T13:14:45.950 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.950 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.950 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.951 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:14:45.951 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:14:45.951 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100402 2024-03-21T13:14:45.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.951 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595353+0000", 2024-03-21T13:14:45.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595353+0000", 2024-03-21T13:14:45.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595354+0000", 2024-03-21T13:14:45.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6400000000000002e-07 2024-03-21T13:14:45.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595374+0000", 2024-03-21T13:14:45.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0288e-05 2024-03-21T13:14:45.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595381+0000", 2024-03-21T13:14:45.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3690000000000001e-06 2024-03-21T13:14:45.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595388+0000", 2024-03-21T13:14:45.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-03-21T13:14:45.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595445+0000", 2024-03-21T13:14:45.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7123000000000002e-05 2024-03-21T13:14:45.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595461+0000", 2024-03-21T13:14:45.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5758000000000001e-05 2024-03-21T13:14:45.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584764+0000", 2024-03-21T13:14:45.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98930377599999997 2024-03-21T13:14:45.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584774+0000", 2024-03-21T13:14:45.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7000000000000003e-06 2024-03-21T13:14:45.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.664696+0000", 2024-03-21T13:14:45.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.079921769 2024-03-21T13:14:45.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.664847+0000", 2024-03-21T13:14:45.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000151249 2024-03-21T13:14:45.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,3,7", 2024-03-21T13:14:45.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.665217+0000", 2024-03-21T13:14:45.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00037034999999999998 2024-03-21T13:14:45.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.957 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673509+0000", 2024-03-21T13:14:45.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082912909999999992 2024-03-21T13:14:45.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.957 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673550+0000", 2024-03-21T13:14:45.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1844000000000001e-05 2024-03-21T13:14:45.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673597+0000", 2024-03-21T13:14:45.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.651e-05 2024-03-21T13:14:45.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673749+0000", 2024-03-21T13:14:45.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000152082 2024-03-21T13:14:45.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673821+0000", 2024-03-21T13:14:45.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2371000000000002e-05 2024-03-21T13:14:45.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673933+0000", 2024-03-21T13:14:45.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000111629 2024-03-21T13:14:45.959 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.959 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.959 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.960 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:100408 9.12 9:4f7d3001:::benchmark_data_smithi163_131650_object100407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:14:45.960 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:02.605697+0000", 2024-03-21T13:14:45.960 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 283.04691346499999, 2024-03-21T13:14:45.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0683809150000001, 2024-03-21T13:14:45.960 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.960 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.960 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.960 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:14:45.960 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:14:45.960 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100408 2024-03-21T13:14:45.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.960 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605697+0000", 2024-03-21T13:14:45.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605697+0000", 2024-03-21T13:14:45.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605701+0000", 2024-03-21T13:14:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.985e-06 2024-03-21T13:14:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605730+0000", 2024-03-21T13:14:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8864999999999999e-05 2024-03-21T13:14:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605741+0000", 2024-03-21T13:14:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.148e-05 2024-03-21T13:14:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605748+0000", 2024-03-21T13:14:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3839999999999999e-06 2024-03-21T13:14:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605818+0000", 2024-03-21T13:14:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9108000000000003e-05 2024-03-21T13:14:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605829+0000", 2024-03-21T13:14:45.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1874e-05 2024-03-21T13:14:45.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584799+0000", 2024-03-21T13:14:45.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.97896959400000005 2024-03-21T13:14:45.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584808+0000", 2024-03-21T13:14:45.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2320000000000007e-06 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.667194+0000", 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.082386031 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.667343+0000", 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000148672 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,3,7", 2024-03-21T13:14:45.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.667701+0000", 2024-03-21T13:14:45.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035776999999999998 2024-03-21T13:14:45.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673640+0000", 2024-03-21T13:14:45.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.005939405 2024-03-21T13:14:45.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673684+0000", 2024-03-21T13:14:45.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4180000000000001e-05 2024-03-21T13:14:45.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673730+0000", 2024-03-21T13:14:45.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5911000000000001e-05 2024-03-21T13:14:45.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673945+0000", 2024-03-21T13:14:45.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000214606 2024-03-21T13:14:45.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673989+0000", 2024-03-21T13:14:45.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3665000000000003e-05 2024-03-21T13:14:45.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.674078+0000", 2024-03-21T13:14:45.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9153000000000005e-05 2024-03-21T13:14:45.968 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.968 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.968 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.969 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:187739 9.12 9:48cfd7ca:::benchmark_data_smithi163_131650_object66209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-03-21T13:14:45.969 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:11:17.703816+0000", 2024-03-21T13:14:45.969 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 207.94879459200001, 2024-03-21T13:14:45.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98840552800000003, 2024-03-21T13:14:45.969 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:14:45.969 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:14:45.969 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:14:45.969 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:14:45.969 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:14:45.969 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 187739 2024-03-21T13:14:45.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.969 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:14:45.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:14:45.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703816+0000", 2024-03-21T13:14:45.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:14:45.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703816+0000", 2024-03-21T13:14:45.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:14:45.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:14:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703817+0000", 2024-03-21T13:14:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1850000000000001e-06 2024-03-21T13:14:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:14:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703822+0000", 2024-03-21T13:14:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1120000000000004e-06 2024-03-21T13:14:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703836+0000", 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4517e-05 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703846+0000", 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1209999999999996e-06 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703908+0000", 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2137000000000005e-05 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:14:45.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703918+0000", 2024-03-21T13:14:45.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9569999999999997e-06 2024-03-21T13:14:45.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:14:45.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.686943+0000", 2024-03-21T13:14:45.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98302562299999996 2024-03-21T13:14:45.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:14:45.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.687082+0000", 2024-03-21T13:14:45.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000138381 2024-03-21T13:14:45.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,7", 2024-03-21T13:14:45.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.687230+0000", 2024-03-21T13:14:45.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014813700000000001 2024-03-21T13:14:45.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.689821+0000", 2024-03-21T13:14:45.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0025908120000000001 2024-03-21T13:14:45.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:14:45.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.690005+0000", 2024-03-21T13:14:45.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018441599999999999 2024-03-21T13:14:45.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:14:45.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.692114+0000", 2024-03-21T13:14:45.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002109016 2024-03-21T13:14:45.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:14:45.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.692176+0000", 2024-03-21T13:14:45.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2414000000000002e-05 2024-03-21T13:14:45.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:14:45.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:14:45.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:14:45.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.692221+0000", 2024-03-21T13:14:45.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4700000000000002e-05 2024-03-21T13:14:45.976 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.976 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.977 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.977 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:14:45.977 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:14:45.977 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:14:45.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:45.978 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:45 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2868049637' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:46.016 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:46.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:46.118 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:46.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:46.158 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:46.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:46.259 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:46.291 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:46.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:46.393 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:46.429 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:46.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:46.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:46.532 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:46.573 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:46.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:46.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:46.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:46.709 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:46.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:46.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:46 smithi163 ceph-mon[29621]: pgmap v756: 148 pgs: 148 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 607 B/s rd, 0 B/s wr, 868 op/s 2024-03-21T13:14:46.758 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:46 smithi163 ceph-mon[40675]: pgmap v756: 148 pgs: 148 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 607 B/s rd, 0 B/s wr, 868 op/s 2024-03-21T13:14:46.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:46.845 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:46.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:46.946 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:46.984 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:46.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:46.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:47.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:46 smithi204 ceph-mon[33153]: pgmap v756: 148 pgs: 148 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 607 B/s rd, 0 B/s wr, 868 op/s 2024-03-21T13:14:47.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:47.125 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:47.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:47.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:47.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:47.262 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:47.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:47.364 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:47.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:47.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:47.499 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:47.532 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:47.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:47.634 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:47.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:47.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:47.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:47.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:47 smithi163 ceph-mon[40675]: pgmap v757: 148 pgs: 148 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-03-21T13:14:47.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:47 smithi163 ceph-mon[29621]: pgmap v757: 148 pgs: 148 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-03-21T13:14:47.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:47.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:47.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:47.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:47.912 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:47.949 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:47.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:48.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:47 smithi204 ceph-mon[33153]: pgmap v757: 148 pgs: 148 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-03-21T13:14:48.051 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:48.082 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:48.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:48.184 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:48.209 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-21T13:14:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:48.227 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:48.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:48.328 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:48.380 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:48.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:48.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:48.564 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:48.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:48.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:48.666 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:48.700 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:48.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:48.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:48.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:48.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:48.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:48.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:48.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:48.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:48.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:48.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:49.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:49.107 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:49.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:49.209 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:49.243 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:49.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:49.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:49.345 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:49.380 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:49.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:49.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:49.400 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:14:49.482 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:49.527 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:49.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:49.628 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:49.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:49.670 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:49.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:49.772 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:49.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:49.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:49.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:49.910 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:49.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:49.934 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:49.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:50.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:49 smithi163 ceph-mon[29621]: pgmap v758: 148 pgs: 148 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-03-21T13:14:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:49 smithi163 ceph-mon[40675]: pgmap v758: 148 pgs: 148 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-03-21T13:14:50.035 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:50.059 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:50.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:50.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:50.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:50.199 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:50.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:50.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:49 smithi204 ceph-mon[33153]: pgmap v758: 148 pgs: 148 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-03-21T13:14:50.300 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:50.364 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:50.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:50.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:50.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:50.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:50.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:50.601 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:50.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:50.637 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:50.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:50.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:50.643 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:50.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:50.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:50.775 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:50.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:50.876 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:50.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:50.916 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:50.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:51.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:51.055 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:51.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:51.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:51.194 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:51.194 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:51.199 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:51.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:51.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:51.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:51.339 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:51.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:51.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:51.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:51.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:51.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:51.586 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:51.620 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:51.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:51.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:51.687 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 2, 5] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:14:51.687 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:14:51.687 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-21T13:14:51.688 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.989953 2024-03-21T13:14:51.688 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 0 0.9899525306082029 2024-03-21T13:14:51.722 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:51.765 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:51.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:51.866 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:51.908 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:51.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:52.009 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:52.017 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:14:52.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:52.055 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:52.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:52.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:52.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:52.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:52.253 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:14:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:14:52.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:51 smithi204 ceph-mon[33153]: pgmap v759: 148 pgs: 148 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:14:52.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:51 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1420478636' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:14:52.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:51 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:14:52.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:51 smithi163 ceph-mon[29621]: pgmap v759: 148 pgs: 148 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:14:52.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:51 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1420478636' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:14:52.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:51 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:14:52.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:51 smithi163 ceph-mon[40675]: pgmap v759: 148 pgs: 148 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:14:52.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:51 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1420478636' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:14:52.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:51 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:14:52.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:52.334 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:52.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:52.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:52.473 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:52.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:52.575 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:52.605 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:52.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:52.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:52.707 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:52.743 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:52.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:52.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:52.845 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:52.879 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:52.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:52.980 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:53.030 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:53.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:53.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:53.132 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:53.172 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:53.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:53.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:52 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:14:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:52 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:14:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:52 smithi204 ceph-mon[33153]: osdmap e370: 8 total, 8 up, 7 in 2024-03-21T13:14:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:52 smithi204 ceph-mon[33153]: 4.1b scrub starts 2024-03-21T13:14:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:52 smithi204 ceph-mon[33153]: 4.1b scrub ok 2024-03-21T13:14:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:52 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:52 smithi204 ceph-mon[33153]: 4.7 scrub starts 2024-03-21T13:14:53.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:52 smithi204 ceph-mon[33153]: 4.7 scrub ok 2024-03-21T13:14:53.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:14:53.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:14:53.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[29621]: osdmap e370: 8 total, 8 up, 7 in 2024-03-21T13:14:53.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[29621]: 4.1b scrub starts 2024-03-21T13:14:53.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[29621]: 4.1b scrub ok 2024-03-21T13:14:53.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:53.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[29621]: 4.7 scrub starts 2024-03-21T13:14:53.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[29621]: 4.7 scrub ok 2024-03-21T13:14:53.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:14:53.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:14:53.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[40675]: osdmap e370: 8 total, 8 up, 7 in 2024-03-21T13:14:53.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[40675]: 4.1b scrub starts 2024-03-21T13:14:53.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[40675]: 4.1b scrub ok 2024-03-21T13:14:53.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:53.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[40675]: 4.7 scrub starts 2024-03-21T13:14:53.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:52 smithi163 ceph-mon[40675]: 4.7 scrub ok 2024-03-21T13:14:53.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:53.312 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:53.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:53.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:53.414 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:53.451 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:53.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:53.553 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:53.586 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:53.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:53.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:53.688 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:53.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:53.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:53.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:53.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:53.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:53.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:53.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:53.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:53.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:54.100 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:54.140 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:54.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:54.242 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:54.288 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:54.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:54.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:54 smithi163 ceph-mon[29621]: pgmap v761: 148 pgs: 148 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:14:54.331 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:54 smithi163 ceph-mon[29621]: 5.1c scrub starts 2024-03-21T13:14:54.331 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:54 smithi163 ceph-mon[29621]: 5.1c scrub ok 2024-03-21T13:14:54.331 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:54 smithi163 ceph-mon[40675]: pgmap v761: 148 pgs: 148 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:14:54.331 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:54 smithi163 ceph-mon[40675]: 5.1c scrub starts 2024-03-21T13:14:54.331 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:54 smithi163 ceph-mon[40675]: 5.1c scrub ok 2024-03-21T13:14:54.338 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:54.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:54.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:54.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:54.465 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:14:54.496 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:14:54.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:54 smithi204 ceph-mon[33153]: pgmap v761: 148 pgs: 148 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:14:54.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:54 smithi204 ceph-mon[33153]: 5.1c scrub starts 2024-03-21T13:14:54.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:54 smithi204 ceph-mon[33153]: 5.1c scrub ok 2024-03-21T13:14:54.526 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:54.532 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:14:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:54.568 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:54.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:54.574 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:14:54.619 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:14:54.651 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:14:54.670 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:54.690 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:14:54.709 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:54.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:54.736 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:14:54.743 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:14:54.743 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:14:54.743 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:14:54.743 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:14:54.743 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:14:54.743 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:14:54.743 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:14:54.743 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:14:54.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:54.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:54.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:54.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:54.958 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:54.996 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:54.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:55.098 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:55.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:55.147 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:55.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:55.248 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:55.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:55 smithi163 ceph-mon[29621]: 3.f scrub starts 2024-03-21T13:14:55.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:55 smithi163 ceph-mon[29621]: 3.f scrub ok 2024-03-21T13:14:55.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:55 smithi163 ceph-mon[29621]: 6.f scrub starts 2024-03-21T13:14:55.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:55 smithi163 ceph-mon[29621]: 6.f scrub ok 2024-03-21T13:14:55.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:55 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:14:55.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:55 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2761085972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:55.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:55 smithi163 ceph-mon[40675]: 3.f scrub starts 2024-03-21T13:14:55.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:55 smithi163 ceph-mon[40675]: 3.f scrub ok 2024-03-21T13:14:55.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:55 smithi163 ceph-mon[40675]: 6.f scrub starts 2024-03-21T13:14:55.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:55 smithi163 ceph-mon[40675]: 6.f scrub ok 2024-03-21T13:14:55.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:55 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:14:55.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:55 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2761085972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:55.284 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:55.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:55.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:55.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:55.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:55.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:55.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:55 smithi204 ceph-mon[33153]: 3.f scrub starts 2024-03-21T13:14:55.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:55 smithi204 ceph-mon[33153]: 3.f scrub ok 2024-03-21T13:14:55.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:55 smithi204 ceph-mon[33153]: 6.f scrub starts 2024-03-21T13:14:55.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:55 smithi204 ceph-mon[33153]: 6.f scrub ok 2024-03-21T13:14:55.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:55 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:14:55.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:55 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2761085972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:14:55.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:55.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:55.549 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:55.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:55.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:55.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:55.686 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:55.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:55.765 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:14:55.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:55.822 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:55.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:55.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:55.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:55.964 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:55.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:55.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:56.013 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:14:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:14:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:14:56.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:56.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:56.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:56.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:56.216 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:56.253 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:56.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:56.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:56.355 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:56.387 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:56.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:56.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:56.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:56 smithi204 ceph-mon[33153]: 3.1a scrub starts 2024-03-21T13:14:56.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:56 smithi204 ceph-mon[33153]: 3.1a scrub ok 2024-03-21T13:14:56.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:56 smithi204 ceph-mon[33153]: 4.19 scrub starts 2024-03-21T13:14:56.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:56 smithi204 ceph-mon[33153]: 4.19 scrub ok 2024-03-21T13:14:56.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:56 smithi204 ceph-mon[33153]: pgmap v762: 148 pgs: 148 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:14:56.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:56 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:56.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:56 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:56.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[29621]: 3.1a scrub starts 2024-03-21T13:14:56.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[29621]: 3.1a scrub ok 2024-03-21T13:14:56.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[29621]: 4.19 scrub starts 2024-03-21T13:14:56.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[29621]: 4.19 scrub ok 2024-03-21T13:14:56.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[29621]: pgmap v762: 148 pgs: 148 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:14:56.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:56.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:56.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[40675]: 3.1a scrub starts 2024-03-21T13:14:56.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[40675]: 3.1a scrub ok 2024-03-21T13:14:56.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[40675]: 4.19 scrub starts 2024-03-21T13:14:56.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[40675]: 4.19 scrub ok 2024-03-21T13:14:56.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[40675]: pgmap v762: 148 pgs: 148 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:14:56.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:56.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:56 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:56.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:56.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:56.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:56.628 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:56.662 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:56.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:56.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:56.764 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:56.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:56.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:56.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:56.940 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:56.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:56.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:57.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:57.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:57.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:57.178 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:57.202 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:57.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:57.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:57.241 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:14:57.242 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:14:57.304 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:57.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:57.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:57.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:14:57.481 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:57.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:14:57.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:57 smithi163 ceph-mon[29621]: 5.4 scrub starts 2024-03-21T13:14:57.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:57 smithi163 ceph-mon[29621]: 5.4 scrub ok 2024-03-21T13:14:57.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:57 smithi163 ceph-mon[40675]: 5.4 scrub starts 2024-03-21T13:14:57.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:57 smithi163 ceph-mon[40675]: 5.4 scrub ok 2024-03-21T13:14:57.583 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:57.608 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:57.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:57.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:57.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:14:57.744 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:57.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:57.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:14:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:57 smithi204 ceph-mon[33153]: 5.4 scrub starts 2024-03-21T13:14:57.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:57 smithi204 ceph-mon[33153]: 5.4 scrub ok 2024-03-21T13:14:57.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:57.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:57.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:57.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:57.985 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:14:58.013 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:58.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:58.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:14:58.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:14:58.159 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:58.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:58.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:14:58.261 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:58.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:58.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:58.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:58.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:58.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:58.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:58.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:58.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:58.580 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:58.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:58.682 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:58.692 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:14:58.716 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:58.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:58.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:58.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:58 smithi204 ceph-mon[33153]: pgmap v763: 148 pgs: 148 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:14:58.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:58 smithi204 ceph-mon[33153]: 5.17 scrub starts 2024-03-21T13:14:58.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:58 smithi204 ceph-mon[33153]: 3.1 scrub starts 2024-03-21T13:14:58.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:58 smithi204 ceph-mon[33153]: 3.1 scrub ok 2024-03-21T13:14:58.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:58 smithi204 ceph-mon[33153]: 5.17 scrub ok 2024-03-21T13:14:58.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:58 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:14:58.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:58 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:14:58.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[29621]: pgmap v763: 148 pgs: 148 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:14:58.754 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[29621]: 5.17 scrub starts 2024-03-21T13:14:58.754 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[29621]: 3.1 scrub starts 2024-03-21T13:14:58.754 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[29621]: 3.1 scrub ok 2024-03-21T13:14:58.754 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[29621]: 5.17 scrub ok 2024-03-21T13:14:58.754 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:14:58.754 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:14:58.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[40675]: pgmap v763: 148 pgs: 148 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:14:58.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[40675]: 5.17 scrub starts 2024-03-21T13:14:58.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[40675]: 3.1 scrub starts 2024-03-21T13:14:58.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[40675]: 3.1 scrub ok 2024-03-21T13:14:58.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[40675]: 5.17 scrub ok 2024-03-21T13:14:58.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:14:58.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:58 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:14:58.819 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:14:58.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:58.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:14:58.900 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-21T13:14:58.958 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:59.028 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:59.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:59.130 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:59.188 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:59.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:59.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:59.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:59.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:59.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:59.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:59.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:59.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:59.566 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:14:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:59.601 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:14:59.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:14:59.702 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:14:59.738 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:59.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:59.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:14:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:59 smithi204 ceph-mon[33153]: osdmap e371: 8 total, 8 up, 7 in 2024-03-21T13:14:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:59 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:14:59.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:59 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:14:59.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:14:59 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:59.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:59 smithi163 ceph-mon[29621]: osdmap e371: 8 total, 8 up, 7 in 2024-03-21T13:14:59.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:59 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:14:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:59 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:14:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:14:59 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:59 smithi163 ceph-mon[40675]: osdmap e371: 8 total, 8 up, 7 in 2024-03-21T13:14:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:59 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:14:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:59 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:14:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:14:59 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:14:59.840 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:14:59.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:14:59.915 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:14:59.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:14:59.916 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:14:59.975 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:00.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:00.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:00.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:00.155 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:00.195 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:00.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:00.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:00.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:00.334 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:00.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:00.436 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:00.472 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:00.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:00.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:00.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:00.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:00.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:00.714 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:00.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:00.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:00.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:00 smithi204 ceph-mon[33153]: pgmap v765: 148 pgs: 148 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 828 op/s 2024-03-21T13:15:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:00 smithi204 ceph-mon[33153]: 6.13 scrub starts 2024-03-21T13:15:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:00 smithi204 ceph-mon[33153]: 6.1a scrub starts 2024-03-21T13:15:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:00 smithi204 ceph-mon[33153]: 6.1a scrub ok 2024-03-21T13:15:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:00 smithi204 ceph-mon[33153]: 6.13 scrub ok 2024-03-21T13:15:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:00 smithi204 ceph-mon[33153]: 5.1 scrub starts 2024-03-21T13:15:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:00 smithi204 ceph-mon[33153]: 5.1 scrub ok 2024-03-21T13:15:00.760 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:00.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[29621]: pgmap v765: 148 pgs: 148 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 828 op/s 2024-03-21T13:15:00.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[29621]: 6.13 scrub starts 2024-03-21T13:15:00.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[29621]: 6.1a scrub starts 2024-03-21T13:15:00.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[29621]: 6.1a scrub ok 2024-03-21T13:15:00.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[29621]: 6.13 scrub ok 2024-03-21T13:15:00.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[29621]: 5.1 scrub starts 2024-03-21T13:15:00.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[29621]: 5.1 scrub ok 2024-03-21T13:15:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[40675]: pgmap v765: 148 pgs: 148 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 828 op/s 2024-03-21T13:15:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[40675]: 6.13 scrub starts 2024-03-21T13:15:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[40675]: 6.1a scrub starts 2024-03-21T13:15:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[40675]: 6.1a scrub ok 2024-03-21T13:15:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[40675]: 6.13 scrub ok 2024-03-21T13:15:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[40675]: 5.1 scrub starts 2024-03-21T13:15:00.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:00 smithi163 ceph-mon[40675]: 5.1 scrub ok 2024-03-21T13:15:00.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:00.889 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:00.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:00.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:00.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:01.021 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:01.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:01.123 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:01.158 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:01.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:01.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:01.260 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:01.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:01.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:01.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:01.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:01.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:01.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:01.533 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:01.571 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:01.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:01.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:01.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:01.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:01 smithi163 ceph-mon[29621]: 4.1c scrub starts 2024-03-21T13:15:01.725 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:01 smithi163 ceph-mon[29621]: 4.1c scrub ok 2024-03-21T13:15:01.725 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:01 smithi163 ceph-mon[29621]: 5.1e scrub starts 2024-03-21T13:15:01.725 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:01 smithi163 ceph-mon[29621]: 5.1e scrub ok 2024-03-21T13:15:01.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:01 smithi163 ceph-mon[40675]: 4.1c scrub starts 2024-03-21T13:15:01.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:01 smithi163 ceph-mon[40675]: 4.1c scrub ok 2024-03-21T13:15:01.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:01 smithi163 ceph-mon[40675]: 5.1e scrub starts 2024-03-21T13:15:01.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:01 smithi163 ceph-mon[40675]: 5.1e scrub ok 2024-03-21T13:15:01.725 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:01.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:01.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:01.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:01 smithi204 ceph-mon[33153]: 4.1c scrub starts 2024-03-21T13:15:01.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:01 smithi204 ceph-mon[33153]: 4.1c scrub ok 2024-03-21T13:15:01.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:01 smithi204 ceph-mon[33153]: 5.1e scrub starts 2024-03-21T13:15:01.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:01 smithi204 ceph-mon[33153]: 5.1e scrub ok 2024-03-21T13:15:01.827 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:01.859 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:01.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:01.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:01.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:01.997 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:01.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:02.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:02.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:02.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:02.235 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:02.279 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:02.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:02.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:02.381 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:02.419 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:02.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:02.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:02.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:15:02 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:15:02.521 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:02.555 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:02.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:02.656 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:02.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[29621]: pgmap v766: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-03-21T13:15:02.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:15:02.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:15:02.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[29621]: 5.d scrub starts 2024-03-21T13:15:02.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[29621]: 5.d scrub ok 2024-03-21T13:15:02.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[29621]: 10.a scrub starts 2024-03-21T13:15:02.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[29621]: osdmap e372: 8 total, 8 up, 7 in 2024-03-21T13:15:02.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/423004821' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.9899525306082029}]: dispatch 2024-03-21T13:15:02.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.9899525306082029}]: dispatch 2024-03-21T13:15:02.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[40675]: pgmap v766: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-03-21T13:15:02.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:15:02.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:15:02.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[40675]: 5.d scrub starts 2024-03-21T13:15:02.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[40675]: 5.d scrub ok 2024-03-21T13:15:02.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[40675]: 10.a scrub starts 2024-03-21T13:15:02.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[40675]: osdmap e372: 8 total, 8 up, 7 in 2024-03-21T13:15:02.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/423004821' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.9899525306082029}]: dispatch 2024-03-21T13:15:02.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:02 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.9899525306082029}]: dispatch 2024-03-21T13:15:02.691 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:02.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:02 smithi204 ceph-mon[33153]: pgmap v766: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-03-21T13:15:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:02 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:15:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:02 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:15:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:02 smithi204 ceph-mon[33153]: 5.d scrub starts 2024-03-21T13:15:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:02 smithi204 ceph-mon[33153]: 5.d scrub ok 2024-03-21T13:15:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:02 smithi204 ceph-mon[33153]: 10.a scrub starts 2024-03-21T13:15:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:02 smithi204 ceph-mon[33153]: osdmap e372: 8 total, 8 up, 7 in 2024-03-21T13:15:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:02 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/423004821' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.9899525306082029}]: dispatch 2024-03-21T13:15:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:02 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.9899525306082029}]: dispatch 2024-03-21T13:15:02.793 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:02.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:02.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:02.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:02.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:02.972 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:02.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:03.074 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:03.104 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:03.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:03.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:03.246 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:03.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:03.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:03.347 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:03.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:03.385 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:03.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:03.470 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.0 primary-affinity to 0.989953 (8648772) 2024-03-21T13:15:03.486 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:03.520 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:03.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:03.622 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:03.666 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:03.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:03.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:03.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:03 smithi204 ceph-mon[33153]: 6.1b scrub starts 2024-03-21T13:15:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:03 smithi204 ceph-mon[33153]: 6.1b scrub ok 2024-03-21T13:15:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:03 smithi204 ceph-mon[33153]: 3.a scrub starts 2024-03-21T13:15:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:03 smithi204 ceph-mon[33153]: 3.a scrub ok 2024-03-21T13:15:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:03 smithi204 ceph-mon[33153]: 10.a scrub ok 2024-03-21T13:15:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:03 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.9899525306082029}]': finished 2024-03-21T13:15:03.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:03 smithi204 ceph-mon[33153]: osdmap e373: 8 total, 8 up, 7 in 2024-03-21T13:15:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[40675]: 6.1b scrub starts 2024-03-21T13:15:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[40675]: 6.1b scrub ok 2024-03-21T13:15:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[40675]: 3.a scrub starts 2024-03-21T13:15:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[40675]: 3.a scrub ok 2024-03-21T13:15:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[40675]: 10.a scrub ok 2024-03-21T13:15:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.9899525306082029}]': finished 2024-03-21T13:15:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[40675]: osdmap e373: 8 total, 8 up, 7 in 2024-03-21T13:15:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:03.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[29621]: 6.1b scrub starts 2024-03-21T13:15:03.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[29621]: 6.1b scrub ok 2024-03-21T13:15:03.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[29621]: 3.a scrub starts 2024-03-21T13:15:03.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[29621]: 3.a scrub ok 2024-03-21T13:15:03.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[29621]: 10.a scrub ok 2024-03-21T13:15:03.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.9899525306082029}]': finished 2024-03-21T13:15:03.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:03 smithi163 ceph-mon[29621]: osdmap e373: 8 total, 8 up, 7 in 2024-03-21T13:15:03.768 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:03.792 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:03.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:03.894 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:03.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:03.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:03.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:04.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:04.027 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:15:04.043 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:04.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:04.045 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:15:04.064 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:15:04.084 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:15:04.104 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:15:04.123 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:15:04.143 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:15:04.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:04.163 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:15:04.164 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:15:04.164 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:15:04.164 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:15:04.164 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:15:04.165 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:15:04.165 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:15:04.165 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:15:04.165 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:15:04.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:04.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:04.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:04.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:04.294 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:04.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:04.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:04.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:04.420 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:04.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:04.522 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:04.559 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:04.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:04.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:04.661 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:04.700 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:04.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:04.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:04 smithi163 ceph-mon[29621]: pgmap v768: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-03-21T13:15:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:04 smithi163 ceph-mon[29621]: 2.1 scrub starts 2024-03-21T13:15:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:04 smithi163 ceph-mon[29621]: 2.1 scrub ok 2024-03-21T13:15:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:04 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2689088250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:04.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:04 smithi163 ceph-mon[29621]: osdmap e374: 8 total, 8 up, 7 in 2024-03-21T13:15:04.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:04 smithi163 ceph-mon[40675]: pgmap v768: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-03-21T13:15:04.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:04 smithi163 ceph-mon[40675]: 2.1 scrub starts 2024-03-21T13:15:04.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:04 smithi163 ceph-mon[40675]: 2.1 scrub ok 2024-03-21T13:15:04.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:04 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2689088250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:04.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:04 smithi163 ceph-mon[40675]: osdmap e374: 8 total, 8 up, 7 in 2024-03-21T13:15:04.802 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:04.837 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:04.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:04.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:04.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:04.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:04.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:04 smithi204 ceph-mon[33153]: pgmap v768: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-03-21T13:15:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:04 smithi204 ceph-mon[33153]: 2.1 scrub starts 2024-03-21T13:15:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:04 smithi204 ceph-mon[33153]: 2.1 scrub ok 2024-03-21T13:15:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:04 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2689088250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:04 smithi204 ceph-mon[33153]: osdmap e374: 8 total, 8 up, 7 in 2024-03-21T13:15:05.081 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:05.117 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:05.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:05.219 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:05.255 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:05.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:05.357 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:05.394 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:05.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:05.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:05.495 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:05.533 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:05.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:05.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:05.635 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:05.672 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:05.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:05.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:05.774 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:05.809 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:05.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:05.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:05.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:05.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:05 smithi204 ceph-mon[33153]: 4.c scrub starts 2024-03-21T13:15:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:05 smithi204 ceph-mon[33153]: 4.c scrub ok 2024-03-21T13:15:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:05 smithi204 ceph-mon[33153]: 6.7 scrub starts 2024-03-21T13:15:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:05 smithi204 ceph-mon[33153]: 6.7 scrub ok 2024-03-21T13:15:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:05 smithi204 ceph-mon[33153]: 4.1e scrub starts 2024-03-21T13:15:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:05 smithi204 ceph-mon[33153]: 4.1e scrub ok 2024-03-21T13:15:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[29621]: 4.c scrub starts 2024-03-21T13:15:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[29621]: 4.c scrub ok 2024-03-21T13:15:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[29621]: 6.7 scrub starts 2024-03-21T13:15:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[29621]: 6.7 scrub ok 2024-03-21T13:15:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[29621]: 4.1e scrub starts 2024-03-21T13:15:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[29621]: 4.1e scrub ok 2024-03-21T13:15:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:06.012 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:15:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:15:06.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[40675]: 4.c scrub starts 2024-03-21T13:15:06.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[40675]: 4.c scrub ok 2024-03-21T13:15:06.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[40675]: 6.7 scrub starts 2024-03-21T13:15:06.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[40675]: 6.7 scrub ok 2024-03-21T13:15:06.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[40675]: 4.1e scrub starts 2024-03-21T13:15:06.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[40675]: 4.1e scrub ok 2024-03-21T13:15:06.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:06.046 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:06.085 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:06.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:06.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:06.187 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:06.220 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:06.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:06.322 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:06.359 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:06.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:06.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:06.503 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:06.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:06.545 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:15:06.604 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:06.636 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:06.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:06.738 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:06 smithi163 ceph-mon[29621]: pgmap v771: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-21T13:15:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:06 smithi163 ceph-mon[29621]: 6.11 scrub starts 2024-03-21T13:15:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:06 smithi163 ceph-mon[29621]: 6.11 scrub ok 2024-03-21T13:15:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:06 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3979494252' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:15:06.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:06 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:15:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:06 smithi163 ceph-mon[40675]: pgmap v771: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-21T13:15:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:06 smithi163 ceph-mon[40675]: 6.11 scrub starts 2024-03-21T13:15:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:06 smithi163 ceph-mon[40675]: 6.11 scrub ok 2024-03-21T13:15:06.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:06 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3979494252' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:15:06.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:06 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:15:06.772 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:06.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:06.873 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:06.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:06.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:06 smithi204 ceph-mon[33153]: pgmap v771: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-21T13:15:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:06 smithi204 ceph-mon[33153]: 6.11 scrub starts 2024-03-21T13:15:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:06 smithi204 ceph-mon[33153]: 6.11 scrub ok 2024-03-21T13:15:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:06 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3979494252' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:15:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:06 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:15:07.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:07.046 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:07.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:07.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:07.148 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:07.187 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:07.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:07.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:07.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:07.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:07.422 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:07.469 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:07.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:07.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:07.571 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:07.583 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:15:07.583 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:15:07.583 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-03-21T13:15:07.583 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-03-21T13:15:07.583 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:15:07.608 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:07.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:07.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:07.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:07.743 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:07.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:07.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[29621]: 6.a scrub starts 2024-03-21T13:15:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[29621]: 6.a scrub ok 2024-03-21T13:15:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:15:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:15:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:15:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[29621]: osdmap e375: 8 total, 8 up, 7 in 2024-03-21T13:15:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[29621]: 6.3 scrub starts 2024-03-21T13:15:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[29621]: 6.3 scrub ok 2024-03-21T13:15:07.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[29621]: pgmap v773: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 456 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-21T13:15:07.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[40675]: 6.a scrub starts 2024-03-21T13:15:07.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[40675]: 6.a scrub ok 2024-03-21T13:15:07.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:15:07.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:15:07.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:15:07.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[40675]: osdmap e375: 8 total, 8 up, 7 in 2024-03-21T13:15:07.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[40675]: 6.3 scrub starts 2024-03-21T13:15:07.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[40675]: 6.3 scrub ok 2024-03-21T13:15:07.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:07 smithi163 ceph-mon[40675]: pgmap v773: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 456 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-21T13:15:07.845 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:07.880 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:07.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:07.982 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:07 smithi204 ceph-mon[33153]: 6.a scrub starts 2024-03-21T13:15:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:07 smithi204 ceph-mon[33153]: 6.a scrub ok 2024-03-21T13:15:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:07 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:15:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:07 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:15:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:07 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:15:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:07 smithi204 ceph-mon[33153]: osdmap e375: 8 total, 8 up, 7 in 2024-03-21T13:15:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:07 smithi204 ceph-mon[33153]: 6.3 scrub starts 2024-03-21T13:15:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:07 smithi204 ceph-mon[33153]: 6.3 scrub ok 2024-03-21T13:15:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:07 smithi204 ceph-mon[33153]: pgmap v773: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 456 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-21T13:15:08.016 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:08.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:08.118 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:08.156 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:08.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:08.258 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:08.295 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:08.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:08.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:08.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:08.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:08.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:08.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:08.532 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:15:08.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:08.568 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:08.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:08.670 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:08.707 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:08.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:08 smithi163 ceph-mon[29621]: 5.f scrub starts 2024-03-21T13:15:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:08 smithi163 ceph-mon[29621]: 5.f scrub ok 2024-03-21T13:15:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:08 smithi163 ceph-mon[40675]: 5.f scrub starts 2024-03-21T13:15:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:08 smithi163 ceph-mon[40675]: 5.f scrub ok 2024-03-21T13:15:08.809 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:08.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:08.838 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:08.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:08.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:08.973 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:08.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:08 smithi204 ceph-mon[33153]: 5.f scrub starts 2024-03-21T13:15:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:08 smithi204 ceph-mon[33153]: 5.f scrub ok 2024-03-21T13:15:09.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:09.108 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:09.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:09.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:09.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:09.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:09.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:09.350 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:09.380 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:09.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:09.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:09.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:09.515 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:09.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:09.617 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:09.650 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:09.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:09.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:09.752 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:09.788 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:09.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:09.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:09.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:09 smithi163 ceph-mon[29621]: pgmap v774: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 840 op/s 2024-03-21T13:15:09.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:09 smithi163 ceph-mon[29621]: 4.0 scrub starts 2024-03-21T13:15:09.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:09 smithi163 ceph-mon[29621]: 4.0 scrub ok 2024-03-21T13:15:09.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:09 smithi163 ceph-mon[40675]: pgmap v774: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 840 op/s 2024-03-21T13:15:09.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:09 smithi163 ceph-mon[40675]: 4.0 scrub starts 2024-03-21T13:15:09.837 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:09 smithi163 ceph-mon[40675]: 4.0 scrub ok 2024-03-21T13:15:09.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:09.894 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-21T13:15:09.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:09.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:09.968 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:10.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:09 smithi204 ceph-mon[33153]: pgmap v774: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 840 op/s 2024-03-21T13:15:10.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:09 smithi204 ceph-mon[33153]: 4.0 scrub starts 2024-03-21T13:15:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:09 smithi204 ceph-mon[33153]: 4.0 scrub ok 2024-03-21T13:15:10.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:10.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:10.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:10.160 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:10.206 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:10.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:10.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:10.347 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:10.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:10.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:10.407 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:15:10.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:10.481 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:10.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:10.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:10.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:10.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:10.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:10.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:10.723 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:10 smithi163 ceph-mon[29621]: 3.4 scrub starts 2024-03-21T13:15:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:10 smithi163 ceph-mon[29621]: 3.4 scrub ok 2024-03-21T13:15:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:10 smithi163 ceph-mon[29621]: 4.1f deep-scrub starts 2024-03-21T13:15:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:10 smithi163 ceph-mon[29621]: 4.1f deep-scrub ok 2024-03-21T13:15:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:10 smithi163 ceph-mon[29621]: 6.2 scrub starts 2024-03-21T13:15:10.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:10 smithi163 ceph-mon[29621]: 6.2 scrub ok 2024-03-21T13:15:10.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:10 smithi163 ceph-mon[40675]: 3.4 scrub starts 2024-03-21T13:15:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:10 smithi163 ceph-mon[40675]: 3.4 scrub ok 2024-03-21T13:15:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:10 smithi163 ceph-mon[40675]: 4.1f deep-scrub starts 2024-03-21T13:15:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:10 smithi163 ceph-mon[40675]: 4.1f deep-scrub ok 2024-03-21T13:15:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:10 smithi163 ceph-mon[40675]: 6.2 scrub starts 2024-03-21T13:15:10.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:10 smithi163 ceph-mon[40675]: 6.2 scrub ok 2024-03-21T13:15:10.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:10.764 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:10.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:10.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:10.903 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:10.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:10 smithi204 ceph-mon[33153]: 3.4 scrub starts 2024-03-21T13:15:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:10 smithi204 ceph-mon[33153]: 3.4 scrub ok 2024-03-21T13:15:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:10 smithi204 ceph-mon[33153]: 4.1f deep-scrub starts 2024-03-21T13:15:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:10 smithi204 ceph-mon[33153]: 4.1f deep-scrub ok 2024-03-21T13:15:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:10 smithi204 ceph-mon[33153]: 6.2 scrub starts 2024-03-21T13:15:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:10 smithi204 ceph-mon[33153]: 6.2 scrub ok 2024-03-21T13:15:11.004 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:11.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:11.044 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:11.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:11.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:11.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:11.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:11.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:11.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:11.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:11.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:11.420 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:11.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:11.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:11.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:11.566 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:11.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:11.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:11.612 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:11.696 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:11.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:11 smithi163 ceph-mon[29621]: 3.13 scrub starts 2024-03-21T13:15:11.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:11 smithi163 ceph-mon[29621]: 3.13 scrub ok 2024-03-21T13:15:11.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:11 smithi163 ceph-mon[29621]: 4.13 scrub starts 2024-03-21T13:15:11.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:11 smithi163 ceph-mon[29621]: 4.13 scrub ok 2024-03-21T13:15:11.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:11 smithi163 ceph-mon[29621]: pgmap v775: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 957 B/s rd, 0 B/s wr, 1.54k op/s 2024-03-21T13:15:11.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:11 smithi163 ceph-mon[40675]: 3.13 scrub starts 2024-03-21T13:15:11.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:11 smithi163 ceph-mon[40675]: 3.13 scrub ok 2024-03-21T13:15:11.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:11 smithi163 ceph-mon[40675]: 4.13 scrub starts 2024-03-21T13:15:11.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:11 smithi163 ceph-mon[40675]: 4.13 scrub ok 2024-03-21T13:15:11.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:11 smithi163 ceph-mon[40675]: pgmap v775: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 957 B/s rd, 0 B/s wr, 1.54k op/s 2024-03-21T13:15:11.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:11.729 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:11.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:11.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:11.860 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:11.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:11.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:11.962 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:11.985 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 2, 5] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:15:11.985 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-21T13:15:11.999 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:12.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:12.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:11 smithi204 ceph-mon[33153]: 3.13 scrub starts 2024-03-21T13:15:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:11 smithi204 ceph-mon[33153]: 3.13 scrub ok 2024-03-21T13:15:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:11 smithi204 ceph-mon[33153]: 4.13 scrub starts 2024-03-21T13:15:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:11 smithi204 ceph-mon[33153]: 4.13 scrub ok 2024-03-21T13:15:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:11 smithi204 ceph-mon[33153]: pgmap v775: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 957 B/s rd, 0 B/s wr, 1.54k op/s 2024-03-21T13:15:12.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:12.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:12.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:12.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:12.276 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:12.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:12.378 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:12.410 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:12.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:12.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:12.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:15:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:15:12.511 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:12.540 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:12.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:12.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:12.642 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:12.673 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:12.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:12.775 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:12.807 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:12.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:12.896 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:12 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:12.896 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:12 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:12.908 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:12.943 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:12.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:13.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:12 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:13.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:13.089 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:13.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:13.189 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:13.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:13.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:13.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:13.370 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:13.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:13.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:13.511 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:13.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:13.612 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:13.645 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:13.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:13.747 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:13.748 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:15:13.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[29621]: 5.1b scrub starts 2024-03-21T13:15:13.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[29621]: 5.1b scrub ok 2024-03-21T13:15:13.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[29621]: 6.1c scrub starts 2024-03-21T13:15:13.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[29621]: 6.1c scrub ok 2024-03-21T13:15:13.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[29621]: 10.7 scrub starts 2024-03-21T13:15:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[29621]: 6.9 scrub starts 2024-03-21T13:15:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[29621]: 6.9 scrub ok 2024-03-21T13:15:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[29621]: pgmap v776: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 844 B/s rd, 0 B/s wr, 1.36k op/s 2024-03-21T13:15:13.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[40675]: 5.1b scrub starts 2024-03-21T13:15:13.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[40675]: 5.1b scrub ok 2024-03-21T13:15:13.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[40675]: 6.1c scrub starts 2024-03-21T13:15:13.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[40675]: 6.1c scrub ok 2024-03-21T13:15:13.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[40675]: 10.7 scrub starts 2024-03-21T13:15:13.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[40675]: 6.9 scrub starts 2024-03-21T13:15:13.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[40675]: 6.9 scrub ok 2024-03-21T13:15:13.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:13 smithi163 ceph-mon[40675]: pgmap v776: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 844 B/s rd, 0 B/s wr, 1.36k op/s 2024-03-21T13:15:13.782 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:15:13.784 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:13.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:13.816 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:15:13.853 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:15:13.885 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:13.889 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:15:13.921 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:13.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:13.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:13.923 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:15:13.962 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:15:14.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:13 smithi204 ceph-mon[33153]: 5.1b scrub starts 2024-03-21T13:15:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:13 smithi204 ceph-mon[33153]: 5.1b scrub ok 2024-03-21T13:15:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:13 smithi204 ceph-mon[33153]: 6.1c scrub starts 2024-03-21T13:15:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:13 smithi204 ceph-mon[33153]: 6.1c scrub ok 2024-03-21T13:15:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:13 smithi204 ceph-mon[33153]: 10.7 scrub starts 2024-03-21T13:15:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:13 smithi204 ceph-mon[33153]: 6.9 scrub starts 2024-03-21T13:15:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:13 smithi204 ceph-mon[33153]: 6.9 scrub ok 2024-03-21T13:15:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:13 smithi204 ceph-mon[33153]: pgmap v776: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 844 B/s rd, 0 B/s wr, 1.36k op/s 2024-03-21T13:15:14.003 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:15:14.009 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:15:14.009 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:15:14.009 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:15:14.009 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:15:14.009 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:15:14.009 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:15:14.009 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:15:14.009 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:15:14.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:14.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:14.063 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:14.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:14.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:14.197 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:14.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:14.299 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:14.345 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:14.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:14.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:14.447 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:14.481 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:14.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:14.502 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:15:14 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=sqlstore.transactions t=2024-03-21T13:15:14.412223827Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-21T13:15:14.583 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:14.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:14.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:14.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:14.722 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:14.756 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:14.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:14.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:14.859 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:14.898 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:14.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:14.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:14 smithi163 ceph-mon[29621]: 5.a scrub starts 2024-03-21T13:15:14.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:14 smithi163 ceph-mon[29621]: 5.a scrub ok 2024-03-21T13:15:14.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:14 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2687616505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:14.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:14 smithi163 ceph-mon[29621]: 10.7 scrub ok 2024-03-21T13:15:14.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:14 smithi163 ceph-mon[29621]: 6.c deep-scrub starts 2024-03-21T13:15:14.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:14 smithi163 ceph-mon[29621]: 6.c deep-scrub ok 2024-03-21T13:15:14.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:14 smithi163 ceph-mon[40675]: 5.a scrub starts 2024-03-21T13:15:14.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:14 smithi163 ceph-mon[40675]: 5.a scrub ok 2024-03-21T13:15:14.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:14 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2687616505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:14.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:14 smithi163 ceph-mon[40675]: 10.7 scrub ok 2024-03-21T13:15:14.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:14 smithi163 ceph-mon[40675]: 6.c deep-scrub starts 2024-03-21T13:15:14.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:14 smithi163 ceph-mon[40675]: 6.c deep-scrub ok 2024-03-21T13:15:14.999 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:14 smithi204 ceph-mon[33153]: 5.a scrub starts 2024-03-21T13:15:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:14 smithi204 ceph-mon[33153]: 5.a scrub ok 2024-03-21T13:15:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:14 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2687616505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:14 smithi204 ceph-mon[33153]: 10.7 scrub ok 2024-03-21T13:15:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:14 smithi204 ceph-mon[33153]: 6.c deep-scrub starts 2024-03-21T13:15:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:14 smithi204 ceph-mon[33153]: 6.c deep-scrub ok 2024-03-21T13:15:15.038 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:15.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:15.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:15.140 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:15.173 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:15.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:15.275 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:15.298 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:15.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:15.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:15.439 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:15.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:15.540 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:15.578 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:15.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:15.680 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:15.710 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:15.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:15.812 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:15.846 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:15.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:15.894 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:15 smithi163 ceph-mon[29621]: 10.4 scrub starts 2024-03-21T13:15:15.894 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:15 smithi163 ceph-mon[29621]: pgmap v777: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 716 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T13:15:15.894 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:15:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:15:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:15:15.895 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:15 smithi163 ceph-mon[40675]: 10.4 scrub starts 2024-03-21T13:15:15.895 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:15 smithi163 ceph-mon[40675]: pgmap v777: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 716 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T13:15:15.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:15.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:15.992 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:15.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:16.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:15 smithi204 ceph-mon[33153]: 10.4 scrub starts 2024-03-21T13:15:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:15 smithi204 ceph-mon[33153]: pgmap v777: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 716 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T13:15:16.093 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:16.133 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:16.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:16.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:16.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:16.270 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:16.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:16.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:16.409 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:16.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:16.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:16.550 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:16.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:16.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:16.652 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:16.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:16.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:16.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:16.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:16.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:16.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:16.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:16.925 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:16.956 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:16.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:17.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:16 smithi204 ceph-mon[33153]: 5.e scrub starts 2024-03-21T13:15:17.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:16 smithi204 ceph-mon[33153]: 5.e scrub ok 2024-03-21T13:15:17.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:16 smithi204 ceph-mon[33153]: 10.4 scrub ok 2024-03-21T13:15:17.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:16 smithi204 ceph-mon[33153]: 4.14 scrub starts 2024-03-21T13:15:17.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:16 smithi204 ceph-mon[33153]: 4.14 scrub ok 2024-03-21T13:15:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:16 smithi163 ceph-mon[29621]: 5.e scrub starts 2024-03-21T13:15:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:16 smithi163 ceph-mon[29621]: 5.e scrub ok 2024-03-21T13:15:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:16 smithi163 ceph-mon[29621]: 10.4 scrub ok 2024-03-21T13:15:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:16 smithi163 ceph-mon[29621]: 4.14 scrub starts 2024-03-21T13:15:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:16 smithi163 ceph-mon[29621]: 4.14 scrub ok 2024-03-21T13:15:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:16 smithi163 ceph-mon[40675]: 5.e scrub starts 2024-03-21T13:15:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:16 smithi163 ceph-mon[40675]: 5.e scrub ok 2024-03-21T13:15:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:16 smithi163 ceph-mon[40675]: 10.4 scrub ok 2024-03-21T13:15:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:16 smithi163 ceph-mon[40675]: 4.14 scrub starts 2024-03-21T13:15:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:16 smithi163 ceph-mon[40675]: 4.14 scrub ok 2024-03-21T13:15:17.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:17.095 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:17.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:17.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:17.231 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:17.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:17.333 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:17.385 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:17.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:17.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:17.489 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:17.530 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:17.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:17.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:17.631 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:17.670 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:17.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:17.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:17.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:17.808 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:17.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:17.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:15:17.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:15:17.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[29621]: 5.b scrub starts 2024-03-21T13:15:17.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[29621]: 5.b scrub ok 2024-03-21T13:15:17.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[29621]: osdmap e376: 8 total, 8 up, 7 in 2024-03-21T13:15:17.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[29621]: 3.1b scrub starts 2024-03-21T13:15:17.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[29621]: 3.1b scrub ok 2024-03-21T13:15:17.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[29621]: pgmap v779: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-03-21T13:15:17.899 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:15:17.899 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:15:17.899 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[40675]: 5.b scrub starts 2024-03-21T13:15:17.899 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[40675]: 5.b scrub ok 2024-03-21T13:15:17.899 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[40675]: osdmap e376: 8 total, 8 up, 7 in 2024-03-21T13:15:17.899 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[40675]: 3.1b scrub starts 2024-03-21T13:15:17.899 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[40675]: 3.1b scrub ok 2024-03-21T13:15:17.899 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:17 smithi163 ceph-mon[40675]: pgmap v779: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-03-21T13:15:17.910 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:17.949 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:17.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:17 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:15:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:17 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:15:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:17 smithi204 ceph-mon[33153]: 5.b scrub starts 2024-03-21T13:15:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:17 smithi204 ceph-mon[33153]: 5.b scrub ok 2024-03-21T13:15:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:17 smithi204 ceph-mon[33153]: osdmap e376: 8 total, 8 up, 7 in 2024-03-21T13:15:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:17 smithi204 ceph-mon[33153]: 3.1b scrub starts 2024-03-21T13:15:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:17 smithi204 ceph-mon[33153]: 3.1b scrub ok 2024-03-21T13:15:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:17 smithi204 ceph-mon[33153]: pgmap v779: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-03-21T13:15:18.051 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:18.083 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:18.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:18.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:18.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:18.220 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:18.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:18.322 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:18.352 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:18.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:18.453 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:18.486 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:18.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:18.586 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:18.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:18.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:18.666 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:15:18.724 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:18.780 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:18.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:18.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:18.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:18.918 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:18.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:18.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:19.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:18 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1483501200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:15:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:18 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:15:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:18 smithi204 ceph-mon[33153]: 5.13 scrub starts 2024-03-21T13:15:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:18 smithi204 ceph-mon[33153]: 5.13 scrub ok 2024-03-21T13:15:19.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:18 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1483501200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:15:19.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:18 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:15:19.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:18 smithi163 ceph-mon[29621]: 5.13 scrub starts 2024-03-21T13:15:19.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:18 smithi163 ceph-mon[29621]: 5.13 scrub ok 2024-03-21T13:15:19.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:18 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1483501200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:15:19.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:18 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:15:19.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:18 smithi163 ceph-mon[40675]: 5.13 scrub starts 2024-03-21T13:15:19.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:18 smithi163 ceph-mon[40675]: 5.13 scrub ok 2024-03-21T13:15:19.020 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:19.055 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:19.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:19.157 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:19.200 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:19.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:19.264 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:15:19.264 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:15:19.302 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:19.342 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:19.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:19.444 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:19.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:19.479 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:19.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:19.580 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:19.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:19.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:19.714 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:19.746 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:19.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:19.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:19.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:19.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:19.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:19.951 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:15:19.968 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[29621]: 3.1d scrub starts 2024-03-21T13:15:19.968 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[29621]: 3.1d scrub ok 2024-03-21T13:15:19.968 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[29621]: 5.8 scrub starts 2024-03-21T13:15:19.968 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[29621]: 5.8 scrub ok 2024-03-21T13:15:19.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:15:19.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:15:19.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[29621]: osdmap e377: 8 total, 8 up, 7 in 2024-03-21T13:15:19.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[29621]: pgmap v781: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 762 op/s 2024-03-21T13:15:19.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:15:19.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:15:19.969 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[40675]: 3.1d scrub starts 2024-03-21T13:15:19.969 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[40675]: 3.1d scrub ok 2024-03-21T13:15:19.969 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[40675]: 5.8 scrub starts 2024-03-21T13:15:19.969 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[40675]: 5.8 scrub ok 2024-03-21T13:15:19.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:15:19.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:15:19.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[40675]: osdmap e377: 8 total, 8 up, 7 in 2024-03-21T13:15:19.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[40675]: pgmap v781: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 762 op/s 2024-03-21T13:15:19.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:15:19.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:19 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:15:19.980 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:19.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:19 smithi204 ceph-mon[33153]: 3.1d scrub starts 2024-03-21T13:15:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:19 smithi204 ceph-mon[33153]: 3.1d scrub ok 2024-03-21T13:15:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:19 smithi204 ceph-mon[33153]: 5.8 scrub starts 2024-03-21T13:15:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:19 smithi204 ceph-mon[33153]: 5.8 scrub ok 2024-03-21T13:15:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:19 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:15:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:19 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:15:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:19 smithi204 ceph-mon[33153]: osdmap e377: 8 total, 8 up, 7 in 2024-03-21T13:15:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:19 smithi204 ceph-mon[33153]: pgmap v781: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 762 op/s 2024-03-21T13:15:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:19 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:15:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:19 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:15:20.012 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:20.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:20.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:20.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:20.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:20.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:20.253 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:20.286 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:20.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:20.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:20.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:20.423 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:20.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:20.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:20.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:20.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:20.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:20.587 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-21T13:15:20.666 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:20.708 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:20.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:20.810 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:20.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:20.846 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:20.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:20.871 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:20 smithi163 ceph-mon[29621]: osdmap e378: 8 total, 8 up, 7 in 2024-03-21T13:15:20.871 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:20.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:20 smithi163 ceph-mon[40675]: osdmap e378: 8 total, 8 up, 7 in 2024-03-21T13:15:20.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:20.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:20.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:20.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:21.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:20 smithi204 ceph-mon[33153]: osdmap e378: 8 total, 8 up, 7 in 2024-03-21T13:15:21.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:21.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:21.117 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:21.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:21.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:21.249 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:21.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:21.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:21.386 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:21.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:21.435 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:21.487 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:21.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:21.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:21.628 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:21.661 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:21.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:21.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:21.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:21.800 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:21.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:21.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:21.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:21.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:21.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:21.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:21.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:21 smithi163 ceph-mon[29621]: pgmap v783: 148 pgs: 148 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.97k op/s 2024-03-21T13:15:21.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:21 smithi163 ceph-mon[40675]: pgmap v783: 148 pgs: 148 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.97k op/s 2024-03-21T13:15:22.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:21 smithi204 ceph-mon[33153]: pgmap v783: 148 pgs: 148 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.97k op/s 2024-03-21T13:15:22.038 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:15:22.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:22.123 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:22.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:22.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:22.225 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:22.272 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:22.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:22.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:22.374 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:22.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:22.410 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:22.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:22.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:15:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:15:22.512 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:22.545 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:22.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:22.646 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:22.680 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:22.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:22.782 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:22.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:22.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:22.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:22.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:22.953 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:22.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:23.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:22 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:15:23.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:22 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:15:23.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:22 smithi204 ceph-mon[33153]: osdmap e379: 8 total, 8 up, 7 in 2024-03-21T13:15:23.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:22 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:23.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:22 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:15:23.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:22 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:15:23.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:22 smithi163 ceph-mon[29621]: osdmap e379: 8 total, 8 up, 7 in 2024-03-21T13:15:23.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:22 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:23.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:22 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:15:23.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:22 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:15:23.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:22 smithi163 ceph-mon[40675]: osdmap e379: 8 total, 8 up, 7 in 2024-03-21T13:15:23.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:22 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:23.056 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:23.090 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:23.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:23.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:23.192 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:23.228 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:23.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:23.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:23.365 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:23.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:23.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:23.503 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:23.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:23.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:23.605 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:23.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:23.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:23.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:23.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:23.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:23 smithi163 ceph-mon[29621]: pgmap v785: 148 pgs: 148 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-21T13:15:23.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:23 smithi163 ceph-mon[40675]: pgmap v785: 148 pgs: 148 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-21T13:15:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:23.778 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:23.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:23.879 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:23.916 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:23.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:23 smithi204 ceph-mon[33153]: pgmap v785: 148 pgs: 148 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-21T13:15:24.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:24.052 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:24.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:24.154 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:24.187 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:24.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:24.288 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:24.386 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:24.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:24.487 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:24.523 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:24.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:24.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:24.662 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:24.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:24.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:24.764 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:24.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:24.800 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:24.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:24.901 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:24.938 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:24.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:24.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:25.039 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:25.060 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:15:25.061 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:15:25.061 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:15:25.061 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:15:25.065 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:15:25.075 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:25.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:25.096 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:15:25.096 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:24 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4197997109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:25.097 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:24 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4197997109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:25.135 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:15:25.176 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:25.178 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:15:25.188 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:15:25.188 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:15:25.189 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:15:25.189 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:15:25.189 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:15:25.189 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:15:25.189 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:15:25.189 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:15:25.215 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:25.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:25.237 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:24 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4197997109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:25.317 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:25.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:25.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:25.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:25.492 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:25.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:25.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:25.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:25.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:25.635 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:25.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:25.736 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:25.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:25.772 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:25.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:25.873 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:25.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:25.913 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:25.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:25.941 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:15:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:15:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:15:26.015 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:26.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:26.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:26.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:26.159 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:26.198 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:26.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:26.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:25 smithi204 ceph-mon[33153]: pgmap v786: 148 pgs: 148 active+clean; 2.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 814 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T13:15:26.258 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:25 smithi163 ceph-mon[29621]: pgmap v786: 148 pgs: 148 active+clean; 2.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 814 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T13:15:26.258 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:25 smithi163 ceph-mon[40675]: pgmap v786: 148 pgs: 148 active+clean; 2.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 814 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T13:15:26.299 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:26.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:26.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:26.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:26.479 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:26.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:26.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:26.580 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:26.613 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:26.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:26.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:26.715 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:26.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:26.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:26.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:26.850 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:26.889 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:26.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:26.957 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.0 to 1 (10000) 2024-03-21T13:15:26.994 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:27.035 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:27.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:27.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:27.137 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:27.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:27.176 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:27.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:27.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:26 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3272245932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:15:27.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:26 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3272245932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:15:27.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:26 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3272245932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:15:27.277 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:27.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:27.320 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:27.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:27.421 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:27.458 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:27.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:27.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:27.560 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:27.590 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:27.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:27.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:27.692 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:27.729 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:27.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:27.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:27.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:27.845 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-21T13:15:27.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:27.869 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:27.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:27.970 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:28.012 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:28.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:28.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:28.082 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:27 smithi163 ceph-mon[29621]: pgmap v787: 148 pgs: 148 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s 2024-03-21T13:15:28.082 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:27 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3272245932' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:15:28.082 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:27 smithi163 ceph-mon[29621]: osdmap e380: 8 total, 8 up, 7 in 2024-03-21T13:15:28.083 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:27 smithi163 ceph-mon[40675]: pgmap v787: 148 pgs: 148 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s 2024-03-21T13:15:28.083 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:27 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3272245932' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:15:28.083 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:27 smithi163 ceph-mon[40675]: osdmap e380: 8 total, 8 up, 7 in 2024-03-21T13:15:28.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:28.115 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:28.143 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:28.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:28.245 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:28.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:27 smithi204 ceph-mon[33153]: pgmap v787: 148 pgs: 148 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s 2024-03-21T13:15:28.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:27 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3272245932' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:15:28.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:27 smithi204 ceph-mon[33153]: osdmap e380: 8 total, 8 up, 7 in 2024-03-21T13:15:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:28.279 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:28.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:28.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:28.414 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:28.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:28.446 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:15:28.446 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:15:28.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.446 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.25481.0:238904 10.9 e380/330 10:95668baa:::benchmark_data_smithi163_154138_object107105:head v 380'14902, mlcod=380'14899)", 2024-03-21T13:15:28.446 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:15:28.445437+0000", 2024-03-21T13:15:28.447 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.000148154, 2024-03-21T13:15:28.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000170929, 2024-03-21T13:15:28.447 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:15:28.447 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:15:28.447 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:15:28.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:15:28.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.445437+0000", 2024-03-21T13:15:28.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:15:28.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:15:28.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.445437+0000", 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.445438+0000", 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.37e-07 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.445443+0000", 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5949999999999998e-06 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.445452+0000", 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8480000000000007e-06 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.445457+0000", 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2390000000000002e-06 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.445501+0000", 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3553000000000001e-05 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.445523+0000", 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2405e-05 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25481.0:238903 10.3 10:cc481df7:::benchmark_data_smithi163_154138_object107104:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:15:28.439758+0000", 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.0058271820000000002, 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0058947660000000001, 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:15:28.450 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25481", 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 238903 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.439758+0000", 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:15:28.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.439758+0000", 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.439759+0000", 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.44e-06 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.439766+0000", 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8299999999999998e-06 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.439784+0000", 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.785e-05 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.439794+0000", 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0197000000000001e-05 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.439859+0000", 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4616000000000006e-05 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.439976+0000", 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011724 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,5", 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.440058+0000", 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2171999999999999e-05 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.442917+0000", 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002859028 2024-03-21T13:15:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:15:28.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:15:28.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:15:28.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:15:28.443555+0000", 2024-03-21T13:15:28.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00063747900000000004 2024-03-21T13:15:28.455 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:15:28.455 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:15:28.455 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:15:28.455 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:15:28.455 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:15:28.455 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 2 2024-03-21T13:15:28.455 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:15:28.516 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:28.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:28.556 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:28.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:28.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:28.692 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:28.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:28.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:28.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:28.832 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:28.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:28.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:28.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:28.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:29.073 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:29.113 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:29.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:29.214 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:29.248 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:29.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:29.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:29.350 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:29.366 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:15:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:29.386 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:29.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:29.487 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:29.517 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:29.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:29.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-21T13:15:29.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:29.649 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:29.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:29.750 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:29.783 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:29.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:29.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:29.885 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:29.918 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:29.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:30.020 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:30.057 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:30.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:30.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:30.199 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:30.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:30.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:30.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:29 smithi204 ceph-mon[33153]: pgmap v789: 148 pgs: 148 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 803 op/s 2024-03-21T13:15:30.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:29 smithi163 ceph-mon[29621]: pgmap v789: 148 pgs: 148 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 803 op/s 2024-03-21T13:15:30.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:29 smithi163 ceph-mon[40675]: pgmap v789: 148 pgs: 148 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 803 op/s 2024-03-21T13:15:30.301 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:30.337 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:30.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:30.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:30.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:30.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:30.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:30.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:30.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:30.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:30.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:30.640 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:30.714 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:30.752 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:30.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:30.853 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:30.892 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:30.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:30.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:31.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:31.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:31.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:31.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:31.154 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:31.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:31.255 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:31.295 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:31.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:31.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:31.459 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:31.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:31.561 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:31.604 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:31.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:31.676 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:31.684 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:31.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:31.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:31.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:31.839 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:31.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:31.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:31.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:31.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:32.012 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:32.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:32.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:32.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:32.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:32.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:32.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:32.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:31 smithi204 ceph-mon[33153]: pgmap v790: 148 pgs: 148 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-21T13:15:32.253 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:15:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:15:32.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:31 smithi163 ceph-mon[29621]: pgmap v790: 148 pgs: 148 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-21T13:15:32.254 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:31 smithi163 ceph-mon[40675]: pgmap v790: 148 pgs: 148 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-21T13:15:32.284 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:32.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:32.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:32.416 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:32.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:32.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:32.518 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:32.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:32.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:32.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:32.656 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:32.689 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:32.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:32.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:32.825 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:32.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:32.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:32.926 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:32.964 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:32.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:32.982 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:15:33.066 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:33.102 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:33.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:33.203 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:33.242 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:33.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:33.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:32 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/890889941' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:15:33.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:32 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:32 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/890889941' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:15:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:32 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:33.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:32 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/890889941' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:15:33.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:32 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:33.344 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:33.380 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:33.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:33.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:33.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:33.523 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:33.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:33.625 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:33.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:33.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:33.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:33.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:33.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:33.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:33.905 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:33.939 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:33.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:34.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:34.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:34.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:34.179 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:34.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:34.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:33 smithi163 ceph-mon[29621]: pgmap v791: 148 pgs: 148 active+clean; 1.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-21T13:15:34.252 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:33 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:15:34.252 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:33 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/890889941' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:15:34.252 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:33 smithi163 ceph-mon[29621]: osdmap e381: 8 total, 8 up, 7 in 2024-03-21T13:15:34.252 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:33 smithi163 ceph-mon[40675]: pgmap v791: 148 pgs: 148 active+clean; 1.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-21T13:15:34.252 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:33 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:15:34.252 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:33 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/890889941' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:15:34.252 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:33 smithi163 ceph-mon[40675]: osdmap e381: 8 total, 8 up, 7 in 2024-03-21T13:15:34.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:33 smithi204 ceph-mon[33153]: pgmap v791: 148 pgs: 148 active+clean; 1.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-21T13:15:34.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:33 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:15:34.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:33 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/890889941' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:15:34.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:33 smithi204 ceph-mon[33153]: osdmap e381: 8 total, 8 up, 7 in 2024-03-21T13:15:34.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:34.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:34.352 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:34.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:34.453 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:34.493 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:34.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:34.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:34.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:34.644 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:34.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:34.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:34.783 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:34.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:34.885 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:34.920 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:34.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:35.022 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:35.054 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:35.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:35.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:35.156 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:35.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:35.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:35.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:35.356 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:35.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:35.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:35.458 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:35.492 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:35.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:35.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:35.594 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:35.629 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:35.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:35.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:35.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:35.768 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:35.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:35.869 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:35.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:35.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:35.919 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:15:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:15:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:15:36.010 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:36.046 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:36.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:36.147 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:36.174 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:35 smithi163 ceph-mon[29621]: pgmap v793: 148 pgs: 148 active+clean; 1.3 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 923 op/s 2024-03-21T13:15:36.174 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:36.175 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:35 smithi163 ceph-mon[40675]: pgmap v793: 148 pgs: 148 active+clean; 1.3 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 923 op/s 2024-03-21T13:15:36.175 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:36.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:36.194 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:36.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:36.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:35 smithi204 ceph-mon[33153]: pgmap v793: 148 pgs: 148 active+clean; 1.3 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 923 op/s 2024-03-21T13:15:36.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:36.294 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:36.334 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:36.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:36.435 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:36.482 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:36.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:36.583 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:36.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:36.715 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:36.753 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:36.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:36.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:36.855 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:36.890 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:36.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:36.991 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:37.026 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:37.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:37.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:37.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:37.165 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:37.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:37.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:37.267 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:37.296 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:37.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:37.398 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:37.433 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:37.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:37.535 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:37.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:37.674 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:37.714 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:37.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:37.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:37.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:37.850 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:37.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:37.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:37.952 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:37.976 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:37.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:38.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:38.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:38.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:38.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:38.219 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:38.262 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:38.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:38.364 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:38.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:38.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:38.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:38.505 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:38.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:38.542 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:38.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:38.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:38.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:38.678 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:38.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:38.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:38 smithi163 ceph-mon[29621]: pgmap v794: 148 pgs: 148 active+clean; 975 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-21T13:15:38.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:38 smithi163 ceph-mon[40675]: pgmap v794: 148 pgs: 148 active+clean; 975 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-21T13:15:38.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:38 smithi204 ceph-mon[33153]: pgmap v794: 148 pgs: 148 active+clean; 975 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-21T13:15:38.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:38.817 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:38.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:38.919 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:38.959 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:38.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:38.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:39.061 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:39.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:39.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:39.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:39.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:39.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:39.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:39.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:39.371 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:39.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:39.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:39.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:39.477 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.3 to 1 (10000) 2024-03-21T13:15:39.509 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:39.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:39.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:39.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:39.653 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:39.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:39.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:39 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2827512733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:15:39.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:39 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:15:39.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:39.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:39 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2827512733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:15:39.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:39 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:15:39.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:39 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2827512733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:15:39.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:39 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:15:39.791 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:39.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:39.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:39.893 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:39.926 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:39.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:39.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:40.029 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:40.064 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:40.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:40.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:40.199 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:40.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:40.301 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:40.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:40.326 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:40.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:40.427 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:40.466 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:40.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:40.567 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:40.608 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:40.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:40.707 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:15:40.707 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:15:40.707 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-03-21T13:15:40.707 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-03-21T13:15:40.707 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:15:40.709 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:40.725 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:40 smithi163 ceph-mon[29621]: pgmap v795: 148 pgs: 148 active+clean; 975 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s 2024-03-21T13:15:40.725 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:40 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:15:40.725 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:40 smithi163 ceph-mon[29621]: osdmap e382: 8 total, 8 up, 7 in 2024-03-21T13:15:40.726 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:40 smithi163 ceph-mon[40675]: pgmap v795: 148 pgs: 148 active+clean; 975 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s 2024-03-21T13:15:40.726 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:40 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:15:40.726 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:40 smithi163 ceph-mon[40675]: osdmap e382: 8 total, 8 up, 7 in 2024-03-21T13:15:40.747 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:40.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:40.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:40 smithi204 ceph-mon[33153]: pgmap v795: 148 pgs: 148 active+clean; 975 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s 2024-03-21T13:15:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:40 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:15:40.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:40 smithi204 ceph-mon[33153]: osdmap e382: 8 total, 8 up, 7 in 2024-03-21T13:15:40.849 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:40.880 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:40.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:40.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:40.929 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:15:40.982 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:41.036 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:41.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:41.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:41.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:41.178 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:41.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:41.279 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:41.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:41.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:41.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:41.419 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:41.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-21T13:15:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:41.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:41.578 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:41.637 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:41.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:41.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:41.772 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:41.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:41.779 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:15:41.817 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:15:41.852 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:15:41.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:41.885 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:15:41.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:41.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:41.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:41.930 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:15:41.966 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:15:42.013 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:42.014 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:15:42.052 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:15:42.055 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:15:42.055 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:15:42.055 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:15:42.055 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:15:42.055 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:15:42.055 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:15:42.055 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:15:42.055 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:15:42.071 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:42.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:42.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:42.173 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:42.203 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:42.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:42.306 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:42.341 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:42.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:42.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:42.443 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:42.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:42.479 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:42.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:42.480 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:15:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:15:42.580 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:42.614 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:42.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:42.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:42.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:42 smithi204 ceph-mon[33153]: pgmap v797: 148 pgs: 148 active+clean; 574 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-03-21T13:15:42.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:42 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1057477313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:42.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:42.754 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:42.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:42.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:42 smithi163 ceph-mon[29621]: pgmap v797: 148 pgs: 148 active+clean; 574 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-03-21T13:15:42.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:42 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1057477313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:42.758 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:42 smithi163 ceph-mon[40675]: pgmap v797: 148 pgs: 148 active+clean; 574 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-03-21T13:15:42.758 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:42 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1057477313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:42.761 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-21T13:15:42.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:42.896 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:42.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:42.998 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:43.051 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:43.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:43.153 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:43.193 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:43.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:43.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:43.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:43.304 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:43.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:43.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:43.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:43.347 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:43.430 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:43.472 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:43.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:43.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:43.574 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:43.611 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:43.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:43.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:43.643 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:43.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:43.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:43.746 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:43.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:43.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:43.848 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:43.889 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:43.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:43.991 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:44.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:44.041 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:44.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:44.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:44.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:44.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:44.281 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:44.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:44.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:44.418 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:44.435 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:44.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:44.465 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:44.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:44.566 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:44.604 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:44.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:44.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:44.744 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:44.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:44.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:44 smithi204 ceph-mon[33153]: pgmap v798: 148 pgs: 148 active+clean; 574 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-21T13:15:44.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:44 smithi163 ceph-mon[29621]: pgmap v798: 148 pgs: 148 active+clean; 574 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-21T13:15:44.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:44 smithi163 ceph-mon[40675]: pgmap v798: 148 pgs: 148 active+clean; 574 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-21T13:15:44.845 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:44.881 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:44.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:44.983 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:45.013 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:45.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:45.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:45.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:45.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:45.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:45.251 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:45.269 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:15:45.288 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:45.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:45.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:45.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:45.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:45.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:45.562 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:45.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:45.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:45.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:45.699 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:45.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:45.800 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:45.834 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:45.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:45.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:45.936 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:45.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:45.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:45.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:46.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:15:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:15:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:15:46.073 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:46.110 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:46.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:46.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:46.211 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:46.250 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:46.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:46.351 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:46.387 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:46.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:46.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:46.489 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:46.532 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:46.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:46.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:46.634 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:46.667 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:46.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:46.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:46.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:46 smithi204 ceph-mon[33153]: pgmap v799: 148 pgs: 148 active+clean; 522 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-21T13:15:46.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:46 smithi163 ceph-mon[29621]: pgmap v799: 148 pgs: 148 active+clean; 522 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-21T13:15:46.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:46 smithi163 ceph-mon[40675]: pgmap v799: 148 pgs: 148 active+clean; 522 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-21T13:15:46.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:46.805 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:46.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:46.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:46.907 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:46.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:46.947 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:46.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:47.048 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:47.082 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:47.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:47.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:47.184 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:47.222 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:47.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:47.304 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Removed 131796 objects 2024-03-21T13:15:47.304 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Clean up completed and total clean up time :104.9 2024-03-21T13:15:47.323 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:47.347 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:47.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:47.436 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-21T13:15:47.437 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-21T13:15:47.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:47.472 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:47.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:47.573 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:47.596 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:47.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:47.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:47.722 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:47.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:47.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:47.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:47.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:47.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:47.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:47.950 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:47.974 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:47.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:47.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:48.076 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:48.101 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:48.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:48.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:48.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:48.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:48.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:48.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:48.353 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:48.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:48.454 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:48.480 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:48.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:48.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:48.582 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:48.606 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:48.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:48.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:48.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:48.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:48.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:48.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:48 smithi163 ceph-mon[29621]: pgmap v800: 148 pgs: 148 active+clean; 162 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-21T13:15:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:48 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:15:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:48 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:15:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:48 smithi163 ceph-mon[40675]: pgmap v800: 148 pgs: 148 active+clean; 162 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-21T13:15:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:48 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:15:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:48 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:15:48.838 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:48.862 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:48.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:48.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:48.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:48.992 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:48.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:48.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:48 smithi204 ceph-mon[33153]: pgmap v800: 148 pgs: 148 active+clean; 162 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-21T13:15:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:48 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:15:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:48 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:15:49.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:49.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:49.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:49.224 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:49.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:49.249 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:49.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:49.329 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:15:49.329 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:15:49.350 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:49.375 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:49.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:49.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:49.477 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:49.501 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:49.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:49.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:49.609 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:49.630 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:49.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:49.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:49.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:49 smithi163 ceph-mon[29621]: osdmap e383: 8 total, 8 up, 7 in 2024-03-21T13:15:49.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:49 smithi163 ceph-mon[40675]: osdmap e383: 8 total, 8 up, 7 in 2024-03-21T13:15:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:49.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:49.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:49.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:49.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:49.985 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:50.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:49 smithi204 ceph-mon[33153]: osdmap e383: 8 total, 8 up, 7 in 2024-03-21T13:15:50.012 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:50.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:50.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:50.138 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:50.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:50.240 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:50.263 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:50.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:50.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:50.394 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:50.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:50.496 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:50.524 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:50.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:50.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:50.626 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:50.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:50.651 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:50.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:50.752 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:50 smithi163 ceph-mon[29621]: pgmap v802: 148 pgs: 148 active+clean; 162 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-03-21T13:15:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:50.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:50 smithi163 ceph-mon[40675]: pgmap v802: 148 pgs: 148 active+clean; 162 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-03-21T13:15:50.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:50.780 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:50.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:50.882 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:50.907 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:50.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:50.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:50 smithi204 ceph-mon[33153]: pgmap v802: 148 pgs: 148 active+clean; 162 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-03-21T13:15:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:51.009 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:51.034 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:51.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:51.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:51.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:51.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:51.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:51.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:51.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:51.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:51.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:51.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:51.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:51.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:51.527 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:51.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:51.555 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:51.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:51.656 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:51.682 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:51.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:51.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:51.784 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:51.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:51.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:51.913 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:51.937 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:51.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:52.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:52.067 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:52.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:52.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:52.169 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:52.194 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:52.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:52.295 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:52.319 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:52.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:52.421 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:52.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:52.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:52.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:15:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:15:52.551 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:52.576 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:52.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:52.677 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:52.701 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:52.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:52.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:52 smithi163 ceph-mon[29621]: pgmap v803: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 917 op/s 2024-03-21T13:15:52.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:52 smithi163 ceph-mon[40675]: pgmap v803: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 917 op/s 2024-03-21T13:15:52.803 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:52.827 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:52.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:52.928 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:52.952 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:52.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:53.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:52 smithi204 ceph-mon[33153]: pgmap v803: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 917 op/s 2024-03-21T13:15:53.054 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:53.077 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:53.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:53.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:53.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:53.208 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:53.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:53.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:53.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:53.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:53.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:53.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:53.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:53.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:53.464 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:53.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:53.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:53.593 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:53.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:53.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:53.695 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:53.719 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:53.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:53.822 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:53.846 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:53.846 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:53.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:53.847 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:53.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:53.948 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:53.973 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:53.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:54.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:54.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:54.103 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:54.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:54.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:54.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:54.231 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:54.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:54.331 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:54.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:54.360 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:54.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:54.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:54.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:54.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:54.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:54.537 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.4 to 1 (10000) 2024-03-21T13:15:54.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:54.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:54.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:54.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:54.745 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:54.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:54 smithi163 ceph-mon[29621]: pgmap v804: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 917 op/s 2024-03-21T13:15:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:54 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2631883513' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:15:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:54 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:15:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:54 smithi163 ceph-mon[40675]: pgmap v804: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 917 op/s 2024-03-21T13:15:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:54 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2631883513' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:15:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:54 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:15:54.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:54.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:54.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:54.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:54.978 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:55.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:54 smithi204 ceph-mon[33153]: pgmap v804: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 917 op/s 2024-03-21T13:15:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:54 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2631883513' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:15:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:54 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:15:55.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:55.004 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:55.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:55.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:55.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:55.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:55.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:55.263 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:55.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:55.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:55.365 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:55.391 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:55.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:55.493 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:55.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:55.523 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:55.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:55.544 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:15:55.624 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:55.651 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:55.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:55.717 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:15:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:15:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:15:55.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:55 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:15:55.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:55 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:15:55.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:55 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:15:55.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:55 smithi163 ceph-mon[29621]: osdmap e384: 8 total, 8 up, 7 in 2024-03-21T13:15:55.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:55 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2834283210' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:15:55.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:55 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:15:55.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:55 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:15:55.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:55 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:15:55.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:55 smithi163 ceph-mon[40675]: osdmap e384: 8 total, 8 up, 7 in 2024-03-21T13:15:55.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:55 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2834283210' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:15:55.736 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:15:55.753 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:55.755 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:15:55.775 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:15:55.782 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:55.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:55.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:55.794 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:15:55.814 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:15:55.834 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:15:55.855 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:15:55.875 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:15:55.876 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:15:55.876 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:15:55.876 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:15:55.876 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:15:55.876 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:15:55.876 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:15:55.876 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:15:55.876 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:15:55.884 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:55.909 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:55.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:56.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:55 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:15:56.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:55 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:15:56.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:55 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:15:56.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:55 smithi204 ceph-mon[33153]: osdmap e384: 8 total, 8 up, 7 in 2024-03-21T13:15:56.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:55 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2834283210' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:15:56.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:56.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:56.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:56.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:56.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:56.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:56.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:56.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:56.271 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:56.296 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:56.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:56.398 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:56.426 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:56.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:56.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:56.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:56.555 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:56.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:56.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:56.657 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:56.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:56.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:56.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:56.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:56 smithi163 ceph-mon[29621]: pgmap v806: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 322 op/s 2024-03-21T13:15:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:56 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:15:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:56 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2834283210' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:15:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:56 smithi163 ceph-mon[29621]: osdmap e385: 8 total, 8 up, 7 in 2024-03-21T13:15:56.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:56 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2604380543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:56 smithi163 ceph-mon[40675]: pgmap v806: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 322 op/s 2024-03-21T13:15:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:56 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:15:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:56 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2834283210' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:15:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:56 smithi163 ceph-mon[40675]: osdmap e385: 8 total, 8 up, 7 in 2024-03-21T13:15:56.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:56 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2604380543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:56.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:56.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:56.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:56.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:56.837 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-21T13:15:56.912 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:56.936 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:56.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:57.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:56 smithi204 ceph-mon[33153]: pgmap v806: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 322 op/s 2024-03-21T13:15:57.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:56 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:15:57.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:56 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2834283210' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:15:57.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:56 smithi204 ceph-mon[33153]: osdmap e385: 8 total, 8 up, 7 in 2024-03-21T13:15:57.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:56 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2604380543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:15:57.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:57.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:57.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:57.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:57.168 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:57.192 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:57.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:57.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:57.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:57.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:57.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:57.422 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:57.446 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:57.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:57.549 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:57.570 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'unique_pool_3' removed 2024-03-21T13:15:57.595 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:57.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:57.697 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:57.725 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:57.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:57.827 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:57.851 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:57.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:57.953 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:57.977 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:57.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:57.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:58.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:57 smithi204 ceph-mon[33153]: 4.11 scrub starts 2024-03-21T13:15:58.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:57 smithi204 ceph-mon[33153]: 4.11 scrub ok 2024-03-21T13:15:58.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:57 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2910467473' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:15:58.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:57 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:15:58.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:57 smithi163 ceph-mon[29621]: 4.11 scrub starts 2024-03-21T13:15:58.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:57 smithi163 ceph-mon[29621]: 4.11 scrub ok 2024-03-21T13:15:58.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:57 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2910467473' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:15:58.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:57 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:15:58.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:57 smithi163 ceph-mon[40675]: 4.11 scrub starts 2024-03-21T13:15:58.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:57 smithi163 ceph-mon[40675]: 4.11 scrub ok 2024-03-21T13:15:58.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:57 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2910467473' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:15:58.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:57 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:15:58.079 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:58.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:58.103 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:58.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:58.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:58.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:58.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:58.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:15:58.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:58.354 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:58.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:15:58.435 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-21T13:15:58.455 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:15:58.501 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:58.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:15:58.606 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:58.630 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:58.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:58.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:58.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:58.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:58.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:58.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:58 smithi163 ceph-mon[29621]: pgmap v808: 148 pgs: 148 active+clean; 582 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 323 op/s 2024-03-21T13:15:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:58 smithi163 ceph-mon[29621]: 4.2 scrub starts 2024-03-21T13:15:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:58 smithi163 ceph-mon[29621]: 4.2 scrub ok 2024-03-21T13:15:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:58 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:15:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:58 smithi163 ceph-mon[29621]: osdmap e386: 8 total, 8 up, 7 in 2024-03-21T13:15:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:58 smithi163 ceph-mon[40675]: pgmap v808: 148 pgs: 148 active+clean; 582 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 323 op/s 2024-03-21T13:15:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:58 smithi163 ceph-mon[40675]: 4.2 scrub starts 2024-03-21T13:15:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:58 smithi163 ceph-mon[40675]: 4.2 scrub ok 2024-03-21T13:15:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:58 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:15:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:58 smithi163 ceph-mon[40675]: osdmap e386: 8 total, 8 up, 7 in 2024-03-21T13:15:58.859 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:15:58.883 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:58.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:58.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:15:58.986 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:58 smithi204 ceph-mon[33153]: pgmap v808: 148 pgs: 148 active+clean; 582 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 323 op/s 2024-03-21T13:15:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:58 smithi204 ceph-mon[33153]: 4.2 scrub starts 2024-03-21T13:15:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:58 smithi204 ceph-mon[33153]: 4.2 scrub ok 2024-03-21T13:15:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:58 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:15:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:58 smithi204 ceph-mon[33153]: osdmap e386: 8 total, 8 up, 7 in 2024-03-21T13:15:59.005 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:59.009 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:59.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:59.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:59.111 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:59.135 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:59.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:59.236 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:15:59.260 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:59.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:59.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:15:59.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:15:59.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:59.386 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:59.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:15:59.470 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-21T13:15:59.471 INFO:tasks.radosbench:Beginning radosbench... 2024-03-21T13:15:59.471 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-21T13:15:59.471 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-21T13:15:59.475 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:15:59.487 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:15:59.511 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:15:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:59.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:15:59.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:59.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:59.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:59.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:59.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:59 smithi163 ceph-mon[40675]: 6.b scrub starts 2024-03-21T13:15:59.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:59 smithi163 ceph-mon[40675]: 6.b scrub ok 2024-03-21T13:15:59.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:59 smithi163 ceph-mon[40675]: 5.1a scrub starts 2024-03-21T13:15:59.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:59 smithi163 ceph-mon[40675]: 5.1a scrub ok 2024-03-21T13:15:59.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:15:59 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:15:59.704 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:59 smithi163 ceph-mon[29621]: 6.b scrub starts 2024-03-21T13:15:59.704 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:59 smithi163 ceph-mon[29621]: 6.b scrub ok 2024-03-21T13:15:59.704 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:59 smithi163 ceph-mon[29621]: 5.1a scrub starts 2024-03-21T13:15:59.704 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:59 smithi163 ceph-mon[29621]: 5.1a scrub ok 2024-03-21T13:15:59.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:15:59 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:15:59.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:15:59.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:15:59.761 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:59.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:15:59.863 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:15:59.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:15:59.888 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:15:59.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:15:59.990 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:59 smithi204 ceph-mon[33153]: 6.b scrub starts 2024-03-21T13:16:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:59 smithi204 ceph-mon[33153]: 6.b scrub ok 2024-03-21T13:16:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:59 smithi204 ceph-mon[33153]: 5.1a scrub starts 2024-03-21T13:16:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:59 smithi204 ceph-mon[33153]: 5.1a scrub ok 2024-03-21T13:16:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:15:59 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:16:00.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:00.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:00.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:00.140 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:00.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:00.241 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:00.281 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:00.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:00.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:00.407 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:00.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:00.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:00.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:00.532 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:00.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:00.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:00.634 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:00.657 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:00.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:00.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:00.759 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:00.782 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:00.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:00.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:00.884 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:00.907 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:00.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:00.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:00.940 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:16:00.972 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[29621]: 6.1f scrub starts 2024-03-21T13:16:00.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[29621]: pgmap v810: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:16:00.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[29621]: 6.1f scrub ok 2024-03-21T13:16:00.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[29621]: 3.14 scrub starts 2024-03-21T13:16:00.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[29621]: 3.14 scrub ok 2024-03-21T13:16:00.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[29621]: 5.c scrub starts 2024-03-21T13:16:00.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[29621]: 5.c scrub ok 2024-03-21T13:16:00.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[29621]: 3.17 scrub starts 2024-03-21T13:16:00.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[29621]: 3.17 scrub ok 2024-03-21T13:16:00.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:00.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:00.973 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[40675]: 6.1f scrub starts 2024-03-21T13:16:00.974 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[40675]: pgmap v810: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:16:00.974 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[40675]: 6.1f scrub ok 2024-03-21T13:16:00.974 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[40675]: 3.14 scrub starts 2024-03-21T13:16:00.974 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[40675]: 3.14 scrub ok 2024-03-21T13:16:00.974 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[40675]: 5.c scrub starts 2024-03-21T13:16:00.974 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[40675]: 5.c scrub ok 2024-03-21T13:16:00.974 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[40675]: 3.17 scrub starts 2024-03-21T13:16:00.974 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[40675]: 3.17 scrub ok 2024-03-21T13:16:00.974 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:00.974 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:00 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:00 smithi204 ceph-mon[33153]: 6.1f scrub starts 2024-03-21T13:16:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:00 smithi204 ceph-mon[33153]: pgmap v810: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:16:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:00 smithi204 ceph-mon[33153]: 6.1f scrub ok 2024-03-21T13:16:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:00 smithi204 ceph-mon[33153]: 3.14 scrub starts 2024-03-21T13:16:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:00 smithi204 ceph-mon[33153]: 3.14 scrub ok 2024-03-21T13:16:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:00 smithi204 ceph-mon[33153]: 5.c scrub starts 2024-03-21T13:16:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:00 smithi204 ceph-mon[33153]: 5.c scrub ok 2024-03-21T13:16:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:00 smithi204 ceph-mon[33153]: 3.17 scrub starts 2024-03-21T13:16:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:00 smithi204 ceph-mon[33153]: 3.17 scrub ok 2024-03-21T13:16:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:00 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:00 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:01.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:01.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:01.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:01.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:01.159 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:01.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:01.261 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:01.286 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:01.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:01.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:01.388 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:01.412 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:01.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:01.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:01.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:01.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:01.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:01.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:01.639 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:01.664 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:01.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:01.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:01 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:16:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:01 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:16:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:01 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:01 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:16:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:01 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:16:01.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:01 smithi163 ceph-mon[29621]: pgmap v811: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:16:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:01 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:16:01.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:01 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:16:01.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:01 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:01.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:01 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:16:01.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:01 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:16:01.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:01 smithi163 ceph-mon[40675]: pgmap v811: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:16:01.766 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:01.793 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:01.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:01.895 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:01.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:01.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:01.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:01 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:16:02.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:01 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:16:02.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:01 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:02.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:01 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:16:02.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:01 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:16:02.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:01 smithi204 ceph-mon[33153]: pgmap v811: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:16:02.021 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:02.047 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:02.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:02.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:02.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:02.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:02.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:02.276 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:02.300 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:02.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:02.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:02.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:02.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:02.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:02.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:16:02 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:16:02.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:02.552 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:02.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:02.654 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:02.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:02.680 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:02.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:02.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[40675]: 6.4 scrub starts 2024-03-21T13:16:02.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[40675]: 6.4 scrub ok 2024-03-21T13:16:02.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[40675]: osdmap e387: 8 total, 8 up, 7 in 2024-03-21T13:16:02.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[40675]: 3.1f scrub starts 2024-03-21T13:16:02.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[40675]: 3.1f scrub ok 2024-03-21T13:16:02.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:02.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[40675]: 3.2 scrub starts 2024-03-21T13:16:02.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[40675]: 3.2 scrub ok 2024-03-21T13:16:02.704 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/565509564' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:16:02.705 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:16:02.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[29621]: 6.4 scrub starts 2024-03-21T13:16:02.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[29621]: 6.4 scrub ok 2024-03-21T13:16:02.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[29621]: osdmap e387: 8 total, 8 up, 7 in 2024-03-21T13:16:02.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[29621]: 3.1f scrub starts 2024-03-21T13:16:02.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[29621]: 3.1f scrub ok 2024-03-21T13:16:02.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:02.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[29621]: 3.2 scrub starts 2024-03-21T13:16:02.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[29621]: 3.2 scrub ok 2024-03-21T13:16:02.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/565509564' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:16:02.705 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:02 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:16:02.781 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:02.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:02.808 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:02.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:02.876 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:16:02.876 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:16:02.876 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-03-21T13:16:02.876 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:16:02.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:02.933 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:02.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:02 smithi204 ceph-mon[33153]: 6.4 scrub starts 2024-03-21T13:16:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:02 smithi204 ceph-mon[33153]: 6.4 scrub ok 2024-03-21T13:16:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:02 smithi204 ceph-mon[33153]: osdmap e387: 8 total, 8 up, 7 in 2024-03-21T13:16:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:02 smithi204 ceph-mon[33153]: 3.1f scrub starts 2024-03-21T13:16:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:02 smithi204 ceph-mon[33153]: 3.1f scrub ok 2024-03-21T13:16:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:02 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:02 smithi204 ceph-mon[33153]: 3.2 scrub starts 2024-03-21T13:16:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:02 smithi204 ceph-mon[33153]: 3.2 scrub ok 2024-03-21T13:16:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:02 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/565509564' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:16:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:02 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:16:03.035 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:03.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:03.060 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:03.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:03.108 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.6 to 1 (10000) 2024-03-21T13:16:03.161 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:03.185 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:03.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:03.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:03.286 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:03.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:03.311 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:03.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:03.316 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:03.317 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:03.412 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:03.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:03.437 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:03.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:03.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:03.564 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:03.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:03.665 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:03.689 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:03.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:03.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:03.815 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:03.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:03.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:03.942 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:03.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:03.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:03 smithi204 ceph-mon[33153]: 6.15 scrub starts 2024-03-21T13:16:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:03 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:16:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:03 smithi204 ceph-mon[33153]: osdmap e388: 8 total, 8 up, 7 in 2024-03-21T13:16:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:03 smithi204 ceph-mon[33153]: 6.15 scrub ok 2024-03-21T13:16:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:03 smithi204 ceph-mon[33153]: pgmap v814: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:16:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:03 smithi204 ceph-mon[33153]: 6.14 scrub starts 2024-03-21T13:16:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:03 smithi204 ceph-mon[33153]: 6.14 scrub ok 2024-03-21T13:16:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[40675]: 6.15 scrub starts 2024-03-21T13:16:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:16:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[40675]: osdmap e388: 8 total, 8 up, 7 in 2024-03-21T13:16:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[40675]: 6.15 scrub ok 2024-03-21T13:16:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[40675]: pgmap v814: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:16:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[40675]: 6.14 scrub starts 2024-03-21T13:16:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[40675]: 6.14 scrub ok 2024-03-21T13:16:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[29621]: 6.15 scrub starts 2024-03-21T13:16:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:16:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[29621]: osdmap e388: 8 total, 8 up, 7 in 2024-03-21T13:16:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[29621]: 6.15 scrub ok 2024-03-21T13:16:04.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[29621]: pgmap v814: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:16:04.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[29621]: 6.14 scrub starts 2024-03-21T13:16:04.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:03 smithi163 ceph-mon[29621]: 6.14 scrub ok 2024-03-21T13:16:04.044 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:04.068 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:04.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:04.076 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-21T13:16:04.170 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:04.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:04.194 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:04.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:04.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:04.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:04.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:04.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:04.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:04.446 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:04.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:04.547 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:04.571 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:04.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:04.673 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:04.696 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:04.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:04.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:04.750 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:04.781 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-21T13:16:04.798 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:04.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:04.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:04.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:04.948 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:04.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:05.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:04 smithi163 ceph-mon[29621]: 3.1c scrub starts 2024-03-21T13:16:05.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:04 smithi163 ceph-mon[29621]: 3.1c scrub ok 2024-03-21T13:16:05.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:04 smithi163 ceph-mon[40675]: 3.1c scrub starts 2024-03-21T13:16:05.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:04 smithi163 ceph-mon[40675]: 3.1c scrub ok 2024-03-21T13:16:05.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:05.073 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:05.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:05.175 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:05.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:05.200 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:05.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:05.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:04 smithi204 ceph-mon[33153]: 3.1c scrub starts 2024-03-21T13:16:05.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:04 smithi204 ceph-mon[33153]: 3.1c scrub ok 2024-03-21T13:16:05.302 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:05.325 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:05.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:05.427 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:05.452 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:05.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:05.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:05.554 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:05.578 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:05.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:05.636 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:16:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:16:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:16:05.679 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:05.703 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:05.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:05.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:05.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:05.827 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:05.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:05.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:05.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:05.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:05 smithi163 ceph-mon[29621]: 3.9 scrub starts 2024-03-21T13:16:05.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:05 smithi163 ceph-mon[29621]: 3.9 scrub ok 2024-03-21T13:16:05.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:05 smithi163 ceph-mon[29621]: pgmap v815: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 691 B/s rd, 0 op/s 2024-03-21T13:16:05.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:05 smithi163 ceph-mon[29621]: 5.10 scrub starts 2024-03-21T13:16:05.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:05 smithi163 ceph-mon[29621]: 5.10 scrub ok 2024-03-21T13:16:05.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:05.936 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:05 smithi163 ceph-mon[40675]: 3.9 scrub starts 2024-03-21T13:16:05.936 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:05 smithi163 ceph-mon[40675]: 3.9 scrub ok 2024-03-21T13:16:05.936 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:05 smithi163 ceph-mon[40675]: pgmap v815: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 691 B/s rd, 0 op/s 2024-03-21T13:16:05.936 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:05 smithi163 ceph-mon[40675]: 5.10 scrub starts 2024-03-21T13:16:05.936 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:05 smithi163 ceph-mon[40675]: 5.10 scrub ok 2024-03-21T13:16:05.936 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:05.953 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:05.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:06.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:06.079 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:06.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:06.180 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:06.205 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:06.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:06.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:05 smithi204 ceph-mon[33153]: 3.9 scrub starts 2024-03-21T13:16:06.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:05 smithi204 ceph-mon[33153]: 3.9 scrub ok 2024-03-21T13:16:06.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:05 smithi204 ceph-mon[33153]: pgmap v815: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 691 B/s rd, 0 op/s 2024-03-21T13:16:06.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:05 smithi204 ceph-mon[33153]: 5.10 scrub starts 2024-03-21T13:16:06.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:05 smithi204 ceph-mon[33153]: 5.10 scrub ok 2024-03-21T13:16:06.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:06.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:06.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:06.332 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:06.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:06.433 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:06.457 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:06.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:06.558 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:06.583 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:06.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:06.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:06.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:06.709 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:06.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:06.810 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:06.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:06.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:06.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:06.935 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:06.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:06.960 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:06.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:07.061 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:07.081 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:07.086 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:07.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:07.118 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:07.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:07.204 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:06 smithi163 ceph-mon[29621]: 4.5 scrub starts 2024-03-21T13:16:07.204 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:06 smithi163 ceph-mon[29621]: 4.5 scrub ok 2024-03-21T13:16:07.204 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:06 smithi163 ceph-mon[29621]: 3.b scrub starts 2024-03-21T13:16:07.204 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:06 smithi163 ceph-mon[29621]: 3.b scrub ok 2024-03-21T13:16:07.205 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:06 smithi163 ceph-mon[40675]: 4.5 scrub starts 2024-03-21T13:16:07.205 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:06 smithi163 ceph-mon[40675]: 4.5 scrub ok 2024-03-21T13:16:07.205 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:06 smithi163 ceph-mon[40675]: 3.b scrub starts 2024-03-21T13:16:07.205 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:06 smithi163 ceph-mon[40675]: 3.b scrub ok 2024-03-21T13:16:07.212 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:07.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:07.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:07.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:06 smithi204 ceph-mon[33153]: 4.5 scrub starts 2024-03-21T13:16:07.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:06 smithi204 ceph-mon[33153]: 4.5 scrub ok 2024-03-21T13:16:07.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:06 smithi204 ceph-mon[33153]: 3.b scrub starts 2024-03-21T13:16:07.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:06 smithi204 ceph-mon[33153]: 3.b scrub ok 2024-03-21T13:16:07.314 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:07.337 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:07.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:07.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:07.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:07.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:07.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:07.565 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:07.588 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:07.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:07.690 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:07.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:07.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:07.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:07.840 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:07.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:07.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:07.966 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:07.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:07.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:08.069 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:08.094 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:08.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:08.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[29621]: 6.8 scrub starts 2024-03-21T13:16:08.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[29621]: 6.8 scrub ok 2024-03-21T13:16:08.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[29621]: 6.1e scrub starts 2024-03-21T13:16:08.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[29621]: 6.1e scrub ok 2024-03-21T13:16:08.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[29621]: pgmap v816: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:08.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[29621]: 5.3 scrub starts 2024-03-21T13:16:08.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[29621]: 5.3 scrub ok 2024-03-21T13:16:08.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[29621]: 6.1d scrub starts 2024-03-21T13:16:08.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[29621]: 6.1d scrub ok 2024-03-21T13:16:08.161 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[40675]: 6.8 scrub starts 2024-03-21T13:16:08.162 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[40675]: 6.8 scrub ok 2024-03-21T13:16:08.162 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[40675]: 6.1e scrub starts 2024-03-21T13:16:08.162 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[40675]: 6.1e scrub ok 2024-03-21T13:16:08.162 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[40675]: pgmap v816: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:08.162 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[40675]: 5.3 scrub starts 2024-03-21T13:16:08.162 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[40675]: 5.3 scrub ok 2024-03-21T13:16:08.162 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[40675]: 6.1d scrub starts 2024-03-21T13:16:08.162 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:07 smithi163 ceph-mon[40675]: 6.1d scrub ok 2024-03-21T13:16:08.195 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:08.218 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:08.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:08.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:07 smithi204 ceph-mon[33153]: 6.8 scrub starts 2024-03-21T13:16:08.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:07 smithi204 ceph-mon[33153]: 6.8 scrub ok 2024-03-21T13:16:08.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:07 smithi204 ceph-mon[33153]: 6.1e scrub starts 2024-03-21T13:16:08.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:07 smithi204 ceph-mon[33153]: 6.1e scrub ok 2024-03-21T13:16:08.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:07 smithi204 ceph-mon[33153]: pgmap v816: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:08.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:07 smithi204 ceph-mon[33153]: 5.3 scrub starts 2024-03-21T13:16:08.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:07 smithi204 ceph-mon[33153]: 5.3 scrub ok 2024-03-21T13:16:08.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:07 smithi204 ceph-mon[33153]: 6.1d scrub starts 2024-03-21T13:16:08.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:07 smithi204 ceph-mon[33153]: 6.1d scrub ok 2024-03-21T13:16:08.320 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:08.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:08.344 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:08.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:08.445 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:08.469 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:08.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:08.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:08.571 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:08.596 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:08.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:08.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:08.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:08.722 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:08.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:08.824 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:08.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:08.849 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:08.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:08.878 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:16:08.949 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:08.977 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:08.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:08.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:09.079 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:09.104 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:09.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:09.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:09.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[40675]: 4.1a scrub starts 2024-03-21T13:16:09.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[40675]: 4.1a scrub ok 2024-03-21T13:16:09.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[40675]: 5.1d scrub starts 2024-03-21T13:16:09.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[40675]: 5.1d scrub ok 2024-03-21T13:16:09.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[40675]: 2.2 scrub starts 2024-03-21T13:16:09.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[40675]: 2.2 scrub ok 2024-03-21T13:16:09.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2014325858' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:09.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:09.167 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[29621]: 4.1a scrub starts 2024-03-21T13:16:09.167 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[29621]: 4.1a scrub ok 2024-03-21T13:16:09.167 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[29621]: 5.1d scrub starts 2024-03-21T13:16:09.167 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[29621]: 5.1d scrub ok 2024-03-21T13:16:09.167 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[29621]: 2.2 scrub starts 2024-03-21T13:16:09.167 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[29621]: 2.2 scrub ok 2024-03-21T13:16:09.167 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2014325858' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:09.167 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:08 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:09.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:09.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:09.230 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:09.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:09.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:08 smithi204 ceph-mon[33153]: 4.1a scrub starts 2024-03-21T13:16:09.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:08 smithi204 ceph-mon[33153]: 4.1a scrub ok 2024-03-21T13:16:09.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:08 smithi204 ceph-mon[33153]: 5.1d scrub starts 2024-03-21T13:16:09.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:08 smithi204 ceph-mon[33153]: 5.1d scrub ok 2024-03-21T13:16:09.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:08 smithi204 ceph-mon[33153]: 2.2 scrub starts 2024-03-21T13:16:09.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:08 smithi204 ceph-mon[33153]: 2.2 scrub ok 2024-03-21T13:16:09.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:08 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2014325858' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:09.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:08 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:09.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:09.359 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:09.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:09.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:09.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:09.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:09.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:09.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:09.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:09.615 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:09.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:09.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:09.717 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:09.742 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:09.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:09.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:09.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:09.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:09.972 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[29621]: 3.15 scrub starts 2024-03-21T13:16:09.972 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[29621]: 3.15 scrub ok 2024-03-21T13:16:09.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:16:09.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:16:09.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:16:09.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[29621]: osdmap e389: 8 total, 8 up, 7 in 2024-03-21T13:16:09.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[29621]: pgmap v818: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 705 B/s rd, 0 op/s 2024-03-21T13:16:09.973 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[40675]: 3.15 scrub starts 2024-03-21T13:16:09.973 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[40675]: 3.15 scrub ok 2024-03-21T13:16:09.973 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:16:09.973 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:16:09.973 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:16:09.973 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[40675]: osdmap e389: 8 total, 8 up, 7 in 2024-03-21T13:16:09.974 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:09 smithi163 ceph-mon[40675]: pgmap v818: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 705 B/s rd, 0 op/s 2024-03-21T13:16:09.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:10.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:10.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:10.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:10.128 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:10.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:10.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:10.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:10.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:09 smithi204 ceph-mon[33153]: 3.15 scrub starts 2024-03-21T13:16:10.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:09 smithi204 ceph-mon[33153]: 3.15 scrub ok 2024-03-21T13:16:10.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:09 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:16:10.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:09 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:16:10.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:09 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:16:10.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:09 smithi204 ceph-mon[33153]: osdmap e389: 8 total, 8 up, 7 in 2024-03-21T13:16:10.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:09 smithi204 ceph-mon[33153]: pgmap v818: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 705 B/s rd, 0 op/s 2024-03-21T13:16:10.256 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:10.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:10.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:10.358 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:10.384 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:10.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:10.485 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:10.512 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:10.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:10.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:10.614 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:10.638 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:10.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:10.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:10.740 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:10.763 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:10.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:10.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:10.865 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:10.889 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:10.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:10.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:10.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:11.020 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:11.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:11.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:11.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:11.148 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:11.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:11.165 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[29621]: 2.0 scrub starts 2024-03-21T13:16:11.165 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[29621]: 2.0 scrub ok 2024-03-21T13:16:11.165 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[29621]: 4.18 deep-scrub starts 2024-03-21T13:16:11.165 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[29621]: 4.18 deep-scrub ok 2024-03-21T13:16:11.165 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[29621]: 5.14 scrub starts 2024-03-21T13:16:11.165 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[29621]: 5.14 scrub ok 2024-03-21T13:16:11.165 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[29621]: 4.9 scrub starts 2024-03-21T13:16:11.166 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[29621]: 4.9 scrub ok 2024-03-21T13:16:11.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[40675]: 2.0 scrub starts 2024-03-21T13:16:11.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[40675]: 2.0 scrub ok 2024-03-21T13:16:11.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[40675]: 4.18 deep-scrub starts 2024-03-21T13:16:11.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[40675]: 4.18 deep-scrub ok 2024-03-21T13:16:11.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[40675]: 5.14 scrub starts 2024-03-21T13:16:11.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[40675]: 5.14 scrub ok 2024-03-21T13:16:11.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[40675]: 4.9 scrub starts 2024-03-21T13:16:11.166 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:10 smithi163 ceph-mon[40675]: 4.9 scrub ok 2024-03-21T13:16:11.250 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:11.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:10 smithi204 ceph-mon[33153]: 2.0 scrub starts 2024-03-21T13:16:11.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:10 smithi204 ceph-mon[33153]: 2.0 scrub ok 2024-03-21T13:16:11.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:10 smithi204 ceph-mon[33153]: 4.18 deep-scrub starts 2024-03-21T13:16:11.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:10 smithi204 ceph-mon[33153]: 4.18 deep-scrub ok 2024-03-21T13:16:11.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:10 smithi204 ceph-mon[33153]: 5.14 scrub starts 2024-03-21T13:16:11.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:10 smithi204 ceph-mon[33153]: 5.14 scrub ok 2024-03-21T13:16:11.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:10 smithi204 ceph-mon[33153]: 4.9 scrub starts 2024-03-21T13:16:11.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:10 smithi204 ceph-mon[33153]: 4.9 scrub ok 2024-03-21T13:16:11.278 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:11.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:11.380 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:11.405 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:11.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:11.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:11.507 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:11.531 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:11.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:11.633 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:11.657 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:11.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:11.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:11.787 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:11.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:11.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:11.916 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:11.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:11.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:12.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:12.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:12.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:12.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:12.147 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:12.157 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:11 smithi204 ceph-mon[33153]: 5.9 scrub starts 2024-03-21T13:16:12.157 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:11 smithi204 ceph-mon[33153]: 5.9 scrub ok 2024-03-21T13:16:12.157 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:11 smithi204 ceph-mon[33153]: pgmap v819: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:12.157 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:11 smithi204 ceph-mon[33153]: 5.5 scrub starts 2024-03-21T13:16:12.157 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:11 smithi204 ceph-mon[33153]: 5.5 scrub ok 2024-03-21T13:16:12.158 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:11 smithi163 ceph-mon[29621]: 5.9 scrub starts 2024-03-21T13:16:12.158 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:11 smithi163 ceph-mon[29621]: 5.9 scrub ok 2024-03-21T13:16:12.158 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:11 smithi163 ceph-mon[29621]: pgmap v819: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:12.158 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:11 smithi163 ceph-mon[29621]: 5.5 scrub starts 2024-03-21T13:16:12.158 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:11 smithi163 ceph-mon[29621]: 5.5 scrub ok 2024-03-21T13:16:12.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:11 smithi163 ceph-mon[40675]: 5.9 scrub starts 2024-03-21T13:16:12.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:11 smithi163 ceph-mon[40675]: 5.9 scrub ok 2024-03-21T13:16:12.159 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:11 smithi163 ceph-mon[40675]: pgmap v819: 132 pgs: 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:12.159 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:11 smithi163 ceph-mon[40675]: 5.5 scrub starts 2024-03-21T13:16:12.159 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:11 smithi163 ceph-mon[40675]: 5.5 scrub ok 2024-03-21T13:16:12.171 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:12.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:12.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:12.263 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:16:12.272 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:12.330 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:12.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:12.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:12.432 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:12.456 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:12.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:12.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:16:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:16:12.558 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:12.581 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:12.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:12.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:12.683 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:12.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:12.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:12.813 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:12.837 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:12.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:12.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:12.880 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:12 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3342202094' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-21T13:16:12.895 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'unique_pool_4' created 2024-03-21T13:16:12.940 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:12.965 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:12.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:12.996 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:16:13.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:12 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3342202094' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-21T13:16:13.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:12 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-21T13:16:13.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:12 smithi204 ceph-mon[33153]: 3.0 deep-scrub starts 2024-03-21T13:16:13.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:12 smithi204 ceph-mon[33153]: 3.0 deep-scrub ok 2024-03-21T13:16:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:12 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:13.015 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:16:13.034 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:16:13.053 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:16:13.066 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:13.073 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:16:13.090 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:13.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:13.094 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:16:13.115 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:16:13.134 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:16:13.135 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:12 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-21T13:16:13.135 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:12 smithi163 ceph-mon[29621]: 3.0 deep-scrub starts 2024-03-21T13:16:13.135 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:12 smithi163 ceph-mon[29621]: 3.0 deep-scrub ok 2024-03-21T13:16:13.135 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:12 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:13.135 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:12 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3342202094' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-21T13:16:13.135 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:12 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-21T13:16:13.135 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:12 smithi163 ceph-mon[40675]: 3.0 deep-scrub starts 2024-03-21T13:16:13.135 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:12 smithi163 ceph-mon[40675]: 3.0 deep-scrub ok 2024-03-21T13:16:13.135 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:12 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:16:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:16:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:16:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:16:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:16:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:16:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:16:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:16:13.192 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:13.207 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:16:13.207 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:16:13.216 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:13.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:13.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:13.345 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:13.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:13.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:13.473 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:13.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:13.574 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:13.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:13.599 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:13.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:13.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:13.724 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:13.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:13.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:13.826 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:13.851 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:13.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:13.952 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:13.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:13.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:13.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:14.048 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:13 smithi204 ceph-mon[33153]: 5.1f scrub starts 2024-03-21T13:16:14.048 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:13 smithi204 ceph-mon[33153]: 5.1f scrub ok 2024-03-21T13:16:14.048 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:13 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-21T13:16:14.048 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:13 smithi204 ceph-mon[33153]: osdmap e390: 8 total, 8 up, 7 in 2024-03-21T13:16:14.048 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:13 smithi204 ceph-mon[33153]: pgmap v821: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:14.048 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:13 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1610545657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:16:14.051 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:13 smithi163 ceph-mon[29621]: 5.1f scrub starts 2024-03-21T13:16:14.051 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:13 smithi163 ceph-mon[29621]: 5.1f scrub ok 2024-03-21T13:16:14.051 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:13 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-21T13:16:14.051 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:13 smithi163 ceph-mon[29621]: osdmap e390: 8 total, 8 up, 7 in 2024-03-21T13:16:14.051 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:13 smithi163 ceph-mon[29621]: pgmap v821: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:14.051 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:13 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1610545657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:16:14.052 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:13 smithi163 ceph-mon[40675]: 5.1f scrub starts 2024-03-21T13:16:14.052 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:13 smithi163 ceph-mon[40675]: 5.1f scrub ok 2024-03-21T13:16:14.052 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:13 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-21T13:16:14.052 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:13 smithi163 ceph-mon[40675]: osdmap e390: 8 total, 8 up, 7 in 2024-03-21T13:16:14.052 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:13 smithi163 ceph-mon[40675]: pgmap v821: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:14.052 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:13 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1610545657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:16:14.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-21T13:16:14.078 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:14.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:14.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:14.204 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:14.228 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:14.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:14.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:14.354 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:14.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:14.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:14.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:14.480 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:14.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:14.582 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:14.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:14.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:14.710 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:14.734 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:14.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:14.836 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:14.859 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:14.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:14.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:14.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:14.985 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:14.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:15.086 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:15.110 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:15.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:15.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:15.211 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:15.235 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:15.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:15.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:15.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:14 smithi204 ceph-mon[33153]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:15.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:14 smithi204 ceph-mon[33153]: osdmap e391: 8 total, 8 up, 7 in 2024-03-21T13:16:15.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:14 smithi204 ceph-mon[33153]: 6.d deep-scrub starts 2024-03-21T13:16:15.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:14 smithi204 ceph-mon[33153]: 4.d deep-scrub starts 2024-03-21T13:16:15.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:14 smithi204 ceph-mon[33153]: 4.d deep-scrub ok 2024-03-21T13:16:15.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:14 smithi204 ceph-mon[33153]: 6.d deep-scrub ok 2024-03-21T13:16:15.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:14 smithi163 ceph-mon[40675]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:15.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:14 smithi163 ceph-mon[40675]: osdmap e391: 8 total, 8 up, 7 in 2024-03-21T13:16:15.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:14 smithi163 ceph-mon[40675]: 6.d deep-scrub starts 2024-03-21T13:16:15.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:14 smithi163 ceph-mon[40675]: 4.d deep-scrub starts 2024-03-21T13:16:15.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:14 smithi163 ceph-mon[40675]: 4.d deep-scrub ok 2024-03-21T13:16:15.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:14 smithi163 ceph-mon[40675]: 6.d deep-scrub ok 2024-03-21T13:16:15.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:14 smithi163 ceph-mon[29621]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:15.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:14 smithi163 ceph-mon[29621]: osdmap e391: 8 total, 8 up, 7 in 2024-03-21T13:16:15.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:14 smithi163 ceph-mon[29621]: 6.d deep-scrub starts 2024-03-21T13:16:15.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:14 smithi163 ceph-mon[29621]: 4.d deep-scrub starts 2024-03-21T13:16:15.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:14 smithi163 ceph-mon[29621]: 4.d deep-scrub ok 2024-03-21T13:16:15.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:14 smithi163 ceph-mon[29621]: 6.d deep-scrub ok 2024-03-21T13:16:15.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:15.347 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-21T13:16:15.347 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:15.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:15.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:15.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:15.486 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:15.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:15.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:15.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:15.612 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:15.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:15.713 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:15.737 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:15.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:15.839 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:15.863 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:15.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:15.901 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:16:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:16:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:16:15.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:15.988 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:15.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:16.090 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:16.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:16.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:16.158 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:15 smithi163 ceph-mon[29621]: 1.0 scrub starts 2024-03-21T13:16:16.158 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:15 smithi163 ceph-mon[29621]: 1.0 scrub ok 2024-03-21T13:16:16.158 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:15 smithi163 ceph-mon[29621]: osdmap e392: 8 total, 8 up, 7 in 2024-03-21T13:16:16.158 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:15 smithi163 ceph-mon[29621]: pgmap v824: 148 pgs: 15 unknown, 133 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:16:16.158 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:15 smithi163 ceph-mon[29621]: 5.6 scrub starts 2024-03-21T13:16:16.158 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:15 smithi163 ceph-mon[29621]: 5.6 scrub ok 2024-03-21T13:16:16.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:15 smithi163 ceph-mon[40675]: 1.0 scrub starts 2024-03-21T13:16:16.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:15 smithi163 ceph-mon[40675]: 1.0 scrub ok 2024-03-21T13:16:16.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:15 smithi163 ceph-mon[40675]: osdmap e392: 8 total, 8 up, 7 in 2024-03-21T13:16:16.159 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:15 smithi163 ceph-mon[40675]: pgmap v824: 148 pgs: 15 unknown, 133 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:16:16.159 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:15 smithi163 ceph-mon[40675]: 5.6 scrub starts 2024-03-21T13:16:16.159 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:15 smithi163 ceph-mon[40675]: 5.6 scrub ok 2024-03-21T13:16:16.216 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:16.240 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:16.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:16.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:16.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:15 smithi204 ceph-mon[33153]: 1.0 scrub starts 2024-03-21T13:16:16.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:15 smithi204 ceph-mon[33153]: 1.0 scrub ok 2024-03-21T13:16:16.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:15 smithi204 ceph-mon[33153]: osdmap e392: 8 total, 8 up, 7 in 2024-03-21T13:16:16.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:15 smithi204 ceph-mon[33153]: pgmap v824: 148 pgs: 15 unknown, 133 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:16:16.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:15 smithi204 ceph-mon[33153]: 5.6 scrub starts 2024-03-21T13:16:16.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:15 smithi204 ceph-mon[33153]: 5.6 scrub ok 2024-03-21T13:16:16.342 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:16.366 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:16.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:16.468 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:16.491 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:16.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:16.593 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:16.612 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:16:16.616 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:16.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:16.718 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:16.742 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:16.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:16.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:16.802 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:16.844 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:16.868 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:16.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:16.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:16.969 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:16.992 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:16.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:16.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:17.094 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:17.120 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:17.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:17.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:17.221 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:17.245 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:17.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:17.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:16 smithi204 ceph-mon[33153]: 6.1 scrub starts 2024-03-21T13:16:17.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:16 smithi204 ceph-mon[33153]: 6.1 scrub ok 2024-03-21T13:16:17.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:16 smithi204 ceph-mon[33153]: 3.12 scrub starts 2024-03-21T13:16:17.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:16 smithi204 ceph-mon[33153]: 3.12 scrub ok 2024-03-21T13:16:17.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:16 smithi204 ceph-mon[33153]: 4.4 scrub starts 2024-03-21T13:16:17.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:16 smithi204 ceph-mon[33153]: 4.4 scrub ok 2024-03-21T13:16:17.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:16 smithi163 ceph-mon[29621]: 6.1 scrub starts 2024-03-21T13:16:17.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:16 smithi163 ceph-mon[29621]: 6.1 scrub ok 2024-03-21T13:16:17.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:16 smithi163 ceph-mon[29621]: 3.12 scrub starts 2024-03-21T13:16:17.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:16 smithi163 ceph-mon[29621]: 3.12 scrub ok 2024-03-21T13:16:17.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:16 smithi163 ceph-mon[29621]: 4.4 scrub starts 2024-03-21T13:16:17.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:16 smithi163 ceph-mon[29621]: 4.4 scrub ok 2024-03-21T13:16:17.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:16 smithi163 ceph-mon[40675]: 6.1 scrub starts 2024-03-21T13:16:17.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:16 smithi163 ceph-mon[40675]: 6.1 scrub ok 2024-03-21T13:16:17.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:16 smithi163 ceph-mon[40675]: 3.12 scrub starts 2024-03-21T13:16:17.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:16 smithi163 ceph-mon[40675]: 3.12 scrub ok 2024-03-21T13:16:17.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:16 smithi163 ceph-mon[40675]: 4.4 scrub starts 2024-03-21T13:16:17.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:16 smithi163 ceph-mon[40675]: 4.4 scrub ok 2024-03-21T13:16:17.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:17.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:17.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:17.377 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:17.474 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:17.498 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:17.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:17.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:17.623 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:17.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:17.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:17.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:17.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:17.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:17.849 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:17.874 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:17.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:17.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:17.927 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.7 to 1 (10000) 2024-03-21T13:16:17.976 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:18.000 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:18.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:18.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:18.010 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:16:18.010 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:16:18.101 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:18.125 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:18.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:18.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:18.250 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:18.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:18.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:17 smithi204 ceph-mon[33153]: pgmap v825: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:16:18.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:17 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2582083357' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:16:18.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:17 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:16:18.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:17 smithi204 ceph-mon[33153]: 3.5 scrub starts 2024-03-21T13:16:18.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:17 smithi204 ceph-mon[33153]: 3.5 scrub ok 2024-03-21T13:16:18.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:17 smithi204 ceph-mon[33153]: 6.17 deep-scrub starts 2024-03-21T13:16:18.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:17 smithi163 ceph-mon[29621]: pgmap v825: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:16:18.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:17 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2582083357' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:16:18.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:17 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:16:18.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:17 smithi163 ceph-mon[29621]: 3.5 scrub starts 2024-03-21T13:16:18.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:17 smithi163 ceph-mon[29621]: 3.5 scrub ok 2024-03-21T13:16:18.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:17 smithi163 ceph-mon[29621]: 6.17 deep-scrub starts 2024-03-21T13:16:18.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:17 smithi163 ceph-mon[40675]: pgmap v825: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:16:18.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:17 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2582083357' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:16:18.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:17 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:16:18.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:17 smithi163 ceph-mon[40675]: 3.5 scrub starts 2024-03-21T13:16:18.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:17 smithi163 ceph-mon[40675]: 3.5 scrub ok 2024-03-21T13:16:18.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:17 smithi163 ceph-mon[40675]: 6.17 deep-scrub starts 2024-03-21T13:16:18.353 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:18.377 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:18.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:18.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:18.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:18.502 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:18.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:18.604 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:18.628 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:18.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:18.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:18.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:18.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-21T13:16:18.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:18.754 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:18.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:18.856 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:18.881 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:18.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:18.983 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:18.994 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:16:19.007 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:19.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:19.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:19.109 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:19.116 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:18 smithi204 ceph-mon[33153]: 6.17 deep-scrub ok 2024-03-21T13:16:19.116 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:18 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:16:19.116 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:18 smithi204 ceph-mon[33153]: osdmap e393: 8 total, 8 up, 7 in 2024-03-21T13:16:19.116 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:18 smithi204 ceph-mon[33153]: 5.16 scrub starts 2024-03-21T13:16:19.116 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:18 smithi204 ceph-mon[33153]: 5.16 scrub ok 2024-03-21T13:16:19.116 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:18 smithi204 ceph-mon[33153]: 6.e scrub starts 2024-03-21T13:16:19.116 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:18 smithi204 ceph-mon[33153]: 6.e scrub ok 2024-03-21T13:16:19.132 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:19.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:19.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:19.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:19.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[29621]: 6.17 deep-scrub ok 2024-03-21T13:16:19.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:16:19.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[29621]: osdmap e393: 8 total, 8 up, 7 in 2024-03-21T13:16:19.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[29621]: 5.16 scrub starts 2024-03-21T13:16:19.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[29621]: 5.16 scrub ok 2024-03-21T13:16:19.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[29621]: 6.e scrub starts 2024-03-21T13:16:19.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[29621]: 6.e scrub ok 2024-03-21T13:16:19.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[40675]: 6.17 deep-scrub ok 2024-03-21T13:16:19.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:16:19.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[40675]: osdmap e393: 8 total, 8 up, 7 in 2024-03-21T13:16:19.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[40675]: 5.16 scrub starts 2024-03-21T13:16:19.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[40675]: 5.16 scrub ok 2024-03-21T13:16:19.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[40675]: 6.e scrub starts 2024-03-21T13:16:19.245 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:18 smithi163 ceph-mon[40675]: 6.e scrub ok 2024-03-21T13:16:19.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:19.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:19.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:19.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:19.385 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:19.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:19.486 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:19.509 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:19.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:19.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:19.635 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:19.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:19.736 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:19.760 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:19.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:19.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:19.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:19.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:19.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:19.987 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:20.011 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:20.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:20.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:20.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:20.137 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:20.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:20.238 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:20.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:19 smithi204 ceph-mon[33153]: 6.5 deep-scrub starts 2024-03-21T13:16:20.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:19 smithi204 ceph-mon[33153]: 5.0 scrub starts 2024-03-21T13:16:20.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:19 smithi204 ceph-mon[33153]: 5.0 scrub ok 2024-03-21T13:16:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:19 smithi204 ceph-mon[33153]: 6.5 deep-scrub ok 2024-03-21T13:16:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:19 smithi204 ceph-mon[33153]: 4.6 scrub starts 2024-03-21T13:16:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:19 smithi204 ceph-mon[33153]: 4.6 scrub ok 2024-03-21T13:16:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:19 smithi204 ceph-mon[33153]: pgmap v827: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:16:20.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[29621]: 6.5 deep-scrub starts 2024-03-21T13:16:20.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[29621]: 5.0 scrub starts 2024-03-21T13:16:20.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[29621]: 5.0 scrub ok 2024-03-21T13:16:20.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[29621]: 6.5 deep-scrub ok 2024-03-21T13:16:20.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[29621]: 4.6 scrub starts 2024-03-21T13:16:20.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[29621]: 4.6 scrub ok 2024-03-21T13:16:20.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[29621]: pgmap v827: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:16:20.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[40675]: 6.5 deep-scrub starts 2024-03-21T13:16:20.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[40675]: 5.0 scrub starts 2024-03-21T13:16:20.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[40675]: 5.0 scrub ok 2024-03-21T13:16:20.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[40675]: 6.5 deep-scrub ok 2024-03-21T13:16:20.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[40675]: 4.6 scrub starts 2024-03-21T13:16:20.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[40675]: 4.6 scrub ok 2024-03-21T13:16:20.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:19 smithi163 ceph-mon[40675]: pgmap v827: 148 pgs: 148 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:16:20.262 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:20.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:20.364 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:20.390 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:20.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:20.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:20.492 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:20.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:20.517 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:20.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:20.618 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:20.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:20.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:20.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:20.744 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:20.767 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:20.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:20.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:20.869 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:20.893 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:20.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:20.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:20.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:21.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:21.023 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:21.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:21.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:21.150 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:21.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:21.252 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:21.277 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:21.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:16:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:16:21.379 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:21.402 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:21.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:21.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:21.504 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:21.518 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:21.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:21.530 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:21.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:21.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:21 smithi163 ceph-mon[29621]: 6.f scrub starts 2024-03-21T13:16:21.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:21.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:21.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:21 smithi163 ceph-mon[29621]: 6.18 scrub starts 2024-03-21T13:16:21.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:21 smithi163 ceph-mon[29621]: 6.18 scrub ok 2024-03-21T13:16:21.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:21 smithi163 ceph-mon[40675]: 6.f scrub starts 2024-03-21T13:16:21.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:21.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:21.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:21 smithi163 ceph-mon[40675]: 6.18 scrub starts 2024-03-21T13:16:21.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:21 smithi163 ceph-mon[40675]: 6.18 scrub ok 2024-03-21T13:16:21.630 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:21.657 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:21.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:21.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:21.738 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:21 smithi204 ceph-mon[33153]: 6.f scrub starts 2024-03-21T13:16:21.738 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:21.738 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:21.738 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:21 smithi204 ceph-mon[33153]: 6.18 scrub starts 2024-03-21T13:16:21.739 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:21 smithi204 ceph-mon[33153]: 6.18 scrub ok 2024-03-21T13:16:21.759 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:21.782 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:21.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:21.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:21.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:21.909 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:21.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:21.944 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-21T13:16:22.010 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:22.034 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:22.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:22.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:22.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-21T13:16:22.135 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:22.158 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:22.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:22.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:22.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:22.284 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:22.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:22.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:22.376 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:16:22.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:22.411 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:22.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:22.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:22.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:16:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:16:22.498 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:22 smithi204 ceph-mon[33153]: 6.f scrub ok 2024-03-21T13:16:22.498 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:22 smithi204 ceph-mon[33153]: 4.10 scrub starts 2024-03-21T13:16:22.498 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:22 smithi204 ceph-mon[33153]: 4.10 scrub ok 2024-03-21T13:16:22.498 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:22 smithi204 ceph-mon[33153]: pgmap v828: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:16:22.498 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:22 smithi204 ceph-mon[33153]: 5.19 scrub starts 2024-03-21T13:16:22.498 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:22 smithi204 ceph-mon[33153]: 5.19 scrub ok 2024-03-21T13:16:22.499 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:22 smithi204 ceph-mon[33153]: 5.d scrub starts 2024-03-21T13:16:22.499 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:22 smithi204 ceph-mon[33153]: 5.d scrub ok 2024-03-21T13:16:22.499 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:22 smithi204 ceph-mon[33153]: 5.11 scrub starts 2024-03-21T13:16:22.499 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:22 smithi204 ceph-mon[33153]: 5.11 scrub ok 2024-03-21T13:16:22.513 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:22.537 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:22.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:22.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:22.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:22.663 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:22.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:22.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[29621]: 6.f scrub ok 2024-03-21T13:16:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[29621]: 4.10 scrub starts 2024-03-21T13:16:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[29621]: 4.10 scrub ok 2024-03-21T13:16:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[29621]: pgmap v828: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:16:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[29621]: 5.19 scrub starts 2024-03-21T13:16:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[29621]: 5.19 scrub ok 2024-03-21T13:16:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[29621]: 5.d scrub starts 2024-03-21T13:16:22.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[29621]: 5.d scrub ok 2024-03-21T13:16:22.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[29621]: 5.11 scrub starts 2024-03-21T13:16:22.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[29621]: 5.11 scrub ok 2024-03-21T13:16:22.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[40675]: 6.f scrub ok 2024-03-21T13:16:22.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[40675]: 4.10 scrub starts 2024-03-21T13:16:22.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[40675]: 4.10 scrub ok 2024-03-21T13:16:22.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[40675]: pgmap v828: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:16:22.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[40675]: 5.19 scrub starts 2024-03-21T13:16:22.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[40675]: 5.19 scrub ok 2024-03-21T13:16:22.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[40675]: 5.d scrub starts 2024-03-21T13:16:22.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[40675]: 5.d scrub ok 2024-03-21T13:16:22.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[40675]: 5.11 scrub starts 2024-03-21T13:16:22.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:22 smithi163 ceph-mon[40675]: 5.11 scrub ok 2024-03-21T13:16:22.765 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:22.789 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:22.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:22.891 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:22.914 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:22.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:23.016 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:23.043 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:23.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:23.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:23.168 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:23.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:23.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:23.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:23.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:23.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:23.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:23.420 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:23.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:23.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:23.467 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[29621]: 4.15 scrub starts 2024-03-21T13:16:23.467 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[29621]: 4.15 scrub ok 2024-03-21T13:16:23.467 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:23.467 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[29621]: 3.4 scrub starts 2024-03-21T13:16:23.468 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[29621]: 3.4 scrub ok 2024-03-21T13:16:23.468 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[29621]: 3.3 deep-scrub starts 2024-03-21T13:16:23.468 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[29621]: 3.3 deep-scrub ok 2024-03-21T13:16:23.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[40675]: 4.15 scrub starts 2024-03-21T13:16:23.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[40675]: 4.15 scrub ok 2024-03-21T13:16:23.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:23.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[40675]: 3.4 scrub starts 2024-03-21T13:16:23.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[40675]: 3.4 scrub ok 2024-03-21T13:16:23.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[40675]: 3.3 deep-scrub starts 2024-03-21T13:16:23.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:23 smithi163 ceph-mon[40675]: 3.3 deep-scrub ok 2024-03-21T13:16:23.522 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:23.545 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:23.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:23.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:23.648 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:23 smithi204 ceph-mon[33153]: 4.15 scrub starts 2024-03-21T13:16:23.648 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:23 smithi204 ceph-mon[33153]: 4.15 scrub ok 2024-03-21T13:16:23.648 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:23 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:23.648 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:23 smithi204 ceph-mon[33153]: 3.4 scrub starts 2024-03-21T13:16:23.648 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:23 smithi204 ceph-mon[33153]: 3.4 scrub ok 2024-03-21T13:16:23.648 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:23 smithi204 ceph-mon[33153]: 3.3 deep-scrub starts 2024-03-21T13:16:23.648 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:23 smithi204 ceph-mon[33153]: 3.3 deep-scrub ok 2024-03-21T13:16:23.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:23.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:23.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:23.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:23.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:23.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:23.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:23.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:23.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:24.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:24.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:24.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:24.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:24.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:24.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:24.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:24.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:24.275 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:24.294 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:16:24.294 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:16:24.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:24.301 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:24.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:24.364 INFO:teuthology.orchestra.run.smithi163.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-21T13:16:24.368 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:16:24.399 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:16:24.400 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:16:24.400 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[29621]: pgmap v829: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:24.400 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[29621]: 3.16 scrub starts 2024-03-21T13:16:24.400 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[29621]: 3.16 scrub ok 2024-03-21T13:16:24.400 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[29621]: osdmap e394: 8 total, 8 up, 7 in 2024-03-21T13:16:24.400 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2484942422' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:24.400 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[29621]: 4.b scrub starts 2024-03-21T13:16:24.400 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[29621]: 4.b scrub ok 2024-03-21T13:16:24.400 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2452007249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:16:24.400 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:16:24.401 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:16:24.401 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:16:24.401 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[40675]: pgmap v829: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:24.401 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[40675]: 3.16 scrub starts 2024-03-21T13:16:24.401 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[40675]: 3.16 scrub ok 2024-03-21T13:16:24.401 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[40675]: osdmap e394: 8 total, 8 up, 7 in 2024-03-21T13:16:24.401 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2484942422' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:24.401 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[40675]: 4.b scrub starts 2024-03-21T13:16:24.401 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[40675]: 4.b scrub ok 2024-03-21T13:16:24.401 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2452007249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:16:24.401 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:24 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:16:24.402 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:24.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:24.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:24.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:24 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:16:24.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:24 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:16:24.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:24 smithi204 ceph-mon[33153]: pgmap v829: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:24.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:24 smithi204 ceph-mon[33153]: 3.16 scrub starts 2024-03-21T13:16:24.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:24 smithi204 ceph-mon[33153]: 3.16 scrub ok 2024-03-21T13:16:24.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:24 smithi204 ceph-mon[33153]: osdmap e394: 8 total, 8 up, 7 in 2024-03-21T13:16:24.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:24 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2484942422' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:24.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:24 smithi204 ceph-mon[33153]: 4.b scrub starts 2024-03-21T13:16:24.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:24 smithi204 ceph-mon[33153]: 4.b scrub ok 2024-03-21T13:16:24.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:24 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2452007249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:16:24.504 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:24 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:16:24.527 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:24.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:24.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:24.653 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:24.678 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:24.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:24.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:24.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:24.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:24.806 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:24.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:24.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:24.921 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:16:24.934 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:24.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:24.940 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:16:24.959 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:16:24.978 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:16:24.998 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:16:25.017 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:16:25.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:25.037 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:16:25.058 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:16:25.059 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:16:25.059 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:16:25.059 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:16:25.059 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:16:25.059 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:16:25.059 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:16:25.059 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:16:25.059 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:16:25.060 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:25.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:25.069 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-21T13:16:25.122 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:25.162 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:25.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:25.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:25.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:25.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:25.312 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:25.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:25.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:25.354 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:16:25.414 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:25.439 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:25.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:25.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:25.565 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:25.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:25.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:25 smithi204 ceph-mon[33153]: 5.15 deep-scrub starts 2024-03-21T13:16:25.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:25 smithi204 ceph-mon[33153]: 5.15 deep-scrub ok 2024-03-21T13:16:25.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:25 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:16:25.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:25 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2484942422' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:25.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:25 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:16:25.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:25 smithi204 ceph-mon[33153]: osdmap e395: 8 total, 8 up, 7 in 2024-03-21T13:16:25.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:25 smithi204 ceph-mon[33153]: 4.12 scrub starts 2024-03-21T13:16:25.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:25 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1262722715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:16:25.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:25.694 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:25.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:25.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:16:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:16:25.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[29621]: 5.15 deep-scrub starts 2024-03-21T13:16:25.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[29621]: 5.15 deep-scrub ok 2024-03-21T13:16:25.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:16:25.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2484942422' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:25.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:16:25.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[29621]: osdmap e395: 8 total, 8 up, 7 in 2024-03-21T13:16:25.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[29621]: 4.12 scrub starts 2024-03-21T13:16:25.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1262722715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:16:25.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[40675]: 5.15 deep-scrub starts 2024-03-21T13:16:25.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[40675]: 5.15 deep-scrub ok 2024-03-21T13:16:25.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:16:25.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2484942422' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:25.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:16:25.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[40675]: osdmap e395: 8 total, 8 up, 7 in 2024-03-21T13:16:25.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[40675]: 4.12 scrub starts 2024-03-21T13:16:25.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:25 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1262722715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:16:25.804 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:25.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:25.829 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:25.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:25.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:25.954 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:25.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:26.056 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:26.080 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:26.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:26.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:26.207 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:26.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:26.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:26.308 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:26.332 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:26.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:26.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:26.383 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[29621]: 4.f scrub starts 2024-03-21T13:16:26.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[29621]: pgmap v832: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 727 B/s rd, 0 op/s 2024-03-21T13:16:26.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:16:26.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:16:26.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[29621]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:16:26.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[29621]: osdmap e396: 8 total, 8 up, 7 in 2024-03-21T13:16:26.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:16:26.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:16:26.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[40675]: 4.f scrub starts 2024-03-21T13:16:26.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[40675]: pgmap v832: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 727 B/s rd, 0 op/s 2024-03-21T13:16:26.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:16:26.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:16:26.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[40675]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:16:26.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[40675]: osdmap e396: 8 total, 8 up, 7 in 2024-03-21T13:16:26.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:16:26.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:26 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:16:26.434 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:26.459 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:26.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:26.489 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:26 smithi204 ceph-mon[33153]: 4.f scrub starts 2024-03-21T13:16:26.489 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:26 smithi204 ceph-mon[33153]: pgmap v832: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 727 B/s rd, 0 op/s 2024-03-21T13:16:26.489 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:26 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:16:26.489 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:26 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:16:26.489 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:26 smithi204 ceph-mon[33153]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:16:26.490 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:26 smithi204 ceph-mon[33153]: osdmap e396: 8 total, 8 up, 7 in 2024-03-21T13:16:26.490 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:26 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:16:26.490 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:26 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:16:26.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:26.585 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:26.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:26.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:26.687 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:26.711 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:26.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:26.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:26.836 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:26.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:26.938 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:26.962 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:26.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:27.055 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:16:27.055 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-03-21T13:16:27.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-03-21T13:16:27.055 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:16:27.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.055 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:195719 8.4 8:2579c874:::benchmark_data_smithi163_107079_object59846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-21T13:16:27.055 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:22.426227+0000", 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 544.62710334500002, 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0223721509999999, 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 195719 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.426227+0000", 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.426227+0000", 2024-03-21T13:16:27.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.426228+0000", 2024-03-21T13:16:27.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.362e-06 2024-03-21T13:16:27.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.426234+0000", 2024-03-21T13:16:27.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8590000000000001e-06 2024-03-21T13:16:27.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.426244+0000", 2024-03-21T13:16:27.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9350000000000006e-06 2024-03-21T13:16:27.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.426252+0000", 2024-03-21T13:16:27.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0439999999999996e-06 2024-03-21T13:16:27.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.426308+0000", 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6761000000000001e-05 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.426317+0000", 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3699999999999995e-06 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.444481+0000", 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0181643709999999 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.444630+0000", 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014883100000000001 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.444752+0000", 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012232200000000001 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.448330+0000", 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035775239999999999 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.448377+0000", 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7212999999999997e-05 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.448454+0000", 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7101999999999995e-05 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.448545+0000", 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1153e-05 2024-03-21T13:16:27.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.448599+0000", 2024-03-21T13:16:27.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3304000000000003e-05 2024-03-21T13:16:27.062 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.062 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.062 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.062 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:195771 8.4 8:2c1d425d:::benchmark_data_smithi163_107079_object59898:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-21T13:16:27.062 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:22.472250+0000", 2024-03-21T13:16:27.062 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 544.58107953599995, 2024-03-21T13:16:27.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97872499400000001, 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 195771 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.472250+0000", 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.472250+0000", 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.472252+0000", 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3740000000000001e-06 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.472260+0000", 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7959999999999996e-06 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.472271+0000", 2024-03-21T13:16:27.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1982e-05 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.472280+0000", 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5809999999999997e-06 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.472346+0000", 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5609000000000002e-05 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.472356+0000", 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0701e-05 2024-03-21T13:16:27.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.446758+0000", 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97440137500000001 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.446886+0000", 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000128458 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.447007+0000", 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000121017 2024-03-21T13:16:27.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.448439+0000", 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001432226 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.450701+0000", 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022614699999999998 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.450748+0000", 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6665000000000001e-05 2024-03-21T13:16:27.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.450928+0000", 2024-03-21T13:16:27.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018050100000000001 2024-03-21T13:16:27.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.068 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:27.069 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.450975+0000", 2024-03-21T13:16:27.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7239e-05 2024-03-21T13:16:27.070 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.070 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.070 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.070 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:195795 8.4 8:2536a1c7:::benchmark_data_smithi163_107079_object59922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-21T13:16:27.070 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:22.491911+0000", 2024-03-21T13:16:27.070 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 544.56141837999996, 2024-03-21T13:16:27.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96285337100000001, 2024-03-21T13:16:27.070 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.073 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.074 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.074 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:27.074 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:27.074 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 195795 2024-03-21T13:16:27.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.074 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.491911+0000", 2024-03-21T13:16:27.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.491911+0000", 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.491913+0000", 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.793e-06 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.491919+0000", 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2340000000000002e-06 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.491932+0000", 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3616e-05 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.491940+0000", 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1240000000000005e-06 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.491994+0000", 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3625000000000002e-05 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.492002+0000", 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7389999999999999e-06 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.448637+0000", 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95663584899999998 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.448743+0000", 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010579899999999999 2024-03-21T13:16:27.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-03-21T13:16:27.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.448866+0000", 2024-03-21T13:16:27.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012298699999999999 2024-03-21T13:16:27.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.451022+0000", 2024-03-21T13:16:27.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021556890000000001 2024-03-21T13:16:27.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.451034+0000", 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1844e-05 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.454629+0000", 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035953399999999998 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.454721+0000", 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2054999999999999e-05 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.454765+0000", 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3677000000000002e-05 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:195807 8.4 8:2a56aae9:::benchmark_data_smithi163_107079_object59934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:22.500999+0000", 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 544.55233121499998, 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95970059500000005, 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:27.080 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 195807 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.500999+0000", 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.500999+0000", 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.501000+0000", 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1489999999999999e-06 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.501006+0000", 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8850000000000004e-06 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.501020+0000", 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4436e-05 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.501029+0000", 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4630000000000001e-06 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.501092+0000", 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3205000000000002e-05 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.501101+0000", 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0110000000000006e-06 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.451061+0000", 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94995971099999998 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.451167+0000", 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010631700000000001 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.451281+0000", 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114592 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.458564+0000", 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072827359999999997 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460553+0000", 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00198928 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460630+0000", 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6524000000000001e-05 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460669+0000", 2024-03-21T13:16:27.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8952000000000002e-05 2024-03-21T13:16:27.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.086 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460699+0000", 2024-03-21T13:16:27.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0334000000000001e-05 2024-03-21T13:16:27.086 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.086 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.086 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.086 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:195846 8.4 8:27a23651:::benchmark_data_smithi163_107079_object59973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-21T13:16:27.086 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:22.530987+0000", 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 544.52234308100003, 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.929802986, 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 195846 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.530987+0000", 2024-03-21T13:16:27.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.530987+0000", 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.530988+0000", 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3400000000000001e-06 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.530993+0000", 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0130000000000001e-06 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.531002+0000", 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0550000000000005e-06 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.531009+0000", 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.165e-06 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.531070+0000", 2024-03-21T13:16:27.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1023999999999997e-05 2024-03-21T13:16:27.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.531079+0000", 2024-03-21T13:16:27.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4039999999999995e-06 2024-03-21T13:16:27.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.452936+0000", 2024-03-21T13:16:27.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92185714200000002 2024-03-21T13:16:27.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.453050+0000", 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114129 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.453169+0000", 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011890700000000001 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460492+0000", 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073234620000000002 2024-03-21T13:16:27.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460586+0000", 2024-03-21T13:16:27.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3927999999999999e-05 2024-03-21T13:16:27.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460734+0000", 2024-03-21T13:16:27.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014766399999999999 2024-03-21T13:16:27.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460764+0000", 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0164000000000001e-05 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460790+0000", 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5588999999999998e-05 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:195858 8.4 8:23f16eb5:::benchmark_data_smithi163_107079_object59985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:22.537451+0000", 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 544.51587932300004, 2024-03-21T13:16:27.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92353189599999996, 2024-03-21T13:16:27.094 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.094 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.094 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.094 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:27.094 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:27.094 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 195858 2024-03-21T13:16:27.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.094 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.537451+0000", 2024-03-21T13:16:27.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.537451+0000", 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.537452+0000", 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3349999999999999e-06 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.537457+0000", 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0989999999999998e-06 2024-03-21T13:16:27.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.537468+0000", 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0709e-05 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.537475+0000", 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3819999999999998e-06 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.537536+0000", 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0544999999999997e-05 2024-03-21T13:16:27.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.537544+0000", 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0530000000000007e-06 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.454810+0000", 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.91726589700000005 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.454921+0000", 2024-03-21T13:16:27.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111829 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.455048+0000", 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012628199999999999 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460499+0000", 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054510410000000002 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460829+0000", 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033067900000000001 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460862+0000", 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2994999999999998e-05 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460942+0000", 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9216999999999997e-05 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460982+0000", 2024-03-21T13:16:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0833e-05 2024-03-21T13:16:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:195866 8.4 8:238e7e72:::benchmark_data_smithi163_107079_object59993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-21T13:16:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:22.544041+0000", 2024-03-21T13:16:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 544.50928839400001, 2024-03-21T13:16:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92414282599999997, 2024-03-21T13:16:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 195866 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.544041+0000", 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.544041+0000", 2024-03-21T13:16:27.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.544043+0000", 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.077e-06 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.544047+0000", 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5329999999999996e-06 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.544058+0000", 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1072e-05 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.544064+0000", 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1260000000000003e-06 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.544119+0000", 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5028000000000002e-05 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.544126+0000", 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0589999999999997e-06 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.456648+0000", 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.91252123699999999 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.456767+0000", 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119764 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.456896+0000", 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012856200000000001 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460595+0000", 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00369907 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.464549+0000", 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039542450000000003 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.468105+0000", 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035557509999999998 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.468137+0000", 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1643000000000002e-05 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.468184+0000", 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7658999999999999e-05 2024-03-21T13:16:27.106 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:195893 8.4 8:28655e7d:::benchmark_data_smithi163_107079_object60020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:22.563055+0000", 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 544.49027476200001, 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.90832378300000005, 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 195893 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.107 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.563055+0000", 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.563055+0000", 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.563057+0000", 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4670000000000001e-06 2024-03-21T13:16:27.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.563061+0000", 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2699999999999998e-06 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.563072+0000", 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0720000000000001e-05 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.563078+0000", 2024-03-21T13:16:27.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9280000000000004e-06 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.563137+0000", 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.821e-05 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.563146+0000", 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3859999999999996e-06 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.458656+0000", 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.89550964499999997 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.458791+0000", 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000135055 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.458903+0000", 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011195 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.460796+0000", 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001893554 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.464596+0000", 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037997119999999998 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.471286+0000", 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006689988 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.471337+0000", 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1272e-05 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.471379+0000", 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1625999999999998e-05 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:195900 8.4 8:21c7645e:::benchmark_data_smithi163_107079_object60027:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:22.567226+0000", 2024-03-21T13:16:27.113 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 544.48610391499994, 2024-03-21T13:16:27.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.91420859799999998, 2024-03-21T13:16:27.114 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.114 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.114 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.114 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:27.114 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:27.114 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 195900 2024-03-21T13:16:27.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.114 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.567226+0000", 2024-03-21T13:16:27.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.567226+0000", 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.567227+0000", 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3289999999999999e-06 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.567231+0000", 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0899999999999998e-06 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.567240+0000", 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7169999999999997e-06 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.567247+0000", 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7719999999999997e-06 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.567301+0000", 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3943999999999997e-05 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.567309+0000", 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7770000000000002e-06 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.461022+0000", 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.89371387999999996 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.461093+0000", 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0413999999999994e-05 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.461173+0000", 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0122000000000003e-05 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.464561+0000", 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033881580000000001 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.464662+0000", 2024-03-21T13:16:27.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010086699999999999 2024-03-21T13:16:27.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.481359+0000", 2024-03-21T13:16:27.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016697172 2024-03-21T13:16:27.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.481401+0000", 2024-03-21T13:16:27.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1996000000000002e-05 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.481435+0000", 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3359999999999999e-05 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:195907 8.4 8:23aaced3:::benchmark_data_smithi163_107079_object60034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:22.572413+0000", 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 544.48091703, 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.90915148300000004, 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 195907 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.572413+0000", 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.572413+0000", 2024-03-21T13:16:27.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.572414+0000", 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.35e-06 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.572420+0000", 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8130000000000001e-06 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.572432+0000", 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1613e-05 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.572440+0000", 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9510000000000006e-06 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.572505+0000", 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5209000000000005e-05 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.572514+0000", 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8340000000000005e-06 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.462716+0000", 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.890202783 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.462819+0000", 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010293699999999999 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.462944+0000", 2024-03-21T13:16:27.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012504399999999999 2024-03-21T13:16:27.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.468056+0000", 2024-03-21T13:16:27.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051111400000000001 2024-03-21T13:16:27.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.468196+0000", 2024-03-21T13:16:27.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014033399999999999 2024-03-21T13:16:27.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.481476+0000", 2024-03-21T13:16:27.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013279901 2024-03-21T13:16:27.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.481524+0000", 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8646000000000001e-05 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.481564+0000", 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9928000000000001e-05 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:195920 8.4 8:2cbe54e2:::benchmark_data_smithi163_107079_object60047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:22.584426+0000", 2024-03-21T13:16:27.128 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 544.46890387899998, 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.89734890099999998, 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 195920 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.584426+0000", 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.584426+0000", 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.584427+0000", 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3859999999999999e-06 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.584434+0000", 2024-03-21T13:16:27.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4420000000000003e-06 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.584446+0000", 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2170999999999999e-05 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.584456+0000", 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7950000000000002e-06 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.584524+0000", 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7811000000000005e-05 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.584533+0000", 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5310000000000007e-06 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.464697+0000", 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.88016433999999999 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.464808+0000", 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011055799999999999 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.464947+0000", 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013924900000000001 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.468202+0000", 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032550040000000001 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.471241+0000", 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030389670000000001 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.481663+0000", 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010421646999999999 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.481733+0000", 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9783000000000006e-05 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.481775+0000", 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2216999999999999e-05 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:195962 8.4 8:25def4e4:::benchmark_data_smithi163_107079_object60089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:22.625276+0000", 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 544.42805415700002, 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.85669114700000004, 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.135 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 195962 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.625276+0000", 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.625276+0000", 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.625277+0000", 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.446e-06 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.625283+0000", 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5639999999999996e-06 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.625295+0000", 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2442999999999999e-05 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.625303+0000", 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3180000000000007e-06 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.625372+0000", 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8361000000000005e-05 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:22.625381+0000", 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4080000000000004e-06 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.466505+0000", 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.84112367799999999 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.466612+0000", 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010668800000000001 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.466730+0000", 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011864800000000001 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.471190+0000", 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004459691 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.471437+0000", 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024685199999999999 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.481849+0000", 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010412031 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.481916+0000", 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7304999999999993e-05 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:23.481967+0000", 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0714000000000003e-05 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:54267 9.16 9.1e271e76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:19.456157+0000", 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 427.59717330199999, 2024-03-21T13:16:27.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97745112899999997, 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 54267 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.456157+0000", 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.456157+0000", 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.456159+0000", 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.199e-06 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.456320+0000", 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016160299999999999 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.456333+0000", 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.307e-05 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.456342+0000", 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1689999999999992e-06 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.456398+0000", 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6304999999999997e-05 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.456403+0000", 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5180000000000004e-06 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:20.433598+0000", 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97719465299999997 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:20.433608+0000", 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6120000000000005e-06 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.145 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:72931 9.16 9:6aaaafd9:::benchmark_data_smithi163_131650_object72930:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:37.430521+0000", 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.62280882300001, 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98165894499999995, 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 72931 2024-03-21T13:16:27.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430521+0000", 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430521+0000", 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430523+0000", 2024-03-21T13:16:27.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.469e-06 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430561+0000", 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8655999999999997e-05 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430574+0000", 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2709e-05 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430583+0000", 2024-03-21T13:16:27.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3200000000000006e-06 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430703+0000", 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000120108 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430715+0000", 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2119999999999999e-05 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.405884+0000", 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97516835199999996 2024-03-21T13:16:27.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.406053+0000", 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000169715 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6,7", 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.406506+0000", 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00045207599999999999 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.411953+0000", 2024-03-21T13:16:27.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054472940000000001 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.412031+0000", 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7810000000000002e-05 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.412082+0000", 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1245000000000001e-05 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.412140+0000", 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8514000000000003e-05 2024-03-21T13:16:27.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.412180+0000", 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9557000000000001e-05 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:72933 9.16 9:6f53bba1:::benchmark_data_smithi163_131650_object72932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:37.430628+0000", 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.62270209399998, 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98172510499999999, 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.152 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 72933 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430628+0000", 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430628+0000", 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430628+0000", 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2399999999999998e-07 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430674+0000", 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5611000000000001e-05 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430682+0000", 2024-03-21T13:16:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5560000000000008e-06 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430689+0000", 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4529999999999999e-06 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430743+0000", 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3594999999999998e-05 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.430750+0000", 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2439999999999996e-06 2024-03-21T13:16:27.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.408148+0000", 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97739806299999998 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.408277+0000", 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000129453 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6,7", 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.408522+0000", 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024521999999999999 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.412036+0000", 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035135000000000001 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.412223+0000", 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000186589 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.412269+0000", 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6258999999999999e-05 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.412316+0000", 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6795999999999997e-05 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.412353+0000", 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7242e-05 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.158 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:72968 9.16 9:6cfe2966:::benchmark_data_smithi163_131650_object72967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-03-21T13:16:27.159 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:37.455834+0000", 2024-03-21T13:16:27.159 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.59749594900001, 2024-03-21T13:16:27.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95952387100000003, 2024-03-21T13:16:27.159 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.159 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.159 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.159 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:27.159 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:27.159 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 72968 2024-03-21T13:16:27.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.159 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.455834+0000", 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.455834+0000", 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.455836+0000", 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.646e-06 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.455881+0000", 2024-03-21T13:16:27.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5337999999999999e-05 2024-03-21T13:16:27.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.455898+0000", 2024-03-21T13:16:27.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7042e-05 2024-03-21T13:16:27.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.455906+0000", 2024-03-21T13:16:27.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5250000000000005e-06 2024-03-21T13:16:27.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.456047+0000", 2024-03-21T13:16:27.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014048700000000001 2024-03-21T13:16:27.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.456059+0000", 2024-03-21T13:16:27.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2143999999999999e-05 2024-03-21T13:16:27.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.410213+0000", 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95415349800000004 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.410359+0000", 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014685300000000001 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6,7", 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.410621+0000", 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026201000000000002 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.412234+0000", 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001612079 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.415174+0000", 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029407919999999998 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.415247+0000", 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2781000000000007e-05 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.415307+0000", 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9966000000000002e-05 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.415358+0000", 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0710000000000001e-05 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:72969 9.16 9:6d7e65a3:::benchmark_data_smithi163_131650_object72968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:37.455928+0000", 2024-03-21T13:16:27.165 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.597401913, 2024-03-21T13:16:27.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95960509100000002, 2024-03-21T13:16:27.166 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.166 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.166 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.166 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:27.166 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:27.166 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 72969 2024-03-21T13:16:27.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.166 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.455928+0000", 2024-03-21T13:16:27.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.455928+0000", 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.455929+0000", 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5400000000000001e-07 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.455974+0000", 2024-03-21T13:16:27.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5084999999999999e-05 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.455982+0000", 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2549999999999997e-06 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.455988+0000", 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4219999999999997e-06 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.456087+0000", 2024-03-21T13:16:27.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8445999999999999e-05 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.456095+0000", 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7859999999999997e-06 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.412387+0000", 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95629251199999998 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.412531+0000", 2024-03-21T13:16:27.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014425899999999999 2024-03-21T13:16:27.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6,7", 2024-03-21T13:16:27.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.412752+0000", 2024-03-21T13:16:27.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022052700000000001 2024-03-21T13:16:27.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.415188+0000", 2024-03-21T13:16:27.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024356759999999999 2024-03-21T13:16:27.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.415405+0000", 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000217715 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.415445+0000", 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9628000000000001e-05 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.415494+0000", 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8761000000000003e-05 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.415533+0000", 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9365e-05 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:73042 9.16 9:6d07215b:::benchmark_data_smithi163_131650_object73041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:37.508448+0000", 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.54488170100001, 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.91027849400000005, 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:27.172 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 73042 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.508448+0000", 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.508448+0000", 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.508450+0000", 2024-03-21T13:16:27.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.593e-06 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.508481+0000", 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0964999999999999e-05 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.508492+0000", 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1399e-05 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.508501+0000", 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8740000000000001e-06 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.508634+0000", 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013276 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.508654+0000", 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0171000000000001e-05 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.413896+0000", 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.90524256800000003 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.414014+0000", 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117701 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6,7", 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.414234+0000", 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021958599999999999 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.418564+0000", 2024-03-21T13:16:27.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043302590000000004 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.418577+0000", 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3089e-05 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.418617+0000", 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0336999999999997e-05 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.418673+0000", 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5318999999999997e-05 2024-03-21T13:16:27.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.418727+0000", 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3872999999999997e-05 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:73109 9.16 9:6bedc227:::benchmark_data_smithi163_131650_object73108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:37.555097+0000", 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.49823236999998, 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86680632400000002, 2024-03-21T13:16:27.178 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 73109 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.555097+0000", 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.555097+0000", 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.555099+0000", 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5379999999999999e-06 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.555130+0000", 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1430999999999998e-05 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.180 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.555148+0000", 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7062999999999999e-05 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.555156+0000", 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2479999999999996e-06 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.555281+0000", 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012572000000000001 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.555292+0000", 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0148e-05 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.415569+0000", 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86027696399999998 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.415743+0000", 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017419899999999999 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6,7", 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.415973+0000", 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000230535 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.418582+0000", 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026083650000000001 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.418769+0000", 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018692099999999999 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.183 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.421764+0000", 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029949149999999999 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.421862+0000", 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8659000000000005e-05 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.421904+0000", 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1618e-05 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:73151 9.16 9:6923a7e2:::benchmark_data_smithi163_131650_object73150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:37.587635+0000", 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.46569458099998, 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.83767395200000005, 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 73151 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.185 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.587635+0000", 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.587635+0000", 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.587637+0000", 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5349999999999999e-06 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.587690+0000", 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2852000000000002e-05 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.587703+0000", 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.345e-05 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.587711+0000", 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2810000000000008e-06 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.587787+0000", 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5288000000000002e-05 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:37.587797+0000", 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8190000000000003e-06 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.417159+0000", 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.82936271900000003 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:27.188 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.417293+0000", 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013415399999999999 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6,7", 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.417493+0000", 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000199309 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.418779+0000", 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0012861540000000001 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.421960+0000", 2024-03-21T13:16:27.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031812979999999999 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.425176+0000", 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032163320000000001 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.425235+0000", 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8412000000000002e-05 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:38.425309+0000", 2024-03-21T13:16:27.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4349000000000003e-05 2024-03-21T13:16:27.191 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.191 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.191 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.191 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:27.191 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:27.191 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:16:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:27.192 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:27.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:27.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:27.317 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:27.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:27.418 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:27.444 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:27.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:27.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:27.570 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:27.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:27.652 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:27 smithi163 ceph-mon[29621]: osdmap e397: 8 total, 8 up, 7 in 2024-03-21T13:16:27.652 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:27 smithi163 ceph-mon[29621]: osdmap e398: 8 total, 8 up, 7 in 2024-03-21T13:16:27.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:27 smithi163 ceph-mon[40675]: osdmap e397: 8 total, 8 up, 7 in 2024-03-21T13:16:27.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:27 smithi163 ceph-mon[40675]: osdmap e398: 8 total, 8 up, 7 in 2024-03-21T13:16:27.673 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:27.696 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:27.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:27 smithi204 ceph-mon[33153]: osdmap e397: 8 total, 8 up, 7 in 2024-03-21T13:16:27.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:27 smithi204 ceph-mon[33153]: osdmap e398: 8 total, 8 up, 7 in 2024-03-21T13:16:27.799 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:27.824 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:27.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:27.895 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:16:27.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-21T13:16:27.925 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:27.976 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:27.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:28.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:28.101 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:28.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:28.153 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:16:28.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:28.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:28.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:28.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:28.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:28.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:28.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:28.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:28.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:28.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:28.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:28.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:28 smithi163 ceph-mon[29621]: pgmap v836: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:16:28.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:28 smithi163 ceph-mon[40675]: pgmap v836: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:16:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:28.578 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:28.598 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:28 smithi204 ceph-mon[33153]: pgmap v836: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:16:28.602 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:28.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:28.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:28.703 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:28.728 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:28.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:28.829 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:28.855 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:28.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:28.956 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:28.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:28.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:28.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:29.085 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:29.109 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:29.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:29.211 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:29.234 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:29.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:29.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:29.278 DEBUG:teuthology.orchestra.run.smithi163:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-21T13:16:29.281 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-21T13:16:29.281 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-21T13:16:29.329 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:hints = 1 2024-03-21T13:16:29.329 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-21T13:16:29.329 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Object prefix: benchmark_data_smithi163_180069 2024-03-21T13:16:29.329 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:16:29.329 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 0 0 0 0 0 0 - 0 2024-03-21T13:16:29.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:29.374 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:29.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:29.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:29.492 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.2 to 1 (10000) 2024-03-21T13:16:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:29.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:29.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:29.574 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:16:29.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:29 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4170221034' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:16:29.630 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:29.672 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:29.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:29.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:29 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4170221034' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:16:29.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:29 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4170221034' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:16:29.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:29.815 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:29.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:29.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:29.960 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:29.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:30.062 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:30.103 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:30.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:30.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:30.196 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:16:30.196 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:16:30.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:30.196 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.15858.0:2953 11.5 e399/390 11:a36bd7b6:::benchmark_data_smithi163_180069_object2952:head v 399'188, mlcod=399'185)", 2024-03-21T13:16:30.197 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:16:30.194839+0000", 2024-03-21T13:16:30.197 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.00066664199999999995, 2024-03-21T13:16:30.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000696475, 2024-03-21T13:16:30.197 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:30.197 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:16:30.197 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:30.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:30.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:30.197 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:30.194839+0000", 2024-03-21T13:16:30.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:30.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:30.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:30.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:30.194839+0000", 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:30.194839+0000", 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5899999999999996e-07 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:30.194861+0000", 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1525000000000001e-05 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:30.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:30.194884+0000", 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2938999999999998e-05 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:30.194891+0000", 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0249999999999997e-06 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:30.194966+0000", 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5535000000000001e-05 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:30.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:30.200 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:30.194985+0000", 2024-03-21T13:16:30.200 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8700999999999999e-05 2024-03-21T13:16:30.200 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:30.200 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:30.200 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:30.200 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:30.200 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-03-21T13:16:30.200 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 1 2024-03-21T13:16:30.200 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:16:30.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:30.243 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:30.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:30.329 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 1 16 3434 3418 213.613 213.625 0.00642034 0.00466651 2024-03-21T13:16:30.345 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:30.384 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:30.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:30.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:30.486 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:30.523 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:30.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:30.625 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:30.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:30.677 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:30.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:30.690 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-21T13:16:30.701 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:30.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:30 smithi163 ceph-mon[29621]: pgmap v837: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:16:30.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:30 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4170221034' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:16:30.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:30 smithi163 ceph-mon[29621]: osdmap e399: 8 total, 8 up, 7 in 2024-03-21T13:16:30.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:30 smithi163 ceph-mon[40675]: pgmap v837: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:16:30.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:30 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4170221034' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:16:30.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:30 smithi163 ceph-mon[40675]: osdmap e399: 8 total, 8 up, 7 in 2024-03-21T13:16:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:30 smithi204 ceph-mon[33153]: pgmap v837: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:16:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:30 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4170221034' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:16:30.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:30 smithi204 ceph-mon[33153]: osdmap e399: 8 total, 8 up, 7 in 2024-03-21T13:16:30.778 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:30.816 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:30.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:30.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:30.953 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:30.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:31.055 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:31.092 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:31.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:31.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:31.194 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:31.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:31.234 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:31.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:31.329 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 2 16 6674 6658 208.048 202.5 0.00516551 0.0047954 2024-03-21T13:16:31.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:31.378 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:31.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:31.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-21T13:16:31.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:31.536 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:31.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:31.638 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:31.683 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:31.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:31.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:31.784 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:31.828 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:31.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:31.876 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:16:31.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:31.966 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:31.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:31.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:32.068 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:32.110 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:32.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:32.211 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:32.252 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:32.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:32.270 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:16:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:16:32.330 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 3 16 9705 9689 201.839 189.438 0.00557689 0.00494577 2024-03-21T13:16:32.354 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:32.392 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:32.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:32.493 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:32.506 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:16:32.536 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:32.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:32.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:32.676 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:32.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:32 smithi204 ceph-mon[33153]: pgmap v839: 148 pgs: 148 active+clean; 155 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.8 KiB/s rd, 28 MiB/s wr, 442 op/s 2024-03-21T13:16:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:32 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:16:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:32 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:16:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:32 smithi204 ceph-mon[33153]: osdmap e400: 8 total, 8 up, 7 in 2024-03-21T13:16:32.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:32 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3065585142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:32.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:32 smithi163 ceph-mon[29621]: pgmap v839: 148 pgs: 148 active+clean; 155 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.8 KiB/s rd, 28 MiB/s wr, 442 op/s 2024-03-21T13:16:32.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:32 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:16:32.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:32 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:16:32.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:32 smithi163 ceph-mon[29621]: osdmap e400: 8 total, 8 up, 7 in 2024-03-21T13:16:32.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:32 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3065585142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:32.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:32 smithi163 ceph-mon[40675]: pgmap v839: 148 pgs: 148 active+clean; 155 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.8 KiB/s rd, 28 MiB/s wr, 442 op/s 2024-03-21T13:16:32.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:32 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:16:32.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:32 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:16:32.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:32 smithi163 ceph-mon[40675]: osdmap e400: 8 total, 8 up, 7 in 2024-03-21T13:16:32.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:32 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3065585142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:32.779 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:32.818 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:32.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:32.920 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:32.955 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:32.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:32.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:33.057 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:33.099 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:33.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:33.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:33.179 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:33.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:33.246 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:33.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:33.330 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 4 16 12729 12713 198.626 189 0.00684247 0.00502794 2024-03-21T13:16:33.347 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:33.388 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:33.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:33.490 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:33.525 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:33.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:33.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:33.627 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:33.664 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:33.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:33.691 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:33 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:33.691 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:33 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:16:33.691 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:33 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3065585142' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:16:33.691 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:33 smithi204 ceph-mon[33153]: osdmap e401: 8 total, 8 up, 7 in 2024-03-21T13:16:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:33 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:33 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:16:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:33 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3065585142' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:16:33.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:33 smithi163 ceph-mon[29621]: osdmap e401: 8 total, 8 up, 7 in 2024-03-21T13:16:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:33 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:33 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:16:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:33 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3065585142' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:16:33.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:33 smithi163 ceph-mon[40675]: osdmap e401: 8 total, 8 up, 7 in 2024-03-21T13:16:33.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:33.806 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:33.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:33.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:33.947 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:33.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:34.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:34.064 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:16:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-03-21T13:16:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-03-21T13:16:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-03-21T13:16:34.064 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:16:34.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:34.087 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:34.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:34.188 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:34.237 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:34.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:34.329 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 5 16 15531 15515 193.922 175.125 0.0082055 0.00514817 2024-03-21T13:16:34.338 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:34.382 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:34.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:34.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:34.484 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:34.532 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:34.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:34.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:34.634 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:34.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:34.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:34.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:34 smithi204 ceph-mon[33153]: pgmap v842: 148 pgs: 148 active+clean; 274 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 46 MiB/s wr, 729 op/s 2024-03-21T13:16:34.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:34 smithi163 ceph-mon[29621]: pgmap v842: 148 pgs: 148 active+clean; 274 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 46 MiB/s wr, 729 op/s 2024-03-21T13:16:34.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:34 smithi163 ceph-mon[40675]: pgmap v842: 148 pgs: 148 active+clean; 274 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 46 MiB/s wr, 729 op/s 2024-03-21T13:16:34.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:34.810 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:34.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:34.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:34.850 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-21T13:16:34.870 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:16:34.909 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:16:34.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:34.949 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:16:34.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:34.976 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:34.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:34.991 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:16:35.029 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:16:35.069 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:16:35.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:35.114 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:16:35.120 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:35.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:35.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:35.163 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:16:35.166 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:16:35.166 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:16:35.166 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:16:35.166 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:16:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:16:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:16:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:16:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:16:35.222 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:35.272 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:35.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:35.329 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 6 16 18075 18059 188.1 159 0.00515472 0.00531016 2024-03-21T13:16:35.374 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:35.412 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:35.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:35.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:35.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:35.531 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:16:35.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:35.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:35.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:35.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:35 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2739336490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:16:35.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:35.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:35 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2739336490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:16:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:35.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:35.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:35.694 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:35.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:35.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:35.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:35.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:35.834 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:35.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:35.935 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:35.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:35.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:35 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2739336490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:16:36.041 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:16:36.041 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:36.042 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:16:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:16:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:16:36.078 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:36.116 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:36.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:36.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:36.216 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:16:36.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:36.257 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:36.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:36.330 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 7 16 20702 20686 184.676 164.188 0.00704182 0.0054081 2024-03-21T13:16:36.359 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:36.401 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:36.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:36.502 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:36.546 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:36.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:36.648 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:36.685 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:36.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:36.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:36.782 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:36 smithi163 ceph-mon[29621]: pgmap v843: 148 pgs: 148 active+clean; 332 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 884 op/s 2024-03-21T13:16:36.782 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:36 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:16:36.782 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:36 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:16:36.782 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:36 smithi163 ceph-mon[29621]: osdmap e402: 8 total, 8 up, 7 in 2024-03-21T13:16:36.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:36 smithi163 ceph-mon[40675]: pgmap v843: 148 pgs: 148 active+clean; 332 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 884 op/s 2024-03-21T13:16:36.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:36 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:16:36.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:36 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:16:36.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:36 smithi163 ceph-mon[40675]: osdmap e402: 8 total, 8 up, 7 in 2024-03-21T13:16:36.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:36.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:36.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:36.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:36.925 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:36.958 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:36.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:36 smithi204 ceph-mon[33153]: pgmap v843: 148 pgs: 148 active+clean; 332 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 884 op/s 2024-03-21T13:16:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:36 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:16:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:36 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:16:37.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:36 smithi204 ceph-mon[33153]: osdmap e402: 8 total, 8 up, 7 in 2024-03-21T13:16:37.060 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:37.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:37.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:37.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:37.237 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:37.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:37.329 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 8 15 23207 23192 181.168 156.625 0.00890268 0.0055142 2024-03-21T13:16:37.340 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:37.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:37.379 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:37.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:37.480 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:37.525 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:37.530 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:37.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:37.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:37.632 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:37.670 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:37.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:37.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:37.772 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:37.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:37.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:37.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:37.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:37.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:38.053 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:38.098 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:38.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:38.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:38.199 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:38.233 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:38.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:38.330 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 9 16 25433 25417 176.488 139.062 0.00447664 0.00566104 2024-03-21T13:16:38.335 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:38.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:38.371 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:38.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:38.472 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:38.511 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:38.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:38.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:38.650 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:38.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:38.752 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:38.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:38 smithi204 ceph-mon[33153]: pgmap v845: 148 pgs: 148 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 170 MiB/s wr, 2.72k op/s 2024-03-21T13:16:38.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:38 smithi163 ceph-mon[29621]: pgmap v845: 148 pgs: 148 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 170 MiB/s wr, 2.72k op/s 2024-03-21T13:16:38.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:38 smithi163 ceph-mon[40675]: pgmap v845: 148 pgs: 148 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 170 MiB/s wr, 2.72k op/s 2024-03-21T13:16:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:38.776 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:38.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:38.834 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:16:38.878 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:38.912 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:38.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:39.014 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:39.057 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:39.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:39.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:39.159 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:39.192 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:39.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:39.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:39.227 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:16:39.227 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:16:39.294 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:39.330 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 10 16 27363 27347 170.901 120.625 0.00829887 0.00584593 2024-03-21T13:16:39.331 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:39.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:39.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:39.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:39.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:39.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:39.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:39.608 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:39.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:39.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:39.749 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:39.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:39.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:39.851 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:39.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:39.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:39.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:40.017 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:40.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:40.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:40.119 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:40.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:40.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:40.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-21T13:16:40.256 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:40.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:40.296 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:40.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:40.330 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 11 16 29222 29206 165.926 116.188 0.00480623 0.00600717 2024-03-21T13:16:40.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:40.447 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:40.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:40.548 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:40.569 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:16:40.582 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:40.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:40.684 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:40.723 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:40.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:40.777 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:40 smithi204 ceph-mon[33153]: pgmap v846: 148 pgs: 148 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 684 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-21T13:16:40.825 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:40.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:40.850 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:40.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:40.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:40.986 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:40.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:40.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:41.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:40 smithi163 ceph-mon[29621]: pgmap v846: 148 pgs: 148 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 684 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-21T13:16:41.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:40 smithi163 ceph-mon[40675]: pgmap v846: 148 pgs: 148 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 684 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-21T13:16:41.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:41.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:41.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:41.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:41.217 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:41.255 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:41.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:41.330 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 12 16 30970 30954 161.202 109.25 0.00191977 0.00618616 2024-03-21T13:16:41.357 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:41.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:41.392 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:41.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:41.494 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:41.527 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:41.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:41.555 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.5 to 1 (10000) 2024-03-21T13:16:41.628 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:41.656 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:41.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:41.758 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:41.796 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:41.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:41.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:41.862 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:41 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/495193243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:16:41.862 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:41 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:16:41.898 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:41.927 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:41.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:41.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:41 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/495193243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:16:42.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:41 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:16:42.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:41 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/495193243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:16:42.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:41 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:16:42.029 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:42.062 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:42.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:42.164 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:42.188 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:42.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:42.289 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:42.322 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:42.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:42.330 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 13 16 32308 32292 155.234 83.625 0.00245787 0.00643487 2024-03-21T13:16:42.423 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:42.446 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:42.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:42.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:42.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:16:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:16:42.548 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:42.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:42.580 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:42.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:42.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:42.717 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:42.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:42.819 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:42.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:42.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:42 smithi163 ceph-mon[29621]: pgmap v847: 148 pgs: 148 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 163 MiB/s wr, 2.62k op/s 2024-03-21T13:16:42.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:42 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:16:42.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:42 smithi163 ceph-mon[29621]: osdmap e403: 8 total, 8 up, 7 in 2024-03-21T13:16:42.884 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:42 smithi163 ceph-mon[40675]: pgmap v847: 148 pgs: 148 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 163 MiB/s wr, 2.62k op/s 2024-03-21T13:16:42.884 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:42 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:16:42.884 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:42 smithi163 ceph-mon[40675]: osdmap e403: 8 total, 8 up, 7 in 2024-03-21T13:16:42.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:42.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:42.985 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:42.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:43.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:42 smithi204 ceph-mon[33153]: pgmap v847: 148 pgs: 148 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 163 MiB/s wr, 2.62k op/s 2024-03-21T13:16:43.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:42 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:16:43.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:42 smithi204 ceph-mon[33153]: osdmap e403: 8 total, 8 up, 7 in 2024-03-21T13:16:43.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:43.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:43.121 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:43.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:43.222 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:43.253 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:43.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:43.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:43.330 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 14 16 33942 33926 151.44 102.125 0.0117098 0.00659737 2024-03-21T13:16:43.355 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:43.393 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:43.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:43.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:43.494 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:43.530 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:43.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:43.632 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:43.669 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:43.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:43.771 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:43.802 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:43.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:43.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:43.904 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:43.942 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:43.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:44.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:44.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:44.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:44.044 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:44.080 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:44.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:44.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:44.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:44.218 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:44.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:44.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:44.320 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:44.330 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 15 16 35917 35901 149.572 123.438 0.0103163 0.0066799 2024-03-21T13:16:44.358 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:44.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:44.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:44.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:44.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:44.599 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:44.634 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:44.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:44.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:44.736 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:44.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:44.776 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:44.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:44.832 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:44 smithi163 ceph-mon[29621]: pgmap v849: 148 pgs: 148 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 191 MiB/s wr, 3.05k op/s 2024-03-21T13:16:44.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:44 smithi163 ceph-mon[40675]: pgmap v849: 148 pgs: 148 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 191 MiB/s wr, 3.05k op/s 2024-03-21T13:16:44.865 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:44 smithi204 ceph-mon[33153]: pgmap v849: 148 pgs: 148 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 191 MiB/s wr, 3.05k op/s 2024-03-21T13:16:44.877 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:44.888 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-21T13:16:44.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:16:44.919 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:44.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:44.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:45.021 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:45.057 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:45.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:45.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:45.159 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:45.198 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:45.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:45.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:45.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:45.330 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 16 16 37814 37798 147.633 118.562 0.00682248 0.00676834 2024-03-21T13:16:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:45.346 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:45.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:45.435 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:16:45.435 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:16:45.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:45.491 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:45.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:45.593 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:45.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:45.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:45.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:45.671 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:45.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:45.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:45 smithi204 ceph-mon[33153]: pgmap v850: 148 pgs: 148 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 542 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-21T13:16:45.771 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:45.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:45.795 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:45 smithi163 ceph-mon[29621]: pgmap v850: 148 pgs: 148 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 542 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-21T13:16:45.796 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:16:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:16:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:16:45.796 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:45 smithi163 ceph-mon[40675]: pgmap v850: 148 pgs: 148 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 542 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-21T13:16:45.873 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:45.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:45.913 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:45.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:46.014 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:46.060 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:46.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:46.162 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:46.199 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:46.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:46.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:46.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:46.330 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 17 16 39531 39515 145.261 107.312 0.00250373 0.00687868 2024-03-21T13:16:46.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:46.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:46.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:46.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-21T13:16:46.478 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:46.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:46.579 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:46.625 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:46.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:46.727 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:46.767 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:46.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:46.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:46.869 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:46.873 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:16:46.903 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:46.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:46.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:47.005 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:47.042 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:47.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:47.144 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:47.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:47.184 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:47.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:47.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:47.322 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:47.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:47.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:47.330 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 18 16 41240 41224 143.124 106.812 0.00478359 0.00698198 2024-03-21T13:16:47.424 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:47.462 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:47.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:47.506 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:16:47.564 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:47.602 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:47.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:47.704 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:47.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:47.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:47.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:47.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:47 smithi204 ceph-mon[33153]: osdmap e404: 8 total, 8 up, 7 in 2024-03-21T13:16:47.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:47 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/863317426' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:16:47.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:47 smithi163 ceph-mon[29621]: osdmap e404: 8 total, 8 up, 7 in 2024-03-21T13:16:47.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:47 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/863317426' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:16:47.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:47 smithi163 ceph-mon[40675]: osdmap e404: 8 total, 8 up, 7 in 2024-03-21T13:16:47.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:47 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/863317426' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:16:47.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:47.888 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:47.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:47.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:48.027 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:48.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:48.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:48.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:48.166 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:48.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:48.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:48.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:48.313 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:48.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:48.331 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 19 16 42958 42942 141.242 107.375 0.00386604 0.00707427 2024-03-21T13:16:48.414 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:48.452 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:48.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:48.554 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:48.590 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:48.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:48.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:48.671 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:48.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:48.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:48.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:48.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:48.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:48 smithi204 ceph-mon[33153]: pgmap v852: 148 pgs: 148 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.44k op/s 2024-03-21T13:16:48.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:48 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:16:48.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:48 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/863317426' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:16:48.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:48 smithi204 ceph-mon[33153]: osdmap e405: 8 total, 8 up, 7 in 2024-03-21T13:16:48.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:48 smithi204 ceph-mon[33153]: 3.d scrub starts 2024-03-21T13:16:48.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:48 smithi204 ceph-mon[33153]: 3.d scrub ok 2024-03-21T13:16:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:48 smithi163 ceph-mon[29621]: pgmap v852: 148 pgs: 148 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.44k op/s 2024-03-21T13:16:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:48 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:16:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:48 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/863317426' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:16:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:48 smithi163 ceph-mon[29621]: osdmap e405: 8 total, 8 up, 7 in 2024-03-21T13:16:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:48 smithi163 ceph-mon[29621]: 3.d scrub starts 2024-03-21T13:16:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:48 smithi163 ceph-mon[29621]: 3.d scrub ok 2024-03-21T13:16:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:48 smithi163 ceph-mon[40675]: pgmap v852: 148 pgs: 148 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.44k op/s 2024-03-21T13:16:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:48 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:16:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:48 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/863317426' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:16:48.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:48 smithi163 ceph-mon[40675]: osdmap e405: 8 total, 8 up, 7 in 2024-03-21T13:16:48.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:48 smithi163 ceph-mon[40675]: 3.d scrub starts 2024-03-21T13:16:48.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:48 smithi163 ceph-mon[40675]: 3.d scrub ok 2024-03-21T13:16:48.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:48.871 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:48.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:48.973 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:49.016 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:49.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:49.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:49.097 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:16:49.098 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:16:49.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:49.152 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:49.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:49.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:49.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:49.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:49.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:49.331 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:16:49.331566+0000 min lat: 0.00136681 max lat: 0.150399 avg lat: 0.00716842 2024-03-21T13:16:49.331 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:16:49.331 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 20 16 44627 44611 139.396 104.312 0.00892476 0.00716842 2024-03-21T13:16:49.393 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:49.429 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:49.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:49.531 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:49.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:49.564 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:49.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:49.665 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:49.703 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:49.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:49.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:49.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:49 smithi204 ceph-mon[33153]: 3.1a scrub starts 2024-03-21T13:16:49.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:49 smithi204 ceph-mon[33153]: 3.1a scrub ok 2024-03-21T13:16:49.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:49 smithi204 ceph-mon[33153]: 5.b scrub starts 2024-03-21T13:16:49.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:49 smithi204 ceph-mon[33153]: 5.b scrub ok 2024-03-21T13:16:49.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:49 smithi163 ceph-mon[29621]: 3.1a scrub starts 2024-03-21T13:16:49.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:49 smithi163 ceph-mon[29621]: 3.1a scrub ok 2024-03-21T13:16:49.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:49 smithi163 ceph-mon[29621]: 5.b scrub starts 2024-03-21T13:16:49.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:49 smithi163 ceph-mon[29621]: 5.b scrub ok 2024-03-21T13:16:49.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:49 smithi163 ceph-mon[40675]: 3.1a scrub starts 2024-03-21T13:16:49.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:49 smithi163 ceph-mon[40675]: 3.1a scrub ok 2024-03-21T13:16:49.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:49 smithi163 ceph-mon[40675]: 5.b scrub starts 2024-03-21T13:16:49.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:49 smithi163 ceph-mon[40675]: 5.b scrub ok 2024-03-21T13:16:49.806 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:49.850 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:49.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:49.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:49.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-21T13:16:49.952 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:49.994 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:49.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:50.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:50.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:50.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:50.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:50.238 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:50.294 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:50.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:50.331 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 21 16 46230 46214 137.528 100.188 0.00550026 0.00726468 2024-03-21T13:16:50.370 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:16:50.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:50.433 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:50.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:50.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:50.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:50.550 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:50 smithi204 ceph-mon[33153]: pgmap v854: 148 pgs: 148 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 690 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-21T13:16:50.550 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:50 smithi204 ceph-mon[33153]: 6.10 scrub starts 2024-03-21T13:16:50.550 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:50 smithi204 ceph-mon[33153]: 6.10 scrub ok 2024-03-21T13:16:50.550 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:50.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:50.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:50.681 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:50.715 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:16:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:50.724 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:50.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:50.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:50 smithi163 ceph-mon[29621]: pgmap v854: 148 pgs: 148 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 690 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-21T13:16:50.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:50 smithi163 ceph-mon[29621]: 6.10 scrub starts 2024-03-21T13:16:50.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:50 smithi163 ceph-mon[29621]: 6.10 scrub ok 2024-03-21T13:16:50.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:50.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:50 smithi163 ceph-mon[40675]: pgmap v854: 148 pgs: 148 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 690 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-21T13:16:50.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:50 smithi163 ceph-mon[40675]: 6.10 scrub starts 2024-03-21T13:16:50.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:50 smithi163 ceph-mon[40675]: 6.10 scrub ok 2024-03-21T13:16:50.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:50.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:50.864 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:50.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:50.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:50.966 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:51.018 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:51.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:51.120 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:51.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:51.176 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:51.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:51.203 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:16:51.209 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:16:51.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:51.311 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:51.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:51.331 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 22 16 47726 47710 135.526 93.5 0.00898642 0.00737312 2024-03-21T13:16:51.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:51.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:51.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:51.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:51.588 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:51.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:51.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:51.640 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:16:51.640 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:16:51.640 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:16:51.640 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:16:51.640 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:16:51.677 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:16:51.690 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:51.722 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:16:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:51.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:51.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:51 smithi163 ceph-mon[29621]: 6.0 scrub starts 2024-03-21T13:16:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:51 smithi163 ceph-mon[29621]: 6.0 scrub ok 2024-03-21T13:16:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:51 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1244531977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:16:51.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:51 smithi163 ceph-mon[29621]: osdmap e406: 8 total, 8 up, 7 in 2024-03-21T13:16:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:51 smithi163 ceph-mon[40675]: 6.0 scrub starts 2024-03-21T13:16:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:51 smithi163 ceph-mon[40675]: 6.0 scrub ok 2024-03-21T13:16:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:51 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1244531977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:16:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:51 smithi163 ceph-mon[40675]: osdmap e406: 8 total, 8 up, 7 in 2024-03-21T13:16:51.762 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:16:51.763 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:16:51.763 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:16:51.764 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:16:51.764 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:16:51.764 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:16:51.764 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:16:51.764 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:16:51.764 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:16:51.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:51.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:51.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:51.934 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:51 smithi204 ceph-mon[33153]: 6.0 scrub starts 2024-03-21T13:16:51.934 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:51 smithi204 ceph-mon[33153]: 6.0 scrub ok 2024-03-21T13:16:51.934 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:51 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1244531977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:16:51.935 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:51 smithi204 ceph-mon[33153]: osdmap e406: 8 total, 8 up, 7 in 2024-03-21T13:16:51.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:52.025 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:52.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:52.127 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:52.164 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:52.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:52.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:52.197 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":855,"stamp":"2024-03-21T13:16:50.961894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2887850408,"num_objects":44254,"num_object_clones":0,"num_object_copies":132762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44254,"num_whiteouts":0,"num_read":2418,"num_read_kb":2264,"num_write":44957,"num_write_kb":2822045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":1779777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45041,"ondisk_log_size":45041,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14212100,"kb_used_data":8542024,"kb_used_omap":10,"kb_used_meta":5670005,"kb_avail":642003964,"statfs":{"total":671965249536,"available":657412059136,"internally_reserved":0,"allocated":8747032576,"data_stored":8743603889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":5806085227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":939130880,"num_objects":14330,"num_object_clones":0,"num_object_copies":42990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14330,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":14330,"num_write_kb":917120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002806"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":167,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504921+0000","last_change":"2024-03-21T13:14:59.311326+0000","last_active":"2024-03-21T13:16:47.504921+0000","last_peered":"2024-03-21T13:16:47.504921+0000","last_clean":"2024-03-21T13:16:47.504921+0000","last_became_active":"2024-03-21T13:12:29.525755+0000","last_became_peered":"2024-03-21T13:12:29.525755+0000","last_unstale":"2024-03-21T13:16:47.504921+0000","last_undegraded":"2024-03-21T13:16:47.504921+0000","last_fullsized":"2024-03-21T13:16:47.504921+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:59.311141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:14:59.311141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198679+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":175,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.502822+0000","last_change":"2024-03-21T13:16:21.204738+0000","last_active":"2024-03-21T13:16:47.502822+0000","last_peered":"2024-03-21T13:16:47.502822+0000","last_clean":"2024-03-21T13:16:47.502822+0000","last_became_active":"2024-03-21T13:12:29.510271+0000","last_became_peered":"2024-03-21T13:12:29.510271+0000","last_unstale":"2024-03-21T13:16:47.502822+0000","last_undegraded":"2024-03-21T13:16:47.502822+0000","last_fullsized":"2024-03-21T13:16:47.502822+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:21.204657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.204657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:31.150802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":196,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.508911+0000","last_change":"2024-03-21T13:16:02.044946+0000","last_active":"2024-03-21T13:16:47.508911+0000","last_peered":"2024-03-21T13:16:47.508911+0000","last_clean":"2024-03-21T13:16:47.508911+0000","last_became_active":"2024-03-21T13:15:04.462014+0000","last_became_peered":"2024-03-21T13:15:04.462014+0000","last_unstale":"2024-03-21T13:16:47.508911+0000","last_undegraded":"2024-03-21T13:16:47.508911+0000","last_fullsized":"2024-03-21T13:16:47.508911+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:02.044853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:02.044853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:31.722187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"234'19","reported_seq":820,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.499940+0000","last_change":"2024-03-21T13:16:09.694036+0000","last_active":"2024-03-21T13:16:47.499940+0000","last_peered":"2024-03-21T13:16:47.499940+0000","last_clean":"2024-03-21T13:16:47.499940+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:16:47.499940+0000","last_undegraded":"2024-03-21T13:16:47.499940+0000","last_fullsized":"2024-03-21T13:16:47.499940+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:09.693933+0000","last_deep_scrub":"234'19","last_deep_scrub_stamp":"2024-03-21T13:16:09.693933+0000","last_clean_scrub_stamp":"2024-03-21T13:16:09.693933+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.080379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":715,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.546629+0000","last_change":"2024-03-21T13:15:02.457035+0000","last_active":"2024-03-21T13:16:41.546629+0000","last_peered":"2024-03-21T13:16:41.546629+0000","last_clean":"2024-03-21T13:16:41.546629+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:16:41.546629+0000","last_undegraded":"2024-03-21T13:16:41.546629+0000","last_fullsized":"2024-03-21T13:16:41.546629+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:02.456902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:15:02.456902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.588057+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":669,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504320+0000","last_change":"2024-03-21T13:14:05.582597+0000","last_active":"2024-03-21T13:16:47.504320+0000","last_peered":"2024-03-21T13:16:47.504320+0000","last_clean":"2024-03-21T13:16:47.504320+0000","last_became_active":"2024-03-21T13:10:03.591543+0000","last_became_peered":"2024-03-21T13:10:03.591543+0000","last_unstale":"2024-03-21T13:16:47.504320+0000","last_undegraded":"2024-03-21T13:16:47.504320+0000","last_fullsized":"2024-03-21T13:16:47.504320+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:05.582473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:14:05.582473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:50.916138+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":748,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.545809+0000","last_change":"2024-03-21T13:14:11.126777+0000","last_active":"2024-03-21T13:16:41.545809+0000","last_peered":"2024-03-21T13:16:41.545809+0000","last_clean":"2024-03-21T13:16:41.545809+0000","last_became_active":"2024-03-21T13:10:03.610318+0000","last_became_peered":"2024-03-21T13:10:03.610318+0000","last_unstale":"2024-03-21T13:16:41.545809+0000","last_undegraded":"2024-03-21T13:16:41.545809+0000","last_fullsized":"2024-03-21T13:16:41.545809+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_clean_scrub_stamp":"2024-03-21T13:14:11.126531+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:50.715288+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":250,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.506720+0000","last_change":"2024-03-21T13:14:54.623357+0000","last_active":"2024-03-21T13:16:47.506720+0000","last_peered":"2024-03-21T13:16:47.506720+0000","last_clean":"2024-03-21T13:16:47.506720+0000","last_became_active":"2024-03-21T13:11:21.702209+0000","last_became_peered":"2024-03-21T13:11:21.702209+0000","last_unstale":"2024-03-21T13:16:47.506720+0000","last_undegraded":"2024-03-21T13:16:47.506720+0000","last_fullsized":"2024-03-21T13:16:47.506720+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:14:54.619347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:54.619347+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:47.641640+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":425,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.505906+0000","last_change":"2024-03-21T13:16:20.952006+0000","last_active":"2024-03-21T13:16:47.505906+0000","last_peered":"2024-03-21T13:16:47.505906+0000","last_clean":"2024-03-21T13:16:47.505906+0000","last_became_active":"2024-03-21T13:15:04.459860+0000","last_became_peered":"2024-03-21T13:15:04.459860+0000","last_unstale":"2024-03-21T13:16:47.505906+0000","last_undegraded":"2024-03-21T13:16:47.505906+0000","last_fullsized":"2024-03-21T13:16:47.505906+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:20.951893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.951893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:35.358793+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":167,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.502876+0000","last_change":"2024-03-21T13:15:12.302278+0000","last_active":"2024-03-21T13:16:47.502876+0000","last_peered":"2024-03-21T13:16:47.502876+0000","last_clean":"2024-03-21T13:16:47.502876+0000","last_became_active":"2024-03-21T13:12:29.518784+0000","last_became_peered":"2024-03-21T13:12:29.518784+0000","last_unstale":"2024-03-21T13:16:47.502876+0000","last_undegraded":"2024-03-21T13:16:47.502876+0000","last_fullsized":"2024-03-21T13:16:47.502876+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:12.302013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:15:12.302013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198735+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":167,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504737+0000","last_change":"2024-03-21T13:15:18.085333+0000","last_active":"2024-03-21T13:16:47.504737+0000","last_peered":"2024-03-21T13:16:47.504737+0000","last_clean":"2024-03-21T13:16:47.504737+0000","last_became_active":"2024-03-21T13:12:29.526129+0000","last_became_peered":"2024-03-21T13:12:29.526129+0000","last_unstale":"2024-03-21T13:16:47.504737+0000","last_undegraded":"2024-03-21T13:16:47.504737+0000","last_fullsized":"2024-03-21T13:16:47.504737+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:18.085215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:15:18.085215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:47.894308+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":726,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.546945+0000","last_change":"2024-03-21T13:16:07.739334+0000","last_active":"2024-03-21T13:16:41.546945+0000","last_peered":"2024-03-21T13:16:41.546945+0000","last_clean":"2024-03-21T13:16:41.546945+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:16:41.546945+0000","last_undegraded":"2024-03-21T13:16:41.546945+0000","last_fullsized":"2024-03-21T13:16:41.546945+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:07.739235+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.739235+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.631193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":167,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.502922+0000","last_change":"2024-03-21T13:14:05.545713+0000","last_active":"2024-03-21T13:16:47.502922+0000","last_peered":"2024-03-21T13:16:47.502922+0000","last_clean":"2024-03-21T13:16:47.502922+0000","last_became_active":"2024-03-21T13:12:29.508520+0000","last_became_peered":"2024-03-21T13:12:29.508520+0000","last_unstale":"2024-03-21T13:16:47.502922+0000","last_undegraded":"2024-03-21T13:16:47.502922+0000","last_fullsized":"2024-03-21T13:16:47.502922+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:05.545425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:05.545425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198465+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":742,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.506660+0000","last_change":"2024-03-21T13:15:58.658853+0000","last_active":"2024-03-21T13:16:47.506660+0000","last_peered":"2024-03-21T13:16:47.506660+0000","last_clean":"2024-03-21T13:16:47.506660+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:16:47.506660+0000","last_undegraded":"2024-03-21T13:16:47.506660+0000","last_fullsized":"2024-03-21T13:16:47.506660+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:58.658749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:58.658749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:05.447652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":167,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503322+0000","last_change":"2024-03-21T13:16:04.184426+0000","last_active":"2024-03-21T13:16:47.503322+0000","last_peered":"2024-03-21T13:16:47.503322+0000","last_clean":"2024-03-21T13:16:47.503322+0000","last_became_active":"2024-03-21T13:12:29.524473+0000","last_became_peered":"2024-03-21T13:12:29.524473+0000","last_unstale":"2024-03-21T13:16:47.503322+0000","last_undegraded":"2024-03-21T13:16:47.503322+0000","last_fullsized":"2024-03-21T13:16:47.503322+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:04.184314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:04.184314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:12.162233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"234'9","reported_seq":746,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.546227+0000","last_change":"2024-03-21T13:14:52.156420+0000","last_active":"2024-03-21T13:16:41.546227+0000","last_peered":"2024-03-21T13:16:41.546227+0000","last_clean":"2024-03-21T13:16:41.546227+0000","last_became_active":"2024-03-21T13:10:03.595745+0000","last_became_peered":"2024-03-21T13:10:03.595745+0000","last_unstale":"2024-03-21T13:16:41.546227+0000","last_undegraded":"2024-03-21T13:16:41.546227+0000","last_fullsized":"2024-03-21T13:16:41.546227+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'9","last_scrub_stamp":"2024-03-21T13:14:52.156318+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:14:52.156318+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587479+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":773,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.501372+0000","last_change":"2024-03-21T13:16:06.811101+0000","last_active":"2024-03-21T13:16:47.501372+0000","last_peered":"2024-03-21T13:16:47.501372+0000","last_clean":"2024-03-21T13:16:47.501372+0000","last_became_active":"2024-03-21T13:12:29.520189+0000","last_became_peered":"2024-03-21T13:12:29.520189+0000","last_unstale":"2024-03-21T13:16:47.501372+0000","last_undegraded":"2024-03-21T13:16:47.501372+0000","last_fullsized":"2024-03-21T13:16:47.501372+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:06.810993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:23.865785+0000","last_clean_scrub_stamp":"2024-03-21T13:16:06.810993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:18.896276+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":659,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.501229+0000","last_change":"2024-03-21T13:16:07.756974+0000","last_active":"2024-03-21T13:16:47.501229+0000","last_peered":"2024-03-21T13:16:47.501229+0000","last_clean":"2024-03-21T13:16:47.501229+0000","last_became_active":"2024-03-21T13:12:29.520241+0000","last_became_peered":"2024-03-21T13:12:29.520241+0000","last_unstale":"2024-03-21T13:16:47.501229+0000","last_undegraded":"2024-03-21T13:16:47.501229+0000","last_fullsized":"2024-03-21T13:16:47.501229+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:07.756896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.756896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.528460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":196,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.514467+0000","last_change":"2024-03-21T13:15:16.781907+0000","last_active":"2024-03-21T13:16:47.514467+0000","last_peered":"2024-03-21T13:16:47.514467+0000","last_clean":"2024-03-21T13:16:47.514467+0000","last_became_active":"2024-03-21T13:15:04.464586+0000","last_became_peered":"2024-03-21T13:15:04.464586+0000","last_unstale":"2024-03-21T13:16:47.514467+0000","last_undegraded":"2024-03-21T13:16:47.514467+0000","last_fullsized":"2024-03-21T13:16:47.514467+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:16.781632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:15:16.781632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":311,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.505724+0000","last_change":"2024-03-21T13:15:00.580047+0000","last_active":"2024-03-21T13:16:47.505724+0000","last_peered":"2024-03-21T13:16:47.505724+0000","last_clean":"2024-03-21T13:16:47.505724+0000","last_became_active":"2024-03-21T13:10:03.604616+0000","last_became_peered":"2024-03-21T13:10:03.604616+0000","last_unstale":"2024-03-21T13:16:47.505724+0000","last_undegraded":"2024-03-21T13:16:47.505724+0000","last_fullsized":"2024-03-21T13:16:47.505724+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:15:00.579867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:00.579867+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.367113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":740,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.545465+0000","last_change":"2024-03-21T13:15:59.096623+0000","last_active":"2024-03-21T13:16:41.545465+0000","last_peered":"2024-03-21T13:16:41.545465+0000","last_clean":"2024-03-21T13:16:41.545465+0000","last_became_active":"2024-03-21T13:12:29.502808+0000","last_became_peered":"2024-03-21T13:12:29.502808+0000","last_unstale":"2024-03-21T13:16:41.545465+0000","last_undegraded":"2024-03-21T13:16:41.545465+0000","last_fullsized":"2024-03-21T13:16:41.545465+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:59.096520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:59.096520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.250778+0000","scrub_duration":233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":652,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.545639+0000","last_change":"2024-03-21T13:14:53.198725+0000","last_active":"2024-03-21T13:16:41.545639+0000","last_peered":"2024-03-21T13:16:41.545639+0000","last_clean":"2024-03-21T13:16:41.545639+0000","last_became_active":"2024-03-21T13:10:03.612409+0000","last_became_peered":"2024-03-21T13:10:03.612409+0000","last_unstale":"2024-03-21T13:16:41.545639+0000","last_undegraded":"2024-03-21T13:16:41.545639+0000","last_fullsized":"2024-03-21T13:16:41.545639+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:53.198601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:14:53.198601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":789,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:48.967355+0000","last_change":"2024-03-21T13:16:48.967395+0000","last_active":"2024-03-21T13:16:48.967355+0000","last_peered":"2024-03-21T13:16:48.967355+0000","last_clean":"2024-03-21T13:16:48.967355+0000","last_became_active":"2024-03-21T13:10:03.586563+0000","last_became_peered":"2024-03-21T13:10:03.586563+0000","last_unstale":"2024-03-21T13:16:48.967355+0000","last_undegraded":"2024-03-21T13:16:48.967355+0000","last_fullsized":"2024-03-21T13:16:48.967355+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:48.967258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:20.956778+0000","last_clean_scrub_stamp":"2024-03-21T13:16:48.967258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:52.310270+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":783,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.545828+0000","last_change":"2024-03-21T13:14:04.151045+0000","last_active":"2024-03-21T13:16:41.545828+0000","last_peered":"2024-03-21T13:16:41.545828+0000","last_clean":"2024-03-21T13:16:41.545828+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:16:41.545828+0000","last_undegraded":"2024-03-21T13:16:41.545828+0000","last_fullsized":"2024-03-21T13:16:41.545828+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-21T13:14:04.150913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:04.150913+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587124+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":147,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.507387+0000","last_change":"2024-03-21T13:15:12.611589+0000","last_active":"2024-03-21T13:16:47.507387+0000","last_peered":"2024-03-21T13:16:47.507387+0000","last_clean":"2024-03-21T13:16:47.507387+0000","last_became_active":"2024-03-21T13:13:06.420220+0000","last_became_peered":"2024-03-21T13:13:06.420220+0000","last_unstale":"2024-03-21T13:16:47.507387+0000","last_undegraded":"2024-03-21T13:16:47.507387+0000","last_fullsized":"2024-03-21T13:16:47.507387+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:15:12.611167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:12.611167+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:47.641702+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":961,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503518+0000","last_change":"2024-03-21T13:16:12.489669+0000","last_active":"2024-03-21T13:16:47.503518+0000","last_peered":"2024-03-21T13:16:47.503518+0000","last_clean":"2024-03-21T13:16:47.503518+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:16:47.503518+0000","last_undegraded":"2024-03-21T13:16:47.503518+0000","last_fullsized":"2024-03-21T13:16:47.503518+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:16:12.489518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.489518+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:14.011096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":626,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.548204+0000","last_change":"2024-03-21T13:14:02.109905+0000","last_active":"2024-03-21T13:16:41.548204+0000","last_peered":"2024-03-21T13:16:41.548204+0000","last_clean":"2024-03-21T13:16:41.548204+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:16:41.548204+0000","last_undegraded":"2024-03-21T13:16:41.548204+0000","last_fullsized":"2024-03-21T13:16:41.548204+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:14:02.109739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:02.109739+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:50.715022+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":397,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.510793+0000","last_change":"2024-03-21T13:15:04.834478+0000","last_active":"2024-03-21T13:16:47.510793+0000","last_peered":"2024-03-21T13:16:47.510793+0000","last_clean":"2024-03-21T13:16:47.510793+0000","last_became_active":"2024-03-21T13:15:04.458831+0000","last_became_peered":"2024-03-21T13:15:04.458831+0000","last_unstale":"2024-03-21T13:16:47.510793+0000","last_undegraded":"2024-03-21T13:16:47.510793+0000","last_fullsized":"2024-03-21T13:16:47.510793+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:15:04.834347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:04.834347+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":167,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504240+0000","last_change":"2024-03-21T13:16:07.247321+0000","last_active":"2024-03-21T13:16:47.504240+0000","last_peered":"2024-03-21T13:16:47.504240+0000","last_clean":"2024-03-21T13:16:47.504240+0000","last_became_active":"2024-03-21T13:12:29.527129+0000","last_became_peered":"2024-03-21T13:12:29.527129+0000","last_unstale":"2024-03-21T13:16:47.504240+0000","last_undegraded":"2024-03-21T13:16:47.504240+0000","last_fullsized":"2024-03-21T13:16:47.504240+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:07.247194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.247194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:08.116592+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":422,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.511023+0000","last_change":"2024-03-21T13:15:04.464609+0000","last_active":"2024-03-21T13:16:47.511023+0000","last_peered":"2024-03-21T13:16:47.511023+0000","last_clean":"2024-03-21T13:16:47.511023+0000","last_became_active":"2024-03-21T13:15:04.462979+0000","last_became_peered":"2024-03-21T13:15:04.462979+0000","last_unstale":"2024-03-21T13:16:47.511023+0000","last_undegraded":"2024-03-21T13:16:47.511023+0000","last_fullsized":"2024-03-21T13:16:47.511023+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:00.690145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:00.690145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":745,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.550313+0000","last_change":"2024-03-21T13:14:20.212339+0000","last_active":"2024-03-21T13:16:41.550313+0000","last_peered":"2024-03-21T13:16:41.550313+0000","last_clean":"2024-03-21T13:16:41.550313+0000","last_became_active":"2024-03-21T13:10:03.610489+0000","last_became_peered":"2024-03-21T13:10:03.610489+0000","last_unstale":"2024-03-21T13:16:41.550313+0000","last_undegraded":"2024-03-21T13:16:41.550313+0000","last_fullsized":"2024-03-21T13:16:41.550313+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:20.212222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:20.212222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"331'24","reported_seq":825,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503579+0000","last_change":"2024-03-21T13:15:09.670586+0000","last_active":"2024-03-21T13:16:47.503579+0000","last_peered":"2024-03-21T13:16:47.503579+0000","last_clean":"2024-03-21T13:16:47.503579+0000","last_became_active":"2024-03-21T13:12:29.522466+0000","last_became_peered":"2024-03-21T13:12:29.522466+0000","last_unstale":"2024-03-21T13:16:47.503579+0000","last_undegraded":"2024-03-21T13:16:47.503579+0000","last_fullsized":"2024-03-21T13:16:47.503579+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'24","last_scrub_stamp":"2024-03-21T13:15:09.670459+0000","last_deep_scrub":"331'24","last_deep_scrub_stamp":"2024-03-21T13:15:09.670459+0000","last_clean_scrub_stamp":"2024-03-21T13:15:09.670459+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:50.916265+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":518,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.508463+0000","last_change":"2024-03-21T13:15:04.458598+0000","last_active":"2024-03-21T13:16:47.508463+0000","last_peered":"2024-03-21T13:16:47.508463+0000","last_clean":"2024-03-21T13:16:47.508463+0000","last_became_active":"2024-03-21T13:15:04.458144+0000","last_became_peered":"2024-03-21T13:15:04.458144+0000","last_unstale":"2024-03-21T13:16:47.508463+0000","last_undegraded":"2024-03-21T13:16:47.508463+0000","last_fullsized":"2024-03-21T13:16:47.508463+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:20.369324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:14:20.369324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":738,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504599+0000","last_change":"2024-03-21T13:16:21.557896+0000","last_active":"2024-03-21T13:16:47.504599+0000","last_peered":"2024-03-21T13:16:47.504599+0000","last_clean":"2024-03-21T13:16:47.504599+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:16:47.504599+0000","last_undegraded":"2024-03-21T13:16:47.504599+0000","last_fullsized":"2024-03-21T13:16:47.504599+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:21.557805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.557805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:35.861086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":188,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.509675+0000","last_change":"2024-03-21T13:16:00.029375+0000","last_active":"2024-03-21T13:16:47.509675+0000","last_peered":"2024-03-21T13:16:47.509675+0000","last_clean":"2024-03-21T13:16:47.509675+0000","last_became_active":"2024-03-21T13:15:04.462103+0000","last_became_peered":"2024-03-21T13:15:04.462103+0000","last_unstale":"2024-03-21T13:16:47.509675+0000","last_undegraded":"2024-03-21T13:16:47.509675+0000","last_fullsized":"2024-03-21T13:16:47.509675+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:00.029292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:22.955737+0000","last_clean_scrub_stamp":"2024-03-21T13:16:00.029292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:02.701099+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":747,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.550597+0000","last_change":"2024-03-21T13:16:20.734468+0000","last_active":"2024-03-21T13:16:41.550597+0000","last_peered":"2024-03-21T13:16:41.550597+0000","last_clean":"2024-03-21T13:16:41.550597+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:16:41.550597+0000","last_undegraded":"2024-03-21T13:16:41.550597+0000","last_fullsized":"2024-03-21T13:16:41.550597+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-21T13:16:20.734379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.734379+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:47.670110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":715,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.546817+0000","last_change":"2024-03-21T13:14:59.413401+0000","last_active":"2024-03-21T13:16:41.546817+0000","last_peered":"2024-03-21T13:16:41.546817+0000","last_clean":"2024-03-21T13:16:41.546817+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:16:41.546817+0000","last_undegraded":"2024-03-21T13:16:41.546817+0000","last_fullsized":"2024-03-21T13:16:41.546817+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:59.413139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:59.413139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587986+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":83,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.506538+0000","last_change":"2024-03-21T13:16:05.059523+0000","last_active":"2024-03-21T13:16:47.506538+0000","last_peered":"2024-03-21T13:16:47.506538+0000","last_clean":"2024-03-21T13:16:47.506538+0000","last_became_active":"2024-03-21T13:15:04.512071+0000","last_became_peered":"2024-03-21T13:15:04.512071+0000","last_unstale":"2024-03-21T13:16:47.506538+0000","last_undegraded":"2024-03-21T13:16:47.506538+0000","last_fullsized":"2024-03-21T13:16:47.506538+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:05.059443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.059443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.468710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":175,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503968+0000","last_change":"2024-03-21T13:16:23.266104+0000","last_active":"2024-03-21T13:16:47.503968+0000","last_peered":"2024-03-21T13:16:47.503968+0000","last_clean":"2024-03-21T13:16:47.503968+0000","last_became_active":"2024-03-21T13:12:29.507706+0000","last_became_peered":"2024-03-21T13:12:29.507706+0000","last_unstale":"2024-03-21T13:16:47.503968+0000","last_undegraded":"2024-03-21T13:16:47.503968+0000","last_fullsized":"2024-03-21T13:16:47.503968+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:23.266023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:23.266023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:34.197737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":469,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.548363+0000","last_change":"2024-03-21T13:15:55.923610+0000","last_active":"2024-03-21T13:16:41.548363+0000","last_peered":"2024-03-21T13:16:41.548363+0000","last_clean":"2024-03-21T13:16:41.548363+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:16:41.548363+0000","last_undegraded":"2024-03-21T13:16:41.548363+0000","last_fullsized":"2024-03-21T13:16:41.548363+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:15:55.923514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:55.923514+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:12.089112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":176,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:50.025961+0000","last_change":"2024-03-21T13:16:50.026008+0000","last_active":"2024-03-21T13:16:50.025961+0000","last_peered":"2024-03-21T13:16:50.025961+0000","last_clean":"2024-03-21T13:16:50.025961+0000","last_became_active":"2024-03-21T13:15:04.512432+0000","last_became_peered":"2024-03-21T13:15:04.512432+0000","last_unstale":"2024-03-21T13:16:50.025961+0000","last_undegraded":"2024-03-21T13:16:50.025961+0000","last_fullsized":"2024-03-21T13:16:50.025961+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:50.025891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:50.025891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:04.135064+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":737,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.546407+0000","last_change":"2024-03-21T13:15:18.312265+0000","last_active":"2024-03-21T13:16:41.546407+0000","last_peered":"2024-03-21T13:16:41.546407+0000","last_clean":"2024-03-21T13:16:41.546407+0000","last_became_active":"2024-03-21T13:12:29.502203+0000","last_became_peered":"2024-03-21T13:12:29.502203+0000","last_unstale":"2024-03-21T13:16:41.546407+0000","last_undegraded":"2024-03-21T13:16:41.546407+0000","last_fullsized":"2024-03-21T13:16:41.546407+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:18.312139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:18.312139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.588086+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":736,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.507061+0000","last_change":"2024-03-21T13:16:08.584734+0000","last_active":"2024-03-21T13:16:47.507061+0000","last_peered":"2024-03-21T13:16:47.507061+0000","last_clean":"2024-03-21T13:16:47.507061+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:16:47.507061+0000","last_undegraded":"2024-03-21T13:16:47.507061+0000","last_fullsized":"2024-03-21T13:16:47.507061+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:08.584643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:19.383090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:08.584643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:19.593361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":738,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503674+0000","last_change":"2024-03-21T13:16:25.780453+0000","last_active":"2024-03-21T13:16:47.503674+0000","last_peered":"2024-03-21T13:16:47.503674+0000","last_clean":"2024-03-21T13:16:47.503674+0000","last_became_active":"2024-03-21T13:10:03.591857+0000","last_became_peered":"2024-03-21T13:10:03.591857+0000","last_unstale":"2024-03-21T13:16:47.503674+0000","last_undegraded":"2024-03-21T13:16:47.503674+0000","last_fullsized":"2024-03-21T13:16:47.503674+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:13:22.789110+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:13:22.789110+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:50.916043+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":563,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.506165+0000","last_change":"2024-03-21T13:15:05.701091+0000","last_active":"2024-03-21T13:16:47.506165+0000","last_peered":"2024-03-21T13:16:47.506165+0000","last_clean":"2024-03-21T13:16:47.506165+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:16:47.506165+0000","last_undegraded":"2024-03-21T13:16:47.506165+0000","last_fullsized":"2024-03-21T13:16:47.506165+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:05.700057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:05.700057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:47.641733+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":167,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503353+0000","last_change":"2024-03-21T13:13:22.501168+0000","last_active":"2024-03-21T13:16:47.503353+0000","last_peered":"2024-03-21T13:16:47.503353+0000","last_clean":"2024-03-21T13:16:47.503353+0000","last_became_active":"2024-03-21T13:12:29.510100+0000","last_became_peered":"2024-03-21T13:12:29.510100+0000","last_unstale":"2024-03-21T13:16:47.503353+0000","last_undegraded":"2024-03-21T13:16:47.503353+0000","last_fullsized":"2024-03-21T13:16:47.503353+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:22.501055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:22.501055+0000","last_clean_scrub_stamp":"2024-03-21T13:13:22.501055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":449,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.507490+0000","last_change":"2024-03-21T13:15:59.651653+0000","last_active":"2024-03-21T13:16:47.507490+0000","last_peered":"2024-03-21T13:16:47.507490+0000","last_clean":"2024-03-21T13:16:47.507490+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:16:47.507490+0000","last_undegraded":"2024-03-21T13:16:47.507490+0000","last_fullsized":"2024-03-21T13:16:47.507490+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:59.651571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:15:59.651571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:26.236664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":82,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.514069+0000","last_change":"2024-03-21T13:15:10.666307+0000","last_active":"2024-03-21T13:16:47.514069+0000","last_peered":"2024-03-21T13:16:47.514069+0000","last_clean":"2024-03-21T13:16:47.514069+0000","last_became_active":"2024-03-21T13:15:04.463107+0000","last_became_peered":"2024-03-21T13:15:04.463107+0000","last_unstale":"2024-03-21T13:16:47.514069+0000","last_undegraded":"2024-03-21T13:16:47.514069+0000","last_fullsized":"2024-03-21T13:16:47.514069+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:15:10.666149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:10.666149+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":168,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.510443+0000","last_change":"2024-03-21T13:15:04.459987+0000","last_active":"2024-03-21T13:16:47.510443+0000","last_peered":"2024-03-21T13:16:47.510443+0000","last_clean":"2024-03-21T13:16:47.510443+0000","last_became_active":"2024-03-21T13:15:04.458489+0000","last_became_peered":"2024-03-21T13:15:04.458489+0000","last_unstale":"2024-03-21T13:16:47.510443+0000","last_undegraded":"2024-03-21T13:16:47.510443+0000","last_fullsized":"2024-03-21T13:16:47.510443+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:03.518372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:03.518372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"394'12","reported_seq":233,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504868+0000","last_change":"2024-03-21T13:16:24.237890+0000","last_active":"2024-03-21T13:16:47.504868+0000","last_peered":"2024-03-21T13:16:47.504868+0000","last_clean":"2024-03-21T13:16:47.504868+0000","last_became_active":"2024-03-21T13:12:29.521385+0000","last_became_peered":"2024-03-21T13:12:29.521385+0000","last_unstale":"2024-03-21T13:16:47.504868+0000","last_undegraded":"2024-03-21T13:16:47.504868+0000","last_fullsized":"2024-03-21T13:16:47.504868+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'12","last_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_deep_scrub":"394'12","last_deep_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_clean_scrub_stamp":"2024-03-21T13:16:24.237800+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:31.363500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":748,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.506910+0000","last_change":"2024-03-21T13:15:10.647075+0000","last_active":"2024-03-21T13:16:47.506910+0000","last_peered":"2024-03-21T13:16:47.506910+0000","last_clean":"2024-03-21T13:16:47.506910+0000","last_became_active":"2024-03-21T13:10:03.599265+0000","last_became_peered":"2024-03-21T13:10:03.599265+0000","last_unstale":"2024-03-21T13:16:47.506910+0000","last_undegraded":"2024-03-21T13:16:47.506910+0000","last_fullsized":"2024-03-21T13:16:47.506910+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:10.646946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:15:10.646946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:47.641794+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'20","reported_seq":807,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.545933+0000","last_change":"2024-03-21T13:15:16.283969+0000","last_active":"2024-03-21T13:16:41.545933+0000","last_peered":"2024-03-21T13:16:41.545933+0000","last_clean":"2024-03-21T13:16:41.545933+0000","last_became_active":"2024-03-21T13:10:03.613119+0000","last_became_peered":"2024-03-21T13:10:03.613119+0000","last_unstale":"2024-03-21T13:16:41.545933+0000","last_undegraded":"2024-03-21T13:16:41.545933+0000","last_fullsized":"2024-03-21T13:16:41.545933+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:15:16.283831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:16.283831+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587959+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":309,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503109+0000","last_change":"2024-03-21T13:16:17.497436+0000","last_active":"2024-03-21T13:16:47.503109+0000","last_peered":"2024-03-21T13:16:47.503109+0000","last_clean":"2024-03-21T13:16:47.503109+0000","last_became_active":"2024-03-21T13:12:29.507751+0000","last_became_peered":"2024-03-21T13:12:29.507751+0000","last_unstale":"2024-03-21T13:16:47.503109+0000","last_undegraded":"2024-03-21T13:16:47.503109+0000","last_fullsized":"2024-03-21T13:16:47.503109+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:17.497325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:17.497325+0000","last_clean_scrub_stamp":"2024-03-21T13:16:17.497325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:26.268883+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"272'10","reported_seq":966,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.546373+0000","last_change":"2024-03-21T13:16:09.700298+0000","last_active":"2024-03-21T13:16:41.546373+0000","last_peered":"2024-03-21T13:16:41.546373+0000","last_clean":"2024-03-21T13:16:41.546373+0000","last_became_active":"2024-03-21T13:10:03.594651+0000","last_became_peered":"2024-03-21T13:10:03.594651+0000","last_unstale":"2024-03-21T13:16:41.546373+0000","last_undegraded":"2024-03-21T13:16:41.546373+0000","last_fullsized":"2024-03-21T13:16:41.546373+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'10","last_scrub_stamp":"2024-03-21T13:16:09.700215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:09.700215+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:35.185939+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":188,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.509685+0000","last_change":"2024-03-21T13:16:15.939316+0000","last_active":"2024-03-21T13:16:47.509685+0000","last_peered":"2024-03-21T13:16:47.509685+0000","last_clean":"2024-03-21T13:16:47.509685+0000","last_became_active":"2024-03-21T13:15:04.458184+0000","last_became_peered":"2024-03-21T13:15:04.458184+0000","last_unstale":"2024-03-21T13:16:47.509685+0000","last_undegraded":"2024-03-21T13:16:47.509685+0000","last_fullsized":"2024-03-21T13:16:47.509685+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:15.939231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:16:15.939231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:17.625617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"234'15","reported_seq":185,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503534+0000","last_change":"2024-03-21T13:16:22.227639+0000","last_active":"2024-03-21T13:16:47.503534+0000","last_peered":"2024-03-21T13:16:47.503534+0000","last_clean":"2024-03-21T13:16:47.503534+0000","last_became_active":"2024-03-21T13:12:29.510247+0000","last_became_peered":"2024-03-21T13:12:29.510247+0000","last_unstale":"2024-03-21T13:16:47.503534+0000","last_undegraded":"2024-03-21T13:16:47.503534+0000","last_fullsized":"2024-03-21T13:16:47.503534+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:16:22.227536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.227536+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:38.775744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":295,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.505606+0000","last_change":"2024-03-21T13:16:03.289521+0000","last_active":"2024-03-21T13:16:47.505606+0000","last_peered":"2024-03-21T13:16:47.505606+0000","last_clean":"2024-03-21T13:16:47.505606+0000","last_became_active":"2024-03-21T13:10:03.613841+0000","last_became_peered":"2024-03-21T13:10:03.613841+0000","last_unstale":"2024-03-21T13:16:47.505606+0000","last_undegraded":"2024-03-21T13:16:47.505606+0000","last_fullsized":"2024-03-21T13:16:47.505606+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:03.289412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:16:03.289412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:07.624710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":725,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.547774+0000","last_change":"2024-03-21T13:14:57.279251+0000","last_active":"2024-03-21T13:16:41.547774+0000","last_peered":"2024-03-21T13:16:41.547774+0000","last_clean":"2024-03-21T13:16:41.547774+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:16:41.547774+0000","last_undegraded":"2024-03-21T13:16:41.547774+0000","last_fullsized":"2024-03-21T13:16:41.547774+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:57.279130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:14:57.279130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587773+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":742,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.506447+0000","last_change":"2024-03-21T13:14:12.561473+0000","last_active":"2024-03-21T13:16:47.506447+0000","last_peered":"2024-03-21T13:16:47.506447+0000","last_clean":"2024-03-21T13:16:47.506447+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:16:47.506447+0000","last_undegraded":"2024-03-21T13:16:47.506447+0000","last_fullsized":"2024-03-21T13:16:47.506447+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:14:12.561236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:12.561236+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:47.641572+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'15","reported_seq":375,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.510886+0000","last_change":"2024-03-21T13:15:04.460964+0000","last_active":"2024-03-21T13:16:47.510886+0000","last_peered":"2024-03-21T13:16:47.510886+0000","last_clean":"2024-03-21T13:16:47.510886+0000","last_became_active":"2024-03-21T13:15:04.458914+0000","last_became_peered":"2024-03-21T13:15:04.458914+0000","last_unstale":"2024-03-21T13:16:47.510886+0000","last_undegraded":"2024-03-21T13:16:47.510886+0000","last_fullsized":"2024-03-21T13:16:47.510886+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:14:15.192958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:15.192958+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":724,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.507053+0000","last_change":"2024-03-21T13:16:02.716355+0000","last_active":"2024-03-21T13:16:47.507053+0000","last_peered":"2024-03-21T13:16:47.507053+0000","last_clean":"2024-03-21T13:16:47.507053+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:16:47.507053+0000","last_undegraded":"2024-03-21T13:16:47.507053+0000","last_fullsized":"2024-03-21T13:16:47.507053+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:02.716255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:02.716255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.235343+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":175,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503868+0000","last_change":"2024-03-21T13:16:18.220471+0000","last_active":"2024-03-21T13:16:47.503868+0000","last_peered":"2024-03-21T13:16:47.503868+0000","last_clean":"2024-03-21T13:16:47.503868+0000","last_became_active":"2024-03-21T13:12:29.514113+0000","last_became_peered":"2024-03-21T13:12:29.514113+0000","last_unstale":"2024-03-21T13:16:47.503868+0000","last_undegraded":"2024-03-21T13:16:47.503868+0000","last_fullsized":"2024-03-21T13:16:47.503868+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.220368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.220368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.800528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":750,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504279+0000","last_change":"2024-03-21T13:14:14.647062+0000","last_active":"2024-03-21T13:16:47.504279+0000","last_peered":"2024-03-21T13:16:47.504279+0000","last_clean":"2024-03-21T13:16:47.504279+0000","last_became_active":"2024-03-21T13:12:29.520286+0000","last_became_peered":"2024-03-21T13:12:29.520286+0000","last_unstale":"2024-03-21T13:16:47.504279+0000","last_undegraded":"2024-03-21T13:16:47.504279+0000","last_fullsized":"2024-03-21T13:16:47.504279+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:14.646933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:14.646933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:50.916168+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":783,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.547633+0000","last_change":"2024-03-21T13:14:09.141688+0000","last_active":"2024-03-21T13:16:41.547633+0000","last_peered":"2024-03-21T13:16:41.547633+0000","last_clean":"2024-03-21T13:16:41.547633+0000","last_became_active":"2024-03-21T13:10:03.610810+0000","last_became_peered":"2024-03-21T13:10:03.610810+0000","last_unstale":"2024-03-21T13:16:41.547633+0000","last_undegraded":"2024-03-21T13:16:41.547633+0000","last_fullsized":"2024-03-21T13:16:41.547633+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-21T13:14:09.141520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:09.141520+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"405'2744","reported_seq":2783,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:49.879971+0000","last_change":"2024-03-21T13:16:13.903136+0000","last_active":"2024-03-21T13:16:49.879971+0000","last_peered":"2024-03-21T13:16:49.879971+0000","last_clean":"2024-03-21T13:16:49.879971+0000","last_became_active":"2024-03-21T13:16:13.901764+0000","last_became_peered":"2024-03-21T13:16:13.901764+0000","last_unstale":"2024-03-21T13:16:49.879971+0000","last_undegraded":"2024-03-21T13:16:49.879971+0000","last_fullsized":"2024-03-21T13:16:49.879971+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2744,"log_dups_size":0,"ondisk_log_size":2744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:14.340802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179830784,"num_objects":2744,"num_object_clones":0,"num_object_copies":8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2744,"num_write_kb":175616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":168,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503582+0000","last_change":"2024-03-21T13:15:06.391491+0000","last_active":"2024-03-21T13:16:47.503582+0000","last_peered":"2024-03-21T13:16:47.503582+0000","last_clean":"2024-03-21T13:16:47.503582+0000","last_became_active":"2024-03-21T13:12:29.517413+0000","last_became_peered":"2024-03-21T13:12:29.517413+0000","last_unstale":"2024-03-21T13:16:47.503582+0000","last_undegraded":"2024-03-21T13:16:47.503582+0000","last_fullsized":"2024-03-21T13:16:47.503582+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:06.391193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:15:06.391193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198650+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":773,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.509547+0000","last_change":"2024-03-21T13:14:53.565542+0000","last_active":"2024-03-21T13:16:47.509547+0000","last_peered":"2024-03-21T13:16:47.509547+0000","last_clean":"2024-03-21T13:16:47.509547+0000","last_became_active":"2024-03-21T13:06:29.487439+0000","last_became_peered":"2024-03-21T13:06:29.487439+0000","last_unstale":"2024-03-21T13:16:47.509547+0000","last_undegraded":"2024-03-21T13:16:47.509547+0000","last_fullsized":"2024-03-21T13:16:47.509547+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:14:53.565419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:53.565419+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:47.641606+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":184,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503127+0000","last_change":"2024-03-21T13:14:00.557564+0000","last_active":"2024-03-21T13:16:47.503127+0000","last_peered":"2024-03-21T13:16:47.503127+0000","last_clean":"2024-03-21T13:16:47.503127+0000","last_became_active":"2024-03-21T13:12:29.513019+0000","last_became_peered":"2024-03-21T13:12:29.513019+0000","last_unstale":"2024-03-21T13:16:47.503127+0000","last_undegraded":"2024-03-21T13:16:47.503127+0000","last_fullsized":"2024-03-21T13:16:47.503127+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:14:00.557372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:00.557372+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":960,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.506859+0000","last_change":"2024-03-21T13:16:10.554480+0000","last_active":"2024-03-21T13:16:47.506859+0000","last_peered":"2024-03-21T13:16:47.506859+0000","last_clean":"2024-03-21T13:16:47.506859+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:16:47.506859+0000","last_undegraded":"2024-03-21T13:16:47.506859+0000","last_fullsized":"2024-03-21T13:16:47.506859+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:16:10.554378+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:16:10.554378+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.191410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"405'2991","reported_seq":3028,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:50.565961+0000","last_change":"2024-03-21T13:16:13.901750+0000","last_active":"2024-03-21T13:16:50.565961+0000","last_peered":"2024-03-21T13:16:50.565961+0000","last_clean":"2024-03-21T13:16:50.565961+0000","last_became_active":"2024-03-21T13:16:13.901065+0000","last_became_peered":"2024-03-21T13:16:13.901065+0000","last_unstale":"2024-03-21T13:16:50.565961+0000","last_undegraded":"2024-03-21T13:16:50.565961+0000","last_fullsized":"2024-03-21T13:16:50.565961+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2991,"log_dups_size":0,"ondisk_log_size":2991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.886855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196018176,"num_objects":2991,"num_object_clones":0,"num_object_copies":8973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2991,"num_write_kb":191424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":722,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.546767+0000","last_change":"2024-03-21T13:15:58.046280+0000","last_active":"2024-03-21T13:16:41.546767+0000","last_peered":"2024-03-21T13:16:41.546767+0000","last_clean":"2024-03-21T13:16:41.546767+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:16:41.546767+0000","last_undegraded":"2024-03-21T13:16:41.546767+0000","last_fullsized":"2024-03-21T13:16:41.546767+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:58.046156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:15:58.046156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:12.947549+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":743,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.507411+0000","last_change":"2024-03-21T13:14:10.542906+0000","last_active":"2024-03-21T13:16:47.507411+0000","last_peered":"2024-03-21T13:16:47.507411+0000","last_clean":"2024-03-21T13:16:47.507411+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:16:47.507411+0000","last_undegraded":"2024-03-21T13:16:47.507411+0000","last_fullsized":"2024-03-21T13:16:47.507411+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:10.542755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:10.542755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:47.641539+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"346'23","reported_seq":146,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.502019+0000","last_change":"2024-03-21T13:16:10.260270+0000","last_active":"2024-03-21T13:16:47.502019+0000","last_peered":"2024-03-21T13:16:47.502019+0000","last_clean":"2024-03-21T13:16:47.502019+0000","last_became_active":"2024-03-21T13:13:21.819111+0000","last_became_peered":"2024-03-21T13:13:21.819111+0000","last_unstale":"2024-03-21T13:16:47.502019+0000","last_undegraded":"2024-03-21T13:16:47.502019+0000","last_fullsized":"2024-03-21T13:16:47.502019+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'23","last_scrub_stamp":"2024-03-21T13:16:10.260157+0000","last_deep_scrub":"346'23","last_deep_scrub_stamp":"2024-03-21T13:13:20.232807+0000","last_clean_scrub_stamp":"2024-03-21T13:16:10.260157+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:35.914575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":302,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.500420+0000","last_change":"2024-03-21T13:15:18.628643+0000","last_active":"2024-03-21T13:16:47.500420+0000","last_peered":"2024-03-21T13:16:47.500420+0000","last_clean":"2024-03-21T13:16:47.500420+0000","last_became_active":"2024-03-21T13:12:29.499921+0000","last_became_peered":"2024-03-21T13:12:29.499921+0000","last_unstale":"2024-03-21T13:16:47.500420+0000","last_undegraded":"2024-03-21T13:16:47.500420+0000","last_fullsized":"2024-03-21T13:16:47.500420+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:18.628530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:18.628530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.367129+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"405'2967","reported_seq":3006,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:50.394876+0000","last_change":"2024-03-21T13:16:13.903159+0000","last_active":"2024-03-21T13:16:50.394876+0000","last_peered":"2024-03-21T13:16:50.394876+0000","last_clean":"2024-03-21T13:16:50.394876+0000","last_became_active":"2024-03-21T13:16:13.902868+0000","last_became_peered":"2024-03-21T13:16:13.902868+0000","last_unstale":"2024-03-21T13:16:50.394876+0000","last_undegraded":"2024-03-21T13:16:50.394876+0000","last_fullsized":"2024-03-21T13:16:50.394876+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2967,"log_dups_size":0,"ondisk_log_size":2967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:29.385439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194445312,"num_objects":2967,"num_object_clones":0,"num_object_copies":8901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2967,"num_write_kb":189888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":745,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.505788+0000","last_change":"2024-03-21T13:16:06.611026+0000","last_active":"2024-03-21T13:16:47.505788+0000","last_peered":"2024-03-21T13:16:47.505788+0000","last_clean":"2024-03-21T13:16:47.505788+0000","last_became_active":"2024-03-21T13:10:03.598366+0000","last_became_peered":"2024-03-21T13:10:03.598366+0000","last_unstale":"2024-03-21T13:16:47.505788+0000","last_undegraded":"2024-03-21T13:16:47.505788+0000","last_fullsized":"2024-03-21T13:16:47.505788+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:06.610947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:16:06.610947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.715015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":318,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:49.448922+0000","last_change":"2024-03-21T13:16:49.448922+0000","last_active":"2024-03-21T13:16:49.448922+0000","last_peered":"2024-03-21T13:16:49.448922+0000","last_clean":"2024-03-21T13:16:49.448922+0000","last_became_active":"2024-03-21T13:12:29.500199+0000","last_became_peered":"2024-03-21T13:12:29.500199+0000","last_unstale":"2024-03-21T13:16:49.448922+0000","last_undegraded":"2024-03-21T13:16:49.448922+0000","last_fullsized":"2024-03-21T13:16:49.448922+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:49.448804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:49.448804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:02.584547+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":757,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.661117+0000","last_change":"2024-03-21T13:16:47.661139+0000","last_active":"2024-03-21T13:16:47.661117+0000","last_peered":"2024-03-21T13:16:47.661117+0000","last_clean":"2024-03-21T13:16:47.661117+0000","last_became_active":"2024-03-21T13:12:29.508984+0000","last_became_peered":"2024-03-21T13:12:29.508984+0000","last_unstale":"2024-03-21T13:16:47.661117+0000","last_undegraded":"2024-03-21T13:16:47.661117+0000","last_fullsized":"2024-03-21T13:16:47.661117+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:47.661063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:47.661063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:15.875522+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'35","reported_seq":868,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503312+0000","last_change":"2024-03-21T13:13:24.732800+0000","last_active":"2024-03-21T13:16:47.503312+0000","last_peered":"2024-03-21T13:16:47.503312+0000","last_clean":"2024-03-21T13:16:47.503312+0000","last_became_active":"2024-03-21T13:12:29.523359+0000","last_became_peered":"2024-03-21T13:12:29.523359+0000","last_unstale":"2024-03-21T13:16:47.503312+0000","last_undegraded":"2024-03-21T13:16:47.503312+0000","last_fullsized":"2024-03-21T13:16:47.503312+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:13:24.732562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:24.732562+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:50.916110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"405'2893","reported_seq":2932,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:50.368960+0000","last_change":"2024-03-21T13:16:13.904507+0000","last_active":"2024-03-21T13:16:50.368960+0000","last_peered":"2024-03-21T13:16:50.368960+0000","last_clean":"2024-03-21T13:16:50.368960+0000","last_became_active":"2024-03-21T13:16:13.904210+0000","last_became_peered":"2024-03-21T13:16:13.904210+0000","last_unstale":"2024-03-21T13:16:50.368960+0000","last_undegraded":"2024-03-21T13:16:50.368960+0000","last_fullsized":"2024-03-21T13:16:50.368960+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2893,"log_dups_size":0,"ondisk_log_size":2893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.107729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189595648,"num_objects":2893,"num_object_clones":0,"num_object_copies":8679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2893,"num_write_kb":185152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":429,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.509972+0000","last_change":"2024-03-21T13:15:12.891812+0000","last_active":"2024-03-21T13:16:47.509972+0000","last_peered":"2024-03-21T13:16:47.509972+0000","last_clean":"2024-03-21T13:16:47.509972+0000","last_became_active":"2024-03-21T13:15:04.458107+0000","last_became_peered":"2024-03-21T13:15:04.458107+0000","last_unstale":"2024-03-21T13:16:47.509972+0000","last_undegraded":"2024-03-21T13:16:47.509972+0000","last_fullsized":"2024-03-21T13:16:47.509972+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:12.891693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:12.891693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":296,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.500978+0000","last_change":"2024-03-21T13:15:13.550311+0000","last_active":"2024-03-21T13:16:47.500978+0000","last_peered":"2024-03-21T13:16:47.500978+0000","last_clean":"2024-03-21T13:16:47.500978+0000","last_became_active":"2024-03-21T13:10:03.603646+0000","last_became_peered":"2024-03-21T13:10:03.603646+0000","last_unstale":"2024-03-21T13:16:47.500978+0000","last_undegraded":"2024-03-21T13:16:47.500978+0000","last_fullsized":"2024-03-21T13:16:47.500978+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:13.550186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:13.550186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.367145+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":167,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503178+0000","last_change":"2024-03-21T13:14:15.386244+0000","last_active":"2024-03-21T13:16:47.503178+0000","last_peered":"2024-03-21T13:16:47.503178+0000","last_clean":"2024-03-21T13:16:47.503178+0000","last_became_active":"2024-03-21T13:12:29.512741+0000","last_became_peered":"2024-03-21T13:12:29.512741+0000","last_unstale":"2024-03-21T13:16:47.503178+0000","last_undegraded":"2024-03-21T13:16:47.503178+0000","last_fullsized":"2024-03-21T13:16:47.503178+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:15.386132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:15.386132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198436+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"234'19","reported_seq":428,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.514787+0000","last_change":"2024-03-21T13:16:23.841059+0000","last_active":"2024-03-21T13:16:47.514787+0000","last_peered":"2024-03-21T13:16:47.514787+0000","last_clean":"2024-03-21T13:16:47.514787+0000","last_became_active":"2024-03-21T13:15:04.460898+0000","last_became_peered":"2024-03-21T13:15:04.460898+0000","last_unstale":"2024-03-21T13:16:47.514787+0000","last_undegraded":"2024-03-21T13:16:47.514787+0000","last_fullsized":"2024-03-21T13:16:47.514787+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:23.840950+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:16:23.840950+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:46.775417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"403'2376","reported_seq":2412,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:46.159867+0000","last_change":"2024-03-21T13:16:13.902405+0000","last_active":"2024-03-21T13:16:46.159867+0000","last_peered":"2024-03-21T13:16:46.159867+0000","last_clean":"2024-03-21T13:16:46.159867+0000","last_became_active":"2024-03-21T13:16:13.902109+0000","last_became_peered":"2024-03-21T13:16:13.902109+0000","last_unstale":"2024-03-21T13:16:46.159867+0000","last_undegraded":"2024-03-21T13:16:46.159867+0000","last_fullsized":"2024-03-21T13:16:46.159867+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2376,"log_dups_size":0,"ondisk_log_size":2376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:41.304847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155713536,"num_objects":2376,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2376,"num_write_kb":152064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":708,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.545956+0000","last_change":"2024-03-21T13:14:14.199570+0000","last_active":"2024-03-21T13:16:41.545956+0000","last_peered":"2024-03-21T13:16:41.545956+0000","last_clean":"2024-03-21T13:16:41.545956+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:16:41.545956+0000","last_undegraded":"2024-03-21T13:16:41.545956+0000","last_fullsized":"2024-03-21T13:16:41.545956+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:14.199450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:14:14.199450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587421+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":172,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503261+0000","last_change":"2024-03-21T13:16:08.211061+0000","last_active":"2024-03-21T13:16:47.503261+0000","last_peered":"2024-03-21T13:16:47.503261+0000","last_clean":"2024-03-21T13:16:47.503261+0000","last_became_active":"2024-03-21T13:12:29.521275+0000","last_became_peered":"2024-03-21T13:12:29.521275+0000","last_unstale":"2024-03-21T13:16:47.503261+0000","last_undegraded":"2024-03-21T13:16:47.503261+0000","last_fullsized":"2024-03-21T13:16:47.503261+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:16:08.210967+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:16:08.210967+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:37.370051+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":84,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.514378+0000","last_change":"2024-03-21T13:16:22.861225+0000","last_active":"2024-03-21T13:16:47.514378+0000","last_peered":"2024-03-21T13:16:47.514378+0000","last_clean":"2024-03-21T13:16:47.514378+0000","last_became_active":"2024-03-21T13:15:04.462662+0000","last_became_peered":"2024-03-21T13:15:04.462662+0000","last_unstale":"2024-03-21T13:16:47.514378+0000","last_undegraded":"2024-03-21T13:16:47.514378+0000","last_fullsized":"2024-03-21T13:16:47.514378+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:22.861136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:22.861136+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.861136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:34.763292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"272'43","reported_seq":408,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.505575+0000","last_change":"2024-03-21T13:16:16.292178+0000","last_active":"2024-03-21T13:16:47.505575+0000","last_peered":"2024-03-21T13:16:47.505575+0000","last_clean":"2024-03-21T13:16:47.505575+0000","last_became_active":"2024-03-21T13:10:03.603889+0000","last_became_peered":"2024-03-21T13:10:03.603889+0000","last_unstale":"2024-03-21T13:16:47.505575+0000","last_undegraded":"2024-03-21T13:16:47.505575+0000","last_fullsized":"2024-03-21T13:16:47.505575+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'43","last_scrub_stamp":"2024-03-21T13:16:16.292073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:16.292073+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:41.103424+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":180,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.509021+0000","last_change":"2024-03-21T13:16:10.968486+0000","last_active":"2024-03-21T13:16:47.509021+0000","last_peered":"2024-03-21T13:16:47.509021+0000","last_clean":"2024-03-21T13:16:47.509021+0000","last_became_active":"2024-03-21T13:15:04.462159+0000","last_became_peered":"2024-03-21T13:15:04.462159+0000","last_unstale":"2024-03-21T13:16:47.509021+0000","last_undegraded":"2024-03-21T13:16:47.509021+0000","last_fullsized":"2024-03-21T13:16:47.509021+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:10.968402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:15.968926+0000","last_clean_scrub_stamp":"2024-03-21T13:16:10.968402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.606676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"403'2391","reported_seq":2425,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:46.158427+0000","last_change":"2024-03-21T13:16:13.900592+0000","last_active":"2024-03-21T13:16:46.158427+0000","last_peered":"2024-03-21T13:16:46.158427+0000","last_clean":"2024-03-21T13:16:46.158427+0000","last_became_active":"2024-03-21T13:16:13.900119+0000","last_became_peered":"2024-03-21T13:16:13.900119+0000","last_unstale":"2024-03-21T13:16:46.158427+0000","last_undegraded":"2024-03-21T13:16:46.158427+0000","last_fullsized":"2024-03-21T13:16:46.158427+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2391,"log_dups_size":0,"ondisk_log_size":2391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:17.609817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156696576,"num_objects":2391,"num_object_clones":0,"num_object_copies":7173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2391,"num_write_kb":153024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":733,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.507529+0000","last_change":"2024-03-21T13:16:18.683140+0000","last_active":"2024-03-21T13:16:47.507529+0000","last_peered":"2024-03-21T13:16:47.507529+0000","last_clean":"2024-03-21T13:16:47.507529+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:16:47.507529+0000","last_undegraded":"2024-03-21T13:16:47.507529+0000","last_fullsized":"2024-03-21T13:16:47.507529+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.683041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:44.764044+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":296,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.500504+0000","last_change":"2024-03-21T13:16:12.231215+0000","last_active":"2024-03-21T13:16:47.500504+0000","last_peered":"2024-03-21T13:16:47.500504+0000","last_clean":"2024-03-21T13:16:47.500504+0000","last_became_active":"2024-03-21T13:10:03.604303+0000","last_became_peered":"2024-03-21T13:10:03.604303+0000","last_unstale":"2024-03-21T13:16:47.500504+0000","last_undegraded":"2024-03-21T13:16:47.500504+0000","last_fullsized":"2024-03-21T13:16:47.500504+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.231115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.231115+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.231115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:12.524706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":296,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503704+0000","last_change":"2024-03-21T13:15:03.565009+0000","last_active":"2024-03-21T13:16:47.503704+0000","last_peered":"2024-03-21T13:16:47.503704+0000","last_clean":"2024-03-21T13:16:47.503704+0000","last_became_active":"2024-03-21T13:10:03.604427+0000","last_became_peered":"2024-03-21T13:10:03.604427+0000","last_unstale":"2024-03-21T13:16:47.503704+0000","last_undegraded":"2024-03-21T13:16:47.503704+0000","last_fullsized":"2024-03-21T13:16:47.503704+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:03.564645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:15:03.564645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.367093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"243'20","reported_seq":180,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504626+0000","last_change":"2024-03-21T13:14:52.337122+0000","last_active":"2024-03-21T13:16:47.504626+0000","last_peered":"2024-03-21T13:16:47.504626+0000","last_clean":"2024-03-21T13:16:47.504626+0000","last_became_active":"2024-03-21T13:12:29.518048+0000","last_became_peered":"2024-03-21T13:12:29.518048+0000","last_unstale":"2024-03-21T13:16:47.504626+0000","last_undegraded":"2024-03-21T13:16:47.504626+0000","last_fullsized":"2024-03-21T13:16:47.504626+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'20","last_scrub_stamp":"2024-03-21T13:14:52.332617+0000","last_deep_scrub":"243'20","last_deep_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_clean_scrub_stamp":"2024-03-21T13:14:52.332617+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198593+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":302,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503101+0000","last_change":"2024-03-21T13:16:15.247743+0000","last_active":"2024-03-21T13:16:47.503101+0000","last_peered":"2024-03-21T13:16:47.503101+0000","last_clean":"2024-03-21T13:16:47.503101+0000","last_became_active":"2024-03-21T13:12:29.506751+0000","last_became_peered":"2024-03-21T13:12:29.506751+0000","last_unstale":"2024-03-21T13:16:47.503101+0000","last_undegraded":"2024-03-21T13:16:47.503101+0000","last_fullsized":"2024-03-21T13:16:47.503101+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:15.247661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:15.247661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:36.430093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"403'2437","reported_seq":2472,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:46.164166+0000","last_change":"2024-03-21T13:16:13.900783+0000","last_active":"2024-03-21T13:16:46.164166+0000","last_peered":"2024-03-21T13:16:46.164166+0000","last_clean":"2024-03-21T13:16:46.164166+0000","last_became_active":"2024-03-21T13:16:13.900477+0000","last_became_peered":"2024-03-21T13:16:13.900477+0000","last_unstale":"2024-03-21T13:16:46.164166+0000","last_undegraded":"2024-03-21T13:16:46.164166+0000","last_fullsized":"2024-03-21T13:16:46.164166+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2437,"log_dups_size":0,"ondisk_log_size":2437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:38.844318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159711232,"num_objects":2437,"num_object_clones":0,"num_object_copies":7311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2437,"num_write_kb":155968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":168,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504356+0000","last_change":"2024-03-21T13:15:04.365662+0000","last_active":"2024-03-21T13:16:47.504356+0000","last_peered":"2024-03-21T13:16:47.504356+0000","last_clean":"2024-03-21T13:16:47.504356+0000","last_became_active":"2024-03-21T13:12:29.527083+0000","last_became_peered":"2024-03-21T13:12:29.527083+0000","last_unstale":"2024-03-21T13:16:47.504356+0000","last_undegraded":"2024-03-21T13:16:47.504356+0000","last_fullsized":"2024-03-21T13:16:47.504356+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:04.365534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:04.365534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198707+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":996,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.506033+0000","last_change":"2024-03-21T13:16:14.628486+0000","last_active":"2024-03-21T13:16:47.506033+0000","last_peered":"2024-03-21T13:16:47.506033+0000","last_clean":"2024-03-21T13:16:47.506033+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:16:47.506033+0000","last_undegraded":"2024-03-21T13:16:47.506033+0000","last_fullsized":"2024-03-21T13:16:47.506033+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:16:14.628384+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:16:14.628384+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.536448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":241,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.500615+0000","last_change":"2024-03-21T13:16:02.247782+0000","last_active":"2024-03-21T13:16:47.500615+0000","last_peered":"2024-03-21T13:16:47.500615+0000","last_clean":"2024-03-21T13:16:47.500615+0000","last_became_active":"2024-03-21T13:12:29.499989+0000","last_became_peered":"2024-03-21T13:12:29.499989+0000","last_unstale":"2024-03-21T13:16:47.500615+0000","last_undegraded":"2024-03-21T13:16:47.500615+0000","last_fullsized":"2024-03-21T13:16:47.500615+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:02.247693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:02.247693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.746970+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"234'36","reported_seq":1002,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.505612+0000","last_change":"2024-03-21T13:16:05.528913+0000","last_active":"2024-03-21T13:16:47.505612+0000","last_peered":"2024-03-21T13:16:47.505612+0000","last_clean":"2024-03-21T13:16:47.505612+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:16:47.505612+0000","last_undegraded":"2024-03-21T13:16:47.505612+0000","last_fullsized":"2024-03-21T13:16:47.505612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'36","last_scrub_stamp":"2024-03-21T13:16:05.528782+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.528782+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:30.885762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":753,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.507650+0000","last_change":"2024-03-21T13:14:56.674085+0000","last_active":"2024-03-21T13:16:47.507650+0000","last_peered":"2024-03-21T13:16:47.507650+0000","last_clean":"2024-03-21T13:16:47.507650+0000","last_became_active":"2024-03-21T13:12:29.507753+0000","last_became_peered":"2024-03-21T13:12:29.507753+0000","last_unstale":"2024-03-21T13:16:47.507650+0000","last_undegraded":"2024-03-21T13:16:47.507650+0000","last_fullsized":"2024-03-21T13:16:47.507650+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:56.673972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:14:56.673972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:47.641670+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"403'2524","reported_seq":2560,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:46.145254+0000","last_change":"2024-03-21T13:16:13.900952+0000","last_active":"2024-03-21T13:16:46.145254+0000","last_peered":"2024-03-21T13:16:46.145254+0000","last_clean":"2024-03-21T13:16:46.145254+0000","last_became_active":"2024-03-21T13:16:13.900650+0000","last_became_peered":"2024-03-21T13:16:13.900650+0000","last_unstale":"2024-03-21T13:16:46.145254+0000","last_undegraded":"2024-03-21T13:16:46.145254+0000","last_fullsized":"2024-03-21T13:16:46.145254+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2524,"log_dups_size":0,"ondisk_log_size":2524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.174786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165412864,"num_objects":2524,"num_object_clones":0,"num_object_copies":7572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2524,"num_write_kb":161536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":168,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503414+0000","last_change":"2024-03-21T13:16:01.195705+0000","last_active":"2024-03-21T13:16:47.503414+0000","last_peered":"2024-03-21T13:16:47.503414+0000","last_clean":"2024-03-21T13:16:47.503414+0000","last_became_active":"2024-03-21T13:12:29.514542+0000","last_became_peered":"2024-03-21T13:12:29.514542+0000","last_unstale":"2024-03-21T13:16:47.503414+0000","last_undegraded":"2024-03-21T13:16:47.503414+0000","last_fullsized":"2024-03-21T13:16:47.503414+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:01.195618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:01.195618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:22.887658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":188,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.509105+0000","last_change":"2024-03-21T13:15:04.459946+0000","last_active":"2024-03-21T13:16:47.509105+0000","last_peered":"2024-03-21T13:16:47.509105+0000","last_clean":"2024-03-21T13:16:47.509105+0000","last_became_active":"2024-03-21T13:15:04.458084+0000","last_became_peered":"2024-03-21T13:15:04.458084+0000","last_unstale":"2024-03-21T13:16:47.509105+0000","last_undegraded":"2024-03-21T13:16:47.509105+0000","last_fullsized":"2024-03-21T13:16:47.509105+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:57.238591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:14:57.238591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":1842,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:49.272937+0000","last_change":"2024-03-21T13:16:09.547909+0000","last_active":"2024-03-21T13:16:49.272937+0000","last_peered":"2024-03-21T13:16:49.272937+0000","last_clean":"2024-03-21T13:16:49.272937+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:16:49.272937+0000","last_undegraded":"2024-03-21T13:16:49.272937+0000","last_fullsized":"2024-03-21T13:16:49.272937+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:16:09.547783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:16:09.547783+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:28.689027+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1082,"num_read_kb":1077,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":870,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.510526+0000","last_change":"2024-03-21T13:16:18.883144+0000","last_active":"2024-03-21T13:16:47.510526+0000","last_peered":"2024-03-21T13:16:47.510526+0000","last_clean":"2024-03-21T13:16:47.510526+0000","last_became_active":"2024-03-21T13:10:03.606672+0000","last_became_peered":"2024-03-21T13:10:03.606672+0000","last_unstale":"2024-03-21T13:16:47.510526+0000","last_undegraded":"2024-03-21T13:16:47.510526+0000","last_fullsized":"2024-03-21T13:16:47.510526+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-21T13:16:18.882994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.882994+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:43.518929+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":168,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504038+0000","last_change":"2024-03-21T13:14:07.406567+0000","last_active":"2024-03-21T13:16:47.504038+0000","last_peered":"2024-03-21T13:16:47.504038+0000","last_clean":"2024-03-21T13:16:47.504038+0000","last_became_active":"2024-03-21T13:12:29.526100+0000","last_became_peered":"2024-03-21T13:12:29.526100+0000","last_unstale":"2024-03-21T13:16:47.504038+0000","last_undegraded":"2024-03-21T13:16:47.504038+0000","last_fullsized":"2024-03-21T13:16:47.504038+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:07.406434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:14:07.406434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198563+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"405'2856","reported_seq":2895,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:50.469108+0000","last_change":"2024-03-21T13:16:13.899271+0000","last_active":"2024-03-21T13:16:50.469108+0000","last_peered":"2024-03-21T13:16:50.469108+0000","last_clean":"2024-03-21T13:16:50.469108+0000","last_became_active":"2024-03-21T13:16:13.898890+0000","last_became_peered":"2024-03-21T13:16:13.898890+0000","last_unstale":"2024-03-21T13:16:50.469108+0000","last_undegraded":"2024-03-21T13:16:50.469108+0000","last_fullsized":"2024-03-21T13:16:50.469108+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:39.631299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187170816,"num_objects":2856,"num_object_clones":0,"num_object_copies":8568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":182784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":636,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504557+0000","last_change":"2024-03-21T13:16:15.593032+0000","last_active":"2024-03-21T13:16:47.504557+0000","last_peered":"2024-03-21T13:16:47.504557+0000","last_clean":"2024-03-21T13:16:47.504557+0000","last_became_active":"2024-03-21T13:10:03.591088+0000","last_became_peered":"2024-03-21T13:10:03.591088+0000","last_unstale":"2024-03-21T13:16:47.504557+0000","last_undegraded":"2024-03-21T13:16:47.504557+0000","last_fullsized":"2024-03-21T13:16:47.504557+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:15.592931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:15.592931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:36.752324+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":310,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.505704+0000","last_change":"2024-03-21T13:16:22.293702+0000","last_active":"2024-03-21T13:16:47.505704+0000","last_peered":"2024-03-21T13:16:47.505704+0000","last_clean":"2024-03-21T13:16:47.505704+0000","last_became_active":"2024-03-21T13:12:29.507783+0000","last_became_peered":"2024-03-21T13:12:29.507783+0000","last_unstale":"2024-03-21T13:16:47.505704+0000","last_undegraded":"2024-03-21T13:16:47.505704+0000","last_fullsized":"2024-03-21T13:16:47.505704+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:22.293625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.293625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.688902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":186,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504276+0000","last_change":"2024-03-21T13:14:16.366110+0000","last_active":"2024-03-21T13:16:47.504276+0000","last_peered":"2024-03-21T13:16:47.504276+0000","last_clean":"2024-03-21T13:16:47.504276+0000","last_became_active":"2024-03-21T13:12:29.518800+0000","last_became_peered":"2024-03-21T13:12:29.518800+0000","last_unstale":"2024-03-21T13:16:47.504276+0000","last_undegraded":"2024-03-21T13:16:47.504276+0000","last_fullsized":"2024-03-21T13:16:47.504276+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-21T13:14:16.365267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:16.365267+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198535+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":755,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503410+0000","last_change":"2024-03-21T13:13:21.836944+0000","last_active":"2024-03-21T13:16:47.503410+0000","last_peered":"2024-03-21T13:16:47.503410+0000","last_clean":"2024-03-21T13:16:47.503410+0000","last_became_active":"2024-03-21T13:12:29.520657+0000","last_became_peered":"2024-03-21T13:12:29.520657+0000","last_unstale":"2024-03-21T13:16:47.503410+0000","last_undegraded":"2024-03-21T13:16:47.503410+0000","last_fullsized":"2024-03-21T13:16:47.503410+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:21.836810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:21.836810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:50.916075+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"405'2878","reported_seq":2918,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:50.458873+0000","last_change":"2024-03-21T13:16:13.899935+0000","last_active":"2024-03-21T13:16:50.458873+0000","last_peered":"2024-03-21T13:16:50.458873+0000","last_clean":"2024-03-21T13:16:50.458873+0000","last_became_active":"2024-03-21T13:16:13.899614+0000","last_became_peered":"2024-03-21T13:16:13.899614+0000","last_unstale":"2024-03-21T13:16:50.458873+0000","last_undegraded":"2024-03-21T13:16:50.458873+0000","last_fullsized":"2024-03-21T13:16:50.458873+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2878,"log_dups_size":0,"ondisk_log_size":2878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.233033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188612608,"num_objects":2878,"num_object_clones":0,"num_object_copies":8634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2878,"num_write_kb":184192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":422,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.505668+0000","last_change":"2024-03-21T13:15:04.458567+0000","last_active":"2024-03-21T13:16:47.505668+0000","last_peered":"2024-03-21T13:16:47.505668+0000","last_clean":"2024-03-21T13:16:47.505668+0000","last_became_active":"2024-03-21T13:15:04.456786+0000","last_became_peered":"2024-03-21T13:15:04.456786+0000","last_unstale":"2024-03-21T13:16:47.505668+0000","last_undegraded":"2024-03-21T13:16:47.505668+0000","last_fullsized":"2024-03-21T13:16:47.505668+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:01.163722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:01.163722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":168,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504064+0000","last_change":"2024-03-21T13:16:17.169677+0000","last_active":"2024-03-21T13:16:47.504064+0000","last_peered":"2024-03-21T13:16:47.504064+0000","last_clean":"2024-03-21T13:16:47.504064+0000","last_became_active":"2024-03-21T13:12:29.512291+0000","last_became_peered":"2024-03-21T13:12:29.512291+0000","last_unstale":"2024-03-21T13:16:47.504064+0000","last_undegraded":"2024-03-21T13:16:47.504064+0000","last_fullsized":"2024-03-21T13:16:47.504064+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:17.169572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:28.030369+0000","last_clean_scrub_stamp":"2024-03-21T13:16:17.169572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.905934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"234'25","reported_seq":180,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503444+0000","last_change":"2024-03-21T13:15:57.053925+0000","last_active":"2024-03-21T13:16:47.503444+0000","last_peered":"2024-03-21T13:16:47.503444+0000","last_clean":"2024-03-21T13:16:47.503444+0000","last_became_active":"2024-03-21T13:12:29.526114+0000","last_became_peered":"2024-03-21T13:12:29.526114+0000","last_unstale":"2024-03-21T13:16:47.503444+0000","last_undegraded":"2024-03-21T13:16:47.503444+0000","last_fullsized":"2024-03-21T13:16:47.503444+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-21T13:15:57.053797+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:15:57.053797+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:09.141738+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"375'14","reported_seq":186,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.514238+0000","last_change":"2024-03-21T13:16:07.043970+0000","last_active":"2024-03-21T13:16:47.514238+0000","last_peered":"2024-03-21T13:16:47.514238+0000","last_clean":"2024-03-21T13:16:47.514238+0000","last_became_active":"2024-03-21T13:15:04.464590+0000","last_became_peered":"2024-03-21T13:15:04.464590+0000","last_unstale":"2024-03-21T13:16:47.514238+0000","last_undegraded":"2024-03-21T13:16:47.514238+0000","last_fullsized":"2024-03-21T13:16:47.514238+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'14","last_scrub_stamp":"2024-03-21T13:16:07.043880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.043880+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:17.795947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.e","version":"405'2945","reported_seq":2985,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:50.226723+0000","last_change":"2024-03-21T13:16:13.899791+0000","last_active":"2024-03-21T13:16:50.226723+0000","last_peered":"2024-03-21T13:16:50.226723+0000","last_clean":"2024-03-21T13:16:50.226723+0000","last_became_active":"2024-03-21T13:16:13.899480+0000","last_became_peered":"2024-03-21T13:16:13.899480+0000","last_unstale":"2024-03-21T13:16:50.226723+0000","last_undegraded":"2024-03-21T13:16:50.226723+0000","last_fullsized":"2024-03-21T13:16:50.226723+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:38.558703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193003520,"num_objects":2945,"num_object_clones":0,"num_object_copies":8835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":188480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":737,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.507105+0000","last_change":"2024-03-21T13:15:06.824637+0000","last_active":"2024-03-21T13:16:47.507105+0000","last_peered":"2024-03-21T13:16:47.507105+0000","last_clean":"2024-03-21T13:16:47.507105+0000","last_became_active":"2024-03-21T13:10:03.606557+0000","last_became_peered":"2024-03-21T13:10:03.606557+0000","last_unstale":"2024-03-21T13:16:47.507105+0000","last_undegraded":"2024-03-21T13:16:47.507105+0000","last_fullsized":"2024-03-21T13:16:47.507105+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:06.824520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:06.824520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:47.641764+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":789,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.510664+0000","last_change":"2024-03-21T13:14:21.647752+0000","last_active":"2024-03-21T13:16:47.510664+0000","last_peered":"2024-03-21T13:16:47.510664+0000","last_clean":"2024-03-21T13:16:47.510664+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:16:47.510664+0000","last_undegraded":"2024-03-21T13:16:47.510664+0000","last_fullsized":"2024-03-21T13:16:47.510664+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:14:21.647601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:21.647601+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'35","reported_seq":184,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.502702+0000","last_change":"2024-03-21T13:14:12.399995+0000","last_active":"2024-03-21T13:16:47.502702+0000","last_peered":"2024-03-21T13:16:47.502702+0000","last_clean":"2024-03-21T13:16:47.502702+0000","last_became_active":"2024-03-21T13:12:29.525154+0000","last_became_peered":"2024-03-21T13:12:29.525154+0000","last_unstale":"2024-03-21T13:16:47.502702+0000","last_undegraded":"2024-03-21T13:16:47.502702+0000","last_fullsized":"2024-03-21T13:16:47.502702+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:14:12.399785+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:14:12.399785+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198502+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"226'10","reported_seq":959,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:42.984804+0000","last_change":"2024-03-21T13:16:18.672479+0000","last_active":"2024-03-21T13:16:42.984804+0000","last_peered":"2024-03-21T13:16:42.984804+0000","last_clean":"2024-03-21T13:16:42.984804+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:16:42.984804+0000","last_undegraded":"2024-03-21T13:16:42.984804+0000","last_fullsized":"2024-03-21T13:16:42.984804+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T13:16:18.672392+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.672392+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:25.253358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"405'2840","reported_seq":2877,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:50.370092+0000","last_change":"2024-03-21T13:16:13.905080+0000","last_active":"2024-03-21T13:16:50.370092+0000","last_peered":"2024-03-21T13:16:50.370092+0000","last_clean":"2024-03-21T13:16:50.370092+0000","last_became_active":"2024-03-21T13:16:13.904756+0000","last_became_peered":"2024-03-21T13:16:13.904756+0000","last_unstale":"2024-03-21T13:16:50.370092+0000","last_undegraded":"2024-03-21T13:16:50.370092+0000","last_fullsized":"2024-03-21T13:16:50.370092+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2840,"log_dups_size":0,"ondisk_log_size":2840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:22.134211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186122240,"num_objects":2840,"num_object_clones":0,"num_object_copies":8520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2840,"num_write_kb":181760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":653,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.546999+0000","last_change":"2024-03-21T13:15:10.158338+0000","last_active":"2024-03-21T13:16:41.546999+0000","last_peered":"2024-03-21T13:16:41.546999+0000","last_clean":"2024-03-21T13:16:41.546999+0000","last_became_active":"2024-03-21T13:10:03.610471+0000","last_became_peered":"2024-03-21T13:10:03.610471+0000","last_unstale":"2024-03-21T13:16:41.546999+0000","last_undegraded":"2024-03-21T13:16:41.546999+0000","last_fullsized":"2024-03-21T13:16:41.546999+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:10.157920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:15:10.157920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587924+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":728,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.547546+0000","last_change":"2024-03-21T13:14:18.185914+0000","last_active":"2024-03-21T13:16:41.547546+0000","last_peered":"2024-03-21T13:16:41.547546+0000","last_clean":"2024-03-21T13:16:41.547546+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:16:41.547546+0000","last_undegraded":"2024-03-21T13:16:41.547546+0000","last_fullsized":"2024-03-21T13:16:41.547546+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:18.185782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:14:18.185782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587366+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"234'30","reported_seq":775,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.551052+0000","last_change":"2024-03-21T13:15:09.148375+0000","last_active":"2024-03-21T13:16:41.551052+0000","last_peered":"2024-03-21T13:16:41.551052+0000","last_clean":"2024-03-21T13:16:41.551052+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:16:41.551052+0000","last_undegraded":"2024-03-21T13:16:41.551052+0000","last_fullsized":"2024-03-21T13:16:41.551052+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'30","last_scrub_stamp":"2024-03-21T13:15:09.148229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:09.148229+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:50.715530+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":740,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.500272+0000","last_change":"2024-03-21T13:15:00.229608+0000","last_active":"2024-03-21T13:16:47.500272+0000","last_peered":"2024-03-21T13:16:47.500272+0000","last_clean":"2024-03-21T13:16:47.500272+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:16:47.500272+0000","last_undegraded":"2024-03-21T13:16:47.500272+0000","last_fullsized":"2024-03-21T13:16:47.500272+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:00.229502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:00.229502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:47.961662+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"403'2551","reported_seq":2587,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:46.144538+0000","last_change":"2024-03-21T13:16:13.900272+0000","last_active":"2024-03-21T13:16:46.144538+0000","last_peered":"2024-03-21T13:16:46.144538+0000","last_clean":"2024-03-21T13:16:46.144538+0000","last_became_active":"2024-03-21T13:16:13.899979+0000","last_became_peered":"2024-03-21T13:16:13.899979+0000","last_unstale":"2024-03-21T13:16:46.144538+0000","last_undegraded":"2024-03-21T13:16:46.144538+0000","last_fullsized":"2024-03-21T13:16:46.144538+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2551,"log_dups_size":0,"ondisk_log_size":2551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:21.233864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167182336,"num_objects":2551,"num_object_clones":0,"num_object_copies":7653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2551,"num_write_kb":163264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":167,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504152+0000","last_change":"2024-03-21T13:16:14.280619+0000","last_active":"2024-03-21T13:16:47.504152+0000","last_peered":"2024-03-21T13:16:47.504152+0000","last_clean":"2024-03-21T13:16:47.504152+0000","last_became_active":"2024-03-21T13:12:29.518722+0000","last_became_peered":"2024-03-21T13:12:29.518722+0000","last_unstale":"2024-03-21T13:16:47.504152+0000","last_undegraded":"2024-03-21T13:16:47.504152+0000","last_fullsized":"2024-03-21T13:16:47.504152+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:14.280515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:14.280515+0000","last_clean_scrub_stamp":"2024-03-21T13:16:14.280515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:42.015454+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"331'13","reported_seq":224,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504104+0000","last_change":"2024-03-21T13:15:15.203026+0000","last_active":"2024-03-21T13:16:47.504104+0000","last_peered":"2024-03-21T13:16:47.504104+0000","last_clean":"2024-03-21T13:16:47.504104+0000","last_became_active":"2024-03-21T13:12:29.525058+0000","last_became_peered":"2024-03-21T13:12:29.525058+0000","last_unstale":"2024-03-21T13:16:47.504104+0000","last_undegraded":"2024-03-21T13:16:47.504104+0000","last_fullsized":"2024-03-21T13:16:47.504104+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:15:15.202826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:15.202826+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198765+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":734,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.550899+0000","last_change":"2024-03-21T13:14:17.224206+0000","last_active":"2024-03-21T13:16:41.550899+0000","last_peered":"2024-03-21T13:16:41.550899+0000","last_clean":"2024-03-21T13:16:41.550899+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:16:41.550899+0000","last_undegraded":"2024-03-21T13:16:41.550899+0000","last_fullsized":"2024-03-21T13:16:41.550899+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:17.224086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:17.224086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587231+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"234'34","reported_seq":692,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.547630+0000","last_change":"2024-03-21T13:16:26.147953+0000","last_active":"2024-03-21T13:16:41.547630+0000","last_peered":"2024-03-21T13:16:41.547630+0000","last_clean":"2024-03-21T13:16:41.547630+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:16:41.547630+0000","last_undegraded":"2024-03-21T13:16:41.547630+0000","last_fullsized":"2024-03-21T13:16:41.547630+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'34","last_scrub_stamp":"2024-03-21T13:14:00.092594+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:14:00.092594+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"405'2838","reported_seq":2877,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:50.452166+0000","last_change":"2024-03-21T13:16:13.904388+0000","last_active":"2024-03-21T13:16:50.452166+0000","last_peered":"2024-03-21T13:16:50.452166+0000","last_clean":"2024-03-21T13:16:50.452166+0000","last_became_active":"2024-03-21T13:16:13.904096+0000","last_became_peered":"2024-03-21T13:16:13.904096+0000","last_unstale":"2024-03-21T13:16:50.452166+0000","last_undegraded":"2024-03-21T13:16:50.452166+0000","last_fullsized":"2024-03-21T13:16:50.452166+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2838,"log_dups_size":0,"ondisk_log_size":2838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:39.867007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185991168,"num_objects":2838,"num_object_clones":0,"num_object_copies":8514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2838,"num_write_kb":181632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"375'2","reported_seq":738,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.545479+0000","last_change":"2024-03-21T13:15:14.313700+0000","last_active":"2024-03-21T13:16:41.545479+0000","last_peered":"2024-03-21T13:16:41.545479+0000","last_clean":"2024-03-21T13:16:41.545479+0000","last_became_active":"2024-03-21T13:12:29.502816+0000","last_became_peered":"2024-03-21T13:12:29.502816+0000","last_unstale":"2024-03-21T13:16:41.545479+0000","last_undegraded":"2024-03-21T13:16:41.545479+0000","last_fullsized":"2024-03-21T13:16:41.545479+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'2","last_scrub_stamp":"2024-03-21T13:15:14.313376+0000","last_deep_scrub":"375'2","last_deep_scrub_stamp":"2024-03-21T13:15:14.313376+0000","last_clean_scrub_stamp":"2024-03-21T13:15:14.313376+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:50.715786+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":167,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503974+0000","last_change":"2024-03-21T13:15:07.382672+0000","last_active":"2024-03-21T13:16:47.503974+0000","last_peered":"2024-03-21T13:16:47.503974+0000","last_clean":"2024-03-21T13:16:47.503974+0000","last_became_active":"2024-03-21T13:12:29.524087+0000","last_became_peered":"2024-03-21T13:12:29.524087+0000","last_unstale":"2024-03-21T13:16:47.503974+0000","last_undegraded":"2024-03-21T13:16:47.503974+0000","last_fullsized":"2024-03-21T13:16:47.503974+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:07.382536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:07.382536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:51.198622+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":776,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.500356+0000","last_change":"2024-03-21T13:16:04.825360+0000","last_active":"2024-03-21T13:16:47.500356+0000","last_peered":"2024-03-21T13:16:47.500356+0000","last_clean":"2024-03-21T13:16:47.500356+0000","last_became_active":"2024-03-21T13:10:03.586593+0000","last_became_peered":"2024-03-21T13:10:03.586593+0000","last_unstale":"2024-03-21T13:16:47.500356+0000","last_undegraded":"2024-03-21T13:16:47.500356+0000","last_fullsized":"2024-03-21T13:16:47.500356+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:04.825280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:04.825280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:18.635897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":491,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.504025+0000","last_change":"2024-03-21T13:14:11.615096+0000","last_active":"2024-03-21T13:16:47.504025+0000","last_peered":"2024-03-21T13:16:47.504025+0000","last_clean":"2024-03-21T13:16:47.504025+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:16:47.504025+0000","last_undegraded":"2024-03-21T13:16:47.504025+0000","last_fullsized":"2024-03-21T13:16:47.504025+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:14:11.611538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:11.611538+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:50.916201+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"405'2878","reported_seq":2917,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:50.455368+0000","last_change":"2024-03-21T13:16:13.899357+0000","last_active":"2024-03-21T13:16:50.455368+0000","last_peered":"2024-03-21T13:16:50.455368+0000","last_clean":"2024-03-21T13:16:50.455368+0000","last_became_active":"2024-03-21T13:16:13.899059+0000","last_became_peered":"2024-03-21T13:16:13.899059+0000","last_unstale":"2024-03-21T13:16:50.455368+0000","last_undegraded":"2024-03-21T13:16:50.455368+0000","last_fullsized":"2024-03-21T13:16:50.455368+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2878,"log_dups_size":0,"ondisk_log_size":2878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:25.856013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188612608,"num_objects":2878,"num_object_clones":0,"num_object_copies":8634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2878,"num_write_kb":184192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":312,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503210+0000","last_change":"2024-03-21T13:16:20.483084+0000","last_active":"2024-03-21T13:16:47.503210+0000","last_peered":"2024-03-21T13:16:47.503210+0000","last_clean":"2024-03-21T13:16:47.503210+0000","last_became_active":"2024-03-21T13:10:03.602752+0000","last_became_peered":"2024-03-21T13:10:03.602752+0000","last_unstale":"2024-03-21T13:16:47.503210+0000","last_undegraded":"2024-03-21T13:16:47.503210+0000","last_fullsized":"2024-03-21T13:16:47.503210+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:20.482943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.482943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.858705+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":648,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.501538+0000","last_change":"2024-03-21T13:15:59.662672+0000","last_active":"2024-03-21T13:16:47.501538+0000","last_peered":"2024-03-21T13:16:47.501538+0000","last_clean":"2024-03-21T13:16:47.501538+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:16:47.501538+0000","last_undegraded":"2024-03-21T13:16:47.501538+0000","last_fullsized":"2024-03-21T13:16:47.501538+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:59.662591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:59.662591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:10.639394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":753,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503489+0000","last_change":"2024-03-21T13:15:02.678593+0000","last_active":"2024-03-21T13:16:47.503489+0000","last_peered":"2024-03-21T13:16:47.503489+0000","last_clean":"2024-03-21T13:16:47.503489+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:16:47.503489+0000","last_undegraded":"2024-03-21T13:16:47.503489+0000","last_fullsized":"2024-03-21T13:16:47.503489+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:02.678485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:15:02.678485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:50.916230+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"391'39","reported_seq":322,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.500723+0000","last_change":"2024-03-21T13:16:14.266738+0000","last_active":"2024-03-21T13:16:47.500723+0000","last_peered":"2024-03-21T13:16:47.500723+0000","last_clean":"2024-03-21T13:16:47.500723+0000","last_became_active":"2024-03-21T13:10:03.591832+0000","last_became_peered":"2024-03-21T13:10:03.591832+0000","last_unstale":"2024-03-21T13:16:47.500723+0000","last_undegraded":"2024-03-21T13:16:47.500723+0000","last_fullsized":"2024-03-21T13:16:47.500723+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'39","last_scrub_stamp":"2024-03-21T13:16:14.266573+0000","last_deep_scrub":"391'39","last_deep_scrub_stamp":"2024-03-21T13:16:14.266573+0000","last_clean_scrub_stamp":"2024-03-21T13:16:14.266573+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:36.013148+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"405'2947","reported_seq":2986,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:50.438087+0000","last_change":"2024-03-21T13:16:13.904234+0000","last_active":"2024-03-21T13:16:50.438087+0000","last_peered":"2024-03-21T13:16:50.438087+0000","last_clean":"2024-03-21T13:16:50.438087+0000","last_became_active":"2024-03-21T13:16:13.903407+0000","last_became_peered":"2024-03-21T13:16:13.903407+0000","last_unstale":"2024-03-21T13:16:50.438087+0000","last_undegraded":"2024-03-21T13:16:50.438087+0000","last_fullsized":"2024-03-21T13:16:50.438087+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":2947,"log_dups_size":0,"ondisk_log_size":2947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.659743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193134592,"num_objects":2947,"num_object_clones":0,"num_object_copies":8841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":188608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":310,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.503076+0000","last_change":"2024-03-21T13:16:18.322285+0000","last_active":"2024-03-21T13:16:47.503076+0000","last_peered":"2024-03-21T13:16:47.503076+0000","last_clean":"2024-03-21T13:16:47.503076+0000","last_became_active":"2024-03-21T13:12:29.499442+0000","last_became_peered":"2024-03-21T13:12:29.499442+0000","last_unstale":"2024-03-21T13:16:47.503076+0000","last_undegraded":"2024-03-21T13:16:47.503076+0000","last_fullsized":"2024-03-21T13:16:47.503076+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.322198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.322198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:40.851693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"331'13","reported_seq":401,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-21T13:16:47.988633+0000","last_change":"2024-03-21T13:16:21.250482+0000","last_active":"2024-03-21T13:16:47.988633+0000","last_peered":"2024-03-21T13:16:47.988633+0000","last_clean":"2024-03-21T13:16:47.988633+0000","last_became_active":"2024-03-21T13:12:29.507925+0000","last_became_peered":"2024-03-21T13:12:29.507925+0000","last_unstale":"2024-03-21T13:16:47.988633+0000","last_undegraded":"2024-03-21T13:16:47.988633+0000","last_fullsized":"2024-03-21T13:16:47.988633+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:16:21.250398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.250398+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:47.624689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":728,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.546322+0000","last_change":"2024-03-21T13:16:05.813771+0000","last_active":"2024-03-21T13:16:41.546322+0000","last_peered":"2024-03-21T13:16:41.546322+0000","last_clean":"2024-03-21T13:16:41.546322+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:16:41.546322+0000","last_undegraded":"2024-03-21T13:16:41.546322+0000","last_fullsized":"2024-03-21T13:16:41.546322+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:05.813657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.813657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.222745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"234'20","reported_seq":689,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-03-21T13:16:41.545727+0000","last_change":"2024-03-21T13:15:04.211127+0000","last_active":"2024-03-21T13:16:41.545727+0000","last_peered":"2024-03-21T13:16:41.545727+0000","last_clean":"2024-03-21T13:16:41.545727+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:16:41.545727+0000","last_undegraded":"2024-03-21T13:16:41.545727+0000","last_fullsized":"2024-03-21T13:16:41.545727+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:15:04.211011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:04.211011+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:16:45.587859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":2887254016,"num_objects":44056,"num_object_clones":0,"num_object_copies":132168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44056,"num_write_kb":2819584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8897953792,"data_stored":8897953792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44056,"ondisk_log_size":44056,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1082,"num_read_kb":1077,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1100,"num_read_kb":857,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":6656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":748,"ondisk_log_size":748,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168799,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2182852,"kb_used_data":1281212,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":91562300,"statfs":{"total":95995035648,"available":93759795200,"internally_reserved":0,"allocated":1311961088,"data_stored":1311451853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":923269578},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463141,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2558404,"kb_used_data":1652604,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":91186748,"statfs":{"total":95995035648,"available":93375229952,"internally_reserved":0,"allocated":1692266496,"data_stored":1691756970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":927529414},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575101,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738627,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2294972,"kb_used_data":1451636,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":91450180,"statfs":{"total":95995035648,"available":93644984320,"internally_reserved":0,"allocated":1486475264,"data_stored":1485972869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411577,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1911896,"kb_used_data":1110224,"kb_used_omap":1,"kb_used_meta":801662,"kb_avail":91833256,"statfs":{"total":95995035648,"available":94037254144,"internally_reserved":0,"allocated":1136869376,"data_stored":1136402929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":820902341},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182643,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2079396,"kb_used_data":1069468,"kb_used_omap":1,"kb_used_meta":1009918,"kb_avail":91665756,"statfs":{"total":95995035648,"available":93865734144,"internally_reserved":0,"allocated":1095135232,"data_stored":1094643269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1034156487},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019118,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2063012,"kb_used_data":1089180,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":91682140,"statfs":{"total":95995035648,"available":93882511360,"internally_reserved":0,"allocated":1115320320,"data_stored":1114842064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724777,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1121568,"kb_used_data":887700,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":92623584,"statfs":{"total":95995035648,"available":94846550016,"internally_reserved":0,"allocated":909004800,"data_stored":908533935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1519386624,"data_stored":1519386624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1136590848,"data_stored":1136590848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1118044160,"data_stored":1118044160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1126170624,"data_stored":1126170624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":932904960,"data_stored":932904960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1723924480,"data_stored":1723924480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1340932096,"data_stored":1340932096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:16:52.203 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:16:52.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:52.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:52.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:52.331 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 23 16 49351 49335 134.049 101.562 0.0143345 0.00745489 2024-03-21T13:16:52.412 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:52.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:52.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:52.499 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:16:52.499 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-03-21T13:16:52.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-03-21T13:16:52.499 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:16:52.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.499 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:180928 8.3 8:c9037379:::benchmark_data_smithi163_107079_object45055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-21T13:16:52.499 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:08.363985+0000", 2024-03-21T13:16:52.499 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 584.12940851799999, 2024-03-21T13:16:52.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.993946526, 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 180928 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.363985+0000", 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.363985+0000", 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.363986+0000", 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6360000000000001e-06 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.363992+0000", 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3809999999999998e-06 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.364005+0000", 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3112000000000001e-05 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.364015+0000", 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6989999999999998e-06 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.364094+0000", 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9091999999999994e-05 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.364112+0000", 2024-03-21T13:16:52.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7886000000000001e-05 2024-03-21T13:16:52.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.353413+0000", 2024-03-21T13:16:52.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98930138700000003 2024-03-21T13:16:52.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.353519+0000", 2024-03-21T13:16:52.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010623100000000001 2024-03-21T13:16:52.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,4", 2024-03-21T13:16:52.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.353610+0000", 2024-03-21T13:16:52.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0628e-05 2024-03-21T13:16:52.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.357695+0000", 2024-03-21T13:16:52.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040855850000000001 2024-03-21T13:16:52.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.357709+0000", 2024-03-21T13:16:52.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3341e-05 2024-03-21T13:16:52.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.357824+0000", 2024-03-21T13:16:52.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011544000000000001 2024-03-21T13:16:52.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.357889+0000", 2024-03-21T13:16:52.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5016000000000002e-05 2024-03-21T13:16:52.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.357931+0000", 2024-03-21T13:16:52.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2092000000000003e-05 2024-03-21T13:16:52.506 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.506 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.506 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.506 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:180936 8.3 8:c3210a5c:::benchmark_data_smithi163_107079_object45063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-21T13:16:52.506 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:08.375804+0000", 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 584.11758931199995, 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98556327399999999, 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 180936 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375804+0000", 2024-03-21T13:16:52.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375804+0000", 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375805+0000", 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0920000000000001e-06 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375810+0000", 2024-03-21T13:16:52.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2870000000000004e-06 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375820+0000", 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0121e-05 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375826+0000", 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5999999999999997e-06 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375897+0000", 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1359000000000006e-05 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.375907+0000", 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2210000000000003e-06 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.355959+0000", 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98005200599999998 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.356079+0000", 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119867 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,4", 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.356175+0000", 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6142000000000002e-05 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361160+0000", 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0049857160000000003 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361217+0000", 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6706000000000002e-05 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361261+0000", 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4069000000000001e-05 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361319+0000", 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7938e-05 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361367+0000", 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8149999999999998e-05 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15078.0:180962 8.3 8:cd1c3043:::benchmark_data_smithi163_107079_object45089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:07:08.401153+0000", 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 584.09224038800005, 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96533337100000005, 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.513 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15078", 2024-03-21T13:16:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/3797792383", 2024-03-21T13:16:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 180962 2024-03-21T13:16:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401153+0000", 2024-03-21T13:16:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401153+0000", 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401155+0000", 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6440000000000001e-06 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401160+0000", 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9969999999999998e-06 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401171+0000", 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1049999999999999e-05 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401178+0000", 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6459999999999993e-06 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401234+0000", 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5554999999999999e-05 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:08.401243+0000", 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7679999999999997e-06 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.357990+0000", 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95674721299999999 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.358108+0000", 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118019 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,4", 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.358207+0000", 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8783000000000006e-05 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361381+0000", 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003174035 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.361428+0000", 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7685000000000002e-05 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.366384+0000", 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0049555010000000002 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.366438+0000", 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4215000000000002e-05 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:07:09.366486+0000", 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8260000000000002e-05 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.519 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.520 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.520 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:52326 9.0 9:038ef788:::benchmark_data_smithi163_131650_object52325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e246)", 2024-03-21T13:16:52.520 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:17.468332+0000", 2024-03-21T13:16:52.520 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 455.02506153899998, 2024-03-21T13:16:52.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99509188100000001, 2024-03-21T13:16:52.520 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.520 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:16:52.520 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.520 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:52.520 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:52.520 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 52326 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.468332+0000", 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.468332+0000", 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.468335+0000", 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8090000000000001e-06 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.468380+0000", 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5525000000000002e-05 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.468399+0000", 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8855000000000002e-05 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.468410+0000", 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1066999999999999e-05 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.551585+0000", 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.083175313000000001 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:17.551597+0000", 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1311999999999999e-05 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:18.460573+0000", 2024-03-21T13:16:52.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.90897664600000005 2024-03-21T13:16:52.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-03-21T13:16:52.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:18.461383+0000", 2024-03-21T13:16:52.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00080961300000000004 2024-03-21T13:16:52.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:18.463342+0000", 2024-03-21T13:16:52.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019586120000000002 2024-03-21T13:16:52.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:18.463424+0000", 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2128999999999999e-05 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:54275 9.18 9.cf8139d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:09:19.460531+0000", 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 453.03286234699999, 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97871142700000002, 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:16:52.525 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.526 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:52.526 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:52.526 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 54275 2024-03-21T13:16:52.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.526 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460531+0000", 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460531+0000", 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460532+0000", 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.144e-06 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460554+0000", 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1467999999999998e-05 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460563+0000", 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.55e-06 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460570+0000", 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5030000000000002e-06 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460608+0000", 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8875999999999998e-05 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:19.460613+0000", 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6789999999999996e-06 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:20.439232+0000", 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97861839500000003 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:09:20.439242+0000", 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0812000000000001e-05 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100381 9.b 9:dd44d674:::benchmark_data_smithi163_131650_object100380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.561457+0000", 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.931936742, 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1224653619999998, 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.530 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100381 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.561457+0000", 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.561457+0000", 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.561458+0000", 2024-03-21T13:16:52.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.502e-06 2024-03-21T13:16:52.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.561495+0000", 2024-03-21T13:16:52.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7379000000000002e-05 2024-03-21T13:16:52.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.561512+0000", 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6322000000000001e-05 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.561520+0000", 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5469999999999997e-06 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.562792+0000", 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001272198 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.562944+0000", 2024-03-21T13:16:52.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000151242 2024-03-21T13:16:52.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:16:52.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.563101+0000", 2024-03-21T13:16:52.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015759199999999999 2024-03-21T13:16:52.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567303+0000", 2024-03-21T13:16:52.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042017759999999999 2024-03-21T13:16:52.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567324+0000", 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0729000000000001e-05 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.586023+0000", 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018698887000000001 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.586037+0000", 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4079e-05 2024-03-21T13:16:52.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574162+0000", 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98812536900000003 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574175+0000", 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3006e-05 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.683847+0000", 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1096722409999999 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.683922+0000", 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4493000000000007e-05 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100386 9.b 9:d957d1b3:::benchmark_data_smithi163_131650_object100385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.567504+0000", 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.92588911299998, 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.129416671, 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-03-21T13:16:52.537 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100386 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567504+0000", 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567504+0000", 2024-03-21T13:16:52.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.539 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.539 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.539 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.539 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567506+0000", 2024-03-21T13:16:52.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.345e-06 2024-03-21T13:16:52.539 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.539 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.539 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.539 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567541+0000", 2024-03-21T13:16:52.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5472999999999997e-05 2024-03-21T13:16:52.539 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.539 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.539 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:16:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:16:52.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567555+0000", 2024-03-21T13:16:52.540 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3511e-05 2024-03-21T13:16:52.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567563+0000", 2024-03-21T13:16:52.540 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3669999999999997e-06 2024-03-21T13:16:52.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567640+0000", 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7366000000000004e-05 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.567793+0000", 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015228 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.568487+0000", 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00069475500000000003 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571382+0000", 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028947220000000002 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.588432+0000", 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017050108000000001 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.588443+0000", 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1352000000000001e-05 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574213+0000", 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98576913200000005 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574223+0000", 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0275000000000001e-05 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.684130+0000", 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.10990674 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.684673+0000", 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00054377400000000002 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.696860+0000", 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012186591 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.696921+0000", 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.088e-05 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100389 9.0 9:063bd6a6:::benchmark_data_smithi163_131650_object100388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.571614+0000", 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.92177925499999, 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.256985604, 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100389 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.546 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571614+0000", 2024-03-21T13:16:52.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.546 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.546 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571614+0000", 2024-03-21T13:16:52.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.546 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.546 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571616+0000", 2024-03-21T13:16:52.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.748e-06 2024-03-21T13:16:52.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571650+0000", 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4409000000000002e-05 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571664+0000", 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.36e-05 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.571673+0000", 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3770000000000002e-06 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.592173+0000", 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020499746999999999 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.592183+0000", 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8800000000000003e-06 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.571390+0000", 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.979206727 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.571404+0000", 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4791e-05 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.803242+0000", 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2318381460000001 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.803399+0000", 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000156587 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.803796+0000", 2024-03-21T13:16:52.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00039736600000000002 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.818306+0000", 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014509762000000001 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.818452+0000", 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014624500000000001 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.824862+0000", 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064098009999999997 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.824923+0000", 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0291000000000003e-05 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.828600+0000", 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036771270000000001 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.552 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100403 9.b 9:d27bacb7:::benchmark_data_smithi163_131650_object100402:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:16:52.552 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.597070+0000", 2024-03-21T13:16:52.552 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.89632306099998, 2024-03-21T13:16:52.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1000044849999999, 2024-03-21T13:16:52.552 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.552 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.552 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.552 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:52.552 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:52.552 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100403 2024-03-21T13:16:52.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.552 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597070+0000", 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597070+0000", 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597072+0000", 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.325e-06 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597104+0000", 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2159000000000002e-05 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597113+0000", 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0599999999999997e-06 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597119+0000", 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7150000000000001e-06 2024-03-21T13:16:52.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597232+0000", 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112543 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.597242+0000", 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0071000000000001e-05 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574254+0000", 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.977012141 2024-03-21T13:16:52.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574263+0000", 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1130000000000007e-06 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.684729+0000", 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.110465461 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.684879+0000", 2024-03-21T13:16:52.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014982799999999999 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,3", 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.684982+0000", 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103797 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.689118+0000", 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041350960000000004 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.691334+0000", 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022162530000000001 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.691364+0000", 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9961999999999999e-05 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.696956+0000", 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0055922439999999997 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697002+0000", 2024-03-21T13:16:52.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6020000000000003e-05 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697075+0000", 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2696999999999999e-05 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100405 9.0 9:03b1f289:::benchmark_data_smithi163_131650_object100404:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.602198+0000", 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.89119509099999, 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2290228340000002, 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.559 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100405 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602198+0000", 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602198+0000", 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602200+0000", 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5039999999999999e-06 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602231+0000", 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1597000000000003e-05 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602248+0000", 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6399e-05 2024-03-21T13:16:52.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602257+0000", 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6549999999999999e-06 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602323+0000", 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5296999999999995e-05 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.602334+0000", 2024-03-21T13:16:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1402e-05 2024-03-21T13:16:52.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.573102+0000", 2024-03-21T13:16:52.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97076796700000001 2024-03-21T13:16:52.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.573115+0000", 2024-03-21T13:16:52.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2679e-05 2024-03-21T13:16:52.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.813300+0000", 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2401856250000001 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.813442+0000", 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014174399999999999 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.813831+0000", 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038913100000000003 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.818610+0000", 2024-03-21T13:16:52.565 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047784689999999996 2024-03-21T13:16:52.565 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.565 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.565 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.565 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.821637+0000", 2024-03-21T13:16:52.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:52.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030269530000000002 2024-03-21T13:16:52.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.828668+0000", 2024-03-21T13:16:52.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0070315559999999996 2024-03-21T13:16:52.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.828723+0000", 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5105999999999997e-05 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.831221+0000", 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00249775 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100413 9.b 9:de0331ce:::benchmark_data_smithi163_131650_object100412:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:16:52.567 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.608307+0000", 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.88508634599998, 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0889179439999999, 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100413 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608307+0000", 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608307+0000", 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608308+0000", 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.322e-06 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608360+0000", 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1860999999999997e-05 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.570 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608369+0000", 2024-03-21T13:16:52.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6170000000000007e-06 2024-03-21T13:16:52.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.570 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.570 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.570 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608376+0000", 2024-03-21T13:16:52.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5440000000000001e-06 2024-03-21T13:16:52.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.570 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.570 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.570 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608442+0000", 2024-03-21T13:16:52.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5980000000000002e-05 2024-03-21T13:16:52.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.608452+0000", 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0205e-05 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574295+0000", 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96584290500000003 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574304+0000", 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5399999999999996e-06 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.686498+0000", 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1121937829999999 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.686598+0000", 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000100296 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,3", 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.686706+0000", 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107565 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.691297+0000", 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0045910040000000001 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.695175+0000", 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003878102 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.695383+0000", 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000208657 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697114+0000", 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001730678 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697150+0000", 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5581999999999999e-05 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697225+0000", 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5302999999999994e-05 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.574 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25226.0:100418 9.b 9:dd6b4148:::benchmark_data_smithi163_131650_object100417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:10:02.615444+0000", 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 409.87794975499997, 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0820358419999998, 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25226", 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100418 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615444+0000", 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615444+0000", 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615445+0000", 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5400000000000001e-06 2024-03-21T13:16:52.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615499+0000", 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3872000000000002e-05 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615511+0000", 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2054e-05 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615520+0000", 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6950000000000006e-06 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615579+0000", 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9378e-05 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:02.615586+0000", 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5250000000000002e-06 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574333+0000", 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95874714800000005 2024-03-21T13:16:52.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:03.574341+0000", 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.49e-06 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.687885+0000", 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.113543653 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.687993+0000", 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108359 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,3", 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.688151+0000", 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000157947 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.691372+0000", 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032209890000000001 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.695216+0000", 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038435320000000002 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.695415+0000", 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019962300000000001 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697343+0000", 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001927443 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697390+0000", 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7639000000000002e-05 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:10:04.697479+0000", 2024-03-21T13:16:52.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8955000000000005e-05 2024-03-21T13:16:52.582 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.582 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.582 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.582 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:86745 10.7 10:e893f8c5:::benchmark_data_smithi163_154138_object86744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-21T13:16:52.582 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:13:22.351225+0000", 2024-03-21T13:16:52.582 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 210.14216853299999, 2024-03-21T13:16:52.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15515072599999999, 2024-03-21T13:16:52.582 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 86745 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.351225+0000", 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.583 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.351225+0000", 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.351225+0000", 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4499999999999997e-07 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.351281+0000", 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5439999999999998e-05 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.351287+0000", 2024-03-21T13:16:52.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2990000000000002e-06 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.351293+0000", 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8760000000000002e-06 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.361981+0000", 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010688543999999999 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.362131+0000", 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014936600000000001 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,5", 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.362312+0000", 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018144499999999999 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.369070+0000", 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067576429999999998 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.371990+0000", 2024-03-21T13:16:52.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029197260000000001 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.506288+0000", 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13429824900000001 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.506337+0000", 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9629999999999997e-05 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:13:22.506375+0000", 2024-03-21T13:16:52.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8062999999999998e-05 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:191604 10.f 10:ff7dfea1:::benchmark_data_smithi163_154138_object59805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:14:49.850920+0000", 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 122.642472854, 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.23430904699999999, 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:16:52.588 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 191604 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.850920+0000", 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.850920+0000", 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.850921+0000", 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1299999999999999e-07 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.850925+0000", 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8310000000000003e-06 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.868656+0000", 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017730817999999999 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.868673+0000", 2024-03-21T13:16:52.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7277000000000001e-05 2024-03-21T13:16:52.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.869798+0000", 2024-03-21T13:16:52.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011247659999999999 2024-03-21T13:16:52.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.872261+0000", 2024-03-21T13:16:52.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024630089999999999 2024-03-21T13:16:52.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-03-21T13:16:52.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.872375+0000", 2024-03-21T13:16:52.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011369 2024-03-21T13:16:52.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.876215+0000", 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038405689999999998 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.882116+0000", 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005900705 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:50.085077+0000", 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.20296095 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:50.085173+0000", 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5810999999999995e-05 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:50.085229+0000", 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6808000000000002e-05 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:191630 10.e 10:78723f98:::benchmark_data_smithi163_154138_object59831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-21T13:16:52.594 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:14:49.878782+0000", 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 122.61461138200001, 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.20966746, 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 191630 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.878782+0000", 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.878782+0000", 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.878783+0000", 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1699999999999997e-07 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.878787+0000", 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2980000000000002e-06 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.878796+0000", 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.816e-06 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.878803+0000", 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0740000000000004e-06 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.878863+0000", 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0194e-05 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.878984+0000", 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012121900000000001 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,4", 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.879086+0000", 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101964 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.882041+0000", 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002954735 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.883520+0000", 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014789530000000001 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:50.088343+0000", 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.204823528 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:50.088398+0000", 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4154999999999999e-05 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:50.088449+0000", 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1807000000000001e-05 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:191650 10.e 10:77e83d32:::benchmark_data_smithi163_154138_object59851:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:14:49.898863+0000", 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 122.59453036799999, 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.18972760399999999, 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.600 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 191650 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.898863+0000", 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.898863+0000", 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.898864+0000", 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5499999999999998e-07 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.898867+0000", 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5669999999999999e-06 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.898885+0000", 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8168999999999998e-05 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.898892+0000", 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.002e-06 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.898937+0000", 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4552999999999998e-05 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.899023+0000", 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6558999999999996e-05 2024-03-21T13:16:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,4", 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.899089+0000", 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5827000000000005e-05 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.901174+0000", 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002084855 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:49.901472+0000", 2024-03-21T13:16:52.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029801300000000002 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:50.088500+0000", 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.18702753699999999 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:50.088551+0000", 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1208000000000002e-05 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:50.088591+0000", 2024-03-21T13:16:52.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9659000000000001e-05 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:196234 10.f 10:f6491fc0:::benchmark_data_smithi163_154138_object64435:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:14:53.597433+0000", 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 118.895960258, 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.175331774, 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:16:52.606 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 196234 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.597433+0000", 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.597433+0000", 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.597434+0000", 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1650000000000001e-06 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.597440+0000", 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5099999999999998e-06 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.597452+0000", 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2255999999999999e-05 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.597460+0000", 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4339999999999992e-06 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.597542+0000", 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1823999999999995e-05 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.597663+0000", 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000120921 2024-03-21T13:16:52.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-03-21T13:16:52.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.597781+0000", 2024-03-21T13:16:52.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117539 2024-03-21T13:16:52.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.601885+0000", 2024-03-21T13:16:52.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041043410000000001 2024-03-21T13:16:52.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.609036+0000", 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0071512950000000002 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.772673+0000", 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16363697899999999 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.772723+0000", 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9539999999999997e-05 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.772765+0000", 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.197e-05 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:196255 10.f 10:ff8371ec:::benchmark_data_smithi163_154138_object64456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:14:53.611452+0000", 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 118.88194173399999, 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.161481667, 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.612 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 196255 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.611452+0000", 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.611452+0000", 2024-03-21T13:16:52.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.611452+0000", 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7000000000000003e-07 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.611457+0000", 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9269999999999996e-06 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.611472+0000", 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4292e-05 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.611479+0000", 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5299999999999999e-06 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.611900+0000", 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00042039999999999997 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.612012+0000", 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011194800000000001 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.612128+0000", 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116464 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.614736+0000", 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026082480000000001 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.616250+0000", 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001513775 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.772819+0000", 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.156569246 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.772885+0000", 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5526999999999998e-05 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.772933+0000", 2024-03-21T13:16:52.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8439999999999997e-05 2024-03-21T13:16:52.618 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.618 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.618 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.619 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:196282 10.f 10:f3a65811:::benchmark_data_smithi163_154138_object64483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-21T13:16:52.619 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:14:53.631215+0000", 2024-03-21T13:16:52.619 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 118.862178228, 2024-03-21T13:16:52.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.14189938299999999, 2024-03-21T13:16:52.619 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.619 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.619 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.619 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:16:52.619 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:16:52.619 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 196282 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.631215+0000", 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.631215+0000", 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.631216+0000", 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0449999999999999e-06 2024-03-21T13:16:52.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.631220+0000", 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.313e-06 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.631229+0000", 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1070000000000005e-06 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.631235+0000", 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.64e-06 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.631975+0000", 2024-03-21T13:16:52.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00073996700000000001 2024-03-21T13:16:52.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.632095+0000", 2024-03-21T13:16:52.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011965599999999999 2024-03-21T13:16:52.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-03-21T13:16:52.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.632405+0000", 2024-03-21T13:16:52.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031047300000000002 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.635110+0000", 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027049270000000002 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.639929+0000", 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0048191589999999999 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.772986+0000", 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.133057128 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.773064+0000", 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7843000000000007e-05 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.773114+0000", 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0124999999999999e-05 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.624 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25481.0:196287 10.f 10:fb8a26fe:::benchmark_data_smithi163_154138_object64488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:14:53.636485+0000", 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 118.85690824300001, 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13678875400000001, 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25481", 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 196287 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.625 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.636485+0000", 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.636485+0000", 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.636486+0000", 2024-03-21T13:16:52.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4799999999999997e-07 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.636490+0000", 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6600000000000001e-06 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.636497+0000", 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2400000000000001e-06 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.636502+0000", 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7740000000000004e-06 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.636548+0000", 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6468999999999998e-05 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.636660+0000", 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111859 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-03-21T13:16:52.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.636784+0000", 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001236 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.639956+0000", 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031719389999999999 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.644356+0000", 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004400124 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:16:52.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.773153+0000", 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12879766200000001 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.773238+0000", 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4981999999999994e-05 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:14:53.773274+0000", 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5497000000000003e-05 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.630 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:52.631 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:52.631 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:16:52.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:52.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:52.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:52.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:52.770 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:52.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:52.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:52.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:52.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:52.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:53.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:52 smithi204 ceph-mon[33153]: pgmap v855: 148 pgs: 148 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-21T13:16:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:52 smithi204 ceph-mon[33153]: from='client.25390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:16:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:52 smithi204 ceph-mon[33153]: 2.1 scrub starts 2024-03-21T13:16:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:52 smithi204 ceph-mon[33153]: 2.1 scrub ok 2024-03-21T13:16:53.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:53.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:52 smithi163 ceph-mon[29621]: pgmap v855: 148 pgs: 148 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-21T13:16:53.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:52 smithi163 ceph-mon[29621]: from='client.25390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:16:53.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:52 smithi163 ceph-mon[29621]: 2.1 scrub starts 2024-03-21T13:16:53.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:52 smithi163 ceph-mon[29621]: 2.1 scrub ok 2024-03-21T13:16:53.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:52 smithi163 ceph-mon[40675]: pgmap v855: 148 pgs: 148 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-21T13:16:53.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:52 smithi163 ceph-mon[40675]: from='client.25390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:16:53.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:52 smithi163 ceph-mon[40675]: 2.1 scrub starts 2024-03-21T13:16:53.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:52 smithi163 ceph-mon[40675]: 2.1 scrub ok 2024-03-21T13:16:53.047 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:53.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:53.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:53.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:53.184 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:53.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:53.256 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:53.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:53.314 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:53.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:53.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:53.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:53.331 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 24 16 50887 50871 132.463 96 0.0110533 0.00754398 2024-03-21T13:16:53.423 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:16:53.474 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:53.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:53.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:16:53.575 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:53.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:53.616 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:53.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:53.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[29621]: 5.12 scrub starts 2024-03-21T13:16:53.697 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[29621]: 5.12 scrub ok 2024-03-21T13:16:53.697 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:53.697 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[29621]: 3.e scrub starts 2024-03-21T13:16:53.697 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[29621]: 3.e scrub ok 2024-03-21T13:16:53.697 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[29621]: 6.16 scrub starts 2024-03-21T13:16:53.697 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[29621]: 6.16 scrub ok 2024-03-21T13:16:53.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[40675]: 5.12 scrub starts 2024-03-21T13:16:53.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[40675]: 5.12 scrub ok 2024-03-21T13:16:53.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:53.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[40675]: 3.e scrub starts 2024-03-21T13:16:53.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[40675]: 3.e scrub ok 2024-03-21T13:16:53.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[40675]: 6.16 scrub starts 2024-03-21T13:16:53.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:53 smithi163 ceph-mon[40675]: 6.16 scrub ok 2024-03-21T13:16:53.717 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:53.754 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:53.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:53.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:53.898 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:53.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:53.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:54.000 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:53 smithi204 ceph-mon[33153]: 5.12 scrub starts 2024-03-21T13:16:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:53 smithi204 ceph-mon[33153]: 5.12 scrub ok 2024-03-21T13:16:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:53 smithi204 ceph-mon[33153]: 3.e scrub starts 2024-03-21T13:16:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:53 smithi204 ceph-mon[33153]: 3.e scrub ok 2024-03-21T13:16:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:53 smithi204 ceph-mon[33153]: 6.16 scrub starts 2024-03-21T13:16:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:53 smithi204 ceph-mon[33153]: 6.16 scrub ok 2024-03-21T13:16:54.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:54.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:54.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:54.137 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:54.181 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:54.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:54.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-21T13:16:54.283 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:54.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:54.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:54.331 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 25 16 52427 52411 131.014 96.25 0.0070821 0.00762784 2024-03-21T13:16:54.418 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:54.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:54.452 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:54.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:54.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:54.608 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:54.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:54.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:54.669 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:16:54.710 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:54.747 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:54.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:54.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[29621]: pgmap v857: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 789 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-21T13:16:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[29621]: 4.8 scrub starts 2024-03-21T13:16:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[29621]: 4.8 scrub ok 2024-03-21T13:16:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[29621]: 5.1 scrub starts 2024-03-21T13:16:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[29621]: 4.12 scrub starts 2024-03-21T13:16:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[29621]: 5.1 scrub ok 2024-03-21T13:16:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[29621]: 4.12 scrub ok 2024-03-21T13:16:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[29621]: 4.1c scrub starts 2024-03-21T13:16:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[29621]: 4.1c scrub ok 2024-03-21T13:16:54.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[40675]: pgmap v857: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 789 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-21T13:16:54.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[40675]: 4.8 scrub starts 2024-03-21T13:16:54.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[40675]: 4.8 scrub ok 2024-03-21T13:16:54.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[40675]: 5.1 scrub starts 2024-03-21T13:16:54.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[40675]: 4.12 scrub starts 2024-03-21T13:16:54.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[40675]: 5.1 scrub ok 2024-03-21T13:16:54.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[40675]: 4.12 scrub ok 2024-03-21T13:16:54.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[40675]: 4.1c scrub starts 2024-03-21T13:16:54.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:54 smithi163 ceph-mon[40675]: 4.1c scrub ok 2024-03-21T13:16:54.809 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:54 smithi204 ceph-mon[33153]: pgmap v857: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 789 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-21T13:16:54.809 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:54 smithi204 ceph-mon[33153]: 4.8 scrub starts 2024-03-21T13:16:54.809 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:54 smithi204 ceph-mon[33153]: 4.8 scrub ok 2024-03-21T13:16:54.809 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:54 smithi204 ceph-mon[33153]: 5.1 scrub starts 2024-03-21T13:16:54.809 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:54 smithi204 ceph-mon[33153]: 4.12 scrub starts 2024-03-21T13:16:54.809 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:54 smithi204 ceph-mon[33153]: 5.1 scrub ok 2024-03-21T13:16:54.809 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:54 smithi204 ceph-mon[33153]: 4.12 scrub ok 2024-03-21T13:16:54.809 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:54 smithi204 ceph-mon[33153]: 4.1c scrub starts 2024-03-21T13:16:54.809 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:54 smithi204 ceph-mon[33153]: 4.1c scrub ok 2024-03-21T13:16:54.850 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:54.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:54.880 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:54.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:54.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:55.018 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:55.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:55.120 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:55.159 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:55.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:55.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:16:55.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:55.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:55.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:55.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:55.331 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 26 16 53903 53887 129.523 92.25 0.0119959 0.00771476 2024-03-21T13:16:55.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:55.452 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:55.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:55.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:55.589 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:55.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:55.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:55.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:55.726 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:55.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:55.827 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:55.851 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:55 smithi204 ceph-mon[33153]: 4.f scrub starts 2024-03-21T13:16:55.851 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:55 smithi204 ceph-mon[33153]: 4.f scrub ok 2024-03-21T13:16:55.851 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:55 smithi204 ceph-mon[33153]: 3.c scrub starts 2024-03-21T13:16:55.851 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:55 smithi204 ceph-mon[33153]: 3.c scrub ok 2024-03-21T13:16:55.851 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:55 smithi204 ceph-mon[33153]: 5.8 scrub starts 2024-03-21T13:16:55.851 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:55 smithi204 ceph-mon[33153]: 5.8 scrub ok 2024-03-21T13:16:55.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:55.866 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:55.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:55.965 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:55 smithi163 ceph-mon[29621]: 4.f scrub starts 2024-03-21T13:16:55.965 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:55 smithi163 ceph-mon[29621]: 4.f scrub ok 2024-03-21T13:16:55.965 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:55 smithi163 ceph-mon[29621]: 3.c scrub starts 2024-03-21T13:16:55.965 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:55 smithi163 ceph-mon[29621]: 3.c scrub ok 2024-03-21T13:16:55.965 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:55 smithi163 ceph-mon[29621]: 5.8 scrub starts 2024-03-21T13:16:55.965 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:55 smithi163 ceph-mon[29621]: 5.8 scrub ok 2024-03-21T13:16:55.965 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:55 smithi163 ceph-mon[40675]: 4.f scrub starts 2024-03-21T13:16:55.966 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:55 smithi163 ceph-mon[40675]: 4.f scrub ok 2024-03-21T13:16:55.966 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:55 smithi163 ceph-mon[40675]: 3.c scrub starts 2024-03-21T13:16:55.966 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:55 smithi163 ceph-mon[40675]: 3.c scrub ok 2024-03-21T13:16:55.966 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:55 smithi163 ceph-mon[40675]: 5.8 scrub starts 2024-03-21T13:16:55.966 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:55 smithi163 ceph-mon[40675]: 5.8 scrub ok 2024-03-21T13:16:55.966 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:16:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:16:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:16:55.967 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:56.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:56.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:56.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:56.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:56.143 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:56.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:56.245 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:56.302 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:56.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:56.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:56.331 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 27 16 55376 55360 128.135 92.0625 0.00705329 0.00779847 2024-03-21T13:16:56.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:56.444 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:56.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:56.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:56.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:56.580 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:56.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:56.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:56.682 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:56.725 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:56.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:56.826 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:56.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:56 smithi204 ceph-mon[33153]: 3.11 scrub starts 2024-03-21T13:16:56.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:56 smithi204 ceph-mon[33153]: 3.11 scrub ok 2024-03-21T13:16:56.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:56 smithi204 ceph-mon[33153]: pgmap v858: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-21T13:16:56.857 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:56 smithi204 ceph-mon[33153]: 5.a scrub starts 2024-03-21T13:16:56.857 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:56 smithi204 ceph-mon[33153]: 5.a scrub ok 2024-03-21T13:16:56.869 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:56.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:56.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:56 smithi163 ceph-mon[29621]: 3.11 scrub starts 2024-03-21T13:16:56.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:56 smithi163 ceph-mon[29621]: 3.11 scrub ok 2024-03-21T13:16:56.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:56 smithi163 ceph-mon[29621]: pgmap v858: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-21T13:16:56.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:56 smithi163 ceph-mon[29621]: 5.a scrub starts 2024-03-21T13:16:56.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:56 smithi163 ceph-mon[29621]: 5.a scrub ok 2024-03-21T13:16:56.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:56 smithi163 ceph-mon[40675]: 3.11 scrub starts 2024-03-21T13:16:56.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:56 smithi163 ceph-mon[40675]: 3.11 scrub ok 2024-03-21T13:16:56.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:56 smithi163 ceph-mon[40675]: pgmap v858: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-21T13:16:56.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:56 smithi163 ceph-mon[40675]: 5.a scrub starts 2024-03-21T13:16:56.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:56 smithi163 ceph-mon[40675]: 5.a scrub ok 2024-03-21T13:16:56.971 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:57.027 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:57.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:57.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:16:57.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:16:57.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:57.172 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:57.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:16:57.273 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:57.287 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:16:57.302 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:16:57.308 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:57.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:57.331 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 28 16 56742 56726 126.608 85.375 0.0131672 0.00789301 2024-03-21T13:16:57.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:57.447 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:57.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:57.549 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:57.564 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:16:57.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:57.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:57.640 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:57 smithi204 ceph-mon[33153]: 4.1d scrub starts 2024-03-21T13:16:57.640 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:57 smithi204 ceph-mon[33153]: 4.1d scrub ok 2024-03-21T13:16:57.640 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:57 smithi204 ceph-mon[33153]: 6.19 scrub starts 2024-03-21T13:16:57.640 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:57 smithi204 ceph-mon[33153]: 6.19 scrub ok 2024-03-21T13:16:57.640 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:57 smithi204 ceph-mon[33153]: 3.f scrub starts 2024-03-21T13:16:57.640 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:57 smithi204 ceph-mon[33153]: 3.f scrub ok 2024-03-21T13:16:57.640 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:57 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:16:57.640 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:57 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:16:57.640 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:57 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3140577256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:57.640 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:57 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:57.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:16:57.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:57.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:57.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:16:57.799 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[29621]: 4.1d scrub starts 2024-03-21T13:16:57.799 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[29621]: 4.1d scrub ok 2024-03-21T13:16:57.799 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[29621]: 6.19 scrub starts 2024-03-21T13:16:57.799 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[29621]: 6.19 scrub ok 2024-03-21T13:16:57.799 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[29621]: 3.f scrub starts 2024-03-21T13:16:57.799 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[29621]: 3.f scrub ok 2024-03-21T13:16:57.799 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:16:57.799 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:16:57.799 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3140577256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:57.799 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:57.800 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[40675]: 4.1d scrub starts 2024-03-21T13:16:57.800 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[40675]: 4.1d scrub ok 2024-03-21T13:16:57.800 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[40675]: 6.19 scrub starts 2024-03-21T13:16:57.800 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[40675]: 6.19 scrub ok 2024-03-21T13:16:57.800 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[40675]: 3.f scrub starts 2024-03-21T13:16:57.800 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[40675]: 3.f scrub ok 2024-03-21T13:16:57.800 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:16:57.800 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:16:57.800 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3140577256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:57.801 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:57 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:16:57.838 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:57.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:57.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:57.979 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:58.003 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:58.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:58.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:58.132 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-21T13:16:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-03-21T13:16:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15858.0:57834 11.4 11:2506e4aa:::benchmark_data_smithi163_180069_object57833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-21T13:16:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:16:58.126837+0000", 2024-03-21T13:16:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.002027928, 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020685349999999998, 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15858", 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 57834 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.126837+0000", 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.126837+0000", 2024-03-21T13:16:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.126838+0000", 2024-03-21T13:16:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.426e-06 2024-03-21T13:16:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.126862+0000", 2024-03-21T13:16:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3465e-05 2024-03-21T13:16:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.126883+0000", 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1290000000000001e-05 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.126891+0000", 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0409999999999998e-06 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.126989+0000", 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7842000000000003e-05 2024-03-21T13:16:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127115+0000", 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000126606 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,6", 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127243+0000", 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000127532 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.15858.0:57833 11.c e407/390 11:3f16b7a5:::benchmark_data_smithi163_180069_object57832:head v 407'3581, mlcod=407'3579)", 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:16:58.126959+0000", 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0019056870000000001, 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020306059999999999, 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.126959+0000", 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.126959+0000", 2024-03-21T13:16:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.126960+0000", 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6199999999999996e-07 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127004+0000", 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4060999999999997e-05 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127026+0000", 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1716999999999999e-05 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127033+0000", 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7300000000000005e-06 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127118+0000", 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4358999999999996e-05 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127139+0000", 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1123000000000001e-05 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.15858.0:57831 11.e e407/390 11:78f921bd:::benchmark_data_smithi163_180069_object57830:head v 407'3658, mlcod=407'3656)", 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-03-21T13:16:58.127416+0000", 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0014488249999999999, 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016328270000000001, 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127416+0000", 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127416+0000", 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127416+0000", 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8400000000000004e-07 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-03-21T13:16:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127440+0000", 2024-03-21T13:16:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3226000000000001e-05 2024-03-21T13:16:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-03-21T13:16:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127449+0000", 2024-03-21T13:16:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5359999999999998e-06 2024-03-21T13:16:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-03-21T13:16:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127456+0000", 2024-03-21T13:16:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9360000000000002e-06 2024-03-21T13:16:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127529+0000", 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3042999999999998e-05 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-03-21T13:16:58.127547+0000", 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7326e-05 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-03-21T13:16:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 3 2024-03-21T13:16:58.144 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-21T13:16:58.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:58.145 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:58.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:58.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:58.270 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:58.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:58.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:58.332 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 29 16 58068 58052 125.099 82.875 0.0126342 0.00798888 2024-03-21T13:16:58.372 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:58.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:58.413 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:58.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:58.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:58.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:58.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:58.655 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:58.696 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:58.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[29621]: pgmap v859: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-21T13:16:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[29621]: from='client.25799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:16:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:16:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:16:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:16:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[29621]: osdmap e407: 8 total, 8 up, 7 in 2024-03-21T13:16:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[29621]: 4.16 scrub starts 2024-03-21T13:16:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[29621]: 4.16 scrub ok 2024-03-21T13:16:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[40675]: pgmap v859: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-21T13:16:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[40675]: from='client.25799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:16:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:16:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:16:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:16:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[40675]: osdmap e407: 8 total, 8 up, 7 in 2024-03-21T13:16:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[40675]: 4.16 scrub starts 2024-03-21T13:16:58.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:58 smithi163 ceph-mon[40675]: 4.16 scrub ok 2024-03-21T13:16:58.798 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:58.834 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:58.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:58.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:16:58.936 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:16:58.973 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:58.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:58.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:16:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:58 smithi204 ceph-mon[33153]: pgmap v859: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-21T13:16:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:58 smithi204 ceph-mon[33153]: from='client.25799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:16:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:58 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:16:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:58 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:16:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:58 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:16:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:58 smithi204 ceph-mon[33153]: osdmap e407: 8 total, 8 up, 7 in 2024-03-21T13:16:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:58 smithi204 ceph-mon[33153]: 4.16 scrub starts 2024-03-21T13:16:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:58 smithi204 ceph-mon[33153]: 4.16 scrub ok 2024-03-21T13:16:59.074 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:59.109 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:59.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:59.185 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":859,"stamp":"2024-03-21T13:16:56.963715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3560184232,"num_objects":54513,"num_object_clones":0,"num_object_copies":163539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54513,"num_whiteouts":0,"num_read":2423,"num_read_kb":2269,"num_write":55216,"num_write_kb":3478621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":1779777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55300,"ondisk_log_size":55300,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15939756,"kb_used_data":10257140,"kb_used_omap":10,"kb_used_meta":5682549,"kb_avail":640276308,"statfs":{"total":671965249536,"available":655642939392,"internally_reserved":0,"allocated":10503311360,"data_stored":10499835423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":5818930283},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,5,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1050017792,"num_objects":16022,"num_object_clones":0,"num_object_copies":48066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16022,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16022,"num_write_kb":1025408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.462901"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":169,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.496027+0000","last_change":"2024-03-21T13:14:59.311326+0000","last_active":"2024-03-21T13:16:51.496027+0000","last_peered":"2024-03-21T13:16:51.496027+0000","last_clean":"2024-03-21T13:16:51.496027+0000","last_became_active":"2024-03-21T13:12:29.525755+0000","last_became_peered":"2024-03-21T13:12:29.525755+0000","last_unstale":"2024-03-21T13:16:51.496027+0000","last_undegraded":"2024-03-21T13:16:51.496027+0000","last_fullsized":"2024-03-21T13:16:51.496027+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:59.311141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:14:59.311141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":177,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.494996+0000","last_change":"2024-03-21T13:16:21.204738+0000","last_active":"2024-03-21T13:16:51.494996+0000","last_peered":"2024-03-21T13:16:51.494996+0000","last_clean":"2024-03-21T13:16:51.494996+0000","last_became_active":"2024-03-21T13:12:29.510271+0000","last_became_peered":"2024-03-21T13:12:29.510271+0000","last_unstale":"2024-03-21T13:16:51.494996+0000","last_undegraded":"2024-03-21T13:16:51.494996+0000","last_fullsized":"2024-03-21T13:16:51.494996+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:21.204657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.204657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:31.150802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":198,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.489892+0000","last_change":"2024-03-21T13:16:02.044946+0000","last_active":"2024-03-21T13:16:51.489892+0000","last_peered":"2024-03-21T13:16:51.489892+0000","last_clean":"2024-03-21T13:16:51.489892+0000","last_became_active":"2024-03-21T13:15:04.462014+0000","last_became_peered":"2024-03-21T13:15:04.462014+0000","last_unstale":"2024-03-21T13:16:51.489892+0000","last_undegraded":"2024-03-21T13:16:51.489892+0000","last_fullsized":"2024-03-21T13:16:51.489892+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:02.044853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:02.044853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:31.722187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"234'19","reported_seq":822,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.489349+0000","last_change":"2024-03-21T13:16:09.694036+0000","last_active":"2024-03-21T13:16:51.489349+0000","last_peered":"2024-03-21T13:16:51.489349+0000","last_clean":"2024-03-21T13:16:51.489349+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:16:51.489349+0000","last_undegraded":"2024-03-21T13:16:51.489349+0000","last_fullsized":"2024-03-21T13:16:51.489349+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:09.693933+0000","last_deep_scrub":"234'19","last_deep_scrub_stamp":"2024-03-21T13:16:09.693933+0000","last_clean_scrub_stamp":"2024-03-21T13:16:09.693933+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.080379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":721,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483129+0000","last_change":"2024-03-21T13:15:02.457035+0000","last_active":"2024-03-21T13:16:51.483129+0000","last_peered":"2024-03-21T13:16:51.483129+0000","last_clean":"2024-03-21T13:16:51.483129+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:16:51.483129+0000","last_undegraded":"2024-03-21T13:16:51.483129+0000","last_fullsized":"2024-03-21T13:16:51.483129+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:02.456902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:15:02.456902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":671,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483892+0000","last_change":"2024-03-21T13:14:05.582597+0000","last_active":"2024-03-21T13:16:51.483892+0000","last_peered":"2024-03-21T13:16:51.483892+0000","last_clean":"2024-03-21T13:16:51.483892+0000","last_became_active":"2024-03-21T13:10:03.591543+0000","last_became_peered":"2024-03-21T13:10:03.591543+0000","last_unstale":"2024-03-21T13:16:51.483892+0000","last_undegraded":"2024-03-21T13:16:51.483892+0000","last_fullsized":"2024-03-21T13:16:51.483892+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:05.582473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:14:05.582473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":754,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.481921+0000","last_change":"2024-03-21T13:14:11.126777+0000","last_active":"2024-03-21T13:16:51.481921+0000","last_peered":"2024-03-21T13:16:51.481921+0000","last_clean":"2024-03-21T13:16:51.481921+0000","last_became_active":"2024-03-21T13:10:03.610318+0000","last_became_peered":"2024-03-21T13:10:03.610318+0000","last_unstale":"2024-03-21T13:16:51.481921+0000","last_undegraded":"2024-03-21T13:16:51.481921+0000","last_fullsized":"2024-03-21T13:16:51.481921+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_clean_scrub_stamp":"2024-03-21T13:14:11.126531+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":252,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.492331+0000","last_change":"2024-03-21T13:14:54.623357+0000","last_active":"2024-03-21T13:16:51.492331+0000","last_peered":"2024-03-21T13:16:51.492331+0000","last_clean":"2024-03-21T13:16:51.492331+0000","last_became_active":"2024-03-21T13:11:21.702209+0000","last_became_peered":"2024-03-21T13:11:21.702209+0000","last_unstale":"2024-03-21T13:16:51.492331+0000","last_undegraded":"2024-03-21T13:16:51.492331+0000","last_fullsized":"2024-03-21T13:16:51.492331+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:14:54.619347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:54.619347+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":427,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.492943+0000","last_change":"2024-03-21T13:16:20.952006+0000","last_active":"2024-03-21T13:16:51.492943+0000","last_peered":"2024-03-21T13:16:51.492943+0000","last_clean":"2024-03-21T13:16:51.492943+0000","last_became_active":"2024-03-21T13:15:04.459860+0000","last_became_peered":"2024-03-21T13:15:04.459860+0000","last_unstale":"2024-03-21T13:16:51.492943+0000","last_undegraded":"2024-03-21T13:16:51.492943+0000","last_fullsized":"2024-03-21T13:16:51.492943+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:20.951893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.951893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:35.358793+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":169,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.495299+0000","last_change":"2024-03-21T13:15:12.302278+0000","last_active":"2024-03-21T13:16:51.495299+0000","last_peered":"2024-03-21T13:16:51.495299+0000","last_clean":"2024-03-21T13:16:51.495299+0000","last_became_active":"2024-03-21T13:12:29.518784+0000","last_became_peered":"2024-03-21T13:12:29.518784+0000","last_unstale":"2024-03-21T13:16:51.495299+0000","last_undegraded":"2024-03-21T13:16:51.495299+0000","last_fullsized":"2024-03-21T13:16:51.495299+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:12.302013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:15:12.302013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":169,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.498101+0000","last_change":"2024-03-21T13:15:18.085333+0000","last_active":"2024-03-21T13:16:51.498101+0000","last_peered":"2024-03-21T13:16:51.498101+0000","last_clean":"2024-03-21T13:16:51.498101+0000","last_became_active":"2024-03-21T13:12:29.526129+0000","last_became_peered":"2024-03-21T13:12:29.526129+0000","last_unstale":"2024-03-21T13:16:51.498101+0000","last_undegraded":"2024-03-21T13:16:51.498101+0000","last_fullsized":"2024-03-21T13:16:51.498101+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:18.085215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:15:18.085215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":732,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.484094+0000","last_change":"2024-03-21T13:16:07.739334+0000","last_active":"2024-03-21T13:16:51.484094+0000","last_peered":"2024-03-21T13:16:51.484094+0000","last_clean":"2024-03-21T13:16:51.484094+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:16:51.484094+0000","last_undegraded":"2024-03-21T13:16:51.484094+0000","last_fullsized":"2024-03-21T13:16:51.484094+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:07.739235+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.739235+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.631193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":169,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.495082+0000","last_change":"2024-03-21T13:14:05.545713+0000","last_active":"2024-03-21T13:16:51.495082+0000","last_peered":"2024-03-21T13:16:51.495082+0000","last_clean":"2024-03-21T13:16:51.495082+0000","last_became_active":"2024-03-21T13:12:29.508520+0000","last_became_peered":"2024-03-21T13:12:29.508520+0000","last_unstale":"2024-03-21T13:16:51.495082+0000","last_undegraded":"2024-03-21T13:16:51.495082+0000","last_fullsized":"2024-03-21T13:16:51.495082+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:05.545425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:05.545425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":744,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.492571+0000","last_change":"2024-03-21T13:15:58.658853+0000","last_active":"2024-03-21T13:16:51.492571+0000","last_peered":"2024-03-21T13:16:51.492571+0000","last_clean":"2024-03-21T13:16:51.492571+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:16:51.492571+0000","last_undegraded":"2024-03-21T13:16:51.492571+0000","last_fullsized":"2024-03-21T13:16:51.492571+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:58.658749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:58.658749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:05.447652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":169,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.495279+0000","last_change":"2024-03-21T13:16:04.184426+0000","last_active":"2024-03-21T13:16:51.495279+0000","last_peered":"2024-03-21T13:16:51.495279+0000","last_clean":"2024-03-21T13:16:51.495279+0000","last_became_active":"2024-03-21T13:12:29.524473+0000","last_became_peered":"2024-03-21T13:12:29.524473+0000","last_unstale":"2024-03-21T13:16:51.495279+0000","last_undegraded":"2024-03-21T13:16:51.495279+0000","last_fullsized":"2024-03-21T13:16:51.495279+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:04.184314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:04.184314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:12.162233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"234'9","reported_seq":752,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.482820+0000","last_change":"2024-03-21T13:14:52.156420+0000","last_active":"2024-03-21T13:16:51.482820+0000","last_peered":"2024-03-21T13:16:51.482820+0000","last_clean":"2024-03-21T13:16:51.482820+0000","last_became_active":"2024-03-21T13:10:03.595745+0000","last_became_peered":"2024-03-21T13:10:03.595745+0000","last_unstale":"2024-03-21T13:16:51.482820+0000","last_undegraded":"2024-03-21T13:16:51.482820+0000","last_fullsized":"2024-03-21T13:16:51.482820+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'9","last_scrub_stamp":"2024-03-21T13:14:52.156318+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:14:52.156318+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":775,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.493498+0000","last_change":"2024-03-21T13:16:06.811101+0000","last_active":"2024-03-21T13:16:51.493498+0000","last_peered":"2024-03-21T13:16:51.493498+0000","last_clean":"2024-03-21T13:16:51.493498+0000","last_became_active":"2024-03-21T13:12:29.520189+0000","last_became_peered":"2024-03-21T13:12:29.520189+0000","last_unstale":"2024-03-21T13:16:51.493498+0000","last_undegraded":"2024-03-21T13:16:51.493498+0000","last_fullsized":"2024-03-21T13:16:51.493498+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:06.810993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:23.865785+0000","last_clean_scrub_stamp":"2024-03-21T13:16:06.810993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:18.896276+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":661,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.494042+0000","last_change":"2024-03-21T13:16:07.756974+0000","last_active":"2024-03-21T13:16:51.494042+0000","last_peered":"2024-03-21T13:16:51.494042+0000","last_clean":"2024-03-21T13:16:51.494042+0000","last_became_active":"2024-03-21T13:12:29.520241+0000","last_became_peered":"2024-03-21T13:12:29.520241+0000","last_unstale":"2024-03-21T13:16:51.494042+0000","last_undegraded":"2024-03-21T13:16:51.494042+0000","last_fullsized":"2024-03-21T13:16:51.494042+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:07.756896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.756896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.528460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":198,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.493549+0000","last_change":"2024-03-21T13:15:16.781907+0000","last_active":"2024-03-21T13:16:51.493549+0000","last_peered":"2024-03-21T13:16:51.493549+0000","last_clean":"2024-03-21T13:16:51.493549+0000","last_became_active":"2024-03-21T13:15:04.464586+0000","last_became_peered":"2024-03-21T13:15:04.464586+0000","last_unstale":"2024-03-21T13:16:51.493549+0000","last_undegraded":"2024-03-21T13:16:51.493549+0000","last_fullsized":"2024-03-21T13:16:51.493549+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:16.781632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:15:16.781632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":321,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:54.409965+0000","last_change":"2024-03-21T13:16:54.410019+0000","last_active":"2024-03-21T13:16:54.409965+0000","last_peered":"2024-03-21T13:16:54.409965+0000","last_clean":"2024-03-21T13:16:54.409965+0000","last_became_active":"2024-03-21T13:10:03.604616+0000","last_became_peered":"2024-03-21T13:10:03.604616+0000","last_unstale":"2024-03-21T13:16:54.409965+0000","last_undegraded":"2024-03-21T13:16:54.409965+0000","last_fullsized":"2024-03-21T13:16:54.409965+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:16:54.409873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.409873+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:16.016094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":746,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.481783+0000","last_change":"2024-03-21T13:15:59.096623+0000","last_active":"2024-03-21T13:16:51.481783+0000","last_peered":"2024-03-21T13:16:51.481783+0000","last_clean":"2024-03-21T13:16:51.481783+0000","last_became_active":"2024-03-21T13:12:29.502808+0000","last_became_peered":"2024-03-21T13:12:29.502808+0000","last_unstale":"2024-03-21T13:16:51.481783+0000","last_undegraded":"2024-03-21T13:16:51.481783+0000","last_fullsized":"2024-03-21T13:16:51.481783+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:59.096520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:59.096520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.250778+0000","scrub_duration":233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":658,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.480672+0000","last_change":"2024-03-21T13:14:53.198725+0000","last_active":"2024-03-21T13:16:51.480672+0000","last_peered":"2024-03-21T13:16:51.480672+0000","last_clean":"2024-03-21T13:16:51.480672+0000","last_became_active":"2024-03-21T13:10:03.612409+0000","last_became_peered":"2024-03-21T13:10:03.612409+0000","last_unstale":"2024-03-21T13:16:51.480672+0000","last_undegraded":"2024-03-21T13:16:51.480672+0000","last_fullsized":"2024-03-21T13:16:51.480672+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:53.198601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:14:53.198601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":791,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.489684+0000","last_change":"2024-03-21T13:16:48.967395+0000","last_active":"2024-03-21T13:16:51.489684+0000","last_peered":"2024-03-21T13:16:51.489684+0000","last_clean":"2024-03-21T13:16:51.489684+0000","last_became_active":"2024-03-21T13:10:03.586563+0000","last_became_peered":"2024-03-21T13:10:03.586563+0000","last_unstale":"2024-03-21T13:16:51.489684+0000","last_undegraded":"2024-03-21T13:16:51.489684+0000","last_fullsized":"2024-03-21T13:16:51.489684+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:48.967258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:20.956778+0000","last_clean_scrub_stamp":"2024-03-21T13:16:48.967258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:52.310270+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":797,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:55.684987+0000","last_change":"2024-03-21T13:16:55.685055+0000","last_active":"2024-03-21T13:16:55.684987+0000","last_peered":"2024-03-21T13:16:55.684987+0000","last_clean":"2024-03-21T13:16:55.684987+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:16:55.684987+0000","last_undegraded":"2024-03-21T13:16:55.684987+0000","last_fullsized":"2024-03-21T13:16:55.684987+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-21T13:16:55.684896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:55.684896+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:05.652758+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":149,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.494048+0000","last_change":"2024-03-21T13:15:12.611589+0000","last_active":"2024-03-21T13:16:51.494048+0000","last_peered":"2024-03-21T13:16:51.494048+0000","last_clean":"2024-03-21T13:16:51.494048+0000","last_became_active":"2024-03-21T13:13:06.420220+0000","last_became_peered":"2024-03-21T13:13:06.420220+0000","last_unstale":"2024-03-21T13:16:51.494048+0000","last_undegraded":"2024-03-21T13:16:51.494048+0000","last_fullsized":"2024-03-21T13:16:51.494048+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:15:12.611167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:12.611167+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":964,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483696+0000","last_change":"2024-03-21T13:16:12.489669+0000","last_active":"2024-03-21T13:16:51.483696+0000","last_peered":"2024-03-21T13:16:51.483696+0000","last_clean":"2024-03-21T13:16:51.483696+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:16:51.483696+0000","last_undegraded":"2024-03-21T13:16:51.483696+0000","last_fullsized":"2024-03-21T13:16:51.483696+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:16:12.489518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.489518+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:14.011096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"64'1","reported_seq":632,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.481253+0000","last_change":"2024-03-21T13:14:02.109905+0000","last_active":"2024-03-21T13:16:51.481253+0000","last_peered":"2024-03-21T13:16:51.481253+0000","last_clean":"2024-03-21T13:16:51.481253+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:16:51.481253+0000","last_undegraded":"2024-03-21T13:16:51.481253+0000","last_fullsized":"2024-03-21T13:16:51.481253+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:14:02.109739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:02.109739+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":399,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.492643+0000","last_change":"2024-03-21T13:15:04.834478+0000","last_active":"2024-03-21T13:16:51.492643+0000","last_peered":"2024-03-21T13:16:51.492643+0000","last_clean":"2024-03-21T13:16:51.492643+0000","last_became_active":"2024-03-21T13:15:04.458831+0000","last_became_peered":"2024-03-21T13:15:04.458831+0000","last_unstale":"2024-03-21T13:16:51.492643+0000","last_undegraded":"2024-03-21T13:16:51.492643+0000","last_fullsized":"2024-03-21T13:16:51.492643+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:15:04.834347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:04.834347+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":169,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.498023+0000","last_change":"2024-03-21T13:16:07.247321+0000","last_active":"2024-03-21T13:16:51.498023+0000","last_peered":"2024-03-21T13:16:51.498023+0000","last_clean":"2024-03-21T13:16:51.498023+0000","last_became_active":"2024-03-21T13:12:29.527129+0000","last_became_peered":"2024-03-21T13:12:29.527129+0000","last_unstale":"2024-03-21T13:16:51.498023+0000","last_undegraded":"2024-03-21T13:16:51.498023+0000","last_fullsized":"2024-03-21T13:16:51.498023+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:07.247194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.247194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:08.116592+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":424,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.493128+0000","last_change":"2024-03-21T13:15:04.464609+0000","last_active":"2024-03-21T13:16:51.493128+0000","last_peered":"2024-03-21T13:16:51.493128+0000","last_clean":"2024-03-21T13:16:51.493128+0000","last_became_active":"2024-03-21T13:15:04.462979+0000","last_became_peered":"2024-03-21T13:15:04.462979+0000","last_unstale":"2024-03-21T13:16:51.493128+0000","last_undegraded":"2024-03-21T13:16:51.493128+0000","last_fullsized":"2024-03-21T13:16:51.493128+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:00.690145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:00.690145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":751,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.488083+0000","last_change":"2024-03-21T13:14:20.212339+0000","last_active":"2024-03-21T13:16:51.488083+0000","last_peered":"2024-03-21T13:16:51.488083+0000","last_clean":"2024-03-21T13:16:51.488083+0000","last_became_active":"2024-03-21T13:10:03.610489+0000","last_became_peered":"2024-03-21T13:10:03.610489+0000","last_unstale":"2024-03-21T13:16:51.488083+0000","last_undegraded":"2024-03-21T13:16:51.488083+0000","last_fullsized":"2024-03-21T13:16:51.488083+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:20.212222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:20.212222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"331'24","reported_seq":827,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483749+0000","last_change":"2024-03-21T13:15:09.670586+0000","last_active":"2024-03-21T13:16:51.483749+0000","last_peered":"2024-03-21T13:16:51.483749+0000","last_clean":"2024-03-21T13:16:51.483749+0000","last_became_active":"2024-03-21T13:12:29.522466+0000","last_became_peered":"2024-03-21T13:12:29.522466+0000","last_unstale":"2024-03-21T13:16:51.483749+0000","last_undegraded":"2024-03-21T13:16:51.483749+0000","last_fullsized":"2024-03-21T13:16:51.483749+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'24","last_scrub_stamp":"2024-03-21T13:15:09.670459+0000","last_deep_scrub":"331'24","last_deep_scrub_stamp":"2024-03-21T13:15:09.670459+0000","last_clean_scrub_stamp":"2024-03-21T13:15:09.670459+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":520,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.490023+0000","last_change":"2024-03-21T13:15:04.458598+0000","last_active":"2024-03-21T13:16:51.490023+0000","last_peered":"2024-03-21T13:16:51.490023+0000","last_clean":"2024-03-21T13:16:51.490023+0000","last_became_active":"2024-03-21T13:15:04.458144+0000","last_became_peered":"2024-03-21T13:15:04.458144+0000","last_unstale":"2024-03-21T13:16:51.490023+0000","last_undegraded":"2024-03-21T13:16:51.490023+0000","last_fullsized":"2024-03-21T13:16:51.490023+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:20.369324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:14:20.369324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":740,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483579+0000","last_change":"2024-03-21T13:16:21.557896+0000","last_active":"2024-03-21T13:16:51.483579+0000","last_peered":"2024-03-21T13:16:51.483579+0000","last_clean":"2024-03-21T13:16:51.483579+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:16:51.483579+0000","last_undegraded":"2024-03-21T13:16:51.483579+0000","last_fullsized":"2024-03-21T13:16:51.483579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:21.557805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.557805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:35.861086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":190,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.490229+0000","last_change":"2024-03-21T13:16:00.029375+0000","last_active":"2024-03-21T13:16:51.490229+0000","last_peered":"2024-03-21T13:16:51.490229+0000","last_clean":"2024-03-21T13:16:51.490229+0000","last_became_active":"2024-03-21T13:15:04.462103+0000","last_became_peered":"2024-03-21T13:15:04.462103+0000","last_unstale":"2024-03-21T13:16:51.490229+0000","last_undegraded":"2024-03-21T13:16:51.490229+0000","last_fullsized":"2024-03-21T13:16:51.490229+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:00.029292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:22.955737+0000","last_clean_scrub_stamp":"2024-03-21T13:16:00.029292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:02.701099+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":753,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.489275+0000","last_change":"2024-03-21T13:16:20.734468+0000","last_active":"2024-03-21T13:16:51.489275+0000","last_peered":"2024-03-21T13:16:51.489275+0000","last_clean":"2024-03-21T13:16:51.489275+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:16:51.489275+0000","last_undegraded":"2024-03-21T13:16:51.489275+0000","last_fullsized":"2024-03-21T13:16:51.489275+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-21T13:16:20.734379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.734379+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:47.670110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":721,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483221+0000","last_change":"2024-03-21T13:14:59.413401+0000","last_active":"2024-03-21T13:16:51.483221+0000","last_peered":"2024-03-21T13:16:51.483221+0000","last_clean":"2024-03-21T13:16:51.483221+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:16:51.483221+0000","last_undegraded":"2024-03-21T13:16:51.483221+0000","last_fullsized":"2024-03-21T13:16:51.483221+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:59.413139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:59.413139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":85,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.493273+0000","last_change":"2024-03-21T13:16:05.059523+0000","last_active":"2024-03-21T13:16:51.493273+0000","last_peered":"2024-03-21T13:16:51.493273+0000","last_clean":"2024-03-21T13:16:51.493273+0000","last_became_active":"2024-03-21T13:15:04.512071+0000","last_became_peered":"2024-03-21T13:15:04.512071+0000","last_unstale":"2024-03-21T13:16:51.493273+0000","last_undegraded":"2024-03-21T13:16:51.493273+0000","last_fullsized":"2024-03-21T13:16:51.493273+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:05.059443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.059443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.468710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":177,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.496999+0000","last_change":"2024-03-21T13:16:23.266104+0000","last_active":"2024-03-21T13:16:51.496999+0000","last_peered":"2024-03-21T13:16:51.496999+0000","last_clean":"2024-03-21T13:16:51.496999+0000","last_became_active":"2024-03-21T13:12:29.507706+0000","last_became_peered":"2024-03-21T13:12:29.507706+0000","last_unstale":"2024-03-21T13:16:51.496999+0000","last_undegraded":"2024-03-21T13:16:51.496999+0000","last_fullsized":"2024-03-21T13:16:51.496999+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:23.266023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:23.266023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:34.197737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":475,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.481505+0000","last_change":"2024-03-21T13:15:55.923610+0000","last_active":"2024-03-21T13:16:51.481505+0000","last_peered":"2024-03-21T13:16:51.481505+0000","last_clean":"2024-03-21T13:16:51.481505+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:16:51.481505+0000","last_undegraded":"2024-03-21T13:16:51.481505+0000","last_fullsized":"2024-03-21T13:16:51.481505+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:15:55.923514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:55.923514+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:12.089112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":178,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.489770+0000","last_change":"2024-03-21T13:16:50.026008+0000","last_active":"2024-03-21T13:16:51.489770+0000","last_peered":"2024-03-21T13:16:51.489770+0000","last_clean":"2024-03-21T13:16:51.489770+0000","last_became_active":"2024-03-21T13:15:04.512432+0000","last_became_peered":"2024-03-21T13:15:04.512432+0000","last_unstale":"2024-03-21T13:16:51.489770+0000","last_undegraded":"2024-03-21T13:16:51.489770+0000","last_fullsized":"2024-03-21T13:16:51.489770+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:50.025891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:50.025891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:04.135064+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":743,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.482988+0000","last_change":"2024-03-21T13:15:18.312265+0000","last_active":"2024-03-21T13:16:51.482988+0000","last_peered":"2024-03-21T13:16:51.482988+0000","last_clean":"2024-03-21T13:16:51.482988+0000","last_became_active":"2024-03-21T13:12:29.502203+0000","last_became_peered":"2024-03-21T13:12:29.502203+0000","last_unstale":"2024-03-21T13:16:51.482988+0000","last_undegraded":"2024-03-21T13:16:51.482988+0000","last_fullsized":"2024-03-21T13:16:51.482988+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:18.312139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:18.312139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":738,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.493080+0000","last_change":"2024-03-21T13:16:08.584734+0000","last_active":"2024-03-21T13:16:51.493080+0000","last_peered":"2024-03-21T13:16:51.493080+0000","last_clean":"2024-03-21T13:16:51.493080+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:16:51.493080+0000","last_undegraded":"2024-03-21T13:16:51.493080+0000","last_fullsized":"2024-03-21T13:16:51.493080+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:08.584643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:19.383090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:08.584643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:19.593361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":748,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:53.867830+0000","last_change":"2024-03-21T13:16:53.867897+0000","last_active":"2024-03-21T13:16:53.867830+0000","last_peered":"2024-03-21T13:16:53.867830+0000","last_clean":"2024-03-21T13:16:53.867830+0000","last_became_active":"2024-03-21T13:10:03.591857+0000","last_became_peered":"2024-03-21T13:10:03.591857+0000","last_unstale":"2024-03-21T13:16:53.867830+0000","last_undegraded":"2024-03-21T13:16:53.867830+0000","last_fullsized":"2024-03-21T13:16:53.867830+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:53.867747+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.867747+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:12.880058+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":565,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.491332+0000","last_change":"2024-03-21T13:15:05.701091+0000","last_active":"2024-03-21T13:16:51.491332+0000","last_peered":"2024-03-21T13:16:51.491332+0000","last_clean":"2024-03-21T13:16:51.491332+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:16:51.491332+0000","last_undegraded":"2024-03-21T13:16:51.491332+0000","last_fullsized":"2024-03-21T13:16:51.491332+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:05.700057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:05.700057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":177,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:52.230066+0000","last_change":"2024-03-21T13:16:52.230103+0000","last_active":"2024-03-21T13:16:52.230066+0000","last_peered":"2024-03-21T13:16:52.230066+0000","last_clean":"2024-03-21T13:16:52.230066+0000","last_became_active":"2024-03-21T13:12:29.510100+0000","last_became_peered":"2024-03-21T13:12:29.510100+0000","last_unstale":"2024-03-21T13:16:52.230066+0000","last_undegraded":"2024-03-21T13:16:52.230066+0000","last_fullsized":"2024-03-21T13:16:52.230066+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.229670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:22.501055+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.229670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:08.462478+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":451,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.494216+0000","last_change":"2024-03-21T13:15:59.651653+0000","last_active":"2024-03-21T13:16:51.494216+0000","last_peered":"2024-03-21T13:16:51.494216+0000","last_clean":"2024-03-21T13:16:51.494216+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:16:51.494216+0000","last_undegraded":"2024-03-21T13:16:51.494216+0000","last_fullsized":"2024-03-21T13:16:51.494216+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:59.651571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:15:59.651571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:26.236664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":84,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.490221+0000","last_change":"2024-03-21T13:15:10.666307+0000","last_active":"2024-03-21T13:16:51.490221+0000","last_peered":"2024-03-21T13:16:51.490221+0000","last_clean":"2024-03-21T13:16:51.490221+0000","last_became_active":"2024-03-21T13:15:04.463107+0000","last_became_peered":"2024-03-21T13:15:04.463107+0000","last_unstale":"2024-03-21T13:16:51.490221+0000","last_undegraded":"2024-03-21T13:16:51.490221+0000","last_fullsized":"2024-03-21T13:16:51.490221+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:15:10.666149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:10.666149+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":178,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:53.027164+0000","last_change":"2024-03-21T13:16:53.027206+0000","last_active":"2024-03-21T13:16:53.027164+0000","last_peered":"2024-03-21T13:16:53.027164+0000","last_clean":"2024-03-21T13:16:53.027164+0000","last_became_active":"2024-03-21T13:15:04.458489+0000","last_became_peered":"2024-03-21T13:15:04.458489+0000","last_unstale":"2024-03-21T13:16:53.027164+0000","last_undegraded":"2024-03-21T13:16:53.027164+0000","last_fullsized":"2024-03-21T13:16:53.027164+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:53.026764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.026764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:53.710604+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"394'12","reported_seq":236,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.498221+0000","last_change":"2024-03-21T13:16:24.237890+0000","last_active":"2024-03-21T13:16:51.498221+0000","last_peered":"2024-03-21T13:16:51.498221+0000","last_clean":"2024-03-21T13:16:51.498221+0000","last_became_active":"2024-03-21T13:12:29.521385+0000","last_became_peered":"2024-03-21T13:12:29.521385+0000","last_unstale":"2024-03-21T13:16:51.498221+0000","last_undegraded":"2024-03-21T13:16:51.498221+0000","last_fullsized":"2024-03-21T13:16:51.498221+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'12","last_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_deep_scrub":"394'12","last_deep_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_clean_scrub_stamp":"2024-03-21T13:16:24.237800+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:31.363500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":750,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.491745+0000","last_change":"2024-03-21T13:15:10.647075+0000","last_active":"2024-03-21T13:16:51.491745+0000","last_peered":"2024-03-21T13:16:51.491745+0000","last_clean":"2024-03-21T13:16:51.491745+0000","last_became_active":"2024-03-21T13:10:03.599265+0000","last_became_peered":"2024-03-21T13:10:03.599265+0000","last_unstale":"2024-03-21T13:16:51.491745+0000","last_undegraded":"2024-03-21T13:16:51.491745+0000","last_fullsized":"2024-03-21T13:16:51.491745+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:10.646946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:15:10.646946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'20","reported_seq":813,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.480958+0000","last_change":"2024-03-21T13:15:16.283969+0000","last_active":"2024-03-21T13:16:51.480958+0000","last_peered":"2024-03-21T13:16:51.480958+0000","last_clean":"2024-03-21T13:16:51.480958+0000","last_became_active":"2024-03-21T13:10:03.613119+0000","last_became_peered":"2024-03-21T13:10:03.613119+0000","last_unstale":"2024-03-21T13:16:51.480958+0000","last_undegraded":"2024-03-21T13:16:51.480958+0000","last_fullsized":"2024-03-21T13:16:51.480958+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:15:16.283831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:16.283831+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":311,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.484896+0000","last_change":"2024-03-21T13:16:17.497436+0000","last_active":"2024-03-21T13:16:51.484896+0000","last_peered":"2024-03-21T13:16:51.484896+0000","last_clean":"2024-03-21T13:16:51.484896+0000","last_became_active":"2024-03-21T13:12:29.507751+0000","last_became_peered":"2024-03-21T13:12:29.507751+0000","last_unstale":"2024-03-21T13:16:51.484896+0000","last_undegraded":"2024-03-21T13:16:51.484896+0000","last_fullsized":"2024-03-21T13:16:51.484896+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:17.497325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:17.497325+0000","last_clean_scrub_stamp":"2024-03-21T13:16:17.497325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:26.268883+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"272'10","reported_seq":974,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.481103+0000","last_change":"2024-03-21T13:16:09.700298+0000","last_active":"2024-03-21T13:16:51.481103+0000","last_peered":"2024-03-21T13:16:51.481103+0000","last_clean":"2024-03-21T13:16:51.481103+0000","last_became_active":"2024-03-21T13:10:03.594651+0000","last_became_peered":"2024-03-21T13:10:03.594651+0000","last_unstale":"2024-03-21T13:16:51.481103+0000","last_undegraded":"2024-03-21T13:16:51.481103+0000","last_fullsized":"2024-03-21T13:16:51.481103+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'10","last_scrub_stamp":"2024-03-21T13:16:09.700215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:09.700215+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:35.185939+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":190,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.490522+0000","last_change":"2024-03-21T13:16:15.939316+0000","last_active":"2024-03-21T13:16:51.490522+0000","last_peered":"2024-03-21T13:16:51.490522+0000","last_clean":"2024-03-21T13:16:51.490522+0000","last_became_active":"2024-03-21T13:15:04.458184+0000","last_became_peered":"2024-03-21T13:15:04.458184+0000","last_unstale":"2024-03-21T13:16:51.490522+0000","last_undegraded":"2024-03-21T13:16:51.490522+0000","last_fullsized":"2024-03-21T13:16:51.490522+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:15.939231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:16:15.939231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:17.625617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"234'15","reported_seq":187,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.497574+0000","last_change":"2024-03-21T13:16:22.227639+0000","last_active":"2024-03-21T13:16:51.497574+0000","last_peered":"2024-03-21T13:16:51.497574+0000","last_clean":"2024-03-21T13:16:51.497574+0000","last_became_active":"2024-03-21T13:12:29.510247+0000","last_became_peered":"2024-03-21T13:12:29.510247+0000","last_unstale":"2024-03-21T13:16:51.497574+0000","last_undegraded":"2024-03-21T13:16:51.497574+0000","last_fullsized":"2024-03-21T13:16:51.497574+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:16:22.227536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.227536+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:38.775744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":297,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.486524+0000","last_change":"2024-03-21T13:16:03.289521+0000","last_active":"2024-03-21T13:16:51.486524+0000","last_peered":"2024-03-21T13:16:51.486524+0000","last_clean":"2024-03-21T13:16:51.486524+0000","last_became_active":"2024-03-21T13:10:03.613841+0000","last_became_peered":"2024-03-21T13:10:03.613841+0000","last_unstale":"2024-03-21T13:16:51.486524+0000","last_undegraded":"2024-03-21T13:16:51.486524+0000","last_fullsized":"2024-03-21T13:16:51.486524+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:03.289412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:16:03.289412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:07.624710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":731,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.486831+0000","last_change":"2024-03-21T13:14:57.279251+0000","last_active":"2024-03-21T13:16:51.486831+0000","last_peered":"2024-03-21T13:16:51.486831+0000","last_clean":"2024-03-21T13:16:51.486831+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:16:51.486831+0000","last_undegraded":"2024-03-21T13:16:51.486831+0000","last_fullsized":"2024-03-21T13:16:51.486831+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:57.279130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:14:57.279130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":752,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:54.652960+0000","last_change":"2024-03-21T13:16:54.653009+0000","last_active":"2024-03-21T13:16:54.652960+0000","last_peered":"2024-03-21T13:16:54.652960+0000","last_clean":"2024-03-21T13:16:54.652960+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:16:54.652960+0000","last_undegraded":"2024-03-21T13:16:54.652960+0000","last_fullsized":"2024-03-21T13:16:54.652960+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:16:54.652821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.652821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:57.529839+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'15","reported_seq":377,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.492953+0000","last_change":"2024-03-21T13:15:04.460964+0000","last_active":"2024-03-21T13:16:51.492953+0000","last_peered":"2024-03-21T13:16:51.492953+0000","last_clean":"2024-03-21T13:16:51.492953+0000","last_became_active":"2024-03-21T13:15:04.458914+0000","last_became_peered":"2024-03-21T13:15:04.458914+0000","last_unstale":"2024-03-21T13:16:51.492953+0000","last_undegraded":"2024-03-21T13:16:51.492953+0000","last_fullsized":"2024-03-21T13:16:51.492953+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:14:15.192958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:15.192958+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":726,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.492968+0000","last_change":"2024-03-21T13:16:02.716355+0000","last_active":"2024-03-21T13:16:51.492968+0000","last_peered":"2024-03-21T13:16:51.492968+0000","last_clean":"2024-03-21T13:16:51.492968+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:16:51.492968+0000","last_undegraded":"2024-03-21T13:16:51.492968+0000","last_fullsized":"2024-03-21T13:16:51.492968+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:02.716255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:02.716255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.235343+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":177,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.496906+0000","last_change":"2024-03-21T13:16:18.220471+0000","last_active":"2024-03-21T13:16:51.496906+0000","last_peered":"2024-03-21T13:16:51.496906+0000","last_clean":"2024-03-21T13:16:51.496906+0000","last_became_active":"2024-03-21T13:12:29.514113+0000","last_became_peered":"2024-03-21T13:12:29.514113+0000","last_unstale":"2024-03-21T13:16:51.496906+0000","last_undegraded":"2024-03-21T13:16:51.496906+0000","last_fullsized":"2024-03-21T13:16:51.496906+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.220368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.220368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.800528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":752,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483801+0000","last_change":"2024-03-21T13:14:14.647062+0000","last_active":"2024-03-21T13:16:51.483801+0000","last_peered":"2024-03-21T13:16:51.483801+0000","last_clean":"2024-03-21T13:16:51.483801+0000","last_became_active":"2024-03-21T13:12:29.520286+0000","last_became_peered":"2024-03-21T13:12:29.520286+0000","last_unstale":"2024-03-21T13:16:51.483801+0000","last_undegraded":"2024-03-21T13:16:51.483801+0000","last_fullsized":"2024-03-21T13:16:51.483801+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:14.646933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:14.646933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":789,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.486658+0000","last_change":"2024-03-21T13:14:09.141688+0000","last_active":"2024-03-21T13:16:51.486658+0000","last_peered":"2024-03-21T13:16:51.486658+0000","last_clean":"2024-03-21T13:16:51.486658+0000","last_became_active":"2024-03-21T13:10:03.610810+0000","last_became_peered":"2024-03-21T13:10:03.610810+0000","last_unstale":"2024-03-21T13:16:51.486658+0000","last_undegraded":"2024-03-21T13:16:51.486658+0000","last_fullsized":"2024-03-21T13:16:51.486658+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-21T13:14:09.141520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:09.141520+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"406'3200","reported_seq":3241,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:54.891677+0000","last_change":"2024-03-21T13:16:13.903136+0000","last_active":"2024-03-21T13:16:54.891677+0000","last_peered":"2024-03-21T13:16:54.891677+0000","last_clean":"2024-03-21T13:16:54.891677+0000","last_became_active":"2024-03-21T13:16:13.901764+0000","last_became_peered":"2024-03-21T13:16:13.901764+0000","last_unstale":"2024-03-21T13:16:54.891677+0000","last_undegraded":"2024-03-21T13:16:54.891677+0000","last_fullsized":"2024-03-21T13:16:54.891677+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3200,"log_dups_size":0,"ondisk_log_size":3200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:14.340802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":3200,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3200,"num_write_kb":204800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":170,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.496643+0000","last_change":"2024-03-21T13:15:06.391491+0000","last_active":"2024-03-21T13:16:51.496643+0000","last_peered":"2024-03-21T13:16:51.496643+0000","last_clean":"2024-03-21T13:16:51.496643+0000","last_became_active":"2024-03-21T13:12:29.517413+0000","last_became_peered":"2024-03-21T13:12:29.517413+0000","last_unstale":"2024-03-21T13:16:51.496643+0000","last_undegraded":"2024-03-21T13:16:51.496643+0000","last_fullsized":"2024-03-21T13:16:51.496643+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:06.391193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:15:06.391193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":775,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.494716+0000","last_change":"2024-03-21T13:14:53.565542+0000","last_active":"2024-03-21T13:16:51.494716+0000","last_peered":"2024-03-21T13:16:51.494716+0000","last_clean":"2024-03-21T13:16:51.494716+0000","last_became_active":"2024-03-21T13:06:29.487439+0000","last_became_peered":"2024-03-21T13:06:29.487439+0000","last_unstale":"2024-03-21T13:16:51.494716+0000","last_undegraded":"2024-03-21T13:16:51.494716+0000","last_fullsized":"2024-03-21T13:16:51.494716+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:14:53.565419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:53.565419+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":194,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:53.176935+0000","last_change":"2024-03-21T13:16:53.176991+0000","last_active":"2024-03-21T13:16:53.176935+0000","last_peered":"2024-03-21T13:16:53.176935+0000","last_clean":"2024-03-21T13:16:53.176935+0000","last_became_active":"2024-03-21T13:12:29.513019+0000","last_became_peered":"2024-03-21T13:12:29.513019+0000","last_unstale":"2024-03-21T13:16:53.176935+0000","last_undegraded":"2024-03-21T13:16:53.176935+0000","last_fullsized":"2024-03-21T13:16:53.176935+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:16:53.176836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.176836+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:13.256411+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":963,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.492460+0000","last_change":"2024-03-21T13:16:10.554480+0000","last_active":"2024-03-21T13:16:51.492460+0000","last_peered":"2024-03-21T13:16:51.492460+0000","last_clean":"2024-03-21T13:16:51.492460+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:16:51.492460+0000","last_undegraded":"2024-03-21T13:16:51.492460+0000","last_fullsized":"2024-03-21T13:16:51.492460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:16:10.554378+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:16:10.554378+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.191410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"406'3478","reported_seq":3519,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:55.574841+0000","last_change":"2024-03-21T13:16:13.901750+0000","last_active":"2024-03-21T13:16:55.574841+0000","last_peered":"2024-03-21T13:16:55.574841+0000","last_clean":"2024-03-21T13:16:55.574841+0000","last_became_active":"2024-03-21T13:16:13.901065+0000","last_became_peered":"2024-03-21T13:16:13.901065+0000","last_unstale":"2024-03-21T13:16:55.574841+0000","last_undegraded":"2024-03-21T13:16:55.574841+0000","last_fullsized":"2024-03-21T13:16:55.574841+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3478,"log_dups_size":0,"ondisk_log_size":3478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.886855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227934208,"num_objects":3478,"num_object_clones":0,"num_object_copies":10434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3478,"num_write_kb":222592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":728,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483162+0000","last_change":"2024-03-21T13:15:58.046280+0000","last_active":"2024-03-21T13:16:51.483162+0000","last_peered":"2024-03-21T13:16:51.483162+0000","last_clean":"2024-03-21T13:16:51.483162+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:16:51.483162+0000","last_undegraded":"2024-03-21T13:16:51.483162+0000","last_fullsized":"2024-03-21T13:16:51.483162+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:58.046156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:15:58.046156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:12.947549+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":753,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:52.653462+0000","last_change":"2024-03-21T13:16:52.653497+0000","last_active":"2024-03-21T13:16:52.653462+0000","last_peered":"2024-03-21T13:16:52.653462+0000","last_clean":"2024-03-21T13:16:52.653462+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:16:52.653462+0000","last_undegraded":"2024-03-21T13:16:52.653462+0000","last_fullsized":"2024-03-21T13:16:52.653462+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.653388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.653388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:19.348258+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"346'23","reported_seq":148,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483936+0000","last_change":"2024-03-21T13:16:10.260270+0000","last_active":"2024-03-21T13:16:51.483936+0000","last_peered":"2024-03-21T13:16:51.483936+0000","last_clean":"2024-03-21T13:16:51.483936+0000","last_became_active":"2024-03-21T13:13:21.819111+0000","last_became_peered":"2024-03-21T13:13:21.819111+0000","last_unstale":"2024-03-21T13:16:51.483936+0000","last_undegraded":"2024-03-21T13:16:51.483936+0000","last_fullsized":"2024-03-21T13:16:51.483936+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'23","last_scrub_stamp":"2024-03-21T13:16:10.260157+0000","last_deep_scrub":"346'23","last_deep_scrub_stamp":"2024-03-21T13:13:20.232807+0000","last_clean_scrub_stamp":"2024-03-21T13:16:10.260157+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:35.914575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":312,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:55.431107+0000","last_change":"2024-03-21T13:16:55.431147+0000","last_active":"2024-03-21T13:16:55.431107+0000","last_peered":"2024-03-21T13:16:55.431107+0000","last_clean":"2024-03-21T13:16:55.431107+0000","last_became_active":"2024-03-21T13:12:29.499921+0000","last_became_peered":"2024-03-21T13:12:29.499921+0000","last_unstale":"2024-03-21T13:16:55.431107+0000","last_undegraded":"2024-03-21T13:16:55.431107+0000","last_fullsized":"2024-03-21T13:16:55.431107+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:55.431004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:55.431004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:23.679031+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"406'3449","reported_seq":3490,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:55.383195+0000","last_change":"2024-03-21T13:16:13.903159+0000","last_active":"2024-03-21T13:16:55.383195+0000","last_peered":"2024-03-21T13:16:55.383195+0000","last_clean":"2024-03-21T13:16:55.383195+0000","last_became_active":"2024-03-21T13:16:13.902868+0000","last_became_peered":"2024-03-21T13:16:13.902868+0000","last_unstale":"2024-03-21T13:16:55.383195+0000","last_undegraded":"2024-03-21T13:16:55.383195+0000","last_fullsized":"2024-03-21T13:16:55.383195+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3449,"log_dups_size":0,"ondisk_log_size":3449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:29.385439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226033664,"num_objects":3449,"num_object_clones":0,"num_object_copies":10347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3449,"num_write_kb":220736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":747,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.491923+0000","last_change":"2024-03-21T13:16:06.611026+0000","last_active":"2024-03-21T13:16:51.491923+0000","last_peered":"2024-03-21T13:16:51.491923+0000","last_clean":"2024-03-21T13:16:51.491923+0000","last_became_active":"2024-03-21T13:10:03.598366+0000","last_became_peered":"2024-03-21T13:10:03.598366+0000","last_unstale":"2024-03-21T13:16:51.491923+0000","last_undegraded":"2024-03-21T13:16:51.491923+0000","last_fullsized":"2024-03-21T13:16:51.491923+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:06.610947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:16:06.610947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.715015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":320,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.484732+0000","last_change":"2024-03-21T13:16:49.448922+0000","last_active":"2024-03-21T13:16:51.484732+0000","last_peered":"2024-03-21T13:16:51.484732+0000","last_clean":"2024-03-21T13:16:51.484732+0000","last_became_active":"2024-03-21T13:12:29.500199+0000","last_became_peered":"2024-03-21T13:12:29.500199+0000","last_unstale":"2024-03-21T13:16:51.484732+0000","last_undegraded":"2024-03-21T13:16:51.484732+0000","last_fullsized":"2024-03-21T13:16:51.484732+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:49.448804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:49.448804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:02.584547+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":759,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.493412+0000","last_change":"2024-03-21T13:16:47.661139+0000","last_active":"2024-03-21T13:16:51.493412+0000","last_peered":"2024-03-21T13:16:51.493412+0000","last_clean":"2024-03-21T13:16:51.493412+0000","last_became_active":"2024-03-21T13:12:29.508984+0000","last_became_peered":"2024-03-21T13:12:29.508984+0000","last_unstale":"2024-03-21T13:16:51.493412+0000","last_undegraded":"2024-03-21T13:16:51.493412+0000","last_fullsized":"2024-03-21T13:16:51.493412+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:47.661063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:47.661063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:15.875522+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'35","reported_seq":870,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483181+0000","last_change":"2024-03-21T13:13:24.732800+0000","last_active":"2024-03-21T13:16:51.483181+0000","last_peered":"2024-03-21T13:16:51.483181+0000","last_clean":"2024-03-21T13:16:51.483181+0000","last_became_active":"2024-03-21T13:12:29.523359+0000","last_became_peered":"2024-03-21T13:12:29.523359+0000","last_unstale":"2024-03-21T13:16:51.483181+0000","last_undegraded":"2024-03-21T13:16:51.483181+0000","last_fullsized":"2024-03-21T13:16:51.483181+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:13:24.732562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:13:24.732562+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"406'3380","reported_seq":3421,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:55.365167+0000","last_change":"2024-03-21T13:16:13.904507+0000","last_active":"2024-03-21T13:16:55.365167+0000","last_peered":"2024-03-21T13:16:55.365167+0000","last_clean":"2024-03-21T13:16:55.365167+0000","last_became_active":"2024-03-21T13:16:13.904210+0000","last_became_peered":"2024-03-21T13:16:13.904210+0000","last_unstale":"2024-03-21T13:16:55.365167+0000","last_undegraded":"2024-03-21T13:16:55.365167+0000","last_fullsized":"2024-03-21T13:16:55.365167+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3380,"log_dups_size":0,"ondisk_log_size":3380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.107729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221511680,"num_objects":3380,"num_object_clones":0,"num_object_copies":10140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3380,"num_write_kb":216320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":431,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.494334+0000","last_change":"2024-03-21T13:15:12.891812+0000","last_active":"2024-03-21T13:16:51.494334+0000","last_peered":"2024-03-21T13:16:51.494334+0000","last_clean":"2024-03-21T13:16:51.494334+0000","last_became_active":"2024-03-21T13:15:04.458107+0000","last_became_peered":"2024-03-21T13:15:04.458107+0000","last_unstale":"2024-03-21T13:16:51.494334+0000","last_undegraded":"2024-03-21T13:16:51.494334+0000","last_fullsized":"2024-03-21T13:16:51.494334+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:12.891693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:12.891693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":298,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.485998+0000","last_change":"2024-03-21T13:15:13.550311+0000","last_active":"2024-03-21T13:16:51.485998+0000","last_peered":"2024-03-21T13:16:51.485998+0000","last_clean":"2024-03-21T13:16:51.485998+0000","last_became_active":"2024-03-21T13:10:03.603646+0000","last_became_peered":"2024-03-21T13:10:03.603646+0000","last_unstale":"2024-03-21T13:16:51.485998+0000","last_undegraded":"2024-03-21T13:16:51.485998+0000","last_fullsized":"2024-03-21T13:16:51.485998+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:13.550186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:13.550186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":177,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:54.243506+0000","last_change":"2024-03-21T13:16:54.243549+0000","last_active":"2024-03-21T13:16:54.243506+0000","last_peered":"2024-03-21T13:16:54.243506+0000","last_clean":"2024-03-21T13:16:54.243506+0000","last_became_active":"2024-03-21T13:12:29.512741+0000","last_became_peered":"2024-03-21T13:12:29.512741+0000","last_unstale":"2024-03-21T13:16:54.243506+0000","last_undegraded":"2024-03-21T13:16:54.243506+0000","last_fullsized":"2024-03-21T13:16:54.243506+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:54.243395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.243395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:03.270582+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"234'19","reported_seq":430,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.494062+0000","last_change":"2024-03-21T13:16:23.841059+0000","last_active":"2024-03-21T13:16:51.494062+0000","last_peered":"2024-03-21T13:16:51.494062+0000","last_clean":"2024-03-21T13:16:51.494062+0000","last_became_active":"2024-03-21T13:15:04.460898+0000","last_became_peered":"2024-03-21T13:15:04.460898+0000","last_unstale":"2024-03-21T13:16:51.494062+0000","last_undegraded":"2024-03-21T13:16:51.494062+0000","last_fullsized":"2024-03-21T13:16:51.494062+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:23.840950+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:16:23.840950+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:46.775417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"406'3346","reported_seq":3388,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:56.163731+0000","last_change":"2024-03-21T13:16:13.902405+0000","last_active":"2024-03-21T13:16:56.163731+0000","last_peered":"2024-03-21T13:16:56.163731+0000","last_clean":"2024-03-21T13:16:56.163731+0000","last_became_active":"2024-03-21T13:16:13.902109+0000","last_became_peered":"2024-03-21T13:16:13.902109+0000","last_unstale":"2024-03-21T13:16:56.163731+0000","last_undegraded":"2024-03-21T13:16:56.163731+0000","last_fullsized":"2024-03-21T13:16:56.163731+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3346,"log_dups_size":0,"ondisk_log_size":3346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:41.304847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219283456,"num_objects":3346,"num_object_clones":0,"num_object_copies":10038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3346,"num_write_kb":214144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":714,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.482160+0000","last_change":"2024-03-21T13:14:14.199570+0000","last_active":"2024-03-21T13:16:51.482160+0000","last_peered":"2024-03-21T13:16:51.482160+0000","last_clean":"2024-03-21T13:16:51.482160+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:16:51.482160+0000","last_undegraded":"2024-03-21T13:16:51.482160+0000","last_fullsized":"2024-03-21T13:16:51.482160+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:14.199450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:14:14.199450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":174,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.496122+0000","last_change":"2024-03-21T13:16:08.211061+0000","last_active":"2024-03-21T13:16:51.496122+0000","last_peered":"2024-03-21T13:16:51.496122+0000","last_clean":"2024-03-21T13:16:51.496122+0000","last_became_active":"2024-03-21T13:12:29.521275+0000","last_became_peered":"2024-03-21T13:12:29.521275+0000","last_unstale":"2024-03-21T13:16:51.496122+0000","last_undegraded":"2024-03-21T13:16:51.496122+0000","last_fullsized":"2024-03-21T13:16:51.496122+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:16:08.210967+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:16:08.210967+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:37.370051+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":86,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.490524+0000","last_change":"2024-03-21T13:16:22.861225+0000","last_active":"2024-03-21T13:16:51.490524+0000","last_peered":"2024-03-21T13:16:51.490524+0000","last_clean":"2024-03-21T13:16:51.490524+0000","last_became_active":"2024-03-21T13:15:04.462662+0000","last_became_peered":"2024-03-21T13:15:04.462662+0000","last_unstale":"2024-03-21T13:16:51.490524+0000","last_undegraded":"2024-03-21T13:16:51.490524+0000","last_fullsized":"2024-03-21T13:16:51.490524+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:22.861136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:22.861136+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.861136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:34.763292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"272'43","reported_seq":411,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.486421+0000","last_change":"2024-03-21T13:16:16.292178+0000","last_active":"2024-03-21T13:16:51.486421+0000","last_peered":"2024-03-21T13:16:51.486421+0000","last_clean":"2024-03-21T13:16:51.486421+0000","last_became_active":"2024-03-21T13:10:03.603889+0000","last_became_peered":"2024-03-21T13:10:03.603889+0000","last_unstale":"2024-03-21T13:16:51.486421+0000","last_undegraded":"2024-03-21T13:16:51.486421+0000","last_fullsized":"2024-03-21T13:16:51.486421+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'43","last_scrub_stamp":"2024-03-21T13:16:16.292073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:16.292073+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:41.103424+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":182,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.489478+0000","last_change":"2024-03-21T13:16:10.968486+0000","last_active":"2024-03-21T13:16:51.489478+0000","last_peered":"2024-03-21T13:16:51.489478+0000","last_clean":"2024-03-21T13:16:51.489478+0000","last_became_active":"2024-03-21T13:15:04.462159+0000","last_became_peered":"2024-03-21T13:15:04.462159+0000","last_unstale":"2024-03-21T13:16:51.489478+0000","last_undegraded":"2024-03-21T13:16:51.489478+0000","last_fullsized":"2024-03-21T13:16:51.489478+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:10.968402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:15.968926+0000","last_clean_scrub_stamp":"2024-03-21T13:16:10.968402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.606676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"406'3403","reported_seq":3445,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:56.170921+0000","last_change":"2024-03-21T13:16:13.900592+0000","last_active":"2024-03-21T13:16:56.170921+0000","last_peered":"2024-03-21T13:16:56.170921+0000","last_clean":"2024-03-21T13:16:56.170921+0000","last_became_active":"2024-03-21T13:16:13.900119+0000","last_became_peered":"2024-03-21T13:16:13.900119+0000","last_unstale":"2024-03-21T13:16:56.170921+0000","last_undegraded":"2024-03-21T13:16:56.170921+0000","last_fullsized":"2024-03-21T13:16:56.170921+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3403,"log_dups_size":0,"ondisk_log_size":3403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:17.609817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223019008,"num_objects":3403,"num_object_clones":0,"num_object_copies":10209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3403,"num_write_kb":217792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":735,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.493582+0000","last_change":"2024-03-21T13:16:18.683140+0000","last_active":"2024-03-21T13:16:51.493582+0000","last_peered":"2024-03-21T13:16:51.493582+0000","last_clean":"2024-03-21T13:16:51.493582+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:16:51.493582+0000","last_undegraded":"2024-03-21T13:16:51.493582+0000","last_fullsized":"2024-03-21T13:16:51.493582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.683041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:44.764044+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":298,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.484269+0000","last_change":"2024-03-21T13:16:12.231215+0000","last_active":"2024-03-21T13:16:51.484269+0000","last_peered":"2024-03-21T13:16:51.484269+0000","last_clean":"2024-03-21T13:16:51.484269+0000","last_became_active":"2024-03-21T13:10:03.604303+0000","last_became_peered":"2024-03-21T13:10:03.604303+0000","last_unstale":"2024-03-21T13:16:51.484269+0000","last_undegraded":"2024-03-21T13:16:51.484269+0000","last_fullsized":"2024-03-21T13:16:51.484269+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.231115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.231115+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.231115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:12.524706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":306,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:52.465515+0000","last_change":"2024-03-21T13:16:52.465547+0000","last_active":"2024-03-21T13:16:52.465515+0000","last_peered":"2024-03-21T13:16:52.465515+0000","last_clean":"2024-03-21T13:16:52.465515+0000","last_became_active":"2024-03-21T13:10:03.604427+0000","last_became_peered":"2024-03-21T13:10:03.604427+0000","last_unstale":"2024-03-21T13:16:52.465515+0000","last_undegraded":"2024-03-21T13:16:52.465515+0000","last_fullsized":"2024-03-21T13:16:52.465515+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.465427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.465427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:59.952130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"243'20","reported_seq":182,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.497168+0000","last_change":"2024-03-21T13:14:52.337122+0000","last_active":"2024-03-21T13:16:51.497168+0000","last_peered":"2024-03-21T13:16:51.497168+0000","last_clean":"2024-03-21T13:16:51.497168+0000","last_became_active":"2024-03-21T13:12:29.518048+0000","last_became_peered":"2024-03-21T13:12:29.518048+0000","last_unstale":"2024-03-21T13:16:51.497168+0000","last_undegraded":"2024-03-21T13:16:51.497168+0000","last_fullsized":"2024-03-21T13:16:51.497168+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'20","last_scrub_stamp":"2024-03-21T13:14:52.332617+0000","last_deep_scrub":"243'20","last_deep_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_clean_scrub_stamp":"2024-03-21T13:14:52.332617+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":304,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.484563+0000","last_change":"2024-03-21T13:16:15.247743+0000","last_active":"2024-03-21T13:16:51.484563+0000","last_peered":"2024-03-21T13:16:51.484563+0000","last_clean":"2024-03-21T13:16:51.484563+0000","last_became_active":"2024-03-21T13:12:29.506751+0000","last_became_peered":"2024-03-21T13:12:29.506751+0000","last_unstale":"2024-03-21T13:16:51.484563+0000","last_undegraded":"2024-03-21T13:16:51.484563+0000","last_fullsized":"2024-03-21T13:16:51.484563+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:15.247661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:15.247661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:36.430093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"406'3456","reported_seq":3498,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:56.153658+0000","last_change":"2024-03-21T13:16:13.900783+0000","last_active":"2024-03-21T13:16:56.153658+0000","last_peered":"2024-03-21T13:16:56.153658+0000","last_clean":"2024-03-21T13:16:56.153658+0000","last_became_active":"2024-03-21T13:16:13.900477+0000","last_became_peered":"2024-03-21T13:16:13.900477+0000","last_unstale":"2024-03-21T13:16:56.153658+0000","last_undegraded":"2024-03-21T13:16:56.153658+0000","last_fullsized":"2024-03-21T13:16:56.153658+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:38.844318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":3456,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3456,"num_write_kb":221184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":170,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.497056+0000","last_change":"2024-03-21T13:15:04.365662+0000","last_active":"2024-03-21T13:16:51.497056+0000","last_peered":"2024-03-21T13:16:51.497056+0000","last_clean":"2024-03-21T13:16:51.497056+0000","last_became_active":"2024-03-21T13:12:29.527083+0000","last_became_peered":"2024-03-21T13:12:29.527083+0000","last_unstale":"2024-03-21T13:16:51.497056+0000","last_undegraded":"2024-03-21T13:16:51.497056+0000","last_fullsized":"2024-03-21T13:16:51.497056+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:04.365534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:04.365534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":998,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.493437+0000","last_change":"2024-03-21T13:16:14.628486+0000","last_active":"2024-03-21T13:16:51.493437+0000","last_peered":"2024-03-21T13:16:51.493437+0000","last_clean":"2024-03-21T13:16:51.493437+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:16:51.493437+0000","last_undegraded":"2024-03-21T13:16:51.493437+0000","last_fullsized":"2024-03-21T13:16:51.493437+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:16:14.628384+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:16:14.628384+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.536448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":243,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.485736+0000","last_change":"2024-03-21T13:16:02.247782+0000","last_active":"2024-03-21T13:16:51.485736+0000","last_peered":"2024-03-21T13:16:51.485736+0000","last_clean":"2024-03-21T13:16:51.485736+0000","last_became_active":"2024-03-21T13:12:29.499989+0000","last_became_peered":"2024-03-21T13:12:29.499989+0000","last_unstale":"2024-03-21T13:16:51.485736+0000","last_undegraded":"2024-03-21T13:16:51.485736+0000","last_fullsized":"2024-03-21T13:16:51.485736+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:02.247693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:02.247693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.746970+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"234'36","reported_seq":1005,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483962+0000","last_change":"2024-03-21T13:16:05.528913+0000","last_active":"2024-03-21T13:16:51.483962+0000","last_peered":"2024-03-21T13:16:51.483962+0000","last_clean":"2024-03-21T13:16:51.483962+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:16:51.483962+0000","last_undegraded":"2024-03-21T13:16:51.483962+0000","last_fullsized":"2024-03-21T13:16:51.483962+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'36","last_scrub_stamp":"2024-03-21T13:16:05.528782+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.528782+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:30.885762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":755,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.494377+0000","last_change":"2024-03-21T13:14:56.674085+0000","last_active":"2024-03-21T13:16:51.494377+0000","last_peered":"2024-03-21T13:16:51.494377+0000","last_clean":"2024-03-21T13:16:51.494377+0000","last_became_active":"2024-03-21T13:12:29.507753+0000","last_became_peered":"2024-03-21T13:12:29.507753+0000","last_unstale":"2024-03-21T13:16:51.494377+0000","last_undegraded":"2024-03-21T13:16:51.494377+0000","last_fullsized":"2024-03-21T13:16:51.494377+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:56.673972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:14:56.673972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"406'3499","reported_seq":3541,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:56.154144+0000","last_change":"2024-03-21T13:16:13.900952+0000","last_active":"2024-03-21T13:16:56.154144+0000","last_peered":"2024-03-21T13:16:56.154144+0000","last_clean":"2024-03-21T13:16:56.154144+0000","last_became_active":"2024-03-21T13:16:13.900650+0000","last_became_peered":"2024-03-21T13:16:13.900650+0000","last_unstale":"2024-03-21T13:16:56.154144+0000","last_undegraded":"2024-03-21T13:16:56.154144+0000","last_fullsized":"2024-03-21T13:16:56.154144+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3499,"log_dups_size":0,"ondisk_log_size":3499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.174786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229310464,"num_objects":3499,"num_object_clones":0,"num_object_copies":10497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3499,"num_write_kb":223936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":170,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.495364+0000","last_change":"2024-03-21T13:16:01.195705+0000","last_active":"2024-03-21T13:16:51.495364+0000","last_peered":"2024-03-21T13:16:51.495364+0000","last_clean":"2024-03-21T13:16:51.495364+0000","last_became_active":"2024-03-21T13:12:29.514542+0000","last_became_peered":"2024-03-21T13:12:29.514542+0000","last_unstale":"2024-03-21T13:16:51.495364+0000","last_undegraded":"2024-03-21T13:16:51.495364+0000","last_fullsized":"2024-03-21T13:16:51.495364+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:01.195618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:01.195618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:22.887658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":190,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.497015+0000","last_change":"2024-03-21T13:15:04.459946+0000","last_active":"2024-03-21T13:16:51.497015+0000","last_peered":"2024-03-21T13:16:51.497015+0000","last_clean":"2024-03-21T13:16:51.497015+0000","last_became_active":"2024-03-21T13:15:04.458084+0000","last_became_peered":"2024-03-21T13:15:04.458084+0000","last_unstale":"2024-03-21T13:16:51.497015+0000","last_undegraded":"2024-03-21T13:16:51.497015+0000","last_fullsized":"2024-03-21T13:16:51.497015+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:57.238591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:34.066532+0000","last_clean_scrub_stamp":"2024-03-21T13:14:57.238591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":1849,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:54.282253+0000","last_change":"2024-03-21T13:16:09.547909+0000","last_active":"2024-03-21T13:16:54.282253+0000","last_peered":"2024-03-21T13:16:54.282253+0000","last_clean":"2024-03-21T13:16:54.282253+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:16:54.282253+0000","last_undegraded":"2024-03-21T13:16:54.282253+0000","last_fullsized":"2024-03-21T13:16:54.282253+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:16:09.547783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:16:09.547783+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:28.689027+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1087,"num_read_kb":1082,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":872,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.492395+0000","last_change":"2024-03-21T13:16:18.883144+0000","last_active":"2024-03-21T13:16:51.492395+0000","last_peered":"2024-03-21T13:16:51.492395+0000","last_clean":"2024-03-21T13:16:51.492395+0000","last_became_active":"2024-03-21T13:10:03.606672+0000","last_became_peered":"2024-03-21T13:10:03.606672+0000","last_unstale":"2024-03-21T13:16:51.492395+0000","last_undegraded":"2024-03-21T13:16:51.492395+0000","last_fullsized":"2024-03-21T13:16:51.492395+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-21T13:16:18.882994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.882994+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:43.518929+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":170,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.496968+0000","last_change":"2024-03-21T13:14:07.406567+0000","last_active":"2024-03-21T13:16:51.496968+0000","last_peered":"2024-03-21T13:16:51.496968+0000","last_clean":"2024-03-21T13:16:51.496968+0000","last_became_active":"2024-03-21T13:12:29.526100+0000","last_became_peered":"2024-03-21T13:12:29.526100+0000","last_unstale":"2024-03-21T13:16:51.496968+0000","last_undegraded":"2024-03-21T13:16:51.496968+0000","last_fullsized":"2024-03-21T13:16:51.496968+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:07.406434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:14:07.406434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"406'3318","reported_seq":3360,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:55.450686+0000","last_change":"2024-03-21T13:16:13.899271+0000","last_active":"2024-03-21T13:16:55.450686+0000","last_peered":"2024-03-21T13:16:55.450686+0000","last_clean":"2024-03-21T13:16:55.450686+0000","last_became_active":"2024-03-21T13:16:13.898890+0000","last_became_peered":"2024-03-21T13:16:13.898890+0000","last_unstale":"2024-03-21T13:16:55.450686+0000","last_undegraded":"2024-03-21T13:16:55.450686+0000","last_fullsized":"2024-03-21T13:16:55.450686+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3318,"log_dups_size":0,"ondisk_log_size":3318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:39.631299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217448448,"num_objects":3318,"num_object_clones":0,"num_object_copies":9954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3318,"num_write_kb":212352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":638,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483524+0000","last_change":"2024-03-21T13:16:15.593032+0000","last_active":"2024-03-21T13:16:51.483524+0000","last_peered":"2024-03-21T13:16:51.483524+0000","last_clean":"2024-03-21T13:16:51.483524+0000","last_became_active":"2024-03-21T13:10:03.591088+0000","last_became_peered":"2024-03-21T13:10:03.591088+0000","last_unstale":"2024-03-21T13:16:51.483524+0000","last_undegraded":"2024-03-21T13:16:51.483524+0000","last_fullsized":"2024-03-21T13:16:51.483524+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:15.592931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:15.592931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:36.752324+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":312,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.487772+0000","last_change":"2024-03-21T13:16:22.293702+0000","last_active":"2024-03-21T13:16:51.487772+0000","last_peered":"2024-03-21T13:16:51.487772+0000","last_clean":"2024-03-21T13:16:51.487772+0000","last_became_active":"2024-03-21T13:12:29.507783+0000","last_became_peered":"2024-03-21T13:12:29.507783+0000","last_unstale":"2024-03-21T13:16:51.487772+0000","last_undegraded":"2024-03-21T13:16:51.487772+0000","last_fullsized":"2024-03-21T13:16:51.487772+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:22.293625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.293625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.688902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":188,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.495549+0000","last_change":"2024-03-21T13:14:16.366110+0000","last_active":"2024-03-21T13:16:51.495549+0000","last_peered":"2024-03-21T13:16:51.495549+0000","last_clean":"2024-03-21T13:16:51.495549+0000","last_became_active":"2024-03-21T13:12:29.518800+0000","last_became_peered":"2024-03-21T13:12:29.518800+0000","last_unstale":"2024-03-21T13:16:51.495549+0000","last_undegraded":"2024-03-21T13:16:51.495549+0000","last_fullsized":"2024-03-21T13:16:51.495549+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-21T13:14:16.365267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:16.365267+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":757,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483217+0000","last_change":"2024-03-21T13:13:21.836944+0000","last_active":"2024-03-21T13:16:51.483217+0000","last_peered":"2024-03-21T13:16:51.483217+0000","last_clean":"2024-03-21T13:16:51.483217+0000","last_became_active":"2024-03-21T13:12:29.520657+0000","last_became_peered":"2024-03-21T13:12:29.520657+0000","last_unstale":"2024-03-21T13:16:51.483217+0000","last_undegraded":"2024-03-21T13:16:51.483217+0000","last_fullsized":"2024-03-21T13:16:51.483217+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:13:21.836810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:13:21.836810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"406'3361","reported_seq":3403,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:55.457901+0000","last_change":"2024-03-21T13:16:13.899935+0000","last_active":"2024-03-21T13:16:55.457901+0000","last_peered":"2024-03-21T13:16:55.457901+0000","last_clean":"2024-03-21T13:16:55.457901+0000","last_became_active":"2024-03-21T13:16:13.899614+0000","last_became_peered":"2024-03-21T13:16:13.899614+0000","last_unstale":"2024-03-21T13:16:55.457901+0000","last_undegraded":"2024-03-21T13:16:55.457901+0000","last_fullsized":"2024-03-21T13:16:55.457901+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3361,"log_dups_size":0,"ondisk_log_size":3361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.233033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220266496,"num_objects":3361,"num_object_clones":0,"num_object_copies":10083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3361,"num_write_kb":215104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":432,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.489821+0000","last_change":"2024-03-21T13:16:51.070000+0000","last_active":"2024-03-21T13:16:51.489821+0000","last_peered":"2024-03-21T13:16:51.489821+0000","last_clean":"2024-03-21T13:16:51.489821+0000","last_became_active":"2024-03-21T13:15:04.456786+0000","last_became_peered":"2024-03-21T13:15:04.456786+0000","last_unstale":"2024-03-21T13:16:51.489821+0000","last_undegraded":"2024-03-21T13:16:51.489821+0000","last_fullsized":"2024-03-21T13:16:51.489821+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:51.069883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:51.069883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:10.981868+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":170,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.497732+0000","last_change":"2024-03-21T13:16:17.169677+0000","last_active":"2024-03-21T13:16:51.497732+0000","last_peered":"2024-03-21T13:16:51.497732+0000","last_clean":"2024-03-21T13:16:51.497732+0000","last_became_active":"2024-03-21T13:12:29.512291+0000","last_became_peered":"2024-03-21T13:12:29.512291+0000","last_unstale":"2024-03-21T13:16:51.497732+0000","last_undegraded":"2024-03-21T13:16:51.497732+0000","last_fullsized":"2024-03-21T13:16:51.497732+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:17.169572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:28.030369+0000","last_clean_scrub_stamp":"2024-03-21T13:16:17.169572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.905934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"234'25","reported_seq":182,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.496281+0000","last_change":"2024-03-21T13:15:57.053925+0000","last_active":"2024-03-21T13:16:51.496281+0000","last_peered":"2024-03-21T13:16:51.496281+0000","last_clean":"2024-03-21T13:16:51.496281+0000","last_became_active":"2024-03-21T13:12:29.526114+0000","last_became_peered":"2024-03-21T13:12:29.526114+0000","last_unstale":"2024-03-21T13:16:51.496281+0000","last_undegraded":"2024-03-21T13:16:51.496281+0000","last_fullsized":"2024-03-21T13:16:51.496281+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-21T13:15:57.053797+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:15:57.053797+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:09.141738+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"375'14","reported_seq":189,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.490351+0000","last_change":"2024-03-21T13:16:07.043970+0000","last_active":"2024-03-21T13:16:51.490351+0000","last_peered":"2024-03-21T13:16:51.490351+0000","last_clean":"2024-03-21T13:16:51.490351+0000","last_became_active":"2024-03-21T13:15:04.464590+0000","last_became_peered":"2024-03-21T13:15:04.464590+0000","last_unstale":"2024-03-21T13:16:51.490351+0000","last_undegraded":"2024-03-21T13:16:51.490351+0000","last_fullsized":"2024-03-21T13:16:51.490351+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'14","last_scrub_stamp":"2024-03-21T13:16:07.043880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.043880+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:17.795947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.e","version":"406'3424","reported_seq":3465,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:55.259609+0000","last_change":"2024-03-21T13:16:13.899791+0000","last_active":"2024-03-21T13:16:55.259609+0000","last_peered":"2024-03-21T13:16:55.259609+0000","last_clean":"2024-03-21T13:16:55.259609+0000","last_became_active":"2024-03-21T13:16:13.899480+0000","last_became_peered":"2024-03-21T13:16:13.899480+0000","last_unstale":"2024-03-21T13:16:55.259609+0000","last_undegraded":"2024-03-21T13:16:55.259609+0000","last_fullsized":"2024-03-21T13:16:55.259609+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3424,"log_dups_size":0,"ondisk_log_size":3424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:38.558703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224395264,"num_objects":3424,"num_object_clones":0,"num_object_copies":10272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3424,"num_write_kb":219136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":739,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.492860+0000","last_change":"2024-03-21T13:15:06.824637+0000","last_active":"2024-03-21T13:16:51.492860+0000","last_peered":"2024-03-21T13:16:51.492860+0000","last_clean":"2024-03-21T13:16:51.492860+0000","last_became_active":"2024-03-21T13:10:03.606557+0000","last_became_peered":"2024-03-21T13:10:03.606557+0000","last_unstale":"2024-03-21T13:16:51.492860+0000","last_undegraded":"2024-03-21T13:16:51.492860+0000","last_fullsized":"2024-03-21T13:16:51.492860+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:06.824520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:06.824520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":791,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.492555+0000","last_change":"2024-03-21T13:14:21.647752+0000","last_active":"2024-03-21T13:16:51.492555+0000","last_peered":"2024-03-21T13:16:51.492555+0000","last_clean":"2024-03-21T13:16:51.492555+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:16:51.492555+0000","last_undegraded":"2024-03-21T13:16:51.492555+0000","last_fullsized":"2024-03-21T13:16:51.492555+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:14:21.647601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:21.647601+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'35","reported_seq":186,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.494885+0000","last_change":"2024-03-21T13:14:12.399995+0000","last_active":"2024-03-21T13:16:51.494885+0000","last_peered":"2024-03-21T13:16:51.494885+0000","last_clean":"2024-03-21T13:16:51.494885+0000","last_became_active":"2024-03-21T13:12:29.525154+0000","last_became_peered":"2024-03-21T13:12:29.525154+0000","last_unstale":"2024-03-21T13:16:51.494885+0000","last_undegraded":"2024-03-21T13:16:51.494885+0000","last_fullsized":"2024-03-21T13:16:51.494885+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:14:12.399785+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:14:12.399785+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"226'10","reported_seq":967,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.489783+0000","last_change":"2024-03-21T13:16:18.672479+0000","last_active":"2024-03-21T13:16:51.489783+0000","last_peered":"2024-03-21T13:16:51.489783+0000","last_clean":"2024-03-21T13:16:51.489783+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:16:51.489783+0000","last_undegraded":"2024-03-21T13:16:51.489783+0000","last_fullsized":"2024-03-21T13:16:51.489783+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T13:16:18.672392+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.672392+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:25.253358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"406'3323","reported_seq":3364,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:55.365220+0000","last_change":"2024-03-21T13:16:13.905080+0000","last_active":"2024-03-21T13:16:55.365220+0000","last_peered":"2024-03-21T13:16:55.365220+0000","last_clean":"2024-03-21T13:16:55.365220+0000","last_became_active":"2024-03-21T13:16:13.904756+0000","last_became_peered":"2024-03-21T13:16:13.904756+0000","last_unstale":"2024-03-21T13:16:55.365220+0000","last_undegraded":"2024-03-21T13:16:55.365220+0000","last_fullsized":"2024-03-21T13:16:55.365220+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3323,"log_dups_size":0,"ondisk_log_size":3323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:22.134211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217776128,"num_objects":3323,"num_object_clones":0,"num_object_copies":9969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3323,"num_write_kb":212672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":659,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.484244+0000","last_change":"2024-03-21T13:15:10.158338+0000","last_active":"2024-03-21T13:16:51.484244+0000","last_peered":"2024-03-21T13:16:51.484244+0000","last_clean":"2024-03-21T13:16:51.484244+0000","last_became_active":"2024-03-21T13:10:03.610471+0000","last_became_peered":"2024-03-21T13:10:03.610471+0000","last_unstale":"2024-03-21T13:16:51.484244+0000","last_undegraded":"2024-03-21T13:16:51.484244+0000","last_fullsized":"2024-03-21T13:16:51.484244+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:10.157920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:15:10.157920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":734,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.481928+0000","last_change":"2024-03-21T13:14:18.185914+0000","last_active":"2024-03-21T13:16:51.481928+0000","last_peered":"2024-03-21T13:16:51.481928+0000","last_clean":"2024-03-21T13:16:51.481928+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:16:51.481928+0000","last_undegraded":"2024-03-21T13:16:51.481928+0000","last_fullsized":"2024-03-21T13:16:51.481928+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:18.185782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:14:18.185782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"234'30","reported_seq":781,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.489644+0000","last_change":"2024-03-21T13:15:09.148375+0000","last_active":"2024-03-21T13:16:51.489644+0000","last_peered":"2024-03-21T13:16:51.489644+0000","last_clean":"2024-03-21T13:16:51.489644+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:16:51.489644+0000","last_undegraded":"2024-03-21T13:16:51.489644+0000","last_fullsized":"2024-03-21T13:16:51.489644+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'30","last_scrub_stamp":"2024-03-21T13:15:09.148229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:09.148229+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":750,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:53.866782+0000","last_change":"2024-03-21T13:16:53.866824+0000","last_active":"2024-03-21T13:16:53.866782+0000","last_peered":"2024-03-21T13:16:53.866782+0000","last_clean":"2024-03-21T13:16:53.866782+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:16:53.866782+0000","last_undegraded":"2024-03-21T13:16:53.866782+0000","last_fullsized":"2024-03-21T13:16:53.866782+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:53.866692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.866692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:12.696345+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"406'3567","reported_seq":3609,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:56.156127+0000","last_change":"2024-03-21T13:16:13.900272+0000","last_active":"2024-03-21T13:16:56.156127+0000","last_peered":"2024-03-21T13:16:56.156127+0000","last_clean":"2024-03-21T13:16:56.156127+0000","last_became_active":"2024-03-21T13:16:13.899979+0000","last_became_peered":"2024-03-21T13:16:13.899979+0000","last_unstale":"2024-03-21T13:16:56.156127+0000","last_undegraded":"2024-03-21T13:16:56.156127+0000","last_fullsized":"2024-03-21T13:16:56.156127+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3567,"log_dups_size":0,"ondisk_log_size":3567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:21.233864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233766912,"num_objects":3567,"num_object_clones":0,"num_object_copies":10701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3567,"num_write_kb":228288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":169,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.497930+0000","last_change":"2024-03-21T13:16:14.280619+0000","last_active":"2024-03-21T13:16:51.497930+0000","last_peered":"2024-03-21T13:16:51.497930+0000","last_clean":"2024-03-21T13:16:51.497930+0000","last_became_active":"2024-03-21T13:12:29.518722+0000","last_became_peered":"2024-03-21T13:12:29.518722+0000","last_unstale":"2024-03-21T13:16:51.497930+0000","last_undegraded":"2024-03-21T13:16:51.497930+0000","last_fullsized":"2024-03-21T13:16:51.497930+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:14.280515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:14.280515+0000","last_clean_scrub_stamp":"2024-03-21T13:16:14.280515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:42.015454+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"331'13","reported_seq":227,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.497148+0000","last_change":"2024-03-21T13:15:15.203026+0000","last_active":"2024-03-21T13:16:51.497148+0000","last_peered":"2024-03-21T13:16:51.497148+0000","last_clean":"2024-03-21T13:16:51.497148+0000","last_became_active":"2024-03-21T13:12:29.525058+0000","last_became_peered":"2024-03-21T13:12:29.525058+0000","last_unstale":"2024-03-21T13:16:51.497148+0000","last_undegraded":"2024-03-21T13:16:51.497148+0000","last_fullsized":"2024-03-21T13:16:51.497148+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:15:15.202826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:15.202826+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":740,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.489556+0000","last_change":"2024-03-21T13:14:17.224206+0000","last_active":"2024-03-21T13:16:51.489556+0000","last_peered":"2024-03-21T13:16:51.489556+0000","last_clean":"2024-03-21T13:16:51.489556+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:16:51.489556+0000","last_undegraded":"2024-03-21T13:16:51.489556+0000","last_fullsized":"2024-03-21T13:16:51.489556+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:17.224086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:17.224086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"234'34","reported_seq":706,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:53.621414+0000","last_change":"2024-03-21T13:16:53.621472+0000","last_active":"2024-03-21T13:16:53.621414+0000","last_peered":"2024-03-21T13:16:53.621414+0000","last_clean":"2024-03-21T13:16:53.621414+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:16:53.621414+0000","last_undegraded":"2024-03-21T13:16:53.621414+0000","last_fullsized":"2024-03-21T13:16:53.621414+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'34","last_scrub_stamp":"2024-03-21T13:16:53.621335+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.621335+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:09.903027+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"406'3327","reported_seq":3368,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:55.448012+0000","last_change":"2024-03-21T13:16:13.904388+0000","last_active":"2024-03-21T13:16:55.448012+0000","last_peered":"2024-03-21T13:16:55.448012+0000","last_clean":"2024-03-21T13:16:55.448012+0000","last_became_active":"2024-03-21T13:16:13.904096+0000","last_became_peered":"2024-03-21T13:16:13.904096+0000","last_unstale":"2024-03-21T13:16:55.448012+0000","last_undegraded":"2024-03-21T13:16:55.448012+0000","last_fullsized":"2024-03-21T13:16:55.448012+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3327,"log_dups_size":0,"ondisk_log_size":3327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:39.867007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218038272,"num_objects":3327,"num_object_clones":0,"num_object_copies":9981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3327,"num_write_kb":212928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"375'2","reported_seq":744,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.480502+0000","last_change":"2024-03-21T13:15:14.313700+0000","last_active":"2024-03-21T13:16:51.480502+0000","last_peered":"2024-03-21T13:16:51.480502+0000","last_clean":"2024-03-21T13:16:51.480502+0000","last_became_active":"2024-03-21T13:12:29.502816+0000","last_became_peered":"2024-03-21T13:12:29.502816+0000","last_unstale":"2024-03-21T13:16:51.480502+0000","last_undegraded":"2024-03-21T13:16:51.480502+0000","last_fullsized":"2024-03-21T13:16:51.480502+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'2","last_scrub_stamp":"2024-03-21T13:15:14.313376+0000","last_deep_scrub":"375'2","last_deep_scrub_stamp":"2024-03-21T13:15:14.313376+0000","last_clean_scrub_stamp":"2024-03-21T13:15:14.313376+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":169,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.496882+0000","last_change":"2024-03-21T13:15:07.382672+0000","last_active":"2024-03-21T13:16:51.496882+0000","last_peered":"2024-03-21T13:16:51.496882+0000","last_clean":"2024-03-21T13:16:51.496882+0000","last_became_active":"2024-03-21T13:12:29.524087+0000","last_became_peered":"2024-03-21T13:12:29.524087+0000","last_unstale":"2024-03-21T13:16:51.496882+0000","last_undegraded":"2024-03-21T13:16:51.496882+0000","last_fullsized":"2024-03-21T13:16:51.496882+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:07.382536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:07.382536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":778,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.490037+0000","last_change":"2024-03-21T13:16:04.825360+0000","last_active":"2024-03-21T13:16:51.490037+0000","last_peered":"2024-03-21T13:16:51.490037+0000","last_clean":"2024-03-21T13:16:51.490037+0000","last_became_active":"2024-03-21T13:10:03.586593+0000","last_became_peered":"2024-03-21T13:10:03.586593+0000","last_unstale":"2024-03-21T13:16:51.490037+0000","last_undegraded":"2024-03-21T13:16:51.490037+0000","last_fullsized":"2024-03-21T13:16:51.490037+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:04.825280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:04.825280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:18.635897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":493,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483335+0000","last_change":"2024-03-21T13:14:11.615096+0000","last_active":"2024-03-21T13:16:51.483335+0000","last_peered":"2024-03-21T13:16:51.483335+0000","last_clean":"2024-03-21T13:16:51.483335+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:16:51.483335+0000","last_undegraded":"2024-03-21T13:16:51.483335+0000","last_fullsized":"2024-03-21T13:16:51.483335+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:14:11.611538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:11.611538+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"406'3350","reported_seq":3392,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:55.445247+0000","last_change":"2024-03-21T13:16:13.899357+0000","last_active":"2024-03-21T13:16:55.445247+0000","last_peered":"2024-03-21T13:16:55.445247+0000","last_clean":"2024-03-21T13:16:55.445247+0000","last_became_active":"2024-03-21T13:16:13.899059+0000","last_became_peered":"2024-03-21T13:16:13.899059+0000","last_unstale":"2024-03-21T13:16:55.445247+0000","last_undegraded":"2024-03-21T13:16:55.445247+0000","last_fullsized":"2024-03-21T13:16:55.445247+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:25.856013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219545600,"num_objects":3350,"num_object_clones":0,"num_object_copies":10050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3350,"num_write_kb":214400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":314,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.485610+0000","last_change":"2024-03-21T13:16:20.483084+0000","last_active":"2024-03-21T13:16:51.485610+0000","last_peered":"2024-03-21T13:16:51.485610+0000","last_clean":"2024-03-21T13:16:51.485610+0000","last_became_active":"2024-03-21T13:10:03.602752+0000","last_became_peered":"2024-03-21T13:10:03.602752+0000","last_unstale":"2024-03-21T13:16:51.485610+0000","last_undegraded":"2024-03-21T13:16:51.485610+0000","last_fullsized":"2024-03-21T13:16:51.485610+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:20.482943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.482943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.858705+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":650,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.490241+0000","last_change":"2024-03-21T13:15:59.662672+0000","last_active":"2024-03-21T13:16:51.490241+0000","last_peered":"2024-03-21T13:16:51.490241+0000","last_clean":"2024-03-21T13:16:51.490241+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:16:51.490241+0000","last_undegraded":"2024-03-21T13:16:51.490241+0000","last_fullsized":"2024-03-21T13:16:51.490241+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:59.662591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:59.662591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:10.639394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":755,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.483359+0000","last_change":"2024-03-21T13:15:02.678593+0000","last_active":"2024-03-21T13:16:51.483359+0000","last_peered":"2024-03-21T13:16:51.483359+0000","last_clean":"2024-03-21T13:16:51.483359+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:16:51.483359+0000","last_undegraded":"2024-03-21T13:16:51.483359+0000","last_fullsized":"2024-03-21T13:16:51.483359+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:02.678485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:15:02.678485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"391'39","reported_seq":324,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.484936+0000","last_change":"2024-03-21T13:16:14.266738+0000","last_active":"2024-03-21T13:16:51.484936+0000","last_peered":"2024-03-21T13:16:51.484936+0000","last_clean":"2024-03-21T13:16:51.484936+0000","last_became_active":"2024-03-21T13:10:03.591832+0000","last_became_peered":"2024-03-21T13:10:03.591832+0000","last_unstale":"2024-03-21T13:16:51.484936+0000","last_undegraded":"2024-03-21T13:16:51.484936+0000","last_fullsized":"2024-03-21T13:16:51.484936+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'39","last_scrub_stamp":"2024-03-21T13:16:14.266573+0000","last_deep_scrub":"391'39","last_deep_scrub_stamp":"2024-03-21T13:16:14.266573+0000","last_clean_scrub_stamp":"2024-03-21T13:16:14.266573+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:36.013148+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"406'3434","reported_seq":3475,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:55.438341+0000","last_change":"2024-03-21T13:16:13.904234+0000","last_active":"2024-03-21T13:16:55.438341+0000","last_peered":"2024-03-21T13:16:55.438341+0000","last_clean":"2024-03-21T13:16:55.438341+0000","last_became_active":"2024-03-21T13:16:13.903407+0000","last_became_peered":"2024-03-21T13:16:13.903407+0000","last_unstale":"2024-03-21T13:16:55.438341+0000","last_undegraded":"2024-03-21T13:16:55.438341+0000","last_fullsized":"2024-03-21T13:16:55.438341+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.659743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225050624,"num_objects":3434,"num_object_clones":0,"num_object_copies":10302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3434,"num_write_kb":219776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":312,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.484452+0000","last_change":"2024-03-21T13:16:18.322285+0000","last_active":"2024-03-21T13:16:51.484452+0000","last_peered":"2024-03-21T13:16:51.484452+0000","last_clean":"2024-03-21T13:16:51.484452+0000","last_became_active":"2024-03-21T13:12:29.499442+0000","last_became_peered":"2024-03-21T13:12:29.499442+0000","last_unstale":"2024-03-21T13:16:51.484452+0000","last_undegraded":"2024-03-21T13:16:51.484452+0000","last_fullsized":"2024-03-21T13:16:51.484452+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.322198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.322198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:40.851693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"331'13","reported_seq":404,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.484792+0000","last_change":"2024-03-21T13:16:21.250482+0000","last_active":"2024-03-21T13:16:51.484792+0000","last_peered":"2024-03-21T13:16:51.484792+0000","last_clean":"2024-03-21T13:16:51.484792+0000","last_became_active":"2024-03-21T13:12:29.507925+0000","last_became_peered":"2024-03-21T13:12:29.507925+0000","last_unstale":"2024-03-21T13:16:51.484792+0000","last_undegraded":"2024-03-21T13:16:51.484792+0000","last_fullsized":"2024-03-21T13:16:51.484792+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:16:21.250398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.250398+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:47.624689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":734,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.482911+0000","last_change":"2024-03-21T13:16:05.813771+0000","last_active":"2024-03-21T13:16:51.482911+0000","last_peered":"2024-03-21T13:16:51.482911+0000","last_clean":"2024-03-21T13:16:51.482911+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:16:51.482911+0000","last_undegraded":"2024-03-21T13:16:51.482911+0000","last_fullsized":"2024-03-21T13:16:51.482911+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:05.813657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.813657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.222745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"234'20","reported_seq":695,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-21T13:16:51.480763+0000","last_change":"2024-03-21T13:15:04.211127+0000","last_active":"2024-03-21T13:16:51.480763+0000","last_peered":"2024-03-21T13:16:51.480763+0000","last_clean":"2024-03-21T13:16:51.480763+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:16:51.480763+0000","last_undegraded":"2024-03-21T13:16:51.480763+0000","last_fullsized":"2024-03-21T13:16:51.480763+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:15:04.211011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:04.211011+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3559587840,"num_objects":54315,"num_object_clones":0,"num_object_copies":162945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54315,"num_write_kb":3476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10639704064,"data_stored":10639704064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54315,"ondisk_log_size":54315,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1087,"num_read_kb":1082,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1100,"num_read_kb":857,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":6656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":748,"ondisk_log_size":748,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168800,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2392144,"kb_used_data":1494664,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":91353008,"statfs":{"total":95995035648,"available":93545480192,"internally_reserved":0,"allocated":1530535936,"data_stored":1530021451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":919009738},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463142,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2838992,"kb_used_data":1933192,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":90906160,"statfs":{"total":95995035648,"available":93087907840,"internally_reserved":0,"allocated":1979588608,"data_stored":1979073832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":927529414},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575102,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738628,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2558472,"kb_used_data":1701248,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":91186680,"statfs":{"total":95995035648,"available":93375160320,"internally_reserved":0,"allocated":1742077952,"data_stored":1741570307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877787589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411578,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2086884,"kb_used_data":1293532,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":91658268,"statfs":{"total":95995035648,"available":93858066432,"internally_reserved":0,"allocated":1324576768,"data_stored":1324105071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182645,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2541000,"kb_used_data":1521344,"kb_used_omap":1,"kb_used_meta":1019646,"kb_avail":91204152,"statfs":{"total":95995035648,"available":93393051648,"internally_reserved":0,"allocated":1557856256,"data_stored":1557348543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1044117959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019119,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2251760,"kb_used_data":1268200,"kb_used_omap":1,"kb_used_meta":983550,"kb_avail":91493392,"statfs":{"total":95995035648,"available":93689233408,"internally_reserved":0,"allocated":1298636800,"data_stored":1298153294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1007155653},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724778,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1270504,"kb_used_data":1044960,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":92474648,"statfs":{"total":95995035648,"available":94694039552,"internally_reserved":0,"allocated":1070039040,"data_stored":1069562925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":230947269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1764229120,"data_stored":1764229120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1325203456,"data_stored":1325203456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1578893312,"data_stored":1578893312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1311244288,"data_stored":1311244288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1090125824,"data_stored":1090125824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2009202688,"data_stored":2009202688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1560805376,"data_stored":1560805376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:16:59.191 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-21T13:16:59.191 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:16:59.192 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:16:59.212 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:59.276 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:59.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:59.332 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 30 16 59570 59554 124.058 93.875 0.0101671 0.00805535 2024-03-21T13:16:59.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:59.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:59.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:59.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:59.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:16:59.553 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:59.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:16:59.654 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:16:59.690 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:59.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:16:59.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:16:59.826 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:16:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:59.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:16:59.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:59 smithi204 ceph-mon[33153]: 6.12 scrub starts 2024-03-21T13:16:59.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:16:59 smithi204 ceph-mon[33153]: 6.12 scrub ok 2024-03-21T13:16:59.835 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-21T13:16:59.881 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:59 smithi163 ceph-mon[40675]: 6.12 scrub starts 2024-03-21T13:16:59.881 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:16:59 smithi163 ceph-mon[40675]: 6.12 scrub ok 2024-03-21T13:16:59.883 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:59 smithi163 ceph-mon[29621]: 6.12 scrub starts 2024-03-21T13:16:59.883 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:16:59 smithi163 ceph-mon[29621]: 6.12 scrub ok 2024-03-21T13:16:59.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:16:59.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:16:59.969 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:16:59.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:00.069 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:00.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:00.107 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:00.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:00.208 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:00.263 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:00.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:00.274 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:17:00.332 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 31 16 60962 60946 122.863 87 0.0123415 0.0081342 2024-03-21T13:17:00.364 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:00.415 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:00.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:00.517 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:00.551 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:00.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:00.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:00.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:00.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:00.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:00.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:00 smithi204 ceph-mon[33153]: 3.19 deep-scrub starts 2024-03-21T13:17:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:00 smithi204 ceph-mon[33153]: 3.19 deep-scrub ok 2024-03-21T13:17:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:00 smithi204 ceph-mon[33153]: pgmap v861: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-21T13:17:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:00 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:17:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:00 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:17:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:00 smithi204 ceph-mon[33153]: osdmap e408: 8 total, 8 up, 7 in 2024-03-21T13:17:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:00 smithi204 ceph-mon[33153]: 3.1 deep-scrub starts 2024-03-21T13:17:00.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:00 smithi204 ceph-mon[33153]: 3.1 deep-scrub ok 2024-03-21T13:17:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[40675]: 3.19 deep-scrub starts 2024-03-21T13:17:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[40675]: 3.19 deep-scrub ok 2024-03-21T13:17:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[40675]: pgmap v861: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-21T13:17:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:17:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:17:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[40675]: osdmap e408: 8 total, 8 up, 7 in 2024-03-21T13:17:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[40675]: 3.1 deep-scrub starts 2024-03-21T13:17:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[40675]: 3.1 deep-scrub ok 2024-03-21T13:17:00.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[29621]: 3.19 deep-scrub starts 2024-03-21T13:17:00.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[29621]: 3.19 deep-scrub ok 2024-03-21T13:17:00.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[29621]: pgmap v861: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-21T13:17:00.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:17:00.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:17:00.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[29621]: osdmap e408: 8 total, 8 up, 7 in 2024-03-21T13:17:00.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[29621]: 3.1 deep-scrub starts 2024-03-21T13:17:00.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:00 smithi163 ceph-mon[29621]: 3.1 deep-scrub ok 2024-03-21T13:17:00.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:00.839 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:00.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:00.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:00.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:01.030 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:01.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:01.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:01.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:01.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:01.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:01.277 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:01.324 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:01.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:01.332 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 32 16 62248 62232 121.535 80.375 0.00722306 0.00822211 2024-03-21T13:17:01.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:01.465 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:01.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:01.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:01.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:01.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:01.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:01.676 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:17:01.681 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:17:01.713 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:01.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:01 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:17:01.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:01 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:17:01.723 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:17:01.764 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:17:01.765 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:01.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:01.807 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:17:01.849 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:17:01.868 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:01.891 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:17:01.902 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:01.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:01.927 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:17:01.934 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:17:01.949 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:01 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:17:01.976 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:17:01.980 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:17:01.980 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:17:01.980 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:17:01.981 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:17:01.981 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:17:01.981 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:17:01.981 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:17:01.981 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:17:02.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:02.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:02.070 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:02.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:02.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:02.213 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:02.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:02.315 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:02.332 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 33 16 63474 63458 120.174 76.625 0.0118041 0.00831481 2024-03-21T13:17:02.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:02.345 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:02.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:02.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:02.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:02.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:02.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:02.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:17:02 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:17:02.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:02.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:02.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:02.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[29621]: 3.8 deep-scrub starts 2024-03-21T13:17:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[29621]: 3.8 deep-scrub ok 2024-03-21T13:17:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[29621]: pgmap v863: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-21T13:17:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[29621]: 4.1 scrub starts 2024-03-21T13:17:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[29621]: 4.1 scrub ok 2024-03-21T13:17:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3702893184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:17:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[29621]: 4.19 scrub starts 2024-03-21T13:17:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[29621]: 4.19 scrub ok 2024-03-21T13:17:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:17:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:17:02.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:17:02.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[40675]: 3.8 deep-scrub starts 2024-03-21T13:17:02.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[40675]: 3.8 deep-scrub ok 2024-03-21T13:17:02.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[40675]: pgmap v863: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-21T13:17:02.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[40675]: 4.1 scrub starts 2024-03-21T13:17:02.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[40675]: 4.1 scrub ok 2024-03-21T13:17:02.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3702893184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:17:02.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[40675]: 4.19 scrub starts 2024-03-21T13:17:02.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[40675]: 4.19 scrub ok 2024-03-21T13:17:02.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:17:02.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:17:02.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:02 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:17:02.763 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:02.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:02.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:02.901 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:02.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:03.002 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:03.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:02 smithi204 ceph-mon[33153]: 3.8 deep-scrub starts 2024-03-21T13:17:03.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:02 smithi204 ceph-mon[33153]: 3.8 deep-scrub ok 2024-03-21T13:17:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:02 smithi204 ceph-mon[33153]: pgmap v863: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-21T13:17:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:02 smithi204 ceph-mon[33153]: 4.1 scrub starts 2024-03-21T13:17:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:02 smithi204 ceph-mon[33153]: 4.1 scrub ok 2024-03-21T13:17:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:02 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3702893184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:17:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:02 smithi204 ceph-mon[33153]: 4.19 scrub starts 2024-03-21T13:17:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:02 smithi204 ceph-mon[33153]: 4.19 scrub ok 2024-03-21T13:17:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:02 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:17:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:02 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:17:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:02 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:17:03.041 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:03.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:03.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:03.143 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:03.175 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:03.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:03.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:03.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:03.313 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:03.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:03.332 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 34 16 64889 64873 119.24 88.4375 0.0144402 0.00838152 2024-03-21T13:17:03.415 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:03.453 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:03.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:03.555 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:03.585 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:03.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:03.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:03.632 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:03.632 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:03 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:17:03.632 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:03 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:17:03.632 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:03 smithi204 ceph-mon[33153]: 5.4 scrub starts 2024-03-21T13:17:03.632 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:03 smithi204 ceph-mon[33153]: 5.4 scrub ok 2024-03-21T13:17:03.632 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:03 smithi204 ceph-mon[33153]: 5.2 scrub starts 2024-03-21T13:17:03.632 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:03 smithi204 ceph-mon[33153]: 5.2 scrub ok 2024-03-21T13:17:03.632 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:03 smithi204 ceph-mon[33153]: pgmap v864: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-21T13:17:03.686 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:03.729 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:03.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:03.830 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:03.854 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:03.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:03.956 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:03.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:03.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:03.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:17:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:17:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[29621]: 5.4 scrub starts 2024-03-21T13:17:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[29621]: 5.4 scrub ok 2024-03-21T13:17:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[29621]: 5.2 scrub starts 2024-03-21T13:17:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[29621]: 5.2 scrub ok 2024-03-21T13:17:04.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[29621]: pgmap v864: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-21T13:17:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:17:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:17:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[40675]: 5.4 scrub starts 2024-03-21T13:17:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[40675]: 5.4 scrub ok 2024-03-21T13:17:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[40675]: 5.2 scrub starts 2024-03-21T13:17:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[40675]: 5.2 scrub ok 2024-03-21T13:17:04.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:03 smithi163 ceph-mon[40675]: pgmap v864: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-21T13:17:04.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:04.126 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:17:04.126 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:17:04.129 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:04.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:04.231 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:04.269 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:04.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:04.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:04.332 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 35 15 66075 66060 117.953 74.1875 0.00645858 0.00847336 2024-03-21T13:17:04.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:04.409 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:04.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:04.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:04.511 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:04.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:04.536 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:04.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:04.637 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:04.669 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:04.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:04.771 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:04.795 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:04.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:04.815 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:04 smithi204 ceph-mon[33153]: 4.17 scrub starts 2024-03-21T13:17:04.815 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:04 smithi204 ceph-mon[33153]: 4.17 scrub ok 2024-03-21T13:17:04.815 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:04 smithi204 ceph-mon[33153]: 4.3 scrub starts 2024-03-21T13:17:04.815 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:04 smithi204 ceph-mon[33153]: 4.3 scrub ok 2024-03-21T13:17:04.815 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:04 smithi204 ceph-mon[33153]: osdmap e409: 8 total, 8 up, 7 in 2024-03-21T13:17:04.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-21T13:17:04.895 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:04.900 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:17:04.924 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:04.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:04.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:04 smithi163 ceph-mon[29621]: 4.17 scrub starts 2024-03-21T13:17:04.955 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:04 smithi163 ceph-mon[29621]: 4.17 scrub ok 2024-03-21T13:17:04.955 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:04 smithi163 ceph-mon[29621]: 4.3 scrub starts 2024-03-21T13:17:04.955 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:04 smithi163 ceph-mon[29621]: 4.3 scrub ok 2024-03-21T13:17:04.955 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:04 smithi163 ceph-mon[29621]: osdmap e409: 8 total, 8 up, 7 in 2024-03-21T13:17:04.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:04 smithi163 ceph-mon[40675]: 4.17 scrub starts 2024-03-21T13:17:04.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:04 smithi163 ceph-mon[40675]: 4.17 scrub ok 2024-03-21T13:17:04.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:04 smithi163 ceph-mon[40675]: 4.3 scrub starts 2024-03-21T13:17:04.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:04 smithi163 ceph-mon[40675]: 4.3 scrub ok 2024-03-21T13:17:04.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:04 smithi163 ceph-mon[40675]: osdmap e409: 8 total, 8 up, 7 in 2024-03-21T13:17:05.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:05.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:05.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:05.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:05.209 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:17:05.213 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:05.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:05.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:05.332 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 36 16 67071 67055 116.403 62.1875 0.00741087 0.00858604 2024-03-21T13:17:05.351 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:05.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:05.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:05.453 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:05.490 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:05.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:05.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:05.592 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:05.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:05.617 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:05 smithi204 ceph-mon[33153]: 3.1e scrub starts 2024-03-21T13:17:05.617 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:05 smithi204 ceph-mon[33153]: 3.1e scrub ok 2024-03-21T13:17:05.617 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:05 smithi204 ceph-mon[33153]: pgmap v866: 148 pgs: 148 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 690 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-21T13:17:05.617 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:17:05.617 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:05.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:05.756 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:05.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:05.857 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:05.895 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:05.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:05.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:05.997 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:05 smithi163 ceph-mon[29621]: 3.1e scrub starts 2024-03-21T13:17:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:05 smithi163 ceph-mon[29621]: 3.1e scrub ok 2024-03-21T13:17:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:05 smithi163 ceph-mon[29621]: pgmap v866: 148 pgs: 148 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 690 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-21T13:17:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:17:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:06.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:05 smithi163 ceph-mon[40675]: 3.1e scrub starts 2024-03-21T13:17:06.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:05 smithi163 ceph-mon[40675]: 3.1e scrub ok 2024-03-21T13:17:06.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:05 smithi163 ceph-mon[40675]: pgmap v866: 148 pgs: 148 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 690 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-21T13:17:06.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:17:06.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:06.012 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:17:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:17:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:17:06.033 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:06.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:06.135 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:06.168 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:06.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:06.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:06.269 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:06.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:06.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:06.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:06.332 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 37 16 68145 68129 115.071 67.125 0.00272306 0.00867594 2024-03-21T13:17:06.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:06.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:06.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:06.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:06.538 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:06.568 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:06.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:06.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:06.670 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:06.683 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:06 smithi204 ceph-mon[33153]: 3.18 scrub starts 2024-03-21T13:17:06.683 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:06 smithi204 ceph-mon[33153]: 3.18 scrub ok 2024-03-21T13:17:06.683 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:06 smithi204 ceph-mon[33153]: 6.1c scrub starts 2024-03-21T13:17:06.683 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:06 smithi204 ceph-mon[33153]: 6.1c scrub ok 2024-03-21T13:17:06.683 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:06 smithi204 ceph-mon[33153]: osdmap e410: 8 total, 8 up, 7 in 2024-03-21T13:17:06.706 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:06.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:06.808 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:06.864 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:06.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:06.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:06.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:07.000 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:07.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:07.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:07.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:06 smithi163 ceph-mon[29621]: 3.18 scrub starts 2024-03-21T13:17:07.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:06 smithi163 ceph-mon[29621]: 3.18 scrub ok 2024-03-21T13:17:07.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:06 smithi163 ceph-mon[29621]: 6.1c scrub starts 2024-03-21T13:17:07.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:06 smithi163 ceph-mon[29621]: 6.1c scrub ok 2024-03-21T13:17:07.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:06 smithi163 ceph-mon[29621]: osdmap e410: 8 total, 8 up, 7 in 2024-03-21T13:17:07.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:06 smithi163 ceph-mon[40675]: 3.18 scrub starts 2024-03-21T13:17:07.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:06 smithi163 ceph-mon[40675]: 3.18 scrub ok 2024-03-21T13:17:07.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:06 smithi163 ceph-mon[40675]: 6.1c scrub starts 2024-03-21T13:17:07.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:06 smithi163 ceph-mon[40675]: 6.1c scrub ok 2024-03-21T13:17:07.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:06 smithi163 ceph-mon[40675]: osdmap e410: 8 total, 8 up, 7 in 2024-03-21T13:17:07.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:07.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:07.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:07.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:07.153 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:17:07.153 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:17:07.240 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:07.264 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:07.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:07.333 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 38 16 69170 69154 113.729 64.0625 0.0164848 0.0087884 2024-03-21T13:17:07.366 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:07.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:07.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:07.507 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:07.547 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:07.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:07.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:07.649 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:07.686 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:07.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:07.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:07.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:07.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:07.868 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:07 smithi204 ceph-mon[33153]: 3.7 scrub starts 2024-03-21T13:17:07.868 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:07 smithi204 ceph-mon[33153]: 3.7 scrub ok 2024-03-21T13:17:07.868 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:07 smithi204 ceph-mon[33153]: 6.9 scrub starts 2024-03-21T13:17:07.868 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:07 smithi204 ceph-mon[33153]: pgmap v868: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-21T13:17:07.869 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:07 smithi204 ceph-mon[33153]: 6.9 scrub ok 2024-03-21T13:17:07.895 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-21T13:17:07.919 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:07 smithi163 ceph-mon[29621]: 3.7 scrub starts 2024-03-21T13:17:07.919 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:07 smithi163 ceph-mon[29621]: 3.7 scrub ok 2024-03-21T13:17:07.919 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:07 smithi163 ceph-mon[29621]: 6.9 scrub starts 2024-03-21T13:17:07.919 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:07 smithi163 ceph-mon[29621]: pgmap v868: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-21T13:17:07.919 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:07 smithi163 ceph-mon[29621]: 6.9 scrub ok 2024-03-21T13:17:07.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:07 smithi163 ceph-mon[40675]: 3.7 scrub starts 2024-03-21T13:17:07.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:07 smithi163 ceph-mon[40675]: 3.7 scrub ok 2024-03-21T13:17:07.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:07 smithi163 ceph-mon[40675]: 6.9 scrub starts 2024-03-21T13:17:07.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:07 smithi163 ceph-mon[40675]: pgmap v868: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-21T13:17:07.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:07 smithi163 ceph-mon[40675]: 6.9 scrub ok 2024-03-21T13:17:07.925 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:07.948 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:07.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:08.050 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:08.094 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:08.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:08.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:08.196 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:08.223 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:08.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:08.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:08.325 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:08.332 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 39 16 70354 70338 112.71 74 0.00575546 0.00886687 2024-03-21T13:17:08.363 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:08.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:08.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:08.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:08.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:08.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:08.590 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:08.627 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:08.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:08.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:08.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:08.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:08.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:08.907 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:08.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:08 smithi204 ceph-mon[33153]: 6.6 scrub starts 2024-03-21T13:17:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:08 smithi204 ceph-mon[33153]: 6.6 scrub ok 2024-03-21T13:17:09.008 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:09.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:08 smithi163 ceph-mon[29621]: 6.6 scrub starts 2024-03-21T13:17:09.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:08 smithi163 ceph-mon[29621]: 6.6 scrub ok 2024-03-21T13:17:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:08 smithi163 ceph-mon[40675]: 6.6 scrub starts 2024-03-21T13:17:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:08 smithi163 ceph-mon[40675]: 6.6 scrub ok 2024-03-21T13:17:09.045 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:09.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:09.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:09.147 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:09.175 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:17:09.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:09.219 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:09.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:09.320 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:09.331 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:17:09.331 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:17:09.333 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:17:09.333552+0000 min lat: 0.00136681 max lat: 0.178874 avg lat: 0.00894877 2024-03-21T13:17:09.333 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:17:09.333 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 40 16 71508 71492 111.695 72.125 0.0120503 0.00894877 2024-03-21T13:17:09.370 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:09.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:09.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:09.472 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:09.504 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:09.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:09.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:09.606 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:09.646 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:09.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:09.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:09.748 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:09.786 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:09.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:09.887 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:09.912 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:09.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:09.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:10.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:09 smithi204 ceph-mon[33153]: 6.14 scrub starts 2024-03-21T13:17:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:09 smithi204 ceph-mon[33153]: 6.14 scrub ok 2024-03-21T13:17:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:09 smithi204 ceph-mon[33153]: 5.7 deep-scrub starts 2024-03-21T13:17:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:09 smithi204 ceph-mon[33153]: 5.7 deep-scrub ok 2024-03-21T13:17:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:09 smithi204 ceph-mon[33153]: 6.11 scrub starts 2024-03-21T13:17:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:09 smithi204 ceph-mon[33153]: 6.11 scrub ok 2024-03-21T13:17:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:09 smithi204 ceph-mon[33153]: 4.a scrub starts 2024-03-21T13:17:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:09 smithi204 ceph-mon[33153]: 4.a scrub ok 2024-03-21T13:17:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:09 smithi204 ceph-mon[33153]: 3.1b scrub starts 2024-03-21T13:17:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:09 smithi204 ceph-mon[33153]: 3.1b scrub ok 2024-03-21T13:17:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:09 smithi204 ceph-mon[33153]: pgmap v869: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-21T13:17:10.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[29621]: 6.14 scrub starts 2024-03-21T13:17:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[29621]: 6.14 scrub ok 2024-03-21T13:17:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[29621]: 5.7 deep-scrub starts 2024-03-21T13:17:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[29621]: 5.7 deep-scrub ok 2024-03-21T13:17:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[29621]: 6.11 scrub starts 2024-03-21T13:17:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[29621]: 6.11 scrub ok 2024-03-21T13:17:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[29621]: 4.a scrub starts 2024-03-21T13:17:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[29621]: 4.a scrub ok 2024-03-21T13:17:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[29621]: 3.1b scrub starts 2024-03-21T13:17:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[29621]: 3.1b scrub ok 2024-03-21T13:17:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[29621]: pgmap v869: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-21T13:17:10.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[40675]: 6.14 scrub starts 2024-03-21T13:17:10.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[40675]: 6.14 scrub ok 2024-03-21T13:17:10.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[40675]: 5.7 deep-scrub starts 2024-03-21T13:17:10.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[40675]: 5.7 deep-scrub ok 2024-03-21T13:17:10.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[40675]: 6.11 scrub starts 2024-03-21T13:17:10.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[40675]: 6.11 scrub ok 2024-03-21T13:17:10.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[40675]: 4.a scrub starts 2024-03-21T13:17:10.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[40675]: 4.a scrub ok 2024-03-21T13:17:10.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[40675]: 3.1b scrub starts 2024-03-21T13:17:10.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[40675]: 3.1b scrub ok 2024-03-21T13:17:10.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:09 smithi163 ceph-mon[40675]: pgmap v869: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-21T13:17:10.013 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:10.048 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:10.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:10.150 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:10.187 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:10.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:10.289 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:10.326 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:10.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:10.333 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 41 16 72724 72708 110.824 76 0.0103617 0.0090184 2024-03-21T13:17:10.427 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:10.456 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:10.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:10.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:10.558 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:10.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:10.593 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:10.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:10.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:10.733 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:10.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:10.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:10.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:10.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:10.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:10 smithi204 ceph-mon[33153]: 6.3 scrub starts 2024-03-21T13:17:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:10 smithi204 ceph-mon[33153]: 4.1e deep-scrub starts 2024-03-21T13:17:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:10 smithi204 ceph-mon[33153]: 4.1e deep-scrub ok 2024-03-21T13:17:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:10 smithi204 ceph-mon[33153]: 6.3 scrub ok 2024-03-21T13:17:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:10 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:17:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:10 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:17:11.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:10 smithi163 ceph-mon[29621]: 6.3 scrub starts 2024-03-21T13:17:11.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:10 smithi163 ceph-mon[29621]: 4.1e deep-scrub starts 2024-03-21T13:17:11.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:10 smithi163 ceph-mon[29621]: 4.1e deep-scrub ok 2024-03-21T13:17:11.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:10 smithi163 ceph-mon[29621]: 6.3 scrub ok 2024-03-21T13:17:11.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:10 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:17:11.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:10 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:17:11.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:10 smithi163 ceph-mon[40675]: 6.3 scrub starts 2024-03-21T13:17:11.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:10 smithi163 ceph-mon[40675]: 4.1e deep-scrub starts 2024-03-21T13:17:11.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:10 smithi163 ceph-mon[40675]: 4.1e deep-scrub ok 2024-03-21T13:17:11.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:10 smithi163 ceph-mon[40675]: 6.3 scrub ok 2024-03-21T13:17:11.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:10 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:17:11.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:10 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:17:11.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:11.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:11.116 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:11.155 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:11.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:11.256 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:11.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:11.333 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 42 16 73936 73920 109.989 75.75 0.0114124 0.00908626 2024-03-21T13:17:11.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:11.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:11.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:11.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:11.567 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:11.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:11.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:11.668 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:11.705 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:11.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:11.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:11.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:11.845 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:11.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:11.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:11.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:11 smithi163 ceph-mon[29621]: 4.1b scrub starts 2024-03-21T13:17:11.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:11 smithi163 ceph-mon[29621]: 4.1b scrub ok 2024-03-21T13:17:11.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:11 smithi163 ceph-mon[29621]: osdmap e411: 8 total, 8 up, 7 in 2024-03-21T13:17:11.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:11 smithi163 ceph-mon[29621]: pgmap v871: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-21T13:17:11.918 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:11 smithi163 ceph-mon[29621]: osdmap e412: 8 total, 8 up, 7 in 2024-03-21T13:17:11.918 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:11 smithi163 ceph-mon[40675]: 4.1b scrub starts 2024-03-21T13:17:11.918 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:11 smithi163 ceph-mon[40675]: 4.1b scrub ok 2024-03-21T13:17:11.918 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:11 smithi163 ceph-mon[40675]: osdmap e411: 8 total, 8 up, 7 in 2024-03-21T13:17:11.918 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:11 smithi163 ceph-mon[40675]: pgmap v871: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-21T13:17:11.918 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:11 smithi163 ceph-mon[40675]: osdmap e412: 8 total, 8 up, 7 in 2024-03-21T13:17:11.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:11.981 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:11.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:11 smithi204 ceph-mon[33153]: 4.1b scrub starts 2024-03-21T13:17:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:11 smithi204 ceph-mon[33153]: 4.1b scrub ok 2024-03-21T13:17:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:11 smithi204 ceph-mon[33153]: osdmap e411: 8 total, 8 up, 7 in 2024-03-21T13:17:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:11 smithi204 ceph-mon[33153]: pgmap v871: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-21T13:17:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:11 smithi204 ceph-mon[33153]: osdmap e412: 8 total, 8 up, 7 in 2024-03-21T13:17:12.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:12.116 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:12.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:12.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:12.218 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:12.241 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:17:12.251 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:12.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:12.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:12.253 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:17:12.333 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 43 16 75241 75225 109.328 81.5625 0.0130515 0.00914226 2024-03-21T13:17:12.353 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:12.394 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:12.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:12.495 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:12.531 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:12.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:12.632 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:12.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:17:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:17:12.667 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:12.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:12.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:12.802 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:12.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:12.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:12.940 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:12.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:12 smithi204 ceph-mon[33153]: 4.13 scrub starts 2024-03-21T13:17:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:12 smithi204 ceph-mon[33153]: 4.13 scrub ok 2024-03-21T13:17:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:12 smithi204 ceph-mon[33153]: 5.c scrub starts 2024-03-21T13:17:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:12 smithi204 ceph-mon[33153]: 5.c scrub ok 2024-03-21T13:17:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:12 smithi204 ceph-mon[33153]: from='client.25811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:17:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:12 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:12 smithi163 ceph-mon[29621]: 4.13 scrub starts 2024-03-21T13:17:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:12 smithi163 ceph-mon[29621]: 4.13 scrub ok 2024-03-21T13:17:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:12 smithi163 ceph-mon[29621]: 5.c scrub starts 2024-03-21T13:17:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:12 smithi163 ceph-mon[29621]: 5.c scrub ok 2024-03-21T13:17:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:12 smithi163 ceph-mon[29621]: from='client.25811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:17:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:12 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:13.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:12 smithi163 ceph-mon[40675]: 4.13 scrub starts 2024-03-21T13:17:13.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:12 smithi163 ceph-mon[40675]: 4.13 scrub ok 2024-03-21T13:17:13.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:12 smithi163 ceph-mon[40675]: 5.c scrub starts 2024-03-21T13:17:13.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:12 smithi163 ceph-mon[40675]: 5.c scrub ok 2024-03-21T13:17:13.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:12 smithi163 ceph-mon[40675]: from='client.25811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:17:13.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:12 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:13.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:13.084 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:13.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:13.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:13.186 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:13.227 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:13.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:13.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:13.329 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:13.333 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 44 16 76569 76553 108.729 83 0.010585 0.00919256 2024-03-21T13:17:13.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:13.372 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:13.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:13.473 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:13.507 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:13.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:13.607 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:13.646 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:13.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:13.747 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:13.777 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:13.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:13.875 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:13 smithi204 ceph-mon[33153]: 5.1e scrub starts 2024-03-21T13:17:13.875 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:13 smithi204 ceph-mon[33153]: 5.1e scrub ok 2024-03-21T13:17:13.875 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:13 smithi204 ceph-mon[33153]: 5.18 scrub starts 2024-03-21T13:17:13.875 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:13 smithi204 ceph-mon[33153]: 5.18 scrub ok 2024-03-21T13:17:13.876 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:13 smithi204 ceph-mon[33153]: pgmap v873: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 789 B/s rd, 55 MiB/s wr, 873 op/s 2024-03-21T13:17:13.879 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:13.916 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:13.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:14.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:13 smithi163 ceph-mon[29621]: 5.1e scrub starts 2024-03-21T13:17:14.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:13 smithi163 ceph-mon[29621]: 5.1e scrub ok 2024-03-21T13:17:14.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:13 smithi163 ceph-mon[29621]: 5.18 scrub starts 2024-03-21T13:17:14.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:13 smithi163 ceph-mon[29621]: 5.18 scrub ok 2024-03-21T13:17:14.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:13 smithi163 ceph-mon[29621]: pgmap v873: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 789 B/s rd, 55 MiB/s wr, 873 op/s 2024-03-21T13:17:14.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:13 smithi163 ceph-mon[40675]: 5.1e scrub starts 2024-03-21T13:17:14.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:13 smithi163 ceph-mon[40675]: 5.1e scrub ok 2024-03-21T13:17:14.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:13 smithi163 ceph-mon[40675]: 5.18 scrub starts 2024-03-21T13:17:14.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:13 smithi163 ceph-mon[40675]: 5.18 scrub ok 2024-03-21T13:17:14.013 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:13 smithi163 ceph-mon[40675]: pgmap v873: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 789 B/s rd, 55 MiB/s wr, 873 op/s 2024-03-21T13:17:14.017 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:14.054 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:14.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:14.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:14.156 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:14.191 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:14.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:14.292 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:14.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:14.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:14.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:14.333 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 45 16 77788 77772 108.006 76.1875 0.0263989 0.00925263 2024-03-21T13:17:14.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:14.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:14.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:14.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:14.562 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":872,"stamp":"2024-03-21T13:17:11.482895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790688168,"num_objects":73289,"num_object_clones":0,"num_object_copies":219867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73289,"num_whiteouts":0,"num_read":2438,"num_read_kb":2284,"num_write":73992,"num_write_kb":4680285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":1779777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73082,"ondisk_log_size":73082,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19632716,"kb_used_data":13884816,"kb_used_omap":10,"kb_used_meta":5747765,"kb_avail":636583348,"statfs":{"total":671965249536,"available":651861348352,"internally_reserved":0,"allocated":14218051584,"data_stored":14214424147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":5885711467},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":740818944,"num_objects":11304,"num_object_clones":0,"num_object_copies":33912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11304,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11304,"num_write_kb":723456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.516414"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":177,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.494826+0000","last_change":"2024-03-21T13:14:59.311326+0000","last_active":"2024-03-21T13:17:06.494826+0000","last_peered":"2024-03-21T13:17:06.494826+0000","last_clean":"2024-03-21T13:17:06.494826+0000","last_became_active":"2024-03-21T13:12:29.525755+0000","last_became_peered":"2024-03-21T13:12:29.525755+0000","last_unstale":"2024-03-21T13:17:06.494826+0000","last_undegraded":"2024-03-21T13:17:06.494826+0000","last_fullsized":"2024-03-21T13:17:06.494826+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:59.311141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:14:59.311141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":185,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.493364+0000","last_change":"2024-03-21T13:16:21.204738+0000","last_active":"2024-03-21T13:17:06.493364+0000","last_peered":"2024-03-21T13:17:06.493364+0000","last_clean":"2024-03-21T13:17:06.493364+0000","last_became_active":"2024-03-21T13:12:29.510271+0000","last_became_peered":"2024-03-21T13:12:29.510271+0000","last_unstale":"2024-03-21T13:17:06.493364+0000","last_undegraded":"2024-03-21T13:17:06.493364+0000","last_fullsized":"2024-03-21T13:17:06.493364+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:21.204657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.204657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:31.150802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":209,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.624533+0000","last_change":"2024-03-21T13:16:02.044946+0000","last_active":"2024-03-21T13:17:10.624533+0000","last_peered":"2024-03-21T13:17:10.624533+0000","last_clean":"2024-03-21T13:17:10.624533+0000","last_became_active":"2024-03-21T13:15:04.462014+0000","last_became_peered":"2024-03-21T13:15:04.462014+0000","last_unstale":"2024-03-21T13:17:10.624533+0000","last_undegraded":"2024-03-21T13:17:10.624533+0000","last_fullsized":"2024-03-21T13:17:10.624533+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:02.044853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:02.044853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:31.722187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"234'19","reported_seq":830,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.490838+0000","last_change":"2024-03-21T13:16:09.694036+0000","last_active":"2024-03-21T13:17:06.490838+0000","last_peered":"2024-03-21T13:17:06.490838+0000","last_clean":"2024-03-21T13:17:06.490838+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:17:06.490838+0000","last_undegraded":"2024-03-21T13:17:06.490838+0000","last_fullsized":"2024-03-21T13:17:06.490838+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:09.693933+0000","last_deep_scrub":"234'19","last_deep_scrub_stamp":"2024-03-21T13:16:09.693933+0000","last_clean_scrub_stamp":"2024-03-21T13:16:09.693933+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.080379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":731,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.626921+0000","last_change":"2024-03-21T13:15:02.457035+0000","last_active":"2024-03-21T13:17:10.626921+0000","last_peered":"2024-03-21T13:17:10.626921+0000","last_clean":"2024-03-21T13:17:10.626921+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:17:10.626921+0000","last_undegraded":"2024-03-21T13:17:10.626921+0000","last_fullsized":"2024-03-21T13:17:10.626921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:02.456902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:15:02.456902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":679,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.501608+0000","last_change":"2024-03-21T13:14:05.582597+0000","last_active":"2024-03-21T13:17:06.501608+0000","last_peered":"2024-03-21T13:17:06.501608+0000","last_clean":"2024-03-21T13:17:06.501608+0000","last_became_active":"2024-03-21T13:10:03.591543+0000","last_became_peered":"2024-03-21T13:10:03.591543+0000","last_unstale":"2024-03-21T13:17:06.501608+0000","last_undegraded":"2024-03-21T13:17:06.501608+0000","last_fullsized":"2024-03-21T13:17:06.501608+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:05.582473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:14:05.582473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":772,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627802+0000","last_change":"2024-03-21T13:17:03.752054+0000","last_active":"2024-03-21T13:17:10.627802+0000","last_peered":"2024-03-21T13:17:10.627802+0000","last_clean":"2024-03-21T13:17:10.627802+0000","last_became_active":"2024-03-21T13:10:03.610318+0000","last_became_peered":"2024-03-21T13:10:03.610318+0000","last_unstale":"2024-03-21T13:17:10.627802+0000","last_undegraded":"2024-03-21T13:17:10.627802+0000","last_fullsized":"2024-03-21T13:17:10.627802+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:17:03.751920+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_clean_scrub_stamp":"2024-03-21T13:17:03.751920+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:15.990057+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":268,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.490582+0000","last_change":"2024-03-21T13:17:01.724193+0000","last_active":"2024-03-21T13:17:06.490582+0000","last_peered":"2024-03-21T13:17:06.490582+0000","last_clean":"2024-03-21T13:17:06.490582+0000","last_became_active":"2024-03-21T13:11:21.702209+0000","last_became_peered":"2024-03-21T13:11:21.702209+0000","last_unstale":"2024-03-21T13:17:06.490582+0000","last_undegraded":"2024-03-21T13:17:06.490582+0000","last_fullsized":"2024-03-21T13:17:06.490582+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:17:01.724037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:01.724037+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:14.326589+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":437,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.622764+0000","last_change":"2024-03-21T13:16:20.952006+0000","last_active":"2024-03-21T13:17:10.622764+0000","last_peered":"2024-03-21T13:17:10.622764+0000","last_clean":"2024-03-21T13:17:10.622764+0000","last_became_active":"2024-03-21T13:15:04.459860+0000","last_became_peered":"2024-03-21T13:15:04.459860+0000","last_unstale":"2024-03-21T13:17:10.622764+0000","last_undegraded":"2024-03-21T13:17:10.622764+0000","last_fullsized":"2024-03-21T13:17:10.622764+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:20.951893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.951893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:35.358793+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":177,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.493895+0000","last_change":"2024-03-21T13:15:12.302278+0000","last_active":"2024-03-21T13:17:06.493895+0000","last_peered":"2024-03-21T13:17:06.493895+0000","last_clean":"2024-03-21T13:17:06.493895+0000","last_became_active":"2024-03-21T13:12:29.518784+0000","last_became_peered":"2024-03-21T13:12:29.518784+0000","last_unstale":"2024-03-21T13:17:06.493895+0000","last_undegraded":"2024-03-21T13:17:06.493895+0000","last_fullsized":"2024-03-21T13:17:06.493895+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:12.302013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:15:12.302013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":177,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.494905+0000","last_change":"2024-03-21T13:15:18.085333+0000","last_active":"2024-03-21T13:17:06.494905+0000","last_peered":"2024-03-21T13:17:06.494905+0000","last_clean":"2024-03-21T13:17:06.494905+0000","last_became_active":"2024-03-21T13:12:29.526129+0000","last_became_peered":"2024-03-21T13:12:29.526129+0000","last_unstale":"2024-03-21T13:17:06.494905+0000","last_undegraded":"2024-03-21T13:17:06.494905+0000","last_fullsized":"2024-03-21T13:17:06.494905+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:18.085215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:15:18.085215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":742,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.626449+0000","last_change":"2024-03-21T13:16:07.739334+0000","last_active":"2024-03-21T13:17:10.626449+0000","last_peered":"2024-03-21T13:17:10.626449+0000","last_clean":"2024-03-21T13:17:10.626449+0000","last_became_active":"2024-03-21T13:01:27.368120+0000","last_became_peered":"2024-03-21T13:01:27.368120+0000","last_unstale":"2024-03-21T13:17:10.626449+0000","last_undegraded":"2024-03-21T13:17:10.626449+0000","last_fullsized":"2024-03-21T13:17:10.626449+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:07.739235+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.739235+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.631193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":185,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.493452+0000","last_change":"2024-03-21T13:16:56.397703+0000","last_active":"2024-03-21T13:17:06.493452+0000","last_peered":"2024-03-21T13:17:06.493452+0000","last_clean":"2024-03-21T13:17:06.493452+0000","last_became_active":"2024-03-21T13:12:29.508520+0000","last_became_peered":"2024-03-21T13:12:29.508520+0000","last_unstale":"2024-03-21T13:17:06.493452+0000","last_undegraded":"2024-03-21T13:17:06.493452+0000","last_fullsized":"2024-03-21T13:17:06.493452+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:56.397585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:56.397585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:13.161134+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":752,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.490036+0000","last_change":"2024-03-21T13:15:58.658853+0000","last_active":"2024-03-21T13:17:06.490036+0000","last_peered":"2024-03-21T13:17:06.490036+0000","last_clean":"2024-03-21T13:17:06.490036+0000","last_became_active":"2024-03-21T13:02:14.335741+0000","last_became_peered":"2024-03-21T13:02:14.335741+0000","last_unstale":"2024-03-21T13:17:06.490036+0000","last_undegraded":"2024-03-21T13:17:06.490036+0000","last_fullsized":"2024-03-21T13:17:06.490036+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:58.658749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:58.658749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":177,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.493543+0000","last_change":"2024-03-21T13:16:04.184426+0000","last_active":"2024-03-21T13:17:06.493543+0000","last_peered":"2024-03-21T13:17:06.493543+0000","last_clean":"2024-03-21T13:17:06.493543+0000","last_became_active":"2024-03-21T13:12:29.524473+0000","last_became_peered":"2024-03-21T13:12:29.524473+0000","last_unstale":"2024-03-21T13:17:06.493543+0000","last_undegraded":"2024-03-21T13:17:06.493543+0000","last_fullsized":"2024-03-21T13:17:06.493543+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:04.184314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:04.184314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:12.162233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"234'9","reported_seq":770,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.626588+0000","last_change":"2024-03-21T13:17:09.946959+0000","last_active":"2024-03-21T13:17:10.626588+0000","last_peered":"2024-03-21T13:17:10.626588+0000","last_clean":"2024-03-21T13:17:10.626588+0000","last_became_active":"2024-03-21T13:10:03.595745+0000","last_became_peered":"2024-03-21T13:10:03.595745+0000","last_unstale":"2024-03-21T13:17:10.626588+0000","last_undegraded":"2024-03-21T13:17:10.626588+0000","last_fullsized":"2024-03-21T13:17:10.626588+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'9","last_scrub_stamp":"2024-03-21T13:17:09.946815+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:17:09.946815+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:13.523384+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":783,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.492552+0000","last_change":"2024-03-21T13:16:06.811101+0000","last_active":"2024-03-21T13:17:06.492552+0000","last_peered":"2024-03-21T13:17:06.492552+0000","last_clean":"2024-03-21T13:17:06.492552+0000","last_became_active":"2024-03-21T13:12:29.520189+0000","last_became_peered":"2024-03-21T13:12:29.520189+0000","last_unstale":"2024-03-21T13:17:06.492552+0000","last_undegraded":"2024-03-21T13:17:06.492552+0000","last_fullsized":"2024-03-21T13:17:06.492552+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:06.810993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:23.865785+0000","last_clean_scrub_stamp":"2024-03-21T13:16:06.810993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:18.896276+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":669,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.490750+0000","last_change":"2024-03-21T13:16:07.756974+0000","last_active":"2024-03-21T13:17:06.490750+0000","last_peered":"2024-03-21T13:17:06.490750+0000","last_clean":"2024-03-21T13:17:06.490750+0000","last_became_active":"2024-03-21T13:12:29.520241+0000","last_became_peered":"2024-03-21T13:12:29.520241+0000","last_unstale":"2024-03-21T13:17:06.490750+0000","last_undegraded":"2024-03-21T13:17:06.490750+0000","last_fullsized":"2024-03-21T13:17:06.490750+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:07.756896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.756896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.528460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":216,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.623485+0000","last_change":"2024-03-21T13:17:08.894621+0000","last_active":"2024-03-21T13:17:10.623485+0000","last_peered":"2024-03-21T13:17:10.623485+0000","last_clean":"2024-03-21T13:17:10.623485+0000","last_became_active":"2024-03-21T13:15:04.464586+0000","last_became_peered":"2024-03-21T13:15:04.464586+0000","last_unstale":"2024-03-21T13:17:10.623485+0000","last_undegraded":"2024-03-21T13:17:10.623485+0000","last_fullsized":"2024-03-21T13:17:10.623485+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:08.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:20.209199+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":331,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.621520+0000","last_change":"2024-03-21T13:16:54.410019+0000","last_active":"2024-03-21T13:17:10.621520+0000","last_peered":"2024-03-21T13:17:10.621520+0000","last_clean":"2024-03-21T13:17:10.621520+0000","last_became_active":"2024-03-21T13:10:03.604616+0000","last_became_peered":"2024-03-21T13:10:03.604616+0000","last_unstale":"2024-03-21T13:17:10.621520+0000","last_undegraded":"2024-03-21T13:17:10.621520+0000","last_fullsized":"2024-03-21T13:17:10.621520+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:16:54.409873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.409873+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:16.016094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":756,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627973+0000","last_change":"2024-03-21T13:15:59.096623+0000","last_active":"2024-03-21T13:17:10.627973+0000","last_peered":"2024-03-21T13:17:10.627973+0000","last_clean":"2024-03-21T13:17:10.627973+0000","last_became_active":"2024-03-21T13:12:29.502808+0000","last_became_peered":"2024-03-21T13:12:29.502808+0000","last_unstale":"2024-03-21T13:17:10.627973+0000","last_undegraded":"2024-03-21T13:17:10.627973+0000","last_fullsized":"2024-03-21T13:17:10.627973+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:59.096520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:59.096520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.250778+0000","scrub_duration":233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":668,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627081+0000","last_change":"2024-03-21T13:14:53.198725+0000","last_active":"2024-03-21T13:17:10.627081+0000","last_peered":"2024-03-21T13:17:10.627081+0000","last_clean":"2024-03-21T13:17:10.627081+0000","last_became_active":"2024-03-21T13:10:03.612409+0000","last_became_peered":"2024-03-21T13:10:03.612409+0000","last_unstale":"2024-03-21T13:17:10.627081+0000","last_undegraded":"2024-03-21T13:17:10.627081+0000","last_fullsized":"2024-03-21T13:17:10.627081+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:53.198601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:14:53.198601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":799,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.490448+0000","last_change":"2024-03-21T13:16:48.967395+0000","last_active":"2024-03-21T13:17:06.490448+0000","last_peered":"2024-03-21T13:17:06.490448+0000","last_clean":"2024-03-21T13:17:06.490448+0000","last_became_active":"2024-03-21T13:10:03.586563+0000","last_became_peered":"2024-03-21T13:10:03.586563+0000","last_unstale":"2024-03-21T13:17:06.490448+0000","last_undegraded":"2024-03-21T13:17:06.490448+0000","last_fullsized":"2024-03-21T13:17:06.490448+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:48.967258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:20.956778+0000","last_clean_scrub_stamp":"2024-03-21T13:16:48.967258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:52.310270+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":807,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.628201+0000","last_change":"2024-03-21T13:16:55.685055+0000","last_active":"2024-03-21T13:17:10.628201+0000","last_peered":"2024-03-21T13:17:10.628201+0000","last_clean":"2024-03-21T13:17:10.628201+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:17:10.628201+0000","last_undegraded":"2024-03-21T13:17:10.628201+0000","last_fullsized":"2024-03-21T13:17:10.628201+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-21T13:16:55.684896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:55.684896+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:05.652758+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":165,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.494574+0000","last_change":"2024-03-21T13:17:05.780695+0000","last_active":"2024-03-21T13:17:06.494574+0000","last_peered":"2024-03-21T13:17:06.494574+0000","last_clean":"2024-03-21T13:17:06.494574+0000","last_became_active":"2024-03-21T13:13:06.420220+0000","last_became_peered":"2024-03-21T13:13:06.420220+0000","last_unstale":"2024-03-21T13:17:06.494574+0000","last_undegraded":"2024-03-21T13:17:06.494574+0000","last_fullsized":"2024-03-21T13:17:06.494574+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:17:05.780572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:05.780572+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:22.825319+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":975,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.501229+0000","last_change":"2024-03-21T13:16:12.489669+0000","last_active":"2024-03-21T13:17:06.501229+0000","last_peered":"2024-03-21T13:17:06.501229+0000","last_clean":"2024-03-21T13:17:06.501229+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:17:06.501229+0000","last_undegraded":"2024-03-21T13:17:06.501229+0000","last_fullsized":"2024-03-21T13:17:06.501229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:16:12.489518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.489518+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:14.011096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"407'2","reported_seq":651,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.626687+0000","last_change":"2024-03-21T13:16:58.735925+0000","last_active":"2024-03-21T13:17:10.626687+0000","last_peered":"2024-03-21T13:17:10.626687+0000","last_clean":"2024-03-21T13:17:10.626687+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:17:10.626687+0000","last_undegraded":"2024-03-21T13:17:10.626687+0000","last_fullsized":"2024-03-21T13:17:10.626687+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'2","last_scrub_stamp":"2024-03-21T13:16:58.735421+0000","last_deep_scrub":"407'2","last_deep_scrub_stamp":"2024-03-21T13:16:58.735421+0000","last_clean_scrub_stamp":"2024-03-21T13:16:58.735421+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:03.350682+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"410'25","reported_seq":423,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.624992+0000","last_change":"2024-03-21T13:17:09.876687+0000","last_active":"2024-03-21T13:17:10.624992+0000","last_peered":"2024-03-21T13:17:10.624992+0000","last_clean":"2024-03-21T13:17:10.624992+0000","last_became_active":"2024-03-21T13:15:04.458831+0000","last_became_peered":"2024-03-21T13:15:04.458831+0000","last_unstale":"2024-03-21T13:17:10.624992+0000","last_undegraded":"2024-03-21T13:17:10.624992+0000","last_fullsized":"2024-03-21T13:17:10.624992+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-03-21T13:17:09.876547+0000","last_deep_scrub":"410'25","last_deep_scrub_stamp":"2024-03-21T13:17:09.876547+0000","last_clean_scrub_stamp":"2024-03-21T13:17:09.876547+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:35.978356+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":177,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.494804+0000","last_change":"2024-03-21T13:16:07.247321+0000","last_active":"2024-03-21T13:17:06.494804+0000","last_peered":"2024-03-21T13:17:06.494804+0000","last_clean":"2024-03-21T13:17:06.494804+0000","last_became_active":"2024-03-21T13:12:29.527129+0000","last_became_peered":"2024-03-21T13:12:29.527129+0000","last_unstale":"2024-03-21T13:17:06.494804+0000","last_undegraded":"2024-03-21T13:17:06.494804+0000","last_fullsized":"2024-03-21T13:17:06.494804+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:07.247194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.247194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:08.116592+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":435,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.625228+0000","last_change":"2024-03-21T13:15:04.464609+0000","last_active":"2024-03-21T13:17:10.625228+0000","last_peered":"2024-03-21T13:17:10.625228+0000","last_clean":"2024-03-21T13:17:10.625228+0000","last_became_active":"2024-03-21T13:15:04.462979+0000","last_became_peered":"2024-03-21T13:15:04.462979+0000","last_unstale":"2024-03-21T13:17:10.625228+0000","last_undegraded":"2024-03-21T13:17:10.625228+0000","last_fullsized":"2024-03-21T13:17:10.625228+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:00.690145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:00.690145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":769,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.626216+0000","last_change":"2024-03-21T13:17:04.839217+0000","last_active":"2024-03-21T13:17:10.626216+0000","last_peered":"2024-03-21T13:17:10.626216+0000","last_clean":"2024-03-21T13:17:10.626216+0000","last_became_active":"2024-03-21T13:10:03.610489+0000","last_became_peered":"2024-03-21T13:10:03.610489+0000","last_unstale":"2024-03-21T13:17:10.626216+0000","last_undegraded":"2024-03-21T13:17:10.626216+0000","last_fullsized":"2024-03-21T13:17:10.626216+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:04.838772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:04.838772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:20.258616+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"331'24","reported_seq":835,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.501287+0000","last_change":"2024-03-21T13:15:09.670586+0000","last_active":"2024-03-21T13:17:06.501287+0000","last_peered":"2024-03-21T13:17:06.501287+0000","last_clean":"2024-03-21T13:17:06.501287+0000","last_became_active":"2024-03-21T13:12:29.522466+0000","last_became_peered":"2024-03-21T13:12:29.522466+0000","last_unstale":"2024-03-21T13:17:06.501287+0000","last_undegraded":"2024-03-21T13:17:06.501287+0000","last_fullsized":"2024-03-21T13:17:06.501287+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'24","last_scrub_stamp":"2024-03-21T13:15:09.670459+0000","last_deep_scrub":"331'24","last_deep_scrub_stamp":"2024-03-21T13:15:09.670459+0000","last_clean_scrub_stamp":"2024-03-21T13:15:09.670459+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":538,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.622845+0000","last_change":"2024-03-21T13:16:59.238877+0000","last_active":"2024-03-21T13:17:10.622845+0000","last_peered":"2024-03-21T13:17:10.622845+0000","last_clean":"2024-03-21T13:17:10.622845+0000","last_became_active":"2024-03-21T13:15:04.458144+0000","last_became_peered":"2024-03-21T13:15:04.458144+0000","last_unstale":"2024-03-21T13:17:10.622845+0000","last_undegraded":"2024-03-21T13:17:10.622845+0000","last_fullsized":"2024-03-21T13:17:10.622845+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:59.238664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:16:59.238664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:27.328140+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":748,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.500880+0000","last_change":"2024-03-21T13:16:21.557896+0000","last_active":"2024-03-21T13:17:06.500880+0000","last_peered":"2024-03-21T13:17:06.500880+0000","last_clean":"2024-03-21T13:17:06.500880+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:17:06.500880+0000","last_undegraded":"2024-03-21T13:17:06.500880+0000","last_fullsized":"2024-03-21T13:17:06.500880+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:21.557805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.557805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:35.861086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":200,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.624929+0000","last_change":"2024-03-21T13:16:00.029375+0000","last_active":"2024-03-21T13:17:10.624929+0000","last_peered":"2024-03-21T13:17:10.624929+0000","last_clean":"2024-03-21T13:17:10.624929+0000","last_became_active":"2024-03-21T13:15:04.462103+0000","last_became_peered":"2024-03-21T13:15:04.462103+0000","last_unstale":"2024-03-21T13:17:10.624929+0000","last_undegraded":"2024-03-21T13:17:10.624929+0000","last_fullsized":"2024-03-21T13:17:10.624929+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:00.029292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:22.955737+0000","last_clean_scrub_stamp":"2024-03-21T13:16:00.029292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":763,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.626348+0000","last_change":"2024-03-21T13:16:20.734468+0000","last_active":"2024-03-21T13:17:10.626348+0000","last_peered":"2024-03-21T13:17:10.626348+0000","last_clean":"2024-03-21T13:17:10.626348+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:17:10.626348+0000","last_undegraded":"2024-03-21T13:17:10.626348+0000","last_fullsized":"2024-03-21T13:17:10.626348+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-21T13:16:20.734379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.734379+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:47.670110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":731,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627201+0000","last_change":"2024-03-21T13:14:59.413401+0000","last_active":"2024-03-21T13:17:10.627201+0000","last_peered":"2024-03-21T13:17:10.627201+0000","last_clean":"2024-03-21T13:17:10.627201+0000","last_became_active":"2024-03-21T12:58:43.352572+0000","last_became_peered":"2024-03-21T12:58:43.352572+0000","last_unstale":"2024-03-21T13:17:10.627201+0000","last_undegraded":"2024-03-21T13:17:10.627201+0000","last_fullsized":"2024-03-21T13:17:10.627201+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:59.413139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:59.413139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":95,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.623040+0000","last_change":"2024-03-21T13:16:05.059523+0000","last_active":"2024-03-21T13:17:10.623040+0000","last_peered":"2024-03-21T13:17:10.623040+0000","last_clean":"2024-03-21T13:17:10.623040+0000","last_became_active":"2024-03-21T13:15:04.512071+0000","last_became_peered":"2024-03-21T13:15:04.512071+0000","last_unstale":"2024-03-21T13:17:10.623040+0000","last_undegraded":"2024-03-21T13:17:10.623040+0000","last_fullsized":"2024-03-21T13:17:10.623040+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:05.059443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.059443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.468710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":185,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.493809+0000","last_change":"2024-03-21T13:16:23.266104+0000","last_active":"2024-03-21T13:17:06.493809+0000","last_peered":"2024-03-21T13:17:06.493809+0000","last_clean":"2024-03-21T13:17:06.493809+0000","last_became_active":"2024-03-21T13:12:29.507706+0000","last_became_peered":"2024-03-21T13:12:29.507706+0000","last_unstale":"2024-03-21T13:17:06.493809+0000","last_undegraded":"2024-03-21T13:17:06.493809+0000","last_fullsized":"2024-03-21T13:17:06.493809+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:23.266023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:23.266023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:34.197737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":485,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.626954+0000","last_change":"2024-03-21T13:15:55.923610+0000","last_active":"2024-03-21T13:17:10.626954+0000","last_peered":"2024-03-21T13:17:10.626954+0000","last_clean":"2024-03-21T13:17:10.626954+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:17:10.626954+0000","last_undegraded":"2024-03-21T13:17:10.626954+0000","last_fullsized":"2024-03-21T13:17:10.626954+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:15:55.923514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:55.923514+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:12.089112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":188,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.622581+0000","last_change":"2024-03-21T13:16:50.026008+0000","last_active":"2024-03-21T13:17:10.622581+0000","last_peered":"2024-03-21T13:17:10.622581+0000","last_clean":"2024-03-21T13:17:10.622581+0000","last_became_active":"2024-03-21T13:15:04.512432+0000","last_became_peered":"2024-03-21T13:15:04.512432+0000","last_unstale":"2024-03-21T13:17:10.622581+0000","last_undegraded":"2024-03-21T13:17:10.622581+0000","last_fullsized":"2024-03-21T13:17:10.622581+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:50.025891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:50.025891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:04.135064+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":753,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.626823+0000","last_change":"2024-03-21T13:15:18.312265+0000","last_active":"2024-03-21T13:17:10.626823+0000","last_peered":"2024-03-21T13:17:10.626823+0000","last_clean":"2024-03-21T13:17:10.626823+0000","last_became_active":"2024-03-21T13:12:29.502203+0000","last_became_peered":"2024-03-21T13:12:29.502203+0000","last_unstale":"2024-03-21T13:17:10.626823+0000","last_undegraded":"2024-03-21T13:17:10.626823+0000","last_fullsized":"2024-03-21T13:17:10.626823+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:18.312139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:18.312139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":746,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.492343+0000","last_change":"2024-03-21T13:16:08.584734+0000","last_active":"2024-03-21T13:17:06.492343+0000","last_peered":"2024-03-21T13:17:06.492343+0000","last_clean":"2024-03-21T13:17:06.492343+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:17:06.492343+0000","last_undegraded":"2024-03-21T13:17:06.492343+0000","last_fullsized":"2024-03-21T13:17:06.492343+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:08.584643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:19.383090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:08.584643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:19.593361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":756,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.501311+0000","last_change":"2024-03-21T13:16:53.867897+0000","last_active":"2024-03-21T13:17:06.501311+0000","last_peered":"2024-03-21T13:17:06.501311+0000","last_clean":"2024-03-21T13:17:06.501311+0000","last_became_active":"2024-03-21T13:10:03.591857+0000","last_became_peered":"2024-03-21T13:10:03.591857+0000","last_unstale":"2024-03-21T13:17:06.501311+0000","last_undegraded":"2024-03-21T13:17:06.501311+0000","last_fullsized":"2024-03-21T13:17:06.501311+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:53.867747+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.867747+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:12.880058+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":581,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:08.823503+0000","last_change":"2024-03-21T13:17:08.823533+0000","last_active":"2024-03-21T13:17:08.823503+0000","last_peered":"2024-03-21T13:17:08.823503+0000","last_clean":"2024-03-21T13:17:08.823503+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:17:08.823503+0000","last_undegraded":"2024-03-21T13:17:08.823503+0000","last_fullsized":"2024-03-21T13:17:08.823503+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:08.823336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.823336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:24.082746+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":186,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.495551+0000","last_change":"2024-03-21T13:16:52.230103+0000","last_active":"2024-03-21T13:17:06.495551+0000","last_peered":"2024-03-21T13:17:06.495551+0000","last_clean":"2024-03-21T13:17:06.495551+0000","last_became_active":"2024-03-21T13:12:29.510100+0000","last_became_peered":"2024-03-21T13:12:29.510100+0000","last_unstale":"2024-03-21T13:17:06.495551+0000","last_undegraded":"2024-03-21T13:17:06.495551+0000","last_fullsized":"2024-03-21T13:17:06.495551+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.229670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:22.501055+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.229670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:08.462478+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":459,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.494677+0000","last_change":"2024-03-21T13:15:59.651653+0000","last_active":"2024-03-21T13:17:06.494677+0000","last_peered":"2024-03-21T13:17:06.494677+0000","last_clean":"2024-03-21T13:17:06.494677+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:17:06.494677+0000","last_undegraded":"2024-03-21T13:17:06.494677+0000","last_fullsized":"2024-03-21T13:17:06.494677+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:59.651571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:15:59.651571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:26.236664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":94,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.624215+0000","last_change":"2024-03-21T13:15:10.666307+0000","last_active":"2024-03-21T13:17:10.624215+0000","last_peered":"2024-03-21T13:17:10.624215+0000","last_clean":"2024-03-21T13:17:10.624215+0000","last_became_active":"2024-03-21T13:15:04.463107+0000","last_became_peered":"2024-03-21T13:15:04.463107+0000","last_unstale":"2024-03-21T13:17:10.624215+0000","last_undegraded":"2024-03-21T13:17:10.624215+0000","last_fullsized":"2024-03-21T13:17:10.624215+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:15:10.666149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:10.666149+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":188,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.623944+0000","last_change":"2024-03-21T13:16:53.027206+0000","last_active":"2024-03-21T13:17:10.623944+0000","last_peered":"2024-03-21T13:17:10.623944+0000","last_clean":"2024-03-21T13:17:10.623944+0000","last_became_active":"2024-03-21T13:15:04.458489+0000","last_became_peered":"2024-03-21T13:15:04.458489+0000","last_unstale":"2024-03-21T13:17:10.623944+0000","last_undegraded":"2024-03-21T13:17:10.623944+0000","last_fullsized":"2024-03-21T13:17:10.623944+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:53.026764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.026764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:53.710604+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"394'12","reported_seq":247,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.495049+0000","last_change":"2024-03-21T13:16:24.237890+0000","last_active":"2024-03-21T13:17:06.495049+0000","last_peered":"2024-03-21T13:17:06.495049+0000","last_clean":"2024-03-21T13:17:06.495049+0000","last_became_active":"2024-03-21T13:12:29.521385+0000","last_became_peered":"2024-03-21T13:12:29.521385+0000","last_unstale":"2024-03-21T13:17:06.495049+0000","last_undegraded":"2024-03-21T13:17:06.495049+0000","last_fullsized":"2024-03-21T13:17:06.495049+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'12","last_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_deep_scrub":"394'12","last_deep_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_clean_scrub_stamp":"2024-03-21T13:16:24.237800+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:31.363500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":758,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.489716+0000","last_change":"2024-03-21T13:15:10.647075+0000","last_active":"2024-03-21T13:17:06.489716+0000","last_peered":"2024-03-21T13:17:06.489716+0000","last_clean":"2024-03-21T13:17:06.489716+0000","last_became_active":"2024-03-21T13:10:03.599265+0000","last_became_peered":"2024-03-21T13:10:03.599265+0000","last_unstale":"2024-03-21T13:17:06.489716+0000","last_undegraded":"2024-03-21T13:17:06.489716+0000","last_fullsized":"2024-03-21T13:17:06.489716+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:10.646946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:15:10.646946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'20","reported_seq":823,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627653+0000","last_change":"2024-03-21T13:15:16.283969+0000","last_active":"2024-03-21T13:17:10.627653+0000","last_peered":"2024-03-21T13:17:10.627653+0000","last_clean":"2024-03-21T13:17:10.627653+0000","last_became_active":"2024-03-21T13:10:03.613119+0000","last_became_peered":"2024-03-21T13:10:03.613119+0000","last_unstale":"2024-03-21T13:17:10.627653+0000","last_undegraded":"2024-03-21T13:17:10.627653+0000","last_fullsized":"2024-03-21T13:17:10.627653+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:15:16.283831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:16.283831+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":321,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.621171+0000","last_change":"2024-03-21T13:16:17.497436+0000","last_active":"2024-03-21T13:17:10.621171+0000","last_peered":"2024-03-21T13:17:10.621171+0000","last_clean":"2024-03-21T13:17:10.621171+0000","last_became_active":"2024-03-21T13:12:29.507751+0000","last_became_peered":"2024-03-21T13:12:29.507751+0000","last_unstale":"2024-03-21T13:17:10.621171+0000","last_undegraded":"2024-03-21T13:17:10.621171+0000","last_fullsized":"2024-03-21T13:17:10.621171+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:17.497325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:17.497325+0000","last_clean_scrub_stamp":"2024-03-21T13:16:17.497325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:26.268883+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"272'10","reported_seq":987,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627711+0000","last_change":"2024-03-21T13:16:09.700298+0000","last_active":"2024-03-21T13:17:10.627711+0000","last_peered":"2024-03-21T13:17:10.627711+0000","last_clean":"2024-03-21T13:17:10.627711+0000","last_became_active":"2024-03-21T13:10:03.594651+0000","last_became_peered":"2024-03-21T13:10:03.594651+0000","last_unstale":"2024-03-21T13:17:10.627711+0000","last_undegraded":"2024-03-21T13:17:10.627711+0000","last_fullsized":"2024-03-21T13:17:10.627711+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'10","last_scrub_stamp":"2024-03-21T13:16:09.700215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:09.700215+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:35.185939+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":200,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.623237+0000","last_change":"2024-03-21T13:16:15.939316+0000","last_active":"2024-03-21T13:17:10.623237+0000","last_peered":"2024-03-21T13:17:10.623237+0000","last_clean":"2024-03-21T13:17:10.623237+0000","last_became_active":"2024-03-21T13:15:04.458184+0000","last_became_peered":"2024-03-21T13:15:04.458184+0000","last_unstale":"2024-03-21T13:17:10.623237+0000","last_undegraded":"2024-03-21T13:17:10.623237+0000","last_fullsized":"2024-03-21T13:17:10.623237+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:15.939231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:16:15.939231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:17.625617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"234'15","reported_seq":195,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.494444+0000","last_change":"2024-03-21T13:16:22.227639+0000","last_active":"2024-03-21T13:17:06.494444+0000","last_peered":"2024-03-21T13:17:06.494444+0000","last_clean":"2024-03-21T13:17:06.494444+0000","last_became_active":"2024-03-21T13:12:29.510247+0000","last_became_peered":"2024-03-21T13:12:29.510247+0000","last_unstale":"2024-03-21T13:17:06.494444+0000","last_undegraded":"2024-03-21T13:17:06.494444+0000","last_fullsized":"2024-03-21T13:17:06.494444+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:16:22.227536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.227536+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:38.775744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":316,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.621372+0000","last_change":"2024-03-21T13:17:07.670896+0000","last_active":"2024-03-21T13:17:10.621372+0000","last_peered":"2024-03-21T13:17:10.621372+0000","last_clean":"2024-03-21T13:17:10.621372+0000","last_became_active":"2024-03-21T13:10:03.613841+0000","last_became_peered":"2024-03-21T13:10:03.613841+0000","last_unstale":"2024-03-21T13:17:10.621372+0000","last_undegraded":"2024-03-21T13:17:10.621372+0000","last_fullsized":"2024-03-21T13:17:10.621372+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:07.670782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:17:07.670782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:29.821007+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":741,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.628180+0000","last_change":"2024-03-21T13:14:57.279251+0000","last_active":"2024-03-21T13:17:10.628180+0000","last_peered":"2024-03-21T13:17:10.628180+0000","last_clean":"2024-03-21T13:17:10.628180+0000","last_became_active":"2024-03-21T13:02:14.352989+0000","last_became_peered":"2024-03-21T13:02:14.352989+0000","last_unstale":"2024-03-21T13:17:10.628180+0000","last_undegraded":"2024-03-21T13:17:10.628180+0000","last_fullsized":"2024-03-21T13:17:10.628180+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:57.279130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:14:57.279130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":760,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.490770+0000","last_change":"2024-03-21T13:16:54.653009+0000","last_active":"2024-03-21T13:17:06.490770+0000","last_peered":"2024-03-21T13:17:06.490770+0000","last_clean":"2024-03-21T13:17:06.490770+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:17:06.490770+0000","last_undegraded":"2024-03-21T13:17:06.490770+0000","last_fullsized":"2024-03-21T13:17:06.490770+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:16:54.652821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.652821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:57.529839+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'15","reported_seq":395,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.625092+0000","last_change":"2024-03-21T13:16:58.080245+0000","last_active":"2024-03-21T13:17:10.625092+0000","last_peered":"2024-03-21T13:17:10.625092+0000","last_clean":"2024-03-21T13:17:10.625092+0000","last_became_active":"2024-03-21T13:15:04.458914+0000","last_became_peered":"2024-03-21T13:15:04.458914+0000","last_unstale":"2024-03-21T13:17:10.625092+0000","last_undegraded":"2024-03-21T13:17:10.625092+0000","last_fullsized":"2024-03-21T13:17:10.625092+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:16:58.080100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:58.080100+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:18.958644+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":734,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.491946+0000","last_change":"2024-03-21T13:16:02.716355+0000","last_active":"2024-03-21T13:17:06.491946+0000","last_peered":"2024-03-21T13:17:06.491946+0000","last_clean":"2024-03-21T13:17:06.491946+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:17:06.491946+0000","last_undegraded":"2024-03-21T13:17:06.491946+0000","last_fullsized":"2024-03-21T13:17:06.491946+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:02.716255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:02.716255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.235343+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":185,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.493726+0000","last_change":"2024-03-21T13:16:18.220471+0000","last_active":"2024-03-21T13:17:06.493726+0000","last_peered":"2024-03-21T13:17:06.493726+0000","last_clean":"2024-03-21T13:17:06.493726+0000","last_became_active":"2024-03-21T13:12:29.514113+0000","last_became_peered":"2024-03-21T13:12:29.514113+0000","last_unstale":"2024-03-21T13:17:06.493726+0000","last_undegraded":"2024-03-21T13:17:06.493726+0000","last_fullsized":"2024-03-21T13:17:06.493726+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.220368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.220368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.800528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":760,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.501524+0000","last_change":"2024-03-21T13:14:14.647062+0000","last_active":"2024-03-21T13:17:06.501524+0000","last_peered":"2024-03-21T13:17:06.501524+0000","last_clean":"2024-03-21T13:17:06.501524+0000","last_became_active":"2024-03-21T13:12:29.520286+0000","last_became_peered":"2024-03-21T13:12:29.520286+0000","last_unstale":"2024-03-21T13:17:06.501524+0000","last_undegraded":"2024-03-21T13:17:06.501524+0000","last_fullsized":"2024-03-21T13:17:06.501524+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:14.646933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:14:14.646933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":807,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.628084+0000","last_change":"2024-03-21T13:17:02.736704+0000","last_active":"2024-03-21T13:17:10.628084+0000","last_peered":"2024-03-21T13:17:10.628084+0000","last_clean":"2024-03-21T13:17:10.628084+0000","last_became_active":"2024-03-21T13:10:03.610810+0000","last_became_peered":"2024-03-21T13:10:03.610810+0000","last_unstale":"2024-03-21T13:17:10.628084+0000","last_undegraded":"2024-03-21T13:17:10.628084+0000","last_fullsized":"2024-03-21T13:17:10.628084+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-21T13:17:02.736579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:02.736579+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:21.749488+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"410'4392","reported_seq":4441,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:09.879300+0000","last_change":"2024-03-21T13:16:13.903136+0000","last_active":"2024-03-21T13:17:09.879300+0000","last_peered":"2024-03-21T13:17:09.879300+0000","last_clean":"2024-03-21T13:17:09.879300+0000","last_became_active":"2024-03-21T13:16:13.901764+0000","last_became_peered":"2024-03-21T13:16:13.901764+0000","last_unstale":"2024-03-21T13:17:09.879300+0000","last_undegraded":"2024-03-21T13:17:09.879300+0000","last_fullsized":"2024-03-21T13:17:09.879300+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:14.340802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287834112,"num_objects":4392,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4392,"num_write_kb":281088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":178,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.495335+0000","last_change":"2024-03-21T13:15:06.391491+0000","last_active":"2024-03-21T13:17:06.495335+0000","last_peered":"2024-03-21T13:17:06.495335+0000","last_clean":"2024-03-21T13:17:06.495335+0000","last_became_active":"2024-03-21T13:12:29.517413+0000","last_became_peered":"2024-03-21T13:12:29.517413+0000","last_unstale":"2024-03-21T13:17:06.495335+0000","last_undegraded":"2024-03-21T13:17:06.495335+0000","last_fullsized":"2024-03-21T13:17:06.495335+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:06.391193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:15:06.391193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":791,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.494051+0000","last_change":"2024-03-21T13:16:56.698464+0000","last_active":"2024-03-21T13:17:06.494051+0000","last_peered":"2024-03-21T13:17:06.494051+0000","last_clean":"2024-03-21T13:17:06.494051+0000","last_became_active":"2024-03-21T13:06:29.487439+0000","last_became_peered":"2024-03-21T13:06:29.487439+0000","last_unstale":"2024-03-21T13:17:06.494051+0000","last_undegraded":"2024-03-21T13:17:06.494051+0000","last_fullsized":"2024-03-21T13:17:06.494051+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:16:56.698173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:56.698173+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:07.559656+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":202,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.494373+0000","last_change":"2024-03-21T13:16:53.176991+0000","last_active":"2024-03-21T13:17:06.494373+0000","last_peered":"2024-03-21T13:17:06.494373+0000","last_clean":"2024-03-21T13:17:06.494373+0000","last_became_active":"2024-03-21T13:12:29.513019+0000","last_became_peered":"2024-03-21T13:12:29.513019+0000","last_unstale":"2024-03-21T13:17:06.494373+0000","last_undegraded":"2024-03-21T13:17:06.494373+0000","last_fullsized":"2024-03-21T13:17:06.494373+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:16:53.176836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.176836+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:13.256411+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":974,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.490726+0000","last_change":"2024-03-21T13:16:10.554480+0000","last_active":"2024-03-21T13:17:06.490726+0000","last_peered":"2024-03-21T13:17:06.490726+0000","last_clean":"2024-03-21T13:17:06.490726+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:17:06.490726+0000","last_undegraded":"2024-03-21T13:17:06.490726+0000","last_fullsized":"2024-03-21T13:17:06.490726+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:16:10.554378+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:16:10.554378+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.191410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"410'4664","reported_seq":4715,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.623382+0000","last_change":"2024-03-21T13:16:13.901750+0000","last_active":"2024-03-21T13:17:10.623382+0000","last_peered":"2024-03-21T13:17:10.623382+0000","last_clean":"2024-03-21T13:17:10.623382+0000","last_became_active":"2024-03-21T13:16:13.901065+0000","last_became_peered":"2024-03-21T13:16:13.901065+0000","last_unstale":"2024-03-21T13:17:10.623382+0000","last_undegraded":"2024-03-21T13:17:10.623382+0000","last_fullsized":"2024-03-21T13:17:10.623382+0000","mapping_epoch":390,"log_start":"399'100","ondisk_log_start":"399'100","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.886855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305659904,"num_objects":4664,"num_object_clones":0,"num_object_copies":13992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4664,"num_write_kb":298496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":738,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627103+0000","last_change":"2024-03-21T13:15:58.046280+0000","last_active":"2024-03-21T13:17:10.627103+0000","last_peered":"2024-03-21T13:17:10.627103+0000","last_clean":"2024-03-21T13:17:10.627103+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:17:10.627103+0000","last_undegraded":"2024-03-21T13:17:10.627103+0000","last_fullsized":"2024-03-21T13:17:10.627103+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:58.046156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:15:58.046156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:12.947549+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":761,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.491945+0000","last_change":"2024-03-21T13:16:52.653497+0000","last_active":"2024-03-21T13:17:06.491945+0000","last_peered":"2024-03-21T13:17:06.491945+0000","last_clean":"2024-03-21T13:17:06.491945+0000","last_became_active":"2024-03-21T13:02:14.336700+0000","last_became_peered":"2024-03-21T13:02:14.336700+0000","last_unstale":"2024-03-21T13:17:06.491945+0000","last_undegraded":"2024-03-21T13:17:06.491945+0000","last_fullsized":"2024-03-21T13:17:06.491945+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.653388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.653388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:19.348258+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"346'23","reported_seq":158,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.619826+0000","last_change":"2024-03-21T13:16:10.260270+0000","last_active":"2024-03-21T13:17:10.619826+0000","last_peered":"2024-03-21T13:17:10.619826+0000","last_clean":"2024-03-21T13:17:10.619826+0000","last_became_active":"2024-03-21T13:13:21.819111+0000","last_became_peered":"2024-03-21T13:13:21.819111+0000","last_unstale":"2024-03-21T13:17:10.619826+0000","last_undegraded":"2024-03-21T13:17:10.619826+0000","last_fullsized":"2024-03-21T13:17:10.619826+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'23","last_scrub_stamp":"2024-03-21T13:16:10.260157+0000","last_deep_scrub":"346'23","last_deep_scrub_stamp":"2024-03-21T13:13:20.232807+0000","last_clean_scrub_stamp":"2024-03-21T13:16:10.260157+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:35.914575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":322,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.620114+0000","last_change":"2024-03-21T13:16:55.431147+0000","last_active":"2024-03-21T13:17:10.620114+0000","last_peered":"2024-03-21T13:17:10.620114+0000","last_clean":"2024-03-21T13:17:10.620114+0000","last_became_active":"2024-03-21T13:12:29.499921+0000","last_became_peered":"2024-03-21T13:12:29.499921+0000","last_unstale":"2024-03-21T13:17:10.620114+0000","last_undegraded":"2024-03-21T13:17:10.620114+0000","last_fullsized":"2024-03-21T13:17:10.620114+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:55.431004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:55.431004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:23.679031+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"410'4646","reported_seq":4695,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.347796+0000","last_change":"2024-03-21T13:16:13.903159+0000","last_active":"2024-03-21T13:17:10.347796+0000","last_peered":"2024-03-21T13:17:10.347796+0000","last_clean":"2024-03-21T13:17:10.347796+0000","last_became_active":"2024-03-21T13:16:13.902868+0000","last_became_peered":"2024-03-21T13:16:13.902868+0000","last_unstale":"2024-03-21T13:17:10.347796+0000","last_undegraded":"2024-03-21T13:17:10.347796+0000","last_fullsized":"2024-03-21T13:17:10.347796+0000","mapping_epoch":390,"log_start":"399'200","ondisk_log_start":"399'200","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4446,"log_dups_size":0,"ondisk_log_size":4446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:29.385439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304480256,"num_objects":4646,"num_object_clones":0,"num_object_copies":13938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4646,"num_write_kb":297344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":755,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.490972+0000","last_change":"2024-03-21T13:16:06.611026+0000","last_active":"2024-03-21T13:17:06.490972+0000","last_peered":"2024-03-21T13:17:06.490972+0000","last_clean":"2024-03-21T13:17:06.490972+0000","last_became_active":"2024-03-21T13:10:03.598366+0000","last_became_peered":"2024-03-21T13:10:03.598366+0000","last_unstale":"2024-03-21T13:17:06.490972+0000","last_undegraded":"2024-03-21T13:17:06.490972+0000","last_fullsized":"2024-03-21T13:17:06.490972+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:06.610947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:16:06.610947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.715015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":330,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.620890+0000","last_change":"2024-03-21T13:16:49.448922+0000","last_active":"2024-03-21T13:17:10.620890+0000","last_peered":"2024-03-21T13:17:10.620890+0000","last_clean":"2024-03-21T13:17:10.620890+0000","last_became_active":"2024-03-21T13:12:29.500199+0000","last_became_peered":"2024-03-21T13:12:29.500199+0000","last_unstale":"2024-03-21T13:17:10.620890+0000","last_undegraded":"2024-03-21T13:17:10.620890+0000","last_fullsized":"2024-03-21T13:17:10.620890+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:49.448804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:49.448804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:02.584547+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":767,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.492650+0000","last_change":"2024-03-21T13:16:47.661139+0000","last_active":"2024-03-21T13:17:06.492650+0000","last_peered":"2024-03-21T13:17:06.492650+0000","last_clean":"2024-03-21T13:17:06.492650+0000","last_became_active":"2024-03-21T13:12:29.508984+0000","last_became_peered":"2024-03-21T13:12:29.508984+0000","last_unstale":"2024-03-21T13:17:06.492650+0000","last_undegraded":"2024-03-21T13:17:06.492650+0000","last_fullsized":"2024-03-21T13:17:06.492650+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:47.661063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:47.661063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:15.875522+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'35","reported_seq":886,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:08.873201+0000","last_change":"2024-03-21T13:17:08.873256+0000","last_active":"2024-03-21T13:17:08.873201+0000","last_peered":"2024-03-21T13:17:08.873201+0000","last_clean":"2024-03-21T13:17:08.873201+0000","last_became_active":"2024-03-21T13:12:29.523359+0000","last_became_peered":"2024-03-21T13:12:29.523359+0000","last_unstale":"2024-03-21T13:17:08.873201+0000","last_undegraded":"2024-03-21T13:17:08.873201+0000","last_fullsized":"2024-03-21T13:17:08.873201+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:17:08.873105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.873105+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:09.856433+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"410'4495","reported_seq":4544,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.330006+0000","last_change":"2024-03-21T13:16:13.904507+0000","last_active":"2024-03-21T13:17:10.330006+0000","last_peered":"2024-03-21T13:17:10.330006+0000","last_clean":"2024-03-21T13:17:10.330006+0000","last_became_active":"2024-03-21T13:16:13.904210+0000","last_became_peered":"2024-03-21T13:16:13.904210+0000","last_unstale":"2024-03-21T13:17:10.330006+0000","last_undegraded":"2024-03-21T13:17:10.330006+0000","last_fullsized":"2024-03-21T13:17:10.330006+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.107729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294584320,"num_objects":4495,"num_object_clones":0,"num_object_copies":13485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4495,"num_write_kb":287680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":449,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.624110+0000","last_change":"2024-03-21T13:17:07.141252+0000","last_active":"2024-03-21T13:17:10.624110+0000","last_peered":"2024-03-21T13:17:10.624110+0000","last_clean":"2024-03-21T13:17:10.624110+0000","last_became_active":"2024-03-21T13:15:04.458107+0000","last_became_peered":"2024-03-21T13:15:04.458107+0000","last_unstale":"2024-03-21T13:17:10.624110+0000","last_undegraded":"2024-03-21T13:17:10.624110+0000","last_fullsized":"2024-03-21T13:17:10.624110+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:07.141148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:07.141148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:09.265018+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":316,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.620969+0000","last_change":"2024-03-21T13:16:56.458265+0000","last_active":"2024-03-21T13:17:10.620969+0000","last_peered":"2024-03-21T13:17:10.620969+0000","last_clean":"2024-03-21T13:17:10.620969+0000","last_became_active":"2024-03-21T13:10:03.603646+0000","last_became_peered":"2024-03-21T13:10:03.603646+0000","last_unstale":"2024-03-21T13:17:10.620969+0000","last_undegraded":"2024-03-21T13:17:10.620969+0000","last_fullsized":"2024-03-21T13:17:10.620969+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:56.458050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:56.458050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:09.246703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":186,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.493456+0000","last_change":"2024-03-21T13:16:54.243549+0000","last_active":"2024-03-21T13:17:06.493456+0000","last_peered":"2024-03-21T13:17:06.493456+0000","last_clean":"2024-03-21T13:17:06.493456+0000","last_became_active":"2024-03-21T13:12:29.512741+0000","last_became_peered":"2024-03-21T13:12:29.512741+0000","last_unstale":"2024-03-21T13:17:06.493456+0000","last_undegraded":"2024-03-21T13:17:06.493456+0000","last_fullsized":"2024-03-21T13:17:06.493456+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:54.243395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.243395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:03.270582+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"234'19","reported_seq":440,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.623720+0000","last_change":"2024-03-21T13:16:23.841059+0000","last_active":"2024-03-21T13:17:10.623720+0000","last_peered":"2024-03-21T13:17:10.623720+0000","last_clean":"2024-03-21T13:17:10.623720+0000","last_became_active":"2024-03-21T13:15:04.460898+0000","last_became_peered":"2024-03-21T13:15:04.460898+0000","last_unstale":"2024-03-21T13:17:10.623720+0000","last_undegraded":"2024-03-21T13:17:10.623720+0000","last_fullsized":"2024-03-21T13:17:10.623720+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:23.840950+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:16:23.840950+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:46.775417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"411'4485","reported_seq":4537,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:11.188188+0000","last_change":"2024-03-21T13:16:13.902405+0000","last_active":"2024-03-21T13:17:11.188188+0000","last_peered":"2024-03-21T13:17:11.188188+0000","last_clean":"2024-03-21T13:17:11.188188+0000","last_became_active":"2024-03-21T13:16:13.902109+0000","last_became_peered":"2024-03-21T13:16:13.902109+0000","last_unstale":"2024-03-21T13:17:11.188188+0000","last_undegraded":"2024-03-21T13:17:11.188188+0000","last_fullsized":"2024-03-21T13:17:11.188188+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:41.304847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293928960,"num_objects":4485,"num_object_clones":0,"num_object_copies":13455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4485,"num_write_kb":287040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":732,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627927+0000","last_change":"2024-03-21T13:17:06.844511+0000","last_active":"2024-03-21T13:17:10.627927+0000","last_peered":"2024-03-21T13:17:10.627927+0000","last_clean":"2024-03-21T13:17:10.627927+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:17:10.627927+0000","last_undegraded":"2024-03-21T13:17:10.627927+0000","last_fullsized":"2024-03-21T13:17:10.627927+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:06.844387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:17:06.844387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:18.955021+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":182,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.494951+0000","last_change":"2024-03-21T13:16:08.211061+0000","last_active":"2024-03-21T13:17:06.494951+0000","last_peered":"2024-03-21T13:17:06.494951+0000","last_clean":"2024-03-21T13:17:06.494951+0000","last_became_active":"2024-03-21T13:12:29.521275+0000","last_became_peered":"2024-03-21T13:12:29.521275+0000","last_unstale":"2024-03-21T13:17:06.494951+0000","last_undegraded":"2024-03-21T13:17:06.494951+0000","last_fullsized":"2024-03-21T13:17:06.494951+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:16:08.210967+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:16:08.210967+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:37.370051+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":97,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.623427+0000","last_change":"2024-03-21T13:16:22.861225+0000","last_active":"2024-03-21T13:17:10.623427+0000","last_peered":"2024-03-21T13:17:10.623427+0000","last_clean":"2024-03-21T13:17:10.623427+0000","last_became_active":"2024-03-21T13:15:04.462662+0000","last_became_peered":"2024-03-21T13:15:04.462662+0000","last_unstale":"2024-03-21T13:17:10.623427+0000","last_undegraded":"2024-03-21T13:17:10.623427+0000","last_fullsized":"2024-03-21T13:17:10.623427+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:22.861136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:22.861136+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.861136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:34.763292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"272'43","reported_seq":424,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.621291+0000","last_change":"2024-03-21T13:16:16.292178+0000","last_active":"2024-03-21T13:17:10.621291+0000","last_peered":"2024-03-21T13:17:10.621291+0000","last_clean":"2024-03-21T13:17:10.621291+0000","last_became_active":"2024-03-21T13:10:03.603889+0000","last_became_peered":"2024-03-21T13:10:03.603889+0000","last_unstale":"2024-03-21T13:17:10.621291+0000","last_undegraded":"2024-03-21T13:17:10.621291+0000","last_fullsized":"2024-03-21T13:17:10.621291+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'43","last_scrub_stamp":"2024-03-21T13:16:16.292073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:16.292073+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:41.103424+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":193,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.623625+0000","last_change":"2024-03-21T13:16:10.968486+0000","last_active":"2024-03-21T13:17:10.623625+0000","last_peered":"2024-03-21T13:17:10.623625+0000","last_clean":"2024-03-21T13:17:10.623625+0000","last_became_active":"2024-03-21T13:15:04.462159+0000","last_became_peered":"2024-03-21T13:15:04.462159+0000","last_unstale":"2024-03-21T13:17:10.623625+0000","last_undegraded":"2024-03-21T13:17:10.623625+0000","last_fullsized":"2024-03-21T13:17:10.623625+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:10.968402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:15.968926+0000","last_clean_scrub_stamp":"2024-03-21T13:16:10.968402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.606676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"411'4588","reported_seq":4639,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:11.171942+0000","last_change":"2024-03-21T13:16:13.900592+0000","last_active":"2024-03-21T13:17:11.171942+0000","last_peered":"2024-03-21T13:17:11.171942+0000","last_clean":"2024-03-21T13:17:11.171942+0000","last_became_active":"2024-03-21T13:16:13.900119+0000","last_became_peered":"2024-03-21T13:16:13.900119+0000","last_unstale":"2024-03-21T13:17:11.171942+0000","last_undegraded":"2024-03-21T13:17:11.171942+0000","last_fullsized":"2024-03-21T13:17:11.171942+0000","mapping_epoch":390,"log_start":"399'100","ondisk_log_start":"399'100","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:17.609817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300679168,"num_objects":4588,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4588,"num_write_kb":293632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":743,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.492794+0000","last_change":"2024-03-21T13:16:18.683140+0000","last_active":"2024-03-21T13:17:06.492794+0000","last_peered":"2024-03-21T13:17:06.492794+0000","last_clean":"2024-03-21T13:17:06.492794+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:17:06.492794+0000","last_undegraded":"2024-03-21T13:17:06.492794+0000","last_fullsized":"2024-03-21T13:17:06.492794+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.683041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:44.764044+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":308,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.620198+0000","last_change":"2024-03-21T13:16:12.231215+0000","last_active":"2024-03-21T13:17:10.620198+0000","last_peered":"2024-03-21T13:17:10.620198+0000","last_clean":"2024-03-21T13:17:10.620198+0000","last_became_active":"2024-03-21T13:10:03.604303+0000","last_became_peered":"2024-03-21T13:10:03.604303+0000","last_unstale":"2024-03-21T13:17:10.620198+0000","last_undegraded":"2024-03-21T13:17:10.620198+0000","last_fullsized":"2024-03-21T13:17:10.620198+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.231115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.231115+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.231115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:12.524706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":316,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.620940+0000","last_change":"2024-03-21T13:16:52.465547+0000","last_active":"2024-03-21T13:17:10.620940+0000","last_peered":"2024-03-21T13:17:10.620940+0000","last_clean":"2024-03-21T13:17:10.620940+0000","last_became_active":"2024-03-21T13:10:03.604427+0000","last_became_peered":"2024-03-21T13:10:03.604427+0000","last_unstale":"2024-03-21T13:17:10.620940+0000","last_undegraded":"2024-03-21T13:17:10.620940+0000","last_fullsized":"2024-03-21T13:17:10.620940+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.465427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.465427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:59.952130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"243'20","reported_seq":190,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.496327+0000","last_change":"2024-03-21T13:14:52.337122+0000","last_active":"2024-03-21T13:17:06.496327+0000","last_peered":"2024-03-21T13:17:06.496327+0000","last_clean":"2024-03-21T13:17:06.496327+0000","last_became_active":"2024-03-21T13:12:29.518048+0000","last_became_peered":"2024-03-21T13:12:29.518048+0000","last_unstale":"2024-03-21T13:17:06.496327+0000","last_undegraded":"2024-03-21T13:17:06.496327+0000","last_fullsized":"2024-03-21T13:17:06.496327+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'20","last_scrub_stamp":"2024-03-21T13:14:52.332617+0000","last_deep_scrub":"243'20","last_deep_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_clean_scrub_stamp":"2024-03-21T13:14:52.332617+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":314,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.620908+0000","last_change":"2024-03-21T13:16:15.247743+0000","last_active":"2024-03-21T13:17:10.620908+0000","last_peered":"2024-03-21T13:17:10.620908+0000","last_clean":"2024-03-21T13:17:10.620908+0000","last_became_active":"2024-03-21T13:12:29.506751+0000","last_became_peered":"2024-03-21T13:12:29.506751+0000","last_unstale":"2024-03-21T13:17:10.620908+0000","last_undegraded":"2024-03-21T13:17:10.620908+0000","last_fullsized":"2024-03-21T13:17:10.620908+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:15.247661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:15.247661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:36.430093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"411'4610","reported_seq":4662,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:11.142469+0000","last_change":"2024-03-21T13:16:13.900783+0000","last_active":"2024-03-21T13:17:11.142469+0000","last_peered":"2024-03-21T13:17:11.142469+0000","last_clean":"2024-03-21T13:17:11.142469+0000","last_became_active":"2024-03-21T13:16:13.900477+0000","last_became_peered":"2024-03-21T13:16:13.900477+0000","last_unstale":"2024-03-21T13:17:11.142469+0000","last_undegraded":"2024-03-21T13:17:11.142469+0000","last_fullsized":"2024-03-21T13:17:11.142469+0000","mapping_epoch":390,"log_start":"399'100","ondisk_log_start":"399'100","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4510,"log_dups_size":0,"ondisk_log_size":4510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:38.844318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302120960,"num_objects":4610,"num_object_clones":0,"num_object_copies":13830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4610,"num_write_kb":295040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":178,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.496178+0000","last_change":"2024-03-21T13:15:04.365662+0000","last_active":"2024-03-21T13:17:06.496178+0000","last_peered":"2024-03-21T13:17:06.496178+0000","last_clean":"2024-03-21T13:17:06.496178+0000","last_became_active":"2024-03-21T13:12:29.527083+0000","last_became_peered":"2024-03-21T13:12:29.527083+0000","last_unstale":"2024-03-21T13:17:06.496178+0000","last_undegraded":"2024-03-21T13:17:06.496178+0000","last_fullsized":"2024-03-21T13:17:06.496178+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:04.365534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:04.365534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":1006,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.491164+0000","last_change":"2024-03-21T13:16:14.628486+0000","last_active":"2024-03-21T13:17:06.491164+0000","last_peered":"2024-03-21T13:17:06.491164+0000","last_clean":"2024-03-21T13:17:06.491164+0000","last_became_active":"2024-03-21T12:58:27.160988+0000","last_became_peered":"2024-03-21T12:58:27.160988+0000","last_unstale":"2024-03-21T13:17:06.491164+0000","last_undegraded":"2024-03-21T13:17:06.491164+0000","last_fullsized":"2024-03-21T13:17:06.491164+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:16:14.628384+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:16:14.628384+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.536448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":253,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.620677+0000","last_change":"2024-03-21T13:16:02.247782+0000","last_active":"2024-03-21T13:17:10.620677+0000","last_peered":"2024-03-21T13:17:10.620677+0000","last_clean":"2024-03-21T13:17:10.620677+0000","last_became_active":"2024-03-21T13:12:29.499989+0000","last_became_peered":"2024-03-21T13:12:29.499989+0000","last_unstale":"2024-03-21T13:17:10.620677+0000","last_undegraded":"2024-03-21T13:17:10.620677+0000","last_fullsized":"2024-03-21T13:17:10.620677+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:02.247693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:02.247693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.746970+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"234'36","reported_seq":1016,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.502903+0000","last_change":"2024-03-21T13:16:05.528913+0000","last_active":"2024-03-21T13:17:06.502903+0000","last_peered":"2024-03-21T13:17:06.502903+0000","last_clean":"2024-03-21T13:17:06.502903+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:17:06.502903+0000","last_undegraded":"2024-03-21T13:17:06.502903+0000","last_fullsized":"2024-03-21T13:17:06.502903+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'36","last_scrub_stamp":"2024-03-21T13:16:05.528782+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.528782+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:30.885762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":771,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.494762+0000","last_change":"2024-03-21T13:17:02.759517+0000","last_active":"2024-03-21T13:17:06.494762+0000","last_peered":"2024-03-21T13:17:06.494762+0000","last_clean":"2024-03-21T13:17:06.494762+0000","last_became_active":"2024-03-21T13:12:29.507753+0000","last_became_peered":"2024-03-21T13:12:29.507753+0000","last_unstale":"2024-03-21T13:17:06.494762+0000","last_undegraded":"2024-03-21T13:17:06.494762+0000","last_fullsized":"2024-03-21T13:17:06.494762+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:02.759379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:17:02.759379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:12.075990+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"411'4684","reported_seq":4736,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:11.182997+0000","last_change":"2024-03-21T13:16:13.900952+0000","last_active":"2024-03-21T13:17:11.182997+0000","last_peered":"2024-03-21T13:17:11.182997+0000","last_clean":"2024-03-21T13:17:11.182997+0000","last_became_active":"2024-03-21T13:16:13.900650+0000","last_became_peered":"2024-03-21T13:16:13.900650+0000","last_unstale":"2024-03-21T13:17:11.182997+0000","last_undegraded":"2024-03-21T13:17:11.182997+0000","last_fullsized":"2024-03-21T13:17:11.182997+0000","mapping_epoch":390,"log_start":"399'200","ondisk_log_start":"399'200","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:15.174786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306970624,"num_objects":4684,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4684,"num_write_kb":299776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":178,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.493627+0000","last_change":"2024-03-21T13:16:01.195705+0000","last_active":"2024-03-21T13:17:06.493627+0000","last_peered":"2024-03-21T13:17:06.493627+0000","last_clean":"2024-03-21T13:17:06.493627+0000","last_became_active":"2024-03-21T13:12:29.514542+0000","last_became_peered":"2024-03-21T13:12:29.514542+0000","last_unstale":"2024-03-21T13:17:06.493627+0000","last_undegraded":"2024-03-21T13:17:06.493627+0000","last_fullsized":"2024-03-21T13:17:06.493627+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:01.195618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:01.195618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:22.887658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":209,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.623751+0000","last_change":"2024-03-21T13:17:00.087495+0000","last_active":"2024-03-21T13:17:10.623751+0000","last_peered":"2024-03-21T13:17:10.623751+0000","last_clean":"2024-03-21T13:17:10.623751+0000","last_became_active":"2024-03-21T13:15:04.458084+0000","last_became_peered":"2024-03-21T13:15:04.458084+0000","last_unstale":"2024-03-21T13:17:10.623751+0000","last_undegraded":"2024-03-21T13:17:10.623751+0000","last_fullsized":"2024-03-21T13:17:10.623751+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:00.086941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:00.086941+0000","last_clean_scrub_stamp":"2024-03-21T13:17:00.086941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:08.803635+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":1872,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:09.314865+0000","last_change":"2024-03-21T13:16:09.547909+0000","last_active":"2024-03-21T13:17:09.314865+0000","last_peered":"2024-03-21T13:17:09.314865+0000","last_clean":"2024-03-21T13:17:09.314865+0000","last_became_active":"2024-03-21T13:02:14.341628+0000","last_became_peered":"2024-03-21T13:02:14.341628+0000","last_unstale":"2024-03-21T13:17:09.314865+0000","last_undegraded":"2024-03-21T13:17:09.314865+0000","last_fullsized":"2024-03-21T13:17:09.314865+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:16:09.547783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:16:09.547783+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:28.689027+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1102,"num_read_kb":1097,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":882,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.623956+0000","last_change":"2024-03-21T13:16:18.883144+0000","last_active":"2024-03-21T13:17:10.623956+0000","last_peered":"2024-03-21T13:17:10.623956+0000","last_clean":"2024-03-21T13:17:10.623956+0000","last_became_active":"2024-03-21T13:10:03.606672+0000","last_became_peered":"2024-03-21T13:10:03.606672+0000","last_unstale":"2024-03-21T13:17:10.623956+0000","last_undegraded":"2024-03-21T13:17:10.623956+0000","last_fullsized":"2024-03-21T13:17:10.623956+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-21T13:16:18.882994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.882994+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:43.518929+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":186,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:08.238044+0000","last_change":"2024-03-21T13:17:08.238066+0000","last_active":"2024-03-21T13:17:08.238044+0000","last_peered":"2024-03-21T13:17:08.238044+0000","last_clean":"2024-03-21T13:17:08.238044+0000","last_became_active":"2024-03-21T13:12:29.526100+0000","last_became_peered":"2024-03-21T13:12:29.526100+0000","last_unstale":"2024-03-21T13:17:08.238044+0000","last_undegraded":"2024-03-21T13:17:08.238044+0000","last_fullsized":"2024-03-21T13:17:08.238044+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:08.237986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:08.237986+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.237986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:12.000031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"411'4533","reported_seq":4584,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.625589+0000","last_change":"2024-03-21T13:16:13.899271+0000","last_active":"2024-03-21T13:17:10.625589+0000","last_peered":"2024-03-21T13:17:10.625589+0000","last_clean":"2024-03-21T13:17:10.625589+0000","last_became_active":"2024-03-21T13:16:13.898890+0000","last_became_peered":"2024-03-21T13:16:13.898890+0000","last_unstale":"2024-03-21T13:17:10.625589+0000","last_undegraded":"2024-03-21T13:17:10.625589+0000","last_fullsized":"2024-03-21T13:17:10.625589+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4533,"log_dups_size":0,"ondisk_log_size":4533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:39.631299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297074688,"num_objects":4533,"num_object_clones":0,"num_object_copies":13599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4533,"num_write_kb":290112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":646,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.500757+0000","last_change":"2024-03-21T13:16:15.593032+0000","last_active":"2024-03-21T13:17:06.500757+0000","last_peered":"2024-03-21T13:17:06.500757+0000","last_clean":"2024-03-21T13:17:06.500757+0000","last_became_active":"2024-03-21T13:10:03.591088+0000","last_became_peered":"2024-03-21T13:10:03.591088+0000","last_unstale":"2024-03-21T13:17:06.500757+0000","last_undegraded":"2024-03-21T13:17:06.500757+0000","last_fullsized":"2024-03-21T13:17:06.500757+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:15.592931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:15.592931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:36.752324+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":322,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.621471+0000","last_change":"2024-03-21T13:16:22.293702+0000","last_active":"2024-03-21T13:17:10.621471+0000","last_peered":"2024-03-21T13:17:10.621471+0000","last_clean":"2024-03-21T13:17:10.621471+0000","last_became_active":"2024-03-21T13:12:29.507783+0000","last_became_peered":"2024-03-21T13:12:29.507783+0000","last_unstale":"2024-03-21T13:17:10.621471+0000","last_undegraded":"2024-03-21T13:17:10.621471+0000","last_fullsized":"2024-03-21T13:17:10.621471+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:22.293625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.293625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:23.688902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":204,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.494122+0000","last_change":"2024-03-21T13:17:03.309407+0000","last_active":"2024-03-21T13:17:06.494122+0000","last_peered":"2024-03-21T13:17:06.494122+0000","last_clean":"2024-03-21T13:17:06.494122+0000","last_became_active":"2024-03-21T13:12:29.518800+0000","last_became_peered":"2024-03-21T13:12:29.518800+0000","last_unstale":"2024-03-21T13:17:06.494122+0000","last_undegraded":"2024-03-21T13:17:06.494122+0000","last_fullsized":"2024-03-21T13:17:06.494122+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-21T13:17:03.309259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:03.309259+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:18.146498+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":773,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.500960+0000","last_change":"2024-03-21T13:17:02.830836+0000","last_active":"2024-03-21T13:17:06.500960+0000","last_peered":"2024-03-21T13:17:06.500960+0000","last_clean":"2024-03-21T13:17:06.500960+0000","last_became_active":"2024-03-21T13:12:29.520657+0000","last_became_peered":"2024-03-21T13:12:29.520657+0000","last_unstale":"2024-03-21T13:17:06.500960+0000","last_undegraded":"2024-03-21T13:17:06.500960+0000","last_fullsized":"2024-03-21T13:17:06.500960+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:02.830721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:02.830721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:31.769496+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"410'4570","reported_seq":4622,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627511+0000","last_change":"2024-03-21T13:16:13.899935+0000","last_active":"2024-03-21T13:17:10.627511+0000","last_peered":"2024-03-21T13:17:10.627511+0000","last_clean":"2024-03-21T13:17:10.627511+0000","last_became_active":"2024-03-21T13:16:13.899614+0000","last_became_peered":"2024-03-21T13:16:13.899614+0000","last_unstale":"2024-03-21T13:17:10.627511+0000","last_undegraded":"2024-03-21T13:17:10.627511+0000","last_fullsized":"2024-03-21T13:17:10.627511+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4570,"log_dups_size":0,"ondisk_log_size":4570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.233033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299499520,"num_objects":4570,"num_object_clones":0,"num_object_copies":13710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4570,"num_write_kb":292480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":443,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.622583+0000","last_change":"2024-03-21T13:16:51.070000+0000","last_active":"2024-03-21T13:17:10.622583+0000","last_peered":"2024-03-21T13:17:10.622583+0000","last_clean":"2024-03-21T13:17:10.622583+0000","last_became_active":"2024-03-21T13:15:04.456786+0000","last_became_peered":"2024-03-21T13:15:04.456786+0000","last_unstale":"2024-03-21T13:17:10.622583+0000","last_undegraded":"2024-03-21T13:17:10.622583+0000","last_fullsized":"2024-03-21T13:17:10.622583+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:51.069883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:51.069883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:10.981868+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":178,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.494576+0000","last_change":"2024-03-21T13:16:17.169677+0000","last_active":"2024-03-21T13:17:06.494576+0000","last_peered":"2024-03-21T13:17:06.494576+0000","last_clean":"2024-03-21T13:17:06.494576+0000","last_became_active":"2024-03-21T13:12:29.512291+0000","last_became_peered":"2024-03-21T13:12:29.512291+0000","last_unstale":"2024-03-21T13:17:06.494576+0000","last_undegraded":"2024-03-21T13:17:06.494576+0000","last_fullsized":"2024-03-21T13:17:06.494576+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:17.169572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:28.030369+0000","last_clean_scrub_stamp":"2024-03-21T13:16:17.169572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.905934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"234'25","reported_seq":190,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.495247+0000","last_change":"2024-03-21T13:15:57.053925+0000","last_active":"2024-03-21T13:17:06.495247+0000","last_peered":"2024-03-21T13:17:06.495247+0000","last_clean":"2024-03-21T13:17:06.495247+0000","last_became_active":"2024-03-21T13:12:29.526114+0000","last_became_peered":"2024-03-21T13:12:29.526114+0000","last_unstale":"2024-03-21T13:17:06.495247+0000","last_undegraded":"2024-03-21T13:17:06.495247+0000","last_fullsized":"2024-03-21T13:17:06.495247+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-21T13:15:57.053797+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:15:57.053797+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:09.141738+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"375'14","reported_seq":203,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.623268+0000","last_change":"2024-03-21T13:16:07.043970+0000","last_active":"2024-03-21T13:17:10.623268+0000","last_peered":"2024-03-21T13:17:10.623268+0000","last_clean":"2024-03-21T13:17:10.623268+0000","last_became_active":"2024-03-21T13:15:04.464590+0000","last_became_peered":"2024-03-21T13:15:04.464590+0000","last_unstale":"2024-03-21T13:17:10.623268+0000","last_undegraded":"2024-03-21T13:17:10.623268+0000","last_fullsized":"2024-03-21T13:17:10.623268+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'14","last_scrub_stamp":"2024-03-21T13:16:07.043880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.043880+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:17.795947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.e","version":"410'4582","reported_seq":4632,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.231539+0000","last_change":"2024-03-21T13:16:13.899791+0000","last_active":"2024-03-21T13:17:10.231539+0000","last_peered":"2024-03-21T13:17:10.231539+0000","last_clean":"2024-03-21T13:17:10.231539+0000","last_became_active":"2024-03-21T13:16:13.899480+0000","last_became_peered":"2024-03-21T13:16:13.899480+0000","last_unstale":"2024-03-21T13:17:10.231539+0000","last_undegraded":"2024-03-21T13:17:10.231539+0000","last_fullsized":"2024-03-21T13:17:10.231539+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4582,"log_dups_size":0,"ondisk_log_size":4582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:38.558703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300285952,"num_objects":4582,"num_object_clones":0,"num_object_copies":13746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4582,"num_write_kb":293248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":755,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:09.938627+0000","last_change":"2024-03-21T13:17:09.938651+0000","last_active":"2024-03-21T13:17:09.938627+0000","last_peered":"2024-03-21T13:17:09.938627+0000","last_clean":"2024-03-21T13:17:09.938627+0000","last_became_active":"2024-03-21T13:10:03.606557+0000","last_became_peered":"2024-03-21T13:10:03.606557+0000","last_unstale":"2024-03-21T13:17:09.938627+0000","last_undegraded":"2024-03-21T13:17:09.938627+0000","last_fullsized":"2024-03-21T13:17:09.938627+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:09.938560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:09.938560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:28.686201+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":809,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.624902+0000","last_change":"2024-03-21T13:16:57.064781+0000","last_active":"2024-03-21T13:17:10.624902+0000","last_peered":"2024-03-21T13:17:10.624902+0000","last_clean":"2024-03-21T13:17:10.624902+0000","last_became_active":"2024-03-21T13:02:14.333307+0000","last_became_peered":"2024-03-21T13:02:14.333307+0000","last_unstale":"2024-03-21T13:17:10.624902+0000","last_undegraded":"2024-03-21T13:17:10.624902+0000","last_fullsized":"2024-03-21T13:17:10.624902+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:16:57.064293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:57.064293+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:10.397884+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'35","reported_seq":202,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.493262+0000","last_change":"2024-03-21T13:17:01.334745+0000","last_active":"2024-03-21T13:17:06.493262+0000","last_peered":"2024-03-21T13:17:06.493262+0000","last_clean":"2024-03-21T13:17:06.493262+0000","last_became_active":"2024-03-21T13:12:29.525154+0000","last_became_peered":"2024-03-21T13:12:29.525154+0000","last_unstale":"2024-03-21T13:17:06.493262+0000","last_undegraded":"2024-03-21T13:17:06.493262+0000","last_fullsized":"2024-03-21T13:17:06.493262+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:17:01.334611+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:17:01.334611+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:06.225704+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"226'10","reported_seq":980,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627315+0000","last_change":"2024-03-21T13:16:18.672479+0000","last_active":"2024-03-21T13:17:10.627315+0000","last_peered":"2024-03-21T13:17:10.627315+0000","last_clean":"2024-03-21T13:17:10.627315+0000","last_became_active":"2024-03-21T13:02:14.343022+0000","last_became_peered":"2024-03-21T13:02:14.343022+0000","last_unstale":"2024-03-21T13:17:10.627315+0000","last_undegraded":"2024-03-21T13:17:10.627315+0000","last_fullsized":"2024-03-21T13:17:10.627315+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T13:16:18.672392+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.672392+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:25.253358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"410'4525","reported_seq":4574,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.329762+0000","last_change":"2024-03-21T13:16:13.905080+0000","last_active":"2024-03-21T13:17:10.329762+0000","last_peered":"2024-03-21T13:17:10.329762+0000","last_clean":"2024-03-21T13:17:10.329762+0000","last_became_active":"2024-03-21T13:16:13.904756+0000","last_became_peered":"2024-03-21T13:16:13.904756+0000","last_unstale":"2024-03-21T13:17:10.329762+0000","last_undegraded":"2024-03-21T13:17:10.329762+0000","last_fullsized":"2024-03-21T13:17:10.329762+0000","mapping_epoch":390,"log_start":"399'100","ondisk_log_start":"399'100","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4425,"log_dups_size":0,"ondisk_log_size":4425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:22.134211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296550400,"num_objects":4525,"num_object_clones":0,"num_object_copies":13575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4525,"num_write_kb":289600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":669,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.626569+0000","last_change":"2024-03-21T13:15:10.158338+0000","last_active":"2024-03-21T13:17:10.626569+0000","last_peered":"2024-03-21T13:17:10.626569+0000","last_clean":"2024-03-21T13:17:10.626569+0000","last_became_active":"2024-03-21T13:10:03.610471+0000","last_became_peered":"2024-03-21T13:10:03.610471+0000","last_unstale":"2024-03-21T13:17:10.626569+0000","last_undegraded":"2024-03-21T13:17:10.626569+0000","last_fullsized":"2024-03-21T13:17:10.626569+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:10.157920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:15:10.157920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":752,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627914+0000","last_change":"2024-03-21T13:17:05.843516+0000","last_active":"2024-03-21T13:17:10.627914+0000","last_peered":"2024-03-21T13:17:10.627914+0000","last_clean":"2024-03-21T13:17:10.627914+0000","last_became_active":"2024-03-21T12:58:37.730685+0000","last_became_peered":"2024-03-21T12:58:37.730685+0000","last_unstale":"2024-03-21T13:17:10.627914+0000","last_undegraded":"2024-03-21T13:17:10.627914+0000","last_fullsized":"2024-03-21T13:17:10.627914+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:05.843337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:17:05.843337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:23.558875+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"234'30","reported_seq":791,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627244+0000","last_change":"2024-03-21T13:15:09.148375+0000","last_active":"2024-03-21T13:17:10.627244+0000","last_peered":"2024-03-21T13:17:10.627244+0000","last_clean":"2024-03-21T13:17:10.627244+0000","last_became_active":"2024-03-21T12:58:39.751212+0000","last_became_peered":"2024-03-21T12:58:39.751212+0000","last_unstale":"2024-03-21T13:17:10.627244+0000","last_undegraded":"2024-03-21T13:17:10.627244+0000","last_fullsized":"2024-03-21T13:17:10.627244+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'30","last_scrub_stamp":"2024-03-21T13:15:09.148229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:09.148229+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":758,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.489704+0000","last_change":"2024-03-21T13:16:53.866824+0000","last_active":"2024-03-21T13:17:06.489704+0000","last_peered":"2024-03-21T13:17:06.489704+0000","last_clean":"2024-03-21T13:17:06.489704+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:17:06.489704+0000","last_undegraded":"2024-03-21T13:17:06.489704+0000","last_fullsized":"2024-03-21T13:17:06.489704+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:53.866692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.866692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:12.696345+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"411'4698","reported_seq":4750,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:11.193414+0000","last_change":"2024-03-21T13:16:13.900272+0000","last_active":"2024-03-21T13:17:11.193414+0000","last_peered":"2024-03-21T13:17:11.193414+0000","last_clean":"2024-03-21T13:17:11.193414+0000","last_became_active":"2024-03-21T13:16:13.899979+0000","last_became_peered":"2024-03-21T13:16:13.899979+0000","last_unstale":"2024-03-21T13:17:11.193414+0000","last_undegraded":"2024-03-21T13:17:11.193414+0000","last_fullsized":"2024-03-21T13:17:11.193414+0000","mapping_epoch":390,"log_start":"399'200","ondisk_log_start":"399'200","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4498,"log_dups_size":0,"ondisk_log_size":4498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:21.233864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307888128,"num_objects":4698,"num_object_clones":0,"num_object_copies":14094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4698,"num_write_kb":300672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":178,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.494707+0000","last_change":"2024-03-21T13:16:14.280619+0000","last_active":"2024-03-21T13:17:06.494707+0000","last_peered":"2024-03-21T13:17:06.494707+0000","last_clean":"2024-03-21T13:17:06.494707+0000","last_became_active":"2024-03-21T13:12:29.518722+0000","last_became_peered":"2024-03-21T13:12:29.518722+0000","last_unstale":"2024-03-21T13:17:06.494707+0000","last_undegraded":"2024-03-21T13:17:06.494707+0000","last_fullsized":"2024-03-21T13:17:06.494707+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:14.280515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:14.280515+0000","last_clean_scrub_stamp":"2024-03-21T13:16:14.280515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:42.015454+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"331'13","reported_seq":239,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.493954+0000","last_change":"2024-03-21T13:15:15.203026+0000","last_active":"2024-03-21T13:17:06.493954+0000","last_peered":"2024-03-21T13:17:06.493954+0000","last_clean":"2024-03-21T13:17:06.493954+0000","last_became_active":"2024-03-21T13:12:29.525058+0000","last_became_peered":"2024-03-21T13:12:29.525058+0000","last_unstale":"2024-03-21T13:17:06.493954+0000","last_undegraded":"2024-03-21T13:17:06.493954+0000","last_fullsized":"2024-03-21T13:17:06.493954+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:15:15.202826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:15.202826+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":758,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627205+0000","last_change":"2024-03-21T13:17:00.746992+0000","last_active":"2024-03-21T13:17:10.627205+0000","last_peered":"2024-03-21T13:17:10.627205+0000","last_clean":"2024-03-21T13:17:10.627205+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:17:10.627205+0000","last_undegraded":"2024-03-21T13:17:10.627205+0000","last_fullsized":"2024-03-21T13:17:10.627205+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:00.746886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:00.746886+0000","last_clean_scrub_stamp":"2024-03-21T13:17:00.746886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:22.824064+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"234'34","reported_seq":716,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.628008+0000","last_change":"2024-03-21T13:16:53.621472+0000","last_active":"2024-03-21T13:17:10.628008+0000","last_peered":"2024-03-21T13:17:10.628008+0000","last_clean":"2024-03-21T13:17:10.628008+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:17:10.628008+0000","last_undegraded":"2024-03-21T13:17:10.628008+0000","last_fullsized":"2024-03-21T13:17:10.628008+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'34","last_scrub_stamp":"2024-03-21T13:16:53.621335+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.621335+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:09.903027+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"410'4503","reported_seq":4550,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.511424+0000","last_change":"2024-03-21T13:16:13.904388+0000","last_active":"2024-03-21T13:17:10.511424+0000","last_peered":"2024-03-21T13:17:10.511424+0000","last_clean":"2024-03-21T13:17:10.511424+0000","last_became_active":"2024-03-21T13:16:13.904096+0000","last_became_peered":"2024-03-21T13:16:13.904096+0000","last_unstale":"2024-03-21T13:17:10.511424+0000","last_undegraded":"2024-03-21T13:17:10.511424+0000","last_fullsized":"2024-03-21T13:17:10.511424+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:39.867007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295108608,"num_objects":4503,"num_object_clones":0,"num_object_copies":13509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":288192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"375'2","reported_seq":754,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.626266+0000","last_change":"2024-03-21T13:15:14.313700+0000","last_active":"2024-03-21T13:17:10.626266+0000","last_peered":"2024-03-21T13:17:10.626266+0000","last_clean":"2024-03-21T13:17:10.626266+0000","last_became_active":"2024-03-21T13:12:29.502816+0000","last_became_peered":"2024-03-21T13:12:29.502816+0000","last_unstale":"2024-03-21T13:17:10.626266+0000","last_undegraded":"2024-03-21T13:17:10.626266+0000","last_fullsized":"2024-03-21T13:17:10.626266+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'2","last_scrub_stamp":"2024-03-21T13:15:14.313376+0000","last_deep_scrub":"375'2","last_deep_scrub_stamp":"2024-03-21T13:15:14.313376+0000","last_clean_scrub_stamp":"2024-03-21T13:15:14.313376+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":178,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.496000+0000","last_change":"2024-03-21T13:15:07.382672+0000","last_active":"2024-03-21T13:17:06.496000+0000","last_peered":"2024-03-21T13:17:06.496000+0000","last_clean":"2024-03-21T13:17:06.496000+0000","last_became_active":"2024-03-21T13:12:29.524087+0000","last_became_peered":"2024-03-21T13:12:29.524087+0000","last_unstale":"2024-03-21T13:17:06.496000+0000","last_undegraded":"2024-03-21T13:17:06.496000+0000","last_fullsized":"2024-03-21T13:17:06.496000+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:07.382536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:07.382536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":786,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.489799+0000","last_change":"2024-03-21T13:16:04.825360+0000","last_active":"2024-03-21T13:17:06.489799+0000","last_peered":"2024-03-21T13:17:06.489799+0000","last_clean":"2024-03-21T13:17:06.489799+0000","last_became_active":"2024-03-21T13:10:03.586593+0000","last_became_peered":"2024-03-21T13:10:03.586593+0000","last_unstale":"2024-03-21T13:17:06.489799+0000","last_undegraded":"2024-03-21T13:17:06.489799+0000","last_fullsized":"2024-03-21T13:17:06.489799+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:04.825280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:04.825280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:18.635897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":501,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.503153+0000","last_change":"2024-03-21T13:14:11.615096+0000","last_active":"2024-03-21T13:17:06.503153+0000","last_peered":"2024-03-21T13:17:06.503153+0000","last_clean":"2024-03-21T13:17:06.503153+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:17:06.503153+0000","last_undegraded":"2024-03-21T13:17:06.503153+0000","last_fullsized":"2024-03-21T13:17:06.503153+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:14:11.611538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:14:11.611538+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"411'4521","reported_seq":4573,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.622006+0000","last_change":"2024-03-21T13:16:13.899357+0000","last_active":"2024-03-21T13:17:10.622006+0000","last_peered":"2024-03-21T13:17:10.622006+0000","last_clean":"2024-03-21T13:17:10.622006+0000","last_became_active":"2024-03-21T13:16:13.899059+0000","last_became_peered":"2024-03-21T13:16:13.899059+0000","last_unstale":"2024-03-21T13:17:10.622006+0000","last_undegraded":"2024-03-21T13:17:10.622006+0000","last_fullsized":"2024-03-21T13:17:10.622006+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:25.856013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296288256,"num_objects":4521,"num_object_clones":0,"num_object_copies":13563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4521,"num_write_kb":289344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":324,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.621276+0000","last_change":"2024-03-21T13:16:20.483084+0000","last_active":"2024-03-21T13:17:10.621276+0000","last_peered":"2024-03-21T13:17:10.621276+0000","last_clean":"2024-03-21T13:17:10.621276+0000","last_became_active":"2024-03-21T13:10:03.602752+0000","last_became_peered":"2024-03-21T13:10:03.602752+0000","last_unstale":"2024-03-21T13:17:10.621276+0000","last_undegraded":"2024-03-21T13:17:10.621276+0000","last_fullsized":"2024-03-21T13:17:10.621276+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:20.482943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.482943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.858705+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":658,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.491121+0000","last_change":"2024-03-21T13:15:59.662672+0000","last_active":"2024-03-21T13:17:06.491121+0000","last_peered":"2024-03-21T13:17:06.491121+0000","last_clean":"2024-03-21T13:17:06.491121+0000","last_became_active":"2024-03-21T13:02:14.335141+0000","last_became_peered":"2024-03-21T13:02:14.335141+0000","last_unstale":"2024-03-21T13:17:06.491121+0000","last_undegraded":"2024-03-21T13:17:06.491121+0000","last_fullsized":"2024-03-21T13:17:06.491121+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:59.662591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:59.662591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:10.639394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":763,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:06.501046+0000","last_change":"2024-03-21T13:15:02.678593+0000","last_active":"2024-03-21T13:17:06.501046+0000","last_peered":"2024-03-21T13:17:06.501046+0000","last_clean":"2024-03-21T13:17:06.501046+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:17:06.501046+0000","last_undegraded":"2024-03-21T13:17:06.501046+0000","last_fullsized":"2024-03-21T13:17:06.501046+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:02.678485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:15:02.678485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"391'39","reported_seq":334,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.621685+0000","last_change":"2024-03-21T13:16:14.266738+0000","last_active":"2024-03-21T13:17:10.621685+0000","last_peered":"2024-03-21T13:17:10.621685+0000","last_clean":"2024-03-21T13:17:10.621685+0000","last_became_active":"2024-03-21T13:10:03.591832+0000","last_became_peered":"2024-03-21T13:10:03.591832+0000","last_unstale":"2024-03-21T13:17:10.621685+0000","last_undegraded":"2024-03-21T13:17:10.621685+0000","last_fullsized":"2024-03-21T13:17:10.621685+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'39","last_scrub_stamp":"2024-03-21T13:16:14.266573+0000","last_deep_scrub":"391'39","last_deep_scrub_stamp":"2024-03-21T13:16:14.266573+0000","last_clean_scrub_stamp":"2024-03-21T13:16:14.266573+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:36.013148+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"410'4595","reported_seq":4644,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.513253+0000","last_change":"2024-03-21T13:16:13.904234+0000","last_active":"2024-03-21T13:17:10.513253+0000","last_peered":"2024-03-21T13:17:10.513253+0000","last_clean":"2024-03-21T13:17:10.513253+0000","last_became_active":"2024-03-21T13:16:13.903407+0000","last_became_peered":"2024-03-21T13:16:13.903407+0000","last_unstale":"2024-03-21T13:17:10.513253+0000","last_undegraded":"2024-03-21T13:17:10.513253+0000","last_fullsized":"2024-03-21T13:17:10.513253+0000","mapping_epoch":390,"log_start":"0'0","ondisk_log_start":"0'0","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4595,"log_dups_size":0,"ondisk_log_size":4595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:24.659743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301137920,"num_objects":4595,"num_object_clones":0,"num_object_copies":13785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4595,"num_write_kb":294080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":322,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.620977+0000","last_change":"2024-03-21T13:16:18.322285+0000","last_active":"2024-03-21T13:17:10.620977+0000","last_peered":"2024-03-21T13:17:10.620977+0000","last_clean":"2024-03-21T13:17:10.620977+0000","last_became_active":"2024-03-21T13:12:29.499442+0000","last_became_peered":"2024-03-21T13:12:29.499442+0000","last_unstale":"2024-03-21T13:17:10.620977+0000","last_undegraded":"2024-03-21T13:17:10.620977+0000","last_fullsized":"2024-03-21T13:17:10.620977+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.322198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.322198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:40.851693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"331'13","reported_seq":417,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.621594+0000","last_change":"2024-03-21T13:16:21.250482+0000","last_active":"2024-03-21T13:17:10.621594+0000","last_peered":"2024-03-21T13:17:10.621594+0000","last_clean":"2024-03-21T13:17:10.621594+0000","last_became_active":"2024-03-21T13:12:29.507925+0000","last_became_peered":"2024-03-21T13:12:29.507925+0000","last_unstale":"2024-03-21T13:17:10.621594+0000","last_undegraded":"2024-03-21T13:17:10.621594+0000","last_fullsized":"2024-03-21T13:17:10.621594+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:16:21.250398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.250398+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:47.624689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":744,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.626727+0000","last_change":"2024-03-21T13:16:05.813771+0000","last_active":"2024-03-21T13:17:10.626727+0000","last_peered":"2024-03-21T13:17:10.626727+0000","last_clean":"2024-03-21T13:17:10.626727+0000","last_became_active":"2024-03-21T12:58:37.731971+0000","last_became_peered":"2024-03-21T12:58:37.731971+0000","last_unstale":"2024-03-21T13:17:10.626727+0000","last_undegraded":"2024-03-21T13:17:10.626727+0000","last_fullsized":"2024-03-21T13:17:10.626727+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:05.813657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.813657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:20.222745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"234'20","reported_seq":705,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-21T13:17:10.627230+0000","last_change":"2024-03-21T13:15:04.211127+0000","last_active":"2024-03-21T13:17:10.627230+0000","last_peered":"2024-03-21T13:17:10.627230+0000","last_clean":"2024-03-21T13:17:10.627230+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:17:10.627230+0000","last_undegraded":"2024-03-21T13:17:10.627230+0000","last_fullsized":"2024-03-21T13:17:10.627230+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:15:04.211011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:04.211011+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4790091776,"num_objects":73091,"num_object_clones":0,"num_object_copies":219273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73091,"num_write_kb":4677824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14335672320,"data_stored":14335672320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72091,"ondisk_log_size":72091,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1102,"num_read_kb":1097,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1100,"num_read_kb":857,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":6656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":753,"ondisk_log_size":753,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168803,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2941312,"kb_used_data":2031352,"kb_used_omap":1,"kb_used_meta":909950,"kb_avail":90803840,"statfs":{"total":95995035648,"available":92983132160,"internally_reserved":0,"allocated":2080104448,"data_stored":2079569063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":931789258},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463145,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3539072,"kb_used_data":2615224,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":90206080,"statfs":{"total":95995035648,"available":92371025920,"internally_reserved":0,"allocated":2677989376,"data_stored":2677453700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":946010566},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575105,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738631,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3159864,"kb_used_data":2302640,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":90585288,"statfs":{"total":95995035648,"available":92759334912,"internally_reserved":0,"allocated":2357903360,"data_stored":2357374815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877787589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411581,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2512148,"kb_used_data":1727116,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":91233004,"statfs":{"total":95995035648,"available":93422596096,"internally_reserved":0,"allocated":1768566784,"data_stored":1768074187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182648,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3084808,"kb_used_data":2052668,"kb_used_omap":1,"kb_used_meta":1032126,"kb_avail":90660344,"statfs":{"total":95995035648,"available":92836192256,"internally_reserved":0,"allocated":2101932032,"data_stored":2101398219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1056897479},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019122,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2717920,"kb_used_data":1726040,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":91027232,"statfs":{"total":95995035648,"available":93211885568,"internally_reserved":0,"allocated":1767464960,"data_stored":1766960554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1015675333},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724781,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1677592,"kb_used_data":1429776,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92067560,"statfs":{"total":95995035648,"available":94277181440,"internally_reserved":0,"allocated":1464090624,"data_stored":1463593609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2382561280,"data_stored":2382561280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1791623168,"data_stored":1791623168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2109669376,"data_stored":2109669376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1776680960,"data_stored":1776680960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1479737344,"data_stored":1479737344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2698575872,"data_stored":2698575872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2096824320,"data_stored":2096824320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:17:14.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:14.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:14.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:14.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:14.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:14.742 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:14.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:14.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:14.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:14 smithi163 ceph-mon[29621]: 3.0 scrub starts 2024-03-21T13:17:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:14 smithi163 ceph-mon[29621]: 3.0 scrub ok 2024-03-21T13:17:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:14 smithi163 ceph-mon[29621]: 5.17 deep-scrub starts 2024-03-21T13:17:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:14 smithi163 ceph-mon[29621]: 5.17 deep-scrub ok 2024-03-21T13:17:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:14 smithi163 ceph-mon[29621]: 3.17 scrub starts 2024-03-21T13:17:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:14 smithi163 ceph-mon[29621]: 3.17 scrub ok 2024-03-21T13:17:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:14 smithi163 ceph-mon[40675]: 3.0 scrub starts 2024-03-21T13:17:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:14 smithi163 ceph-mon[40675]: 3.0 scrub ok 2024-03-21T13:17:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:14 smithi163 ceph-mon[40675]: 5.17 deep-scrub starts 2024-03-21T13:17:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:14 smithi163 ceph-mon[40675]: 5.17 deep-scrub ok 2024-03-21T13:17:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:14 smithi163 ceph-mon[40675]: 3.17 scrub starts 2024-03-21T13:17:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:14 smithi163 ceph-mon[40675]: 3.17 scrub ok 2024-03-21T13:17:14.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:14.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:14.888 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:14.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:14.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:14 smithi204 ceph-mon[33153]: 3.0 scrub starts 2024-03-21T13:17:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:14 smithi204 ceph-mon[33153]: 3.0 scrub ok 2024-03-21T13:17:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:14 smithi204 ceph-mon[33153]: 5.17 deep-scrub starts 2024-03-21T13:17:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:14 smithi204 ceph-mon[33153]: 5.17 deep-scrub ok 2024-03-21T13:17:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:14 smithi204 ceph-mon[33153]: 3.17 scrub starts 2024-03-21T13:17:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:14 smithi204 ceph-mon[33153]: 3.17 scrub ok 2024-03-21T13:17:15.029 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:15.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:15.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:15.172 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:15.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:15.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:15.273 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:15.305 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:15.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:15.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:15.333 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 46 16 78990 78974 107.291 75.125 0.00930536 0.00931501 2024-03-21T13:17:15.407 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:15.441 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:15.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:15.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:15.583 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:15.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:15.685 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:15.724 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:15.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:15.826 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:15.860 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:15.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:15.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:15.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:15.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:16.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:15 smithi204 ceph-mon[33153]: pgmap v874: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 47 MiB/s wr, 756 op/s 2024-03-21T13:17:16.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:15 smithi163 ceph-mon[29621]: pgmap v874: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 47 MiB/s wr, 756 op/s 2024-03-21T13:17:16.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:15 smithi163 ceph-mon[40675]: pgmap v874: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 47 MiB/s wr, 756 op/s 2024-03-21T13:17:16.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:17:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:17:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:17:16.097 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:16.135 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:16.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:16.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:16.237 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:16.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:16.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:16.334 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 47 16 80299 80283 106.748 81.8125 0.00765286 0.00936359 2024-03-21T13:17:16.386 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:16.421 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:16.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:16.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:16.523 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:16.555 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:16.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:16.656 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:16.697 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:16.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:16.798 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:16.840 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:16.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:16.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:16.942 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:16.979 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:16.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:17.082 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:17.119 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:17.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:17.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:17.220 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:17.257 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:17.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:17.334 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 48 16 81585 81569 106.198 80.375 0.0133959 0.00941157 2024-03-21T13:17:17.359 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:17.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:17.396 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:17.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:17.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:17.531 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:17.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:17.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:17.633 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:17.672 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:17.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:17.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:17.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:17.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:17 smithi163 ceph-mon[29621]: 5.1c scrub starts 2024-03-21T13:17:17.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:17 smithi163 ceph-mon[29621]: 5.1c scrub ok 2024-03-21T13:17:17.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:17 smithi163 ceph-mon[29621]: 3.10 scrub starts 2024-03-21T13:17:17.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:17 smithi163 ceph-mon[29621]: 3.10 scrub ok 2024-03-21T13:17:17.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:17 smithi163 ceph-mon[40675]: 5.1c scrub starts 2024-03-21T13:17:17.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:17 smithi163 ceph-mon[40675]: 5.1c scrub ok 2024-03-21T13:17:17.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:17 smithi163 ceph-mon[40675]: 3.10 scrub starts 2024-03-21T13:17:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:17 smithi163 ceph-mon[40675]: 3.10 scrub ok 2024-03-21T13:17:17.806 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:17.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:17.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:17.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:17.909 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:17:17.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:17.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:17.952 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:17:17.997 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:17:18.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:17 smithi204 ceph-mon[33153]: 5.1c scrub starts 2024-03-21T13:17:18.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:17 smithi204 ceph-mon[33153]: 5.1c scrub ok 2024-03-21T13:17:18.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:17 smithi204 ceph-mon[33153]: 3.10 scrub starts 2024-03-21T13:17:18.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:17 smithi204 ceph-mon[33153]: 3.10 scrub ok 2024-03-21T13:17:18.034 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:17:18.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:18.071 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:18.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:18.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:18.073 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:17:18.114 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:17:18.163 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:17:18.173 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:18.205 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:18.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:18.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:18.206 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:17:18.212 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:17:18.212 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:17:18.212 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:17:18.212 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:17:18.213 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:17:18.213 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:17:18.213 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:17:18.213 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:17:18.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:18.334 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 49 16 82759 82743 105.528 73.375 0.00679244 0.00947129 2024-03-21T13:17:18.343 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:18.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:18.444 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:18.483 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:18.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:18.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:18.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:18.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:18.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:18.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:18.761 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:18.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:18.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:18.894 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:18.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:18.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:19.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:18 smithi204 ceph-mon[33153]: 3.13 scrub starts 2024-03-21T13:17:19.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:18 smithi204 ceph-mon[33153]: 3.13 scrub ok 2024-03-21T13:17:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:18 smithi204 ceph-mon[33153]: pgmap v875: 148 pgs: 148 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-21T13:17:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:18 smithi204 ceph-mon[33153]: 4.c scrub starts 2024-03-21T13:17:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:18 smithi204 ceph-mon[33153]: 4.c scrub ok 2024-03-21T13:17:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:18 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/377590517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:17:19.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:18 smithi163 ceph-mon[29621]: 3.13 scrub starts 2024-03-21T13:17:19.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:18 smithi163 ceph-mon[29621]: 3.13 scrub ok 2024-03-21T13:17:19.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:18 smithi163 ceph-mon[29621]: pgmap v875: 148 pgs: 148 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-21T13:17:19.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:18 smithi163 ceph-mon[29621]: 4.c scrub starts 2024-03-21T13:17:19.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:18 smithi163 ceph-mon[29621]: 4.c scrub ok 2024-03-21T13:17:19.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:18 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/377590517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:17:19.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:18 smithi163 ceph-mon[40675]: 3.13 scrub starts 2024-03-21T13:17:19.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:18 smithi163 ceph-mon[40675]: 3.13 scrub ok 2024-03-21T13:17:19.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:18 smithi163 ceph-mon[40675]: pgmap v875: 148 pgs: 148 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-21T13:17:19.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:18 smithi163 ceph-mon[40675]: 4.c scrub starts 2024-03-21T13:17:19.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:18 smithi163 ceph-mon[40675]: 4.c scrub ok 2024-03-21T13:17:19.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:18 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/377590517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:17:19.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:19.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:19.134 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:19.170 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:19.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:19.271 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:19.312 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:19.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:19.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:19.334 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 50 16 83943 83927 104.898 74 0.0203411 0.00952811 2024-03-21T13:17:19.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:19.452 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:19.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:19.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:19.554 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:19.568 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 2, 5] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:17:19.568 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:17:19.568 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 3, 4, 6, 7, 2, 5] 2024-03-21T13:17:19.569 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd out 0 2024-03-21T13:17:19.602 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:19.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:19.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:19.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:19.746 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:19.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:19.815 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:17:19.818 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:19 smithi163 ceph-mon[29621]: 4.7 scrub starts 2024-03-21T13:17:19.818 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:19 smithi163 ceph-mon[29621]: 4.7 scrub ok 2024-03-21T13:17:19.818 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:19 smithi163 ceph-mon[29621]: 11.6 scrub starts 2024-03-21T13:17:19.818 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:19 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/901198275' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:17:19.819 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:19 smithi163 ceph-mon[40675]: 4.7 scrub starts 2024-03-21T13:17:19.819 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:19 smithi163 ceph-mon[40675]: 4.7 scrub ok 2024-03-21T13:17:19.819 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:19 smithi163 ceph-mon[40675]: 11.6 scrub starts 2024-03-21T13:17:19.819 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:19 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/901198275' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:17:19.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:19.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:19.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:19.986 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:20.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:19 smithi204 ceph-mon[33153]: 4.7 scrub starts 2024-03-21T13:17:20.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:19 smithi204 ceph-mon[33153]: 4.7 scrub ok 2024-03-21T13:17:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:19 smithi204 ceph-mon[33153]: 11.6 scrub starts 2024-03-21T13:17:20.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:19 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/901198275' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:17:20.020 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:20.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:20.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:20.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:20.156 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:20.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:20.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:20.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:20.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:20.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:20.334 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 51 16 85195 85179 104.375 78.25 0.015236 0.00957615 2024-03-21T13:17:20.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:20.427 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:20.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:20.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:20.567 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:20.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:20.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:20.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:20.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:20.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:20.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:20.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:20.853 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:20.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:20.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[40675]: 5.1a scrub starts 2024-03-21T13:17:20.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[40675]: 5.1a scrub ok 2024-03-21T13:17:20.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[40675]: pgmap v876: 148 pgs: 148 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 613 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-21T13:17:20.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[40675]: 4.0 scrub starts 2024-03-21T13:17:20.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[40675]: 4.0 scrub ok 2024-03-21T13:17:20.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:17:20.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/901198275' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:17:20.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[40675]: osdmap e413: 8 total, 8 up, 7 in 2024-03-21T13:17:20.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:17:20.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:20.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[29621]: 5.1a scrub starts 2024-03-21T13:17:20.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[29621]: 5.1a scrub ok 2024-03-21T13:17:20.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[29621]: pgmap v876: 148 pgs: 148 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 613 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-21T13:17:20.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[29621]: 4.0 scrub starts 2024-03-21T13:17:20.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[29621]: 4.0 scrub ok 2024-03-21T13:17:20.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:17:20.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/901198275' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:17:20.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[29621]: osdmap e413: 8 total, 8 up, 7 in 2024-03-21T13:17:20.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:17:20.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:20.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:20.989 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:20.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:21.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:20 smithi204 ceph-mon[33153]: 5.1a scrub starts 2024-03-21T13:17:21.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:20 smithi204 ceph-mon[33153]: 5.1a scrub ok 2024-03-21T13:17:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:20 smithi204 ceph-mon[33153]: pgmap v876: 148 pgs: 148 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 613 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-21T13:17:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:20 smithi204 ceph-mon[33153]: 4.0 scrub starts 2024-03-21T13:17:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:20 smithi204 ceph-mon[33153]: 4.0 scrub ok 2024-03-21T13:17:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:20 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:17:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:20 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/901198275' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:17:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:20 smithi204 ceph-mon[33153]: osdmap e413: 8 total, 8 up, 7 in 2024-03-21T13:17:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:17:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:21.091 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:21.099 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:17:21.128 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:21.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:21.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:21.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:21.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:21.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:21.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:21.296 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:17:21.334 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 52 16 86424 86408 103.844 76.8125 0.0215547 0.0096245 2024-03-21T13:17:21.354 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:17:21.355 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:17:21.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.355 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:86409 11.6 11:61fda957:::benchmark_data_smithi163_180069_object86408:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-21T13:17:21.355 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:21.317551+0000", 2024-03-21T13:17:21.355 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.027114728000000001, 2024-03-21T13:17:21.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027159368, 2024-03-21T13:17:21.355 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:17:21.355 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:17:21.355 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:17:21.355 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:17:21.356 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:17:21.356 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 86409 2024-03-21T13:17:21.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.356 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:17:21.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:17:21.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.317551+0000", 2024-03-21T13:17:21.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:17:21.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.317551+0000", 2024-03-21T13:17:21.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:17:21.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.317552+0000", 2024-03-21T13:17:21.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6199999999999999e-06 2024-03-21T13:17:21.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:17:21.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.317590+0000", 2024-03-21T13:17:21.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7151999999999999e-05 2024-03-21T13:17:21.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:17:21.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.317603+0000", 2024-03-21T13:17:21.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3157e-05 2024-03-21T13:17:21.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:17:21.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.317611+0000", 2024-03-21T13:17:21.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2810000000000008e-06 2024-03-21T13:17:21.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:17:21.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.331490+0000", 2024-03-21T13:17:21.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013878685999999999 2024-03-21T13:17:21.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:17:21.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.331701+0000", 2024-03-21T13:17:21.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00021174799999999999 2024-03-21T13:17:21.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4", 2024-03-21T13:17:21.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.332247+0000", 2024-03-21T13:17:21.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000545329 2024-03-21T13:17:21.359 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:86412 11.6 11:6978b2f4:::benchmark_data_smithi163_180069_object86411:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-21T13:17:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:21.322687+0000", 2024-03-21T13:17:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.021978310000000001, 2024-03-21T13:17:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022121641000000001, 2024-03-21T13:17:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:17:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:17:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:17:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:17:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:17:21.361 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 86412 2024-03-21T13:17:21.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.361 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:17:21.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:17:21.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.322687+0000", 2024-03-21T13:17:21.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:17:21.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.322687+0000", 2024-03-21T13:17:21.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:17:21.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.322689+0000", 2024-03-21T13:17:21.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3969999999999999e-06 2024-03-21T13:17:21.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:17:21.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.322713+0000", 2024-03-21T13:17:21.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4071999999999999e-05 2024-03-21T13:17:21.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:17:21.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.322724+0000", 2024-03-21T13:17:21.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1738e-05 2024-03-21T13:17:21.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:17:21.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.322732+0000", 2024-03-21T13:17:21.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9950000000000005e-06 2024-03-21T13:17:21.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:17:21.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.336667+0000", 2024-03-21T13:17:21.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013934480000000001 2024-03-21T13:17:21.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:17:21.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.336825+0000", 2024-03-21T13:17:21.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015757499999999999 2024-03-21T13:17:21.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4", 2024-03-21T13:17:21.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.336975+0000", 2024-03-21T13:17:21.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015088800000000001 2024-03-21T13:17:21.364 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.364 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:21.364 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.365 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:86410 11.e e413/390 11:7796b093:::benchmark_data_smithi163_180069_object86409:head v 413'5438, mlcod=413'5436)", 2024-03-21T13:17:21.365 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:21.322936+0000", 2024-03-21T13:17:21.365 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.021729460999999999, 2024-03-21T13:17:21.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.02195341, 2024-03-21T13:17:21.365 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:17:21.365 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:17:21.365 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:17:21.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:17:21.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.322936+0000", 2024-03-21T13:17:21.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:17:21.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.322936+0000", 2024-03-21T13:17:21.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:17:21.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.322937+0000", 2024-03-21T13:17:21.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8599999999999998e-07 2024-03-21T13:17:21.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:17:21.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.322972+0000", 2024-03-21T13:17:21.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5027999999999997e-05 2024-03-21T13:17:21.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:17:21.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.322984+0000", 2024-03-21T13:17:21.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2408e-05 2024-03-21T13:17:21.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:17:21.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.322991+0000", 2024-03-21T13:17:21.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6080000000000001e-06 2024-03-21T13:17:21.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:17:21.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.325305+0000", 2024-03-21T13:17:21.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023140750000000001 2024-03-21T13:17:21.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:17:21.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.325331+0000", 2024-03-21T13:17:21.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6106999999999999e-05 2024-03-21T13:17:21.368 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.368 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:21.369 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.369 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:86418 11.6 11:6e4f3a89:::benchmark_data_smithi163_180069_object86417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-21T13:17:21.369 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:21.331222+0000", 2024-03-21T13:17:21.369 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.013443620999999999, 2024-03-21T13:17:21.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013730119000000001, 2024-03-21T13:17:21.369 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:17:21.369 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:17:21.369 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:17:21.369 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:17:21.369 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:17:21.370 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 86418 2024-03-21T13:17:21.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.370 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:17:21.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:17:21.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.331222+0000", 2024-03-21T13:17:21.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:17:21.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.331222+0000", 2024-03-21T13:17:21.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:17:21.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.331224+0000", 2024-03-21T13:17:21.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.841e-06 2024-03-21T13:17:21.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:17:21.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.331248+0000", 2024-03-21T13:17:21.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3937000000000001e-05 2024-03-21T13:17:21.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:17:21.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.331258+0000", 2024-03-21T13:17:21.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0366999999999999e-05 2024-03-21T13:17:21.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:17:21.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.331266+0000", 2024-03-21T13:17:21.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1049999999999995e-06 2024-03-21T13:17:21.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:17:21.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.339668+0000", 2024-03-21T13:17:21.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084015789999999993 2024-03-21T13:17:21.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:17:21.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.339801+0000", 2024-03-21T13:17:21.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013306400000000001 2024-03-21T13:17:21.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4", 2024-03-21T13:17:21.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.339979+0000", 2024-03-21T13:17:21.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017848699999999999 2024-03-21T13:17:21.373 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.373 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:21.374 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.374 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:86417 11.9 e413/390 11:95eeab33:::benchmark_data_smithi163_180069_object86416:head v 413'5494, mlcod=413'5491)", 2024-03-21T13:17:21.374 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:21.333685+0000", 2024-03-21T13:17:21.374 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.010980098000000001, 2024-03-21T13:17:21.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011348379, 2024-03-21T13:17:21.374 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:17:21.374 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:17:21.374 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:17:21.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:17:21.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.333685+0000", 2024-03-21T13:17:21.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:17:21.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.333685+0000", 2024-03-21T13:17:21.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:17:21.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.333687+0000", 2024-03-21T13:17:21.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0690000000000001e-06 2024-03-21T13:17:21.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:17:21.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.333713+0000", 2024-03-21T13:17:21.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6075e-05 2024-03-21T13:17:21.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:17:21.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.333728+0000", 2024-03-21T13:17:21.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4995e-05 2024-03-21T13:17:21.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:17:21.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.333736+0000", 2024-03-21T13:17:21.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5029999999999997e-06 2024-03-21T13:17:21.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:17:21.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.333829+0000", 2024-03-21T13:17:21.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3170000000000004e-05 2024-03-21T13:17:21.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:17:21.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.333857+0000", 2024-03-21T13:17:21.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7226999999999999e-05 2024-03-21T13:17:21.377 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.377 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:21.378 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.378 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:86430 11.6 11:6fa580f9:::benchmark_data_smithi163_180069_object86429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-21T13:17:21.378 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:21.339990+0000", 2024-03-21T13:17:21.378 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.0046756519999999998, 2024-03-21T13:17:21.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0051063059999999997, 2024-03-21T13:17:21.378 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:17:21.378 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for sub ops", 2024-03-21T13:17:21.378 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:17:21.378 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:17:21.378 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:17:21.379 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 86430 2024-03-21T13:17:21.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.379 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:17:21.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:17:21.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.339990+0000", 2024-03-21T13:17:21.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:17:21.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.339990+0000", 2024-03-21T13:17:21.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:17:21.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.339991+0000", 2024-03-21T13:17:21.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5540000000000001e-06 2024-03-21T13:17:21.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:17:21.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340016+0000", 2024-03-21T13:17:21.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4287999999999999e-05 2024-03-21T13:17:21.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:17:21.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340038+0000", 2024-03-21T13:17:21.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2741000000000001e-05 2024-03-21T13:17:21.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:17:21.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340048+0000", 2024-03-21T13:17:21.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5699999999999999e-06 2024-03-21T13:17:21.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:17:21.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.342567+0000", 2024-03-21T13:17:21.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0025185559999999999 2024-03-21T13:17:21.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:17:21.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.342724+0000", 2024-03-21T13:17:21.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015692599999999999 2024-03-21T13:17:21.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4", 2024-03-21T13:17:21.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.342890+0000", 2024-03-21T13:17:21.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000166393 2024-03-21T13:17:21.382 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.382 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:21.383 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.383 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:86424 11.2 e413/390 11:44e842d0:::benchmark_data_smithi163_180069_object86423:head v 413'5370, mlcod=413'5368)", 2024-03-21T13:17:21.383 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:21.340035+0000", 2024-03-21T13:17:21.383 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.0046306719999999997, 2024-03-21T13:17:21.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0051319249999999999, 2024-03-21T13:17:21.383 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:17:21.383 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:17:21.383 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:17:21.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:17:21.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340035+0000", 2024-03-21T13:17:21.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:17:21.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340035+0000", 2024-03-21T13:17:21.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:17:21.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340036+0000", 2024-03-21T13:17:21.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3900000000000003e-07 2024-03-21T13:17:21.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:17:21.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340067+0000", 2024-03-21T13:17:21.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1090000000000002e-05 2024-03-21T13:17:21.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:17:21.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340078+0000", 2024-03-21T13:17:21.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1233000000000001e-05 2024-03-21T13:17:21.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:17:21.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340086+0000", 2024-03-21T13:17:21.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5719999999999997e-06 2024-03-21T13:17:21.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:17:21.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340389+0000", 2024-03-21T13:17:21.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000303744 2024-03-21T13:17:21.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:17:21.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340412+0000", 2024-03-21T13:17:21.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2354999999999999e-05 2024-03-21T13:17:21.387 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.387 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:21.387 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.387 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.15858.0:86409 11.6 e413/390)", 2024-03-21T13:17:21.387 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:21.340117+0000", 2024-03-21T13:17:21.387 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.0045486529999999997, 2024-03-21T13:17:21.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0051073919999999997, 2024-03-21T13:17:21.387 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:17:21.387 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "queued for pg", 2024-03-21T13:17:21.387 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:17:21.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:17:21.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340117+0000", 2024-03-21T13:17:21.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:17:21.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340117+0000", 2024-03-21T13:17:21.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:17:21.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340117+0000", 2024-03-21T13:17:21.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1100000000000001e-07 2024-03-21T13:17:21.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:17:21.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340121+0000", 2024-03-21T13:17:21.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6210000000000001e-06 2024-03-21T13:17:21.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:17:21.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340128+0000", 2024-03-21T13:17:21.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7900000000000002e-06 2024-03-21T13:17:21.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:17:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340134+0000", 2024-03-21T13:17:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7869999999999998e-06 2024-03-21T13:17:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.15858.0:86409 11.6 e413/390)", 2024-03-21T13:17:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:21.340154+0000", 2024-03-21T13:17:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.004511531, 2024-03-21T13:17:21.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0051140329999999996, 2024-03-21T13:17:21.391 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:17:21.391 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "queued for pg", 2024-03-21T13:17:21.391 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:17:21.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:17:21.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340154+0000", 2024-03-21T13:17:21.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:17:21.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340154+0000", 2024-03-21T13:17:21.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:17:21.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340154+0000", 2024-03-21T13:17:21.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2199999999999999e-07 2024-03-21T13:17:21.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:17:21.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340157+0000", 2024-03-21T13:17:21.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7300000000000001e-06 2024-03-21T13:17:21.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:17:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340163+0000", 2024-03-21T13:17:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2570000000000004e-06 2024-03-21T13:17:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:17:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340168+0000", 2024-03-21T13:17:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0470000000000001e-06 2024-03-21T13:17:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:86426 11.e e413/390)", 2024-03-21T13:17:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:21.340176+0000", 2024-03-21T13:17:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.004489336, 2024-03-21T13:17:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0053588919999999996, 2024-03-21T13:17:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:17:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:17:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:17:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:17:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340176+0000", 2024-03-21T13:17:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:17:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340176+0000", 2024-03-21T13:17:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:17:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340177+0000", 2024-03-21T13:17:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.91e-07 2024-03-21T13:17:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:17:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340209+0000", 2024-03-21T13:17:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2453000000000003e-05 2024-03-21T13:17:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:17:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340216+0000", 2024-03-21T13:17:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6649999999999997e-06 2024-03-21T13:17:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:17:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340221+0000", 2024-03-21T13:17:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1510000000000003e-06 2024-03-21T13:17:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:17:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.345523+0000", 2024-03-21T13:17:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0053017419999999999 2024-03-21T13:17:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:86429 11.f e413/390 11:fa843f13:::benchmark_data_smithi163_180069_object86428:head v 413'5349, mlcod=413'5347)", 2024-03-21T13:17:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:21.340378+0000", 2024-03-21T13:17:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.0042872860000000004, 2024-03-21T13:17:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052228480000000004, 2024-03-21T13:17:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:17:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:17:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:17:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:17:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340378+0000", 2024-03-21T13:17:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:17:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340378+0000", 2024-03-21T13:17:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:17:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340379+0000", 2024-03-21T13:17:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1699999999999997e-07 2024-03-21T13:17:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:17:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340413+0000", 2024-03-21T13:17:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3714000000000003e-05 2024-03-21T13:17:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:17:21.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340424+0000", 2024-03-21T13:17:21.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0756e-05 2024-03-21T13:17:21.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:17:21.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340431+0000", 2024-03-21T13:17:21.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6590000000000007e-06 2024-03-21T13:17:21.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:17:21.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340504+0000", 2024-03-21T13:17:21.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2857999999999999e-05 2024-03-21T13:17:21.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:17:21.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.340525+0000", 2024-03-21T13:17:21.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1104999999999999e-05 2024-03-21T13:17:21.401 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.401 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:21.401 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.401 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:21.402 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:86425 11.9 e413/390 11:9edfb66a:::benchmark_data_smithi163_180069_object86424:head v 413'5495, mlcod=413'5492)", 2024-03-21T13:17:21.403 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:21.344254+0000", 2024-03-21T13:17:21.403 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.00041187299999999999, 2024-03-21T13:17:21.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014132610000000001, 2024-03-21T13:17:21.403 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:17:21.403 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:17:21.403 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:17:21.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:17:21.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.344254+0000", 2024-03-21T13:17:21.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:17:21.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.344254+0000", 2024-03-21T13:17:21.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:17:21.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.344255+0000", 2024-03-21T13:17:21.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5600000000000004e-07 2024-03-21T13:17:21.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:17:21.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.344278+0000", 2024-03-21T13:17:21.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3153999999999999e-05 2024-03-21T13:17:21.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:17:21.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.344291+0000", 2024-03-21T13:17:21.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.344e-05 2024-03-21T13:17:21.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:17:21.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.344299+0000", 2024-03-21T13:17:21.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8709999999999997e-06 2024-03-21T13:17:21.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:17:21.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.345356+0000", 2024-03-21T13:17:21.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010572159999999999 2024-03-21T13:17:21.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:17:21.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.345377+0000", 2024-03-21T13:17:21.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0693e-05 2024-03-21T13:17:21.406 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.406 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:21.406 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.407 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:86427 11.9 e413/390)", 2024-03-21T13:17:21.407 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:21.345194+0000", 2024-03-21T13:17:21.407 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4294967295.9994721, 2024-03-21T13:17:21.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053433599999999995, 2024-03-21T13:17:21.407 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:17:21.407 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "queued for pg", 2024-03-21T13:17:21.407 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:17:21.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:17:21.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.345194+0000", 2024-03-21T13:17:21.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:17:21.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.345194+0000", 2024-03-21T13:17:21.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:17:21.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:17:21.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.345194+0000", 2024-03-21T13:17:21.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.27e-07 2024-03-21T13:17:21.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:17:21.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.345222+0000", 2024-03-21T13:17:21.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7648999999999999e-05 2024-03-21T13:17:21.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:17:21.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.345404+0000", 2024-03-21T13:17:21.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000181838 2024-03-21T13:17:21.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:21.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:21.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:17:21.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:21.345411+0000", 2024-03-21T13:17:21.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4279999999999999e-06 2024-03-21T13:17:21.410 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.410 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:21.410 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.410 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:21.410 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:17:21.410 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 13 2024-03-21T13:17:21.410 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:17:21.455 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:21.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:21.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:21.557 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:21.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:21.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:21.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:21.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:21.743 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:21.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:21.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:21.845 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:21.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:21.890 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:21.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:21.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:22.037 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:22.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:22.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:22.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:22.177 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:22.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:22.278 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:22.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:22.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:22.334 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 53 16 87667 87651 103.35 77.6875 0.00928263 0.00967075 2024-03-21T13:17:22.417 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:22.457 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:22.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:22.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:22.562 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:17:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:17:22.599 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:22.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:22.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:22.640 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-21T13:17:22.666 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:17:22.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:22.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:22.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:22.821 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:22 smithi163 ceph-mon[29621]: pgmap v878: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T13:17:22.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:22 smithi163 ceph-mon[40675]: pgmap v878: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T13:17:22.839 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:22.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:22.895 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:22.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:22.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:23.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:22 smithi204 ceph-mon[33153]: pgmap v878: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T13:17:23.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:23.042 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:23.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:23.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:23.180 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:23.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:23.281 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:23.320 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:23.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:23.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:23.335 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 54 15 88881 88866 102.843 75.9375 0.0100977 0.0097188 2024-03-21T13:17:23.422 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:23.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:23.454 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:23.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:23.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:23.597 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:23.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:23.680 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:17:23.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:23.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:23.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:23.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:23.766 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:17:23.779 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:17:23.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:23 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:23.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:23 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:23.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:23.867 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:23.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:23.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:23 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:24.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:24.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:24.106 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:24.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:24.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:24.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:24.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:24.291 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:24.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:24.335 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 55 16 90154 90138 102.418 79.5 0.00997657 0.00975934 2024-03-21T13:17:24.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:24.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:24.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:24.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:24.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:24.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:24.668 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:24.709 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:24.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:24 smithi204 ceph-mon[33153]: pgmap v879: 148 pgs: 148 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-21T13:17:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:24 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:17:24.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:24 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:17:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:24 smithi163 ceph-mon[29621]: pgmap v879: 148 pgs: 148 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-21T13:17:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:24 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:17:24.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:24 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:17:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:24 smithi163 ceph-mon[40675]: pgmap v879: 148 pgs: 148 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-21T13:17:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:24 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:17:24.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:24 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:17:24.810 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:24.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:24.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:24.951 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:24.989 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:24.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:25.090 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:25.126 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:25.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:25.228 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:25.264 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:25.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:25.335 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 56 16 91387 91371 101.965 77.0625 0.00980198 0.00980247 2024-03-21T13:17:25.366 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:25.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:25.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:25.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:25.506 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:25.543 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:25.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:25.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:25.645 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:25.683 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:25.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:25.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:25.784 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:25.833 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:25.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:25.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:25 smithi163 ceph-mon[29621]: osdmap e414: 8 total, 8 up, 7 in 2024-03-21T13:17:25.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:25 smithi163 ceph-mon[29621]: pgmap v881: 148 pgs: 148 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 826 op/s 2024-03-21T13:17:25.899 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:17:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:17:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:17:25.899 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:25 smithi163 ceph-mon[40675]: osdmap e414: 8 total, 8 up, 7 in 2024-03-21T13:17:25.899 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:25 smithi163 ceph-mon[40675]: pgmap v881: 148 pgs: 148 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 826 op/s 2024-03-21T13:17:25.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:25.973 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:25.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:25.995 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:25 smithi204 ceph-mon[33153]: osdmap e414: 8 total, 8 up, 7 in 2024-03-21T13:17:25.995 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:25 smithi204 ceph-mon[33153]: pgmap v881: 148 pgs: 148 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 826 op/s 2024-03-21T13:17:26.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:26.117 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:26.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:26.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:26.254 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:26.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:26.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:26.335 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 57 16 92645 92629 101.556 78.625 0.00803468 0.00984196 2024-03-21T13:17:26.356 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:26.390 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:26.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:26.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:26.492 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:26.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:26 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a[29617]: 2024-03-21T13:17:26.365+0000 7f8e1b000700 -1 mon.a@0(leader).osd e414 definitely_dead 0 2024-03-21T13:17:26.536 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:26.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:26.629 INFO:teuthology.orchestra.run.smithi163.stderr:marked out osd.0. 2024-03-21T13:17:26.638 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:26.671 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:26.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:26.773 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:26.796 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:26.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:26.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:26.898 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:26.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:26.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:26.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:26 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1024164890' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-21T13:17:26.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:26 smithi163 ceph-mon[29621]: Client client.admin marked osd.0 out, while it was still marked up 2024-03-21T13:17:26.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:26 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1024164890' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-21T13:17:26.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:26 smithi163 ceph-mon[40675]: Client client.admin marked osd.0 out, while it was still marked up 2024-03-21T13:17:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:26 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1024164890' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-21T13:17:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:26 smithi204 ceph-mon[33153]: Client client.admin marked osd.0 out, while it was still marked up 2024-03-21T13:17:27.023 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:27.047 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:27.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:27.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:27.149 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:27.172 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:27.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:27.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:27.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:27.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:27.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:27.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:27.335 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 58 16 93020 93004 100.209 23.4375 0.00480726 0.0098549 2024-03-21T13:17:27.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:27.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:27.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:27.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:27.549 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:27.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:27.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:27.651 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:27.717 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:27.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:27.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:27 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1024164890' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-03-21T13:17:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:27 smithi163 ceph-mon[29621]: osdmap e415: 8 total, 8 up, 6 in 2024-03-21T13:17:27.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:27 smithi163 ceph-mon[29621]: pgmap v883: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.3 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-21T13:17:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:27 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1024164890' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-03-21T13:17:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:27 smithi163 ceph-mon[40675]: osdmap e415: 8 total, 8 up, 6 in 2024-03-21T13:17:27.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:27 smithi163 ceph-mon[40675]: pgmap v883: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.3 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-21T13:17:27.819 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:27.842 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:27.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:27.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:27.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:27.968 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:27.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:27.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:28.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:27 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1024164890' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-03-21T13:17:28.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:27 smithi204 ceph-mon[33153]: osdmap e415: 8 total, 8 up, 6 in 2024-03-21T13:17:28.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:27 smithi204 ceph-mon[33153]: pgmap v883: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.3 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-21T13:17:28.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:28.094 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:28.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:28.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:28.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:28.219 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:28.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:28.321 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:28.335 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 59 16 93020 93004 98.5103 0 - 0.0098549 2024-03-21T13:17:28.345 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:28.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:28.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:28.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:28.471 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:28.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:28.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:28.572 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:28.597 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:28.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:28.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:28.728 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:28.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:28.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:28 smithi163 ceph-mon[29621]: osdmap e416: 8 total, 8 up, 6 in 2024-03-21T13:17:28.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:28 smithi163 ceph-mon[40675]: osdmap e416: 8 total, 8 up, 6 in 2024-03-21T13:17:28.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:28.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:28.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:28.964 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:29.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:28 smithi204 ceph-mon[33153]: osdmap e416: 8 total, 8 up, 6 in 2024-03-21T13:17:29.012 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:29.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:29.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:29.146 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:29.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:29.248 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:29.314 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:29.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:29.347 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:17:29.348133+0000 min lat: 0.00136681 max lat: 2.54676 avg lat: 0.0102927 2024-03-21T13:17:29.347 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:17:29.347 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 60 16 93222 93206 97.0596 6.3125 0.0712445 0.0102927 2024-03-21T13:17:29.416 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:29.451 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:29.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:29.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:29.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:29.583 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:29.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:29.656 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:17:29.685 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:29.729 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:29.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:29.801 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:17:29.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:29.842 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:17:29.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:29.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:29.874 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:17:29.908 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:17:29.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[29621]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 414'5772 2024-03-21T13:17:29.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[29621]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 414'5749 2024-03-21T13:17:29.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[29621]: 11.f starting backfill to osd.4 from (0'0,0'0] MAX to 414'5769 2024-03-21T13:17:29.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[29621]: 11.6 starting backfill to osd.5 from (0'0,0'0] MAX to 414'5832 2024-03-21T13:17:29.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[29621]: osdmap e417: 8 total, 8 up, 6 in 2024-03-21T13:17:29.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[29621]: 11.e starting backfill to osd.3 from (0'0,0'0] MAX to 414'5860 2024-03-21T13:17:29.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[29621]: 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 414'5701 2024-03-21T13:17:29.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[29621]: 11.7 starting backfill to osd.7 from (0'0,0'0] MAX to 414'5688 2024-03-21T13:17:29.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[29621]: pgmap v886: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-21T13:17:29.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2831974007' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:17:29.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:17:29.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[40675]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 414'5772 2024-03-21T13:17:29.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[40675]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 414'5749 2024-03-21T13:17:29.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[40675]: 11.f starting backfill to osd.4 from (0'0,0'0] MAX to 414'5769 2024-03-21T13:17:29.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[40675]: 11.6 starting backfill to osd.5 from (0'0,0'0] MAX to 414'5832 2024-03-21T13:17:29.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[40675]: osdmap e417: 8 total, 8 up, 6 in 2024-03-21T13:17:29.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[40675]: 11.e starting backfill to osd.3 from (0'0,0'0] MAX to 414'5860 2024-03-21T13:17:29.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[40675]: 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 414'5701 2024-03-21T13:17:29.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[40675]: 11.7 starting backfill to osd.7 from (0'0,0'0] MAX to 414'5688 2024-03-21T13:17:29.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[40675]: pgmap v886: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-21T13:17:29.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2831974007' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:17:29.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:29 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:17:29.940 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:17:29.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:29.974 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:17:29.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:29.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:30.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:29 smithi204 ceph-mon[33153]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 414'5772 2024-03-21T13:17:30.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:29 smithi204 ceph-mon[33153]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 414'5749 2024-03-21T13:17:30.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:29 smithi204 ceph-mon[33153]: 11.f starting backfill to osd.4 from (0'0,0'0] MAX to 414'5769 2024-03-21T13:17:30.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:29 smithi204 ceph-mon[33153]: 11.6 starting backfill to osd.5 from (0'0,0'0] MAX to 414'5832 2024-03-21T13:17:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:29 smithi204 ceph-mon[33153]: osdmap e417: 8 total, 8 up, 6 in 2024-03-21T13:17:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:29 smithi204 ceph-mon[33153]: 11.e starting backfill to osd.3 from (0'0,0'0] MAX to 414'5860 2024-03-21T13:17:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:29 smithi204 ceph-mon[33153]: 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 414'5701 2024-03-21T13:17:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:29 smithi204 ceph-mon[33153]: 11.7 starting backfill to osd.7 from (0'0,0'0] MAX to 414'5688 2024-03-21T13:17:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:29 smithi204 ceph-mon[33153]: pgmap v886: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-21T13:17:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:29 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2831974007' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:17:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:29 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:17:30.007 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:17:30.042 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:17:30.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:17:30.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:17:30.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:17:30.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:17:30.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:17:30.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:17:30.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:17:30.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:17:30.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:30.135 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:30.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:30.236 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:30.267 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:30.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:30.347 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 61 16 93457 93441 95.7095 14.6875 0.127245 0.0103966 2024-03-21T13:17:30.369 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:30.401 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:30.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:30.502 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:30.541 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:30.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:30.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:30.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:30.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:30.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:30.786 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:30.828 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:30.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:30.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:30.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:30.960 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:30.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:30.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:30 smithi204 ceph-mon[33153]: 11.9 starting backfill to osd.7 from (0'0,0'0] MAX to 414'5894 2024-03-21T13:17:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:30 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:17:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:30 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:17:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:30 smithi204 ceph-mon[33153]: osdmap e418: 8 total, 8 up, 6 in 2024-03-21T13:17:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:30 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/121781569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:17:31.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:30 smithi163 ceph-mon[29621]: 11.9 starting backfill to osd.7 from (0'0,0'0] MAX to 414'5894 2024-03-21T13:17:31.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:30 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:17:31.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:30 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:17:31.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:30 smithi163 ceph-mon[29621]: osdmap e418: 8 total, 8 up, 6 in 2024-03-21T13:17:31.012 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:30 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/121781569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:17:31.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:30 smithi163 ceph-mon[40675]: 11.9 starting backfill to osd.7 from (0'0,0'0] MAX to 414'5894 2024-03-21T13:17:31.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:30 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:17:31.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:30 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:17:31.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:30 smithi163 ceph-mon[40675]: osdmap e418: 8 total, 8 up, 6 in 2024-03-21T13:17:31.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:30 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/121781569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:17:31.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:31.100 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:31.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:31.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:31.202 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:31.235 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:31.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:31.337 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:31.350 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 62 16 94340 94324 95.0559 55.1875 0.00854616 0.0105154 2024-03-21T13:17:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:31.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:31.468 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:31.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:31.513 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:31.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:31.614 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:31.653 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:31.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:31.755 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:31.791 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:31.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:31.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:31.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:31.927 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:31.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:32.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:31 smithi204 ceph-mon[33153]: osdmap e419: 8 total, 8 up, 6 in 2024-03-21T13:17:32.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:31 smithi204 ceph-mon[33153]: pgmap v889: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 141 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 348 op/s; 32498/280341 objects misplaced (11.592%); 37 MiB/s, 611 objects/s recovering 2024-03-21T13:17:32.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:31 smithi163 ceph-mon[29621]: osdmap e419: 8 total, 8 up, 6 in 2024-03-21T13:17:32.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:31 smithi163 ceph-mon[29621]: pgmap v889: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 141 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 348 op/s; 32498/280341 objects misplaced (11.592%); 37 MiB/s, 611 objects/s recovering 2024-03-21T13:17:32.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:31 smithi163 ceph-mon[40675]: osdmap e419: 8 total, 8 up, 6 in 2024-03-21T13:17:32.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:31 smithi163 ceph-mon[40675]: pgmap v889: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 141 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 348 op/s; 32498/280341 objects misplaced (11.592%); 37 MiB/s, 611 objects/s recovering 2024-03-21T13:17:32.029 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:32.060 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:32.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:32.162 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:32.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:32.189 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:32.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:32.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:32.331 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:32.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:32.347 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 63 16 95156 95140 94.3567 51 0.0101532 0.0105935 2024-03-21T13:17:32.433 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:32.463 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:32.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:32.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:32.564 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:32.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:32.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:32.612 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:17:32.665 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:17:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:17:32.703 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:32.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:32.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:32.846 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:32.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:32.890 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:32.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:32.991 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:33.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:32 smithi204 ceph-mon[33153]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:17:33.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:32 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:33.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:32 smithi163 ceph-mon[29621]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:17:33.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:32 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:33.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:32 smithi163 ceph-mon[40675]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T13:17:33.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:32 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:33.025 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:33.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:33.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:33.168 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:33.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:33.269 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:33.303 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:33.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:33.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:33.347 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 64 16 96167 96151 93.8697 63.1875 0.0132693 0.010648 2024-03-21T13:17:33.405 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:33.430 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:33.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:33.496 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 6, 7, 2, 5] out_osds: [1, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:17:33.496 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:17:33.496 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-21T13:17:33.496 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-21T13:17:33.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:33.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:33.572 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:33.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:33.673 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:33.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:33.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:33.802 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:33.844 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:33.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:33.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:33.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:33 smithi163 ceph-mon[29621]: pgmap v890: 148 pgs: 7 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 139 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 22 MiB/s wr, 359 op/s; 43261/281922 objects misplaced (15.345%); 39 MiB/s, 640 objects/s recovering 2024-03-21T13:17:33.956 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:33 smithi163 ceph-mon[40675]: pgmap v890: 148 pgs: 7 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 139 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 22 MiB/s wr, 359 op/s; 43261/281922 objects misplaced (15.345%); 39 MiB/s, 640 objects/s recovering 2024-03-21T13:17:33.984 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:33.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:33 smithi204 ceph-mon[33153]: pgmap v890: 148 pgs: 7 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 139 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 22 MiB/s wr, 359 op/s; 43261/281922 objects misplaced (15.345%); 39 MiB/s, 640 objects/s recovering 2024-03-21T13:17:34.086 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:34.124 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:34.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:34.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:34.226 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:34.260 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:34.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:34.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:34.348 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 65 16 97076 97060 93.2996 56.8125 0.0155253 0.0107133 2024-03-21T13:17:34.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:34.398 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:34.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:34.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:34.500 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:34.523 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:34.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:34.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:34.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:34.666 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:34.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:34.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:34.767 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:34.806 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:34.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:34.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:34.908 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:34.949 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:34.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:34.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:35.051 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:35.091 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:35.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:35.193 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:35.232 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:35.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:35.333 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:35.348 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 66 16 98053 98037 92.8112 61.0625 0.0162488 0.0107705 2024-03-21T13:17:35.367 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:35.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:35.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:35.469 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:35.506 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:35.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:35.608 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:35.643 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:35.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:35.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:35.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:35.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:35.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:35.831 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:17:35.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:35.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:35.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:35.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:35 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:17:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:35 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:17:36.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:35 smithi204 ceph-mon[33153]: osdmap e420: 8 total, 8 up, 6 in 2024-03-21T13:17:36.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:35 smithi204 ceph-mon[33153]: pgmap v892: 148 pgs: 7 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 139 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 23 MiB/s wr, 364 op/s; 42698/282723 objects misplaced (15.102%); 43 MiB/s, 703 objects/s recovering 2024-03-21T13:17:36.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:17:36.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:36.006 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:17:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:17:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:17:36.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:35 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:17:36.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:35 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:17:36.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:35 smithi163 ceph-mon[29621]: osdmap e420: 8 total, 8 up, 6 in 2024-03-21T13:17:36.007 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:35 smithi163 ceph-mon[29621]: pgmap v892: 148 pgs: 7 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 139 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 23 MiB/s wr, 364 op/s; 42698/282723 objects misplaced (15.102%); 43 MiB/s, 703 objects/s recovering 2024-03-21T13:17:36.007 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:17:36.007 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:36.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:35 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:17:36.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:35 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:17:36.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:35 smithi163 ceph-mon[40675]: osdmap e420: 8 total, 8 up, 6 in 2024-03-21T13:17:36.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:35 smithi163 ceph-mon[40675]: pgmap v892: 148 pgs: 7 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 139 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 23 MiB/s wr, 364 op/s; 42698/282723 objects misplaced (15.102%); 43 MiB/s, 703 objects/s recovering 2024-03-21T13:17:36.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:17:36.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:36.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:36.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:36.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:36.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:36.206 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:36.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:36.308 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:36.343 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:36.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:36.348 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 67 16 99118 99102 92.4194 66.5625 0.0205881 0.0108157 2024-03-21T13:17:36.416 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:17:36.445 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:36.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:36.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:36.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:36.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:36.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:36.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:36.723 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:36.765 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:36.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:36.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:36.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:36.903 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:36.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:36.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:37.005 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:37.042 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:37.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:37.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:37.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:37.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:37.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:37.288 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:37.327 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:37.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:37.350 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 68 16 100212 100196 92.0655 68.375 0.0115581 0.0108574 2024-03-21T13:17:37.427 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:37.465 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:37.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:37.635 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:37.676 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:37.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:37.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:37.778 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:37.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:37.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:37.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:37.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:37.941 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:37.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:37.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:38.043 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:38.085 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:38.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:38.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:38.186 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:38.232 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:38.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:38.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:38.348 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 69 16 101155 101139 91.5854 58.9375 0.019286 0.0109136 2024-03-21T13:17:38.371 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:38.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:38.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:38.407 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:17:38.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:38 smithi163 ceph-mon[29621]: pgmap v893: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.5 KiB/s rd, 56 MiB/s wr, 897 op/s; 40743/295884 objects misplaced (13.770%); 67 MiB/s, 1.08k objects/s recovering 2024-03-21T13:17:38.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:38 smithi163 ceph-mon[40675]: pgmap v893: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.5 KiB/s rd, 56 MiB/s wr, 897 op/s; 40743/295884 objects misplaced (13.770%); 67 MiB/s, 1.08k objects/s recovering 2024-03-21T13:17:38.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:38.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:38 smithi204 ceph-mon[33153]: pgmap v893: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.5 KiB/s rd, 56 MiB/s wr, 897 op/s; 40743/295884 objects misplaced (13.770%); 67 MiB/s, 1.08k objects/s recovering 2024-03-21T13:17:38.513 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:38.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:38.614 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:38.649 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:38.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:38.750 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:38.788 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:38.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:38.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:38.931 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:38.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:38.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:39.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:39.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:39.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:39.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:39.179 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:39.210 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:39.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:39.313 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:39.348 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 70 15 102106 102091 91.1271 59.5 0.014517 0.0109686 2024-03-21T13:17:39.349 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:39.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:39.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:39.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:39.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:39.591 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:39.626 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:39.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:39.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:39.728 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:39.759 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:39.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:39.861 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:39.895 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:39.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:39.996 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:40.041 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:40.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:40.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:40.178 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:40.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:40.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:40.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:40 smithi163 ceph-mon[29621]: pgmap v894: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 737 B/s rd, 42 MiB/s wr, 676 op/s; 40743/295884 objects misplaced (13.770%); 49 MiB/s, 786 objects/s recovering 2024-03-21T13:17:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:40 smithi163 ceph-mon[40675]: pgmap v894: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 737 B/s rd, 42 MiB/s wr, 676 op/s; 40743/295884 objects misplaced (13.770%); 49 MiB/s, 786 objects/s recovering 2024-03-21T13:17:40.280 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:40.324 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:40.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:40.348 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 71 15 102994 102979 90.6253 55.5 0.0234428 0.0110298 2024-03-21T13:17:40.426 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:40.467 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:40.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:40.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:40 smithi204 ceph-mon[33153]: pgmap v894: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 737 B/s rd, 42 MiB/s wr, 676 op/s; 40743/295884 objects misplaced (13.770%); 49 MiB/s, 786 objects/s recovering 2024-03-21T13:17:40.569 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:40.610 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:40.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:40.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:40.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:40.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:40.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:40.875 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:17:40.875 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:17:40.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:40.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:40.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:41.039 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:41.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:41.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:41.140 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:41.176 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:41.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:41.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:41.312 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:41.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:41.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:41.348 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 72 16 103984 103968 90.225 61.8125 0.0264517 0.0110723 2024-03-21T13:17:41.414 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:41.450 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:41.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:41.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:41.551 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:41.588 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:41.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:41.689 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:41.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:41.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:41.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:41.867 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:41.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:41.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:41.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:42.009 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:42.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:42.111 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:42.145 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:42.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:42.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:42.247 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:42.286 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:42.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:42.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:42.304 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:42 smithi204 ceph-mon[33153]: pgmap v895: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 54 MiB/s wr, 858 op/s; 39249/306084 objects misplaced (12.823%); 53 MiB/s, 845 objects/s recovering 2024-03-21T13:17:42.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 73 16 105043 105027 89.8958 66.1875 0.0157957 0.0111196 2024-03-21T13:17:42.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:42.428 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:42.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:42.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:42 smithi163 ceph-mon[29621]: pgmap v895: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 54 MiB/s wr, 858 op/s; 39249/306084 objects misplaced (12.823%); 53 MiB/s, 845 objects/s recovering 2024-03-21T13:17:42.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:42 smithi163 ceph-mon[40675]: pgmap v895: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 54 MiB/s wr, 858 op/s; 39249/306084 objects misplaced (12.823%); 53 MiB/s, 845 objects/s recovering 2024-03-21T13:17:42.529 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:42.567 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:42.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:42.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:42.669 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:42.707 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:42.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:42.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:42.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:17:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:17:42.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:42.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:42.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:42.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:42.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:42.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:42.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:43.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:43.131 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:43.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:43.233 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:43.268 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:43.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:43.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 74 16 106040 106024 89.523 62.3125 0.01851 0.0111653 2024-03-21T13:17:43.370 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:43.405 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:43.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:43.411 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:43.412 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:43.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:43.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:43.555 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:43.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:43.657 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:43.710 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:43.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:43.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:43.852 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:43.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:43.954 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:43.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:43.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:44.091 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-21T13:17:44.094 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:44.153 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:44.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:44.254 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:44.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:44.299 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:44.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:44.346 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:44 smithi163 ceph-mon[29621]: pgmap v896: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 60 MiB/s wr, 960 op/s; 38606/310725 objects misplaced (12.424%); 54 MiB/s, 860 objects/s recovering 2024-03-21T13:17:44.347 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:44 smithi163 ceph-mon[40675]: pgmap v896: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 60 MiB/s wr, 960 op/s; 38606/310725 objects misplaced (12.424%); 54 MiB/s, 860 objects/s recovering 2024-03-21T13:17:44.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 75 16 107066 107050 89.1844 64.125 0.0151443 0.0112077 2024-03-21T13:17:44.355 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:17:44.401 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:44.449 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:44.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:44.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:44.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:44 smithi204 ceph-mon[33153]: pgmap v896: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 60 MiB/s wr, 960 op/s; 38606/310725 objects misplaced (12.424%); 54 MiB/s, 860 objects/s recovering 2024-03-21T13:17:44.551 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:44.606 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:44.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:44.708 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:44.756 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:44.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:44.857 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:44.903 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:44.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:44.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:45.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:45.041 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:45.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:45.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:45.185 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:45.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:45.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:45.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:45.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:45.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:45.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 76 16 107969 107953 88.7535 56.4375 0.00912838 0.0112623 2024-03-21T13:17:45.392 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:45 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:17:45.392 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:45 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:17:45.392 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:45 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:17:45.392 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:45 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:17:45.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:45.467 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:45.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:45.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:45 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:17:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:45 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:17:45.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:45.611 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:45.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:45.713 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:45.757 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:45.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:45.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:45.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:17:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:17:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:17:45.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:45.896 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:45.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:45.979 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:17:45.979 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:17:45.979 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 421, 2024-03-21T13:17:45.979 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "d4bf79c0-e781-11ee-95cd-87774f69a715", 2024-03-21T13:17:45.979 INFO:teuthology.orchestra.run.smithi163.stdout: "created": "2024-03-21T12:52:56.031419+0000", 2024-03-21T13:17:45.979 INFO:teuthology.orchestra.run.smithi163.stdout: "modified": "2024-03-21T13:17:45.067262+0000", 2024-03-21T13:17:45.980 INFO:teuthology.orchestra.run.smithi163.stdout: "last_up_change": "2024-03-21T12:58:23.775302+0000", 2024-03-21T13:17:45.980 INFO:teuthology.orchestra.run.smithi163.stdout: "last_in_change": "2024-03-21T13:17:26.597508+0000", 2024-03-21T13:17:45.980 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-21T13:17:45.980 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_num": 5806080, 2024-03-21T13:17:45.980 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_set": [ 2024-03-21T13:17:45.980 INFO:teuthology.orchestra.run.smithi163.stdout: "nodeep-scrub", 2024-03-21T13:17:45.980 INFO:teuthology.orchestra.run.smithi163.stdout: "noscrub", 2024-03-21T13:17:45.980 INFO:teuthology.orchestra.run.smithi163.stdout: "pglog_hardlimit", 2024-03-21T13:17:45.980 INFO:teuthology.orchestra.run.smithi163.stdout: "purged_snapdirs", 2024-03-21T13:17:45.980 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_deletes", 2024-03-21T13:17:45.980 INFO:teuthology.orchestra.run.smithi163.stdout: "sortbitwise" 2024-03-21T13:17:45.980 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:17:45.980 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_version": 18, 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "full_ratio": 0.94999998807907104, 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_snapshot": "", 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_max": 11, 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd": 8, 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "require_min_compat_client": "luminous", 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "min_compat_client": "luminous", 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "require_osd_release": "squid", 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "allow_crimson": false, 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "pools": [ 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 1, 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".mgr", 2024-03-21T13:17:45.981 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:56:20.557490+0000", 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 1, 2024-03-21T13:17:45.982 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 1, 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 1, 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 1, 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 1, 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "264", 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:17:45.983 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:17:45.984 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:17:45.985 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:17:45.986 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:17:45.986 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:17:45.986 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:17:45.986 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:17:45.986 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-03-21T13:17:45.986 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_max": 32, 2024-03-21T13:17:45.986 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_min": 1 2024-03-21T13:17:45.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:45.986 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:17:45.986 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": {} 2024-03-21T13:17:45.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:45.987 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:17:45.987 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:17:45.987 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 7.8899998664855957, 2024-03-21T13:17:45.987 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 7.8899998664855957, 2024-03-21T13:17:45.987 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.37999999523162842, 2024-03-21T13:17:45.987 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 3, 2024-03-21T13:17:45.987 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 3, 2024-03-21T13:17:45.987 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.94608116149902344, 2024-03-21T13:17:45.987 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-21T13:17:45.987 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-21T13:17:45.987 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:45.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:45.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:45.988 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 2, 2024-03-21T13:17:45.988 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "datapool", 2024-03-21T13:17:45.988 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:35.713383+0000", 2024-03-21T13:17:45.988 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-03-21T13:17:45.988 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-21T13:17:45.988 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:17:45.988 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:17:45.988 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:17:45.988 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:17:45.988 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:17:45.988 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 3, 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 3, 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 3, 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 3, 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 3, 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:17:45.989 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:17:45.990 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:17:45.990 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:17:45.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:45.990 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "67", 2024-03-21T13:17:45.990 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:17:45.990 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:17:45.990 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:17:45.990 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:17:45.990 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:17:45.990 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 2, 2024-03-21T13:17:45.990 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 67, 2024-03-21T13:17:45.990 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:17:45.990 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:17:45.991 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:17:45.992 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd": {} 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.6500000953674316, 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.6500000953674316, 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.62999999523162842, 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.6699999570846558, 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.6699999570846558, 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.94608116149902344, 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:45.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 3, 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".rgw.root", 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:36.436384+0000", 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:17:45.994 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:17:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "65", 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:17:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:45.997 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:17:45.998 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:17:45.998 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:17:45.998 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:17:45.998 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:17:45.998 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:17:45.998 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:17:45.998 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:17:45.998 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:17:45.998 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.25, 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.25, 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.75, 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.690000057220459, 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.690000057220459, 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.94608116149902344, 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-21T13:17:45.999 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 4, 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.log", 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:38.049304+0000", 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:17:46.000 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:17:46.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "67", 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:17:46.002 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.003 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:17:46.004 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.75, 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.75, 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.75, 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 2.059999942779541, 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 2.059999942779541, 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.94608116149902344, 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 5, 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.control", 2024-03-21T13:17:46.005 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:39.990512+0000", 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:17:46.006 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "69", 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:17:46.007 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:17:46.008 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:17:46.009 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:17:46.010 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2, 2024-03-21T13:17:46.011 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2, 2024-03-21T13:17:46.011 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.75, 2024-03-21T13:17:46.011 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.5, 2024-03-21T13:17:46.011 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.5, 2024-03-21T13:17:46.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:46.012 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.94608116149902344, 2024-03-21T13:17:46.012 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-21T13:17:46.012 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-21T13:17:46.012 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.012 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 6, 2024-03-21T13:17:46.012 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.meta", 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T12:58:41.463686+0000", 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:17:46.013 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "71", 2024-03-21T13:17:46.014 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:17:46.015 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:17:46.016 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:17:46.017 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:17:46.017 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:17:46.017 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:17:46.017 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:17:46.017 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:17:46.017 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:17:46.017 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:17:46.017 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:17:46.017 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:17:46.017 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-03-21T13:17:46.017 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_bias": 4 2024-03-21T13:17:46.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.018 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:17:46.018 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-03-21T13:17:46.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.018 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:17:46.018 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:17:46.018 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.5099999904632568, 2024-03-21T13:17:46.018 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.5099999904632568, 2024-03-21T13:17:46.018 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.75, 2024-03-21T13:17:46.018 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-21T13:17:46.018 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-21T13:17:46.018 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.94608116149902344, 2024-03-21T13:17:46.018 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-21T13:17:46.018 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 11, 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "unique_pool_4", 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-03-21T13:16:12.266158+0000", 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-03-21T13:17:46.019 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 16, 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 16, 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 16, 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 16, 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 16, 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-03-21T13:17:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "395", 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-03-21T13:17:46.021 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-03-21T13:17:46.022 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "read_balance": { 2024-03-21T13:17:46.023 INFO:teuthology.orchestra.run.smithi163.stdout: "score_type": "Fair distribution", 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: "score_acting": 2.5099999904632568, 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: "score_stable": 2.5099999904632568, 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: "optimal_score": 0.75, 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity_weighted": 0.94608116149902344, 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "c34a3b25-9e02-46e6-8f9a-d0af2615258f", 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:17:46.024 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 0, 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 0, 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0.9899444580078125, 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 8, 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 390, 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6801", 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:17:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6802", 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6804", 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6803", 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3086410981 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6801/3086410981", 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6802/3086410981", 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6804/3086410981", 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6803/3086410981", 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "36e05f89-c238-41f7-be65-1553b529e8ce", 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:17:46.028 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 0, 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 0, 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 13, 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 110, 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6805", 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:17:46.029 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.030 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.030 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:17:46.030 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.030 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.030 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6806", 2024-03-21T13:17:46.030 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:17:46.030 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.030 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.030 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:17:46.030 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.031 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.031 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6808", 2024-03-21T13:17:46.031 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:17:46.031 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.031 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.031 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:17:46.031 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.031 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.031 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6807", 2024-03-21T13:17:46.031 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3666062784 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6805/3666062784", 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6806/3666062784", 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6808/3666062784", 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6807/3666062784", 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-03-21T13:17:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "968ec78d-25b8-4905-b328-d490c38ca85f", 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 18, 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 416, 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6809", 2024-03-21T13:17:46.034 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:17:46.034 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.034 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.034 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:17:46.034 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.034 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.034 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6810", 2024-03-21T13:17:46.034 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:17:46.034 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.034 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.035 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:17:46.035 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.035 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.035 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6812", 2024-03-21T13:17:46.035 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:17:46.035 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.035 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.035 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:17:46.035 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6811", 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 4009213507 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6809/4009213507", 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6810/4009213507", 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6812/4009213507", 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6811/4009213507", 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:17:46.036 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.037 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-03-21T13:17:46.037 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "67dfa4f6-1709-48f6-971d-c6287233492d", 2024-03-21T13:17:46.037 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:17:46.037 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:17:46.037 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:17:46.037 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:17:46.037 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:17:46.037 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:17:46.037 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 25, 2024-03-21T13:17:46.037 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 416, 2024-03-21T13:17:46.037 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:17:46.038 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:17:46.038 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:17:46.038 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.038 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.038 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6813", 2024-03-21T13:17:46.038 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:17:46.038 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.038 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.038 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:17:46.038 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6814", 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6816", 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.039 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.040 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:17:46.040 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.040 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.040 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6815", 2024-03-21T13:17:46.040 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3981878189 2024-03-21T13:17:46.040 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.040 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.040 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6813/3981878189", 2024-03-21T13:17:46.040 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6814/3981878189", 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6816/3981878189", 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6815/3981878189", 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "7fd311a9-f3c2-456d-8ab1-341cf5b33471", 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0.578704833984375, 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:17:46.041 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 30, 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 416, 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6800", 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.042 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6801", 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6803", 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.043 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6802", 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 719276054 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6800/719276054", 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6801/719276054", 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6803/719276054", 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6802/719276054", 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:17:46.044 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "a87759c8-251c-415d-9387-a49f7d2f36d1", 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 39, 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 416, 2024-03-21T13:17:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:17:46.046 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:17:46.046 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:17:46.046 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.046 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.046 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6804", 2024-03-21T13:17:46.046 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:17:46.046 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.046 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.046 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:17:46.046 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.047 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6805", 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6807", 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6806", 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1346947668 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6804/1346947668", 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6805/1346947668", 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6807/1346947668", 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6806/1346947668", 2024-03-21T13:17:46.049 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "5eb135cd-f677-4d0c-b23a-738f31615b6b", 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:17:46.050 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 47, 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 416, 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6808", 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6809", 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6811", 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.052 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6810", 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 200077988 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6808/200077988", 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6809/200077988", 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6811/200077988", 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6810/200077988", 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:17:46.053 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:17:46.054 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.054 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-03-21T13:17:46.054 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "4f528e10-2c93-4bc2-a54c-06ca4132ed49", 2024-03-21T13:17:46.054 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-03-21T13:17:46.054 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-03-21T13:17:46.054 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-03-21T13:17:46.054 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-03-21T13:17:46.057 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-03-21T13:17:46.057 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-03-21T13:17:46.057 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 56, 2024-03-21T13:17:46.057 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 418, 2024-03-21T13:17:46.058 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-03-21T13:17:46.058 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-03-21T13:17:46.058 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-03-21T13:17:46.058 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.058 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.058 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6812", 2024-03-21T13:17:46.058 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:17:46.058 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.058 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.058 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6813", 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6815", 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:17:46.059 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.204:6814", 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 315664888 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.204:6812/315664888", 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.204:6813/315664888", 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.204:6815/315664888", 2024-03-21T13:17:46.060 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.204:6814/315664888", 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_xinfo": [ 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 65536, 2024-03-21T13:17:46.061 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:17:02.631692+0000", 2024-03-21T13:17:46.062 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:17:46.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.062 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-03-21T13:17:46.062 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:17:46.062 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:17:46.062 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:17:46.062 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:17:46.062 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 65536, 2024-03-21T13:17:46.062 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:17:24.258342+0000", 2024-03-21T13:17:46.062 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:17:46.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:17:34.229596+0000", 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:17:46.063 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:16:25.480783+0000", 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:16:31.452023+0000", 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-03-21T13:17:46.064 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:17:44.539859+0000", 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:16:59.528422+0000", 2024-03-21T13:17:46.065 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540701547738038271, 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-03-21T13:17:10.535335+0000", 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap": [ 2024-03-21T13:17:46.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.13", 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: 0, 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: 5, 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: 6 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "6.11", 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: 7, 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: 4 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.067 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_items": [ 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "3.2", 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 3, 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "3.19", 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 4, 2024-03-21T13:17:46.068 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 7 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.9", 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 3, 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.19", 2024-03-21T13:17:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 3, 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 7 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_primaries": [], 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_temp": [ 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "11.1", 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: 6, 2024-03-21T13:17:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: 0 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "11.2", 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: 2, 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: 0 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "11.6", 2024-03-21T13:17:46.071 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:17:46.072 INFO:teuthology.orchestra.run.smithi163.stdout: 2, 2024-03-21T13:17:46.072 INFO:teuthology.orchestra.run.smithi163.stdout: 4, 2024-03-21T13:17:46.072 INFO:teuthology.orchestra.run.smithi163.stdout: 0 2024-03-21T13:17:46.072 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.072 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "11.7", 2024-03-21T13:17:46.072 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:17:46.072 INFO:teuthology.orchestra.run.smithi163.stdout: 5, 2024-03-21T13:17:46.072 INFO:teuthology.orchestra.run.smithi163.stdout: 4, 2024-03-21T13:17:46.072 INFO:teuthology.orchestra.run.smithi163.stdout: 0 2024-03-21T13:17:46.072 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "11.9", 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: 6, 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: 0 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "11.b", 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: 3, 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: 4, 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: 0 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.074 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "11.e", 2024-03-21T13:17:46.074 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:17:46.074 INFO:teuthology.orchestra.run.smithi163.stdout: 4, 2024-03-21T13:17:46.074 INFO:teuthology.orchestra.run.smithi163.stdout: 7, 2024-03-21T13:17:46.074 INFO:teuthology.orchestra.run.smithi163.stdout: 0 2024-03-21T13:17:46.074 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:17:46.074 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "11.f", 2024-03-21T13:17:46.074 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-03-21T13:17:46.074 INFO:teuthology.orchestra.run.smithi163.stdout: 7, 2024-03-21T13:17:46.074 INFO:teuthology.orchestra.run.smithi163.stdout: 2, 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: 0 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_temp": [], 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: "blocklist": { 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3453855427": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/2571244897": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/3073378854": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1334591766": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/116934309": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3890775846": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1783921524": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:17:46.075 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2553440211": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1812694077": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1070540283": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/2354157786": "2024-03-22T12:53:32.790116+0000", 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1591817392": "2024-03-22T12:59:18.599967+0000", 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/4152198951": "2024-03-22T12:54:00.224135+0000", 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/1798115633": "2024-03-22T12:54:00.224135+0000" 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "range_blocklist": {}, 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profiles": { 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "default": { 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-03-21T13:17:46.076 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van" 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps_queue": [], 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: "new_removed_snaps": [], 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: "new_purged_snaps": [], 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_node_flags": {}, 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: "device_class_flags": {}, 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode": { 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_enabled": false, 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_bucket_count": 0, 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: "degraded_stretch_mode": 0, 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: "recovering_stretch_mode": 0, 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_bucket": 0 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:17:46.077 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:17:46.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:46.079 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:46.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:46.103 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:17:46.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:46.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:46.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:46.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:46.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 77 16 108998 108982 88.4361 64.3125 0.00627789 0.0113026 2024-03-21T13:17:46.367 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:46.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:46.389 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:46 smithi163 ceph-mon[40675]: pgmap v897: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 993 B/s rd, 60 MiB/s wr, 960 op/s; 38276/311601 objects misplaced (12.284%); 55 MiB/s, 885 objects/s recovering 2024-03-21T13:17:46.389 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:46 smithi163 ceph-mon[40675]: osdmap e421: 8 total, 8 up, 6 in 2024-03-21T13:17:46.389 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:46 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/160322078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:17:46.389 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:46 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3479412364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:17:46.389 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:46 smithi163 ceph-mon[29621]: pgmap v897: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 993 B/s rd, 60 MiB/s wr, 960 op/s; 38276/311601 objects misplaced (12.284%); 55 MiB/s, 885 objects/s recovering 2024-03-21T13:17:46.389 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:46 smithi163 ceph-mon[29621]: osdmap e421: 8 total, 8 up, 6 in 2024-03-21T13:17:46.390 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:46 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/160322078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:17:46.390 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:46 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3479412364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:17:46.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:46.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:46 smithi204 ceph-mon[33153]: pgmap v897: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 993 B/s rd, 60 MiB/s wr, 960 op/s; 38276/311601 objects misplaced (12.284%); 55 MiB/s, 885 objects/s recovering 2024-03-21T13:17:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:46 smithi204 ceph-mon[33153]: osdmap e421: 8 total, 8 up, 6 in 2024-03-21T13:17:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:46 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/160322078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:17:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:46 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3479412364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T13:17:46.509 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:46.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:46.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:46.611 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:46.649 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:46.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:46.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:46.752 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:46.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:46.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:46.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:46.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:46.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:46.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:47.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:47.070 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:47.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:47.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:47.172 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:47.208 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:47.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:47.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:47.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:47.347 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:47.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:47.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 78 15 110087 110072 88.1756 68.125 0.018846 0.0113364 2024-03-21T13:17:47.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:47.486 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:47.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:47.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:47 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:17:47.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:47 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/160322078' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:17:47.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:47 smithi163 ceph-mon[29621]: osdmap e422: 8 total, 8 up, 6 in 2024-03-21T13:17:47.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:47 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:17:47.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:47 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/160322078' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:17:47.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:47 smithi163 ceph-mon[40675]: osdmap e422: 8 total, 8 up, 6 in 2024-03-21T13:17:47.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:47 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:17:47.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:47 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/160322078' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:17:47.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:47 smithi204 ceph-mon[33153]: osdmap e422: 8 total, 8 up, 6 in 2024-03-21T13:17:47.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:47.620 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:47.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:47.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:47.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:47.758 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:47.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:47.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:47.860 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:47.900 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:47.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:48.001 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:48.061 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:48.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:48.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:48.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:48.205 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:48.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:48.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:48.343 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:48.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:48.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 79 16 111148 111132 87.8981 66.25 0.0244322 0.0113722 2024-03-21T13:17:48.445 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:48.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:48.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:48.579 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:48.619 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:48.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:48.721 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:48.760 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:48 smithi163 ceph-mon[29621]: pgmap v900: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s; 36306/325680 objects misplaced (11.148%); 64 MiB/s, 1.02k objects/s recovering 2024-03-21T13:17:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:48 smithi163 ceph-mon[40675]: pgmap v900: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s; 36306/325680 objects misplaced (11.148%); 64 MiB/s, 1.02k objects/s recovering 2024-03-21T13:17:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:48.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:48.863 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:48.890 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:48.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:48.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:48.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:48 smithi204 ceph-mon[33153]: pgmap v900: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s; 36306/325680 objects misplaced (11.148%); 64 MiB/s, 1.02k objects/s recovering 2024-03-21T13:17:49.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:49.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:49.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:49.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:49.168 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:49.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:49.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:49.306 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:49.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:49.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:2024-03-21T13:17:49.350122+0000 min lat: 0.00136681 max lat: 2.54676 avg lat: 0.0114097 2024-03-21T13:17:49.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T13:17:49.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 80 16 112186 112170 87.6103 64.875 0.0167393 0.0114097 2024-03-21T13:17:49.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:49.447 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:49.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:49.549 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:49.573 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:49.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:49.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:49.713 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:49.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:49.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:49.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:49.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:49.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:49.960 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:49.996 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:49.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:50.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:49 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:17:50.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:49 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:17:50.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:49 smithi204 ceph-mon[33153]: osdmap e423: 8 total, 8 up, 6 in 2024-03-21T13:17:50.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:49 smithi204 ceph-mon[33153]: 4.e scrub starts 2024-03-21T13:17:50.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:49 smithi204 ceph-mon[33153]: 4.e scrub ok 2024-03-21T13:17:50.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:49 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:17:50.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:49 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:17:50.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:49 smithi163 ceph-mon[29621]: osdmap e423: 8 total, 8 up, 6 in 2024-03-21T13:17:50.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:49 smithi163 ceph-mon[29621]: 4.e scrub starts 2024-03-21T13:17:50.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:49 smithi163 ceph-mon[29621]: 4.e scrub ok 2024-03-21T13:17:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:49 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:17:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:49 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:17:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:49 smithi163 ceph-mon[40675]: osdmap e423: 8 total, 8 up, 6 in 2024-03-21T13:17:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:49 smithi163 ceph-mon[40675]: 4.e scrub starts 2024-03-21T13:17:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:49 smithi163 ceph-mon[40675]: 4.e scrub ok 2024-03-21T13:17:50.054 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:17:50.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:50.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:50.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:50.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:50.238 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:50.280 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:50.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:50.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 81 15 113121 113106 87.2509 58.5 0.0103593 0.0114566 2024-03-21T13:17:50.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:50.420 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:50.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:50.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:50.522 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:50.559 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:50.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:50.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:50.661 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:50.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:50.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:50 smithi163 ceph-mon[29621]: pgmap v902: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 853 B/s rd, 52 MiB/s wr, 831 op/s; 36306/325680 objects misplaced (11.148%); 42 MiB/s, 678 objects/s recovering 2024-03-21T13:17:50.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:50 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:50.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:50 smithi163 ceph-mon[40675]: pgmap v902: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 853 B/s rd, 52 MiB/s wr, 831 op/s; 36306/325680 objects misplaced (11.148%); 42 MiB/s, 678 objects/s recovering 2024-03-21T13:17:50.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:50 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:50.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:50.840 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:50.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:50.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:50.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:50.974 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:50.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:50 smithi204 ceph-mon[33153]: pgmap v902: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 853 B/s rd, 52 MiB/s wr, 831 op/s; 36306/325680 objects misplaced (11.148%); 42 MiB/s, 678 objects/s recovering 2024-03-21T13:17:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:50 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:51.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:51.105 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:51.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:51.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:51.207 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:51.241 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:51.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:51.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:51.343 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:51.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 82 16 114074 114058 86.9124 59.5 0.00736243 0.011501 2024-03-21T13:17:51.378 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:51.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:51.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:51.480 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:51.523 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:51.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:51.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:51.625 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:51.659 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:51.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:51.762 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:51.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:51.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:51.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:51.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:51.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:51.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:52.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:52.069 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:52.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:52.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:52.210 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:52.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:52.312 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:52.350 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 83 16 115067 115051 86.613 62.0625 0.0171169 0.0115404 2024-03-21T13:17:52.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:52.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:52.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:52.457 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:52.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:52.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:52.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:52.595 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:52.649 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:52.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:52.750 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:52.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:52 smithi204 ceph-mon[33153]: pgmap v903: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.7 KiB/s rd, 86 MiB/s wr, 1.37k op/s; 34729/336318 objects misplaced (10.326%); 66 MiB/s, 1.06k objects/s recovering 2024-03-21T13:17:52.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:17:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:17:52.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:52 smithi163 ceph-mon[29621]: pgmap v903: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.7 KiB/s rd, 86 MiB/s wr, 1.37k op/s; 34729/336318 objects misplaced (10.326%); 66 MiB/s, 1.06k objects/s recovering 2024-03-21T13:17:52.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:52 smithi163 ceph-mon[40675]: pgmap v903: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.7 KiB/s rd, 86 MiB/s wr, 1.37k op/s; 34729/336318 objects misplaced (10.326%); 66 MiB/s, 1.06k objects/s recovering 2024-03-21T13:17:52.790 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:52.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:52.892 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:52.922 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:52.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:53.024 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:53.061 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:53.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:53.163 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:53.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:53.199 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:53.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:53.300 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:53.334 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:53.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:53.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 84 16 116018 116002 86.2895 59.4375 0.0139265 0.0115843 2024-03-21T13:17:53.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:53.471 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:17:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:53.472 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:53.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:53.509 INFO:tasks.thrashosds.thrasher:j is {'epoch': 421, 'fsid': 'd4bf79c0-e781-11ee-95cd-87774f69a715', 'created': '2024-03-21T12:52:56.031419+0000', 'modified': '2024-03-21T13:17:45.067262+0000', 'last_up_change': '2024-03-21T12:58:23.775302+0000', 'last_in_change': '2024-03-21T13:17:26.597508+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T12:56:20.557490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9460811614990234, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-21T12:58:35.713383+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.9460811614990234, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-21T12:58:36.436384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 0.75, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9460811614990234, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-21T12:58:38.049304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.75, 'score_stable': 2.75, 'optimal_score': 0.75, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.9460811614990234, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-21T12:58:39.990512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9460811614990234, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-21T12:58:41.463686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9460811614990234, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-21T13:16:12.266158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '395', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9460811614990234, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': 'c34a3b25-9e02-46e6-8f9a-d0af2615258f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.9899444580078125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 390, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6801', 'nonce': 3086410981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6802', 'nonce': 3086410981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6804', 'nonce': 3086410981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6803', 'nonce': 3086410981}]}, 'public_addr': '172.21.15.163:6801/3086410981', 'cluster_addr': '172.21.15.163:6802/3086410981', 'heartbeat_back_addr': '172.21.15.163:6804/3086410981', 'heartbeat_front_addr': '172.21.15.163:6803/3086410981', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '36e05f89-c238-41f7-be65-1553b529e8ce', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6805', 'nonce': 3666062784}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6806', 'nonce': 3666062784}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6808', 'nonce': 3666062784}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6807', 'nonce': 3666062784}]}, 'public_addr': '172.21.15.163:6805/3666062784', 'cluster_addr': '172.21.15.163:6806/3666062784', 'heartbeat_back_addr': '172.21.15.163:6808/3666062784', 'heartbeat_front_addr': '172.21.15.163:6807/3666062784', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '968ec78d-25b8-4905-b328-d490c38ca85f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 416, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6809', 'nonce': 4009213507}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6810', 'nonce': 4009213507}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6812', 'nonce': 4009213507}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6811', 'nonce': 4009213507}]}, 'public_addr': '172.21.15.163:6809/4009213507', 'cluster_addr': '172.21.15.163:6810/4009213507', 'heartbeat_back_addr': '172.21.15.163:6812/4009213507', 'heartbeat_front_addr': '172.21.15.163:6811/4009213507', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '67dfa4f6-1709-48f6-971d-c6287233492d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 416, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6813', 'nonce': 3981878189}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6814', 'nonce': 3981878189}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6816', 'nonce': 3981878189}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6815', 'nonce': 3981878189}]}, 'public_addr': '172.21.15.163:6813/3981878189', 'cluster_addr': '172.21.15.163:6814/3981878189', 'heartbeat_back_addr': '172.21.15.163:6816/3981878189', 'heartbeat_front_addr': '172.21.15.163:6815/3981878189', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7fd311a9-f3c2-456d-8ab1-341cf5b33471', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.578704833984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 416, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 719276054}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6801', 'nonce': 719276054}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6803', 'nonce': 719276054}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 719276054}]}, 'public_addr': '172.21.15.204:6800/719276054', 'cluster_addr': '172.21.15.204:6801/719276054', 'heartbeat_back_addr': '172.21.15.204:6803/719276054', 'heartbeat_front_addr': '172.21.15.204:6802/719276054', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a87759c8-251c-415d-9387-a49f7d2f36d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 416, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 1346947668}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6805', 'nonce': 1346947668}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6807', 'nonce': 1346947668}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 1346947668}]}, 'public_addr': '172.21.15.204:6804/1346947668', 'cluster_addr': '172.21.15.204:6805/1346947668', 'heartbeat_back_addr': '172.21.15.204:6807/1346947668', 'heartbeat_front_addr': '172.21.15.204:6806/1346947668', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5eb135cd-f677-4d0c-b23a-738f31615b6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 416, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 200077988}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6809', 'nonce': 200077988}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6811', 'nonce': 200077988}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 200077988}]}, 'public_addr': '172.21.15.204:6808/200077988', 'cluster_addr': '172.21.15.204:6809/200077988', 'heartbeat_back_addr': '172.21.15.204:6811/200077988', 'heartbeat_front_addr': '172.21.15.204:6810/200077988', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4f528e10-2c93-4bc2-a54c-06ca4132ed49', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 315664888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6813', 'nonce': 315664888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6815', 'nonce': 315664888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 315664888}]}, 'public_addr': '172.21.15.204:6812/315664888', 'cluster_addr': '172.21.15.204:6813/315664888', 'heartbeat_back_addr': '172.21.15.204:6815/315664888', 'heartbeat_front_addr': '172.21.15.204:6814/315664888', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-21T13:17:02.631692+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-21T13:17:24.258342+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:17:34.229596+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:16:25.480783+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:16:31.452023+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:17:44.539859+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:16:59.528422+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T13:17:10.535335+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.13', 'osds': [0, 5, 6]}, {'pgid': '6.11', 'osds': [7, 3, 4]}], 'pg_upmap_items': [{'pgid': '3.2', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.19', 'mappings': [{'from': 4, 'to': 7}]}, {'pgid': '4.9', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.19', 'mappings': [{'from': 3, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '11.1', 'osds': [6, 3, 0]}, {'pgid': '11.2', 'osds': [2, 3, 0]}, {'pgid': '11.6', 'osds': [2, 4, 0]}, {'pgid': '11.7', 'osds': [5, 4, 0]}, {'pgid': '11.9', 'osds': [3, 6, 0]}, {'pgid': '11.b', 'osds': [3, 4, 0]}, {'pgid': '11.e', 'osds': [4, 7, 0]}, {'pgid': '11.f', 'osds': [7, 2, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.163:0/3453855427': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/2571244897': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:6800/3073378854': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/1334591766': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/116934309': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:0/3890775846': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/1783921524': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:6800/2553440211': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1812694077': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1070540283': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/2354157786': '2024-03-22T12:53:32.790116+0000', '172.21.15.163:0/1591817392': '2024-03-22T12:59:18.599967+0000', '172.21.15.163:0/4152198951': '2024-03-22T12:54:00.224135+0000', '172.21.15.163:6800/1798115633': '2024-03-22T12:54:00.224135+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-21T13:17:53.509 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:17:53.572 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:53.641 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:53.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:53.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:53.743 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:53.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:53.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:53.793 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:53.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:53.895 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:17:53.930 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:53.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:17:54.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:54.032 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:54.067 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:54.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:54.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:54.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:54.196 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:54.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:54.297 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:54.333 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:54.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:54.349 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 85 16 116991 116975 85.9898 60.8125 0.00825018 0.0116244 2024-03-21T13:17:54.437 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:54.473 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:54.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:54.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:54.575 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:54.604 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:54.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:54.706 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:54.746 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:54.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:54.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:54.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:54.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:54.989 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:55.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:54 smithi204 ceph-mon[33153]: pgmap v904: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.9 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 1.3 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 33939/341151 objects misplaced (9.948%); 60 MiB/s, 953 objects/s recovering 2024-03-21T13:17:55.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:54 smithi163 ceph-mon[29621]: pgmap v904: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.9 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 1.3 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 33939/341151 objects misplaced (9.948%); 60 MiB/s, 953 objects/s recovering 2024-03-21T13:17:55.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:54 smithi163 ceph-mon[40675]: pgmap v904: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.9 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 1.3 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 33939/341151 objects misplaced (9.948%); 60 MiB/s, 953 objects/s recovering 2024-03-21T13:17:55.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:55.031 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:55.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:55.131 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:55.168 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:55.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:55.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:55.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:55.307 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:55.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:55.350 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 86 16 117948 117932 85.6854 59.8125 0.00972149 0.011666 2024-03-21T13:17:55.408 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:55.442 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:55.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:55.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:55.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:55.583 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:55.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:55.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:55.724 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:55.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:55.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:55.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:55.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:55.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:55.963 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:56.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:56.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:17:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:17:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:17:56.108 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:56.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:56.146 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:56.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:56.246 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:56.299 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:56.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:56.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:56.351 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 87 16 118916 118900 85.3959 60.5 0.0117175 0.0117054 2024-03-21T13:17:56.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:56.438 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:56.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:56.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:56.540 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:56.584 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:56.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:56.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:56.698 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:56 smithi163 ceph-mon[29621]: pgmap v905: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.9 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 576 B/s rd, 49 MiB/s wr, 780 op/s; 33500/342018 objects misplaced (9.795%); 40 MiB/s, 636 objects/s recovering 2024-03-21T13:17:56.698 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:56 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/865125507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:17:56.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:56 smithi163 ceph-mon[40675]: pgmap v905: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.9 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 576 B/s rd, 49 MiB/s wr, 780 op/s; 33500/342018 objects misplaced (9.795%); 40 MiB/s, 636 objects/s recovering 2024-03-21T13:17:56.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:56 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/865125507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:17:56.724 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:56.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:56.825 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:56.862 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:56.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:56.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:56.964 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:57.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:56 smithi204 ceph-mon[33153]: pgmap v905: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 6.9 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 576 B/s rd, 49 MiB/s wr, 780 op/s; 33500/342018 objects misplaced (9.795%); 40 MiB/s, 636 objects/s recovering 2024-03-21T13:17:57.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:56 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/865125507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:17:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:57.004 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:57.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:57.103 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:17:57.103 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:17:57.103 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:17:57.103 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:17:57.103 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:17:57.103 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:17:57.103 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:17:57.103 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:17:57.103 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:17:57.103 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:17:57.103 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:17:57.103 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:17:57.103 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:17:57.104 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:17:57.104 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:17:57.104 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:17:57.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:17:57.143 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:57.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:57.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:17:57.245 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:57.282 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:57.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:57.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:57.350 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 88 15 119882 119867 85.1122 60.4375 0.0143269 0.0117448 2024-03-21T13:17:57.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:57.429 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:57.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:57.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:57.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:57.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:57.569 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:57.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:57.670 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:57.703 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:57.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:57.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:57.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:57.840 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:57.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:57.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:57.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:57.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:57.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:57.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:58.057 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:17:58.085 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:17:58.140 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:58.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:17:58.242 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:58.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:58.291 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:58.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:58.350 INFO:tasks.radosbench.radosbench.0.smithi163.stdout: 89 16 120942 120926 84.8995 66.1875 0.00925184 0.0117739 2024-03-21T13:17:58.392 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:58.432 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:58.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:58.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:58.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:58.572 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:58.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:58.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:58.711 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:58.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:58.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:58.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:58 smithi163 ceph-mon[29621]: pgmap v906: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 7.2 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s; 30950/355542 objects misplaced (8.705%); 52 MiB/s, 825 objects/s recovering 2024-03-21T13:17:58.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:58 smithi163 ceph-mon[40675]: pgmap v906: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 7.2 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s; 30950/355542 objects misplaced (8.705%); 52 MiB/s, 825 objects/s recovering 2024-03-21T13:17:58.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:58.852 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:58.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:58.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:58.954 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:17:58.990 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:58.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:17:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:58 smithi204 ceph-mon[33153]: pgmap v906: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 7.2 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s; 30950/355542 objects misplaced (8.705%); 52 MiB/s, 825 objects/s recovering 2024-03-21T13:17:59.092 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:17:59.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:59.131 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:59.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:17:59.232 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:17:59.271 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:59.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:59.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:17:59.351 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Total time run: 90.0115 2024-03-21T13:17:59.351 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Total writes made: 121981 2024-03-21T13:17:59.351 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Write size: 65536 2024-03-21T13:17:59.351 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Object size: 65536 2024-03-21T13:17:59.351 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Bandwidth (MB/sec): 84.6982 2024-03-21T13:17:59.351 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev Bandwidth: 38.3892 2024-03-21T13:17:59.351 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max bandwidth (MB/sec): 213.625 2024-03-21T13:17:59.351 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min bandwidth (MB/sec): 0 2024-03-21T13:17:59.351 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Average IOPS: 1355 2024-03-21T13:17:59.351 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev IOPS: 614.227 2024-03-21T13:17:59.351 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max IOPS: 3418 2024-03-21T13:17:59.351 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min IOPS: 0 2024-03-21T13:17:59.352 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Average Latency(s): 0.0118022 2024-03-21T13:17:59.352 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Stddev Latency(s): 0.0279716 2024-03-21T13:17:59.352 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Max latency(s): 2.54676 2024-03-21T13:17:59.352 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Min latency(s): 0.00136681 2024-03-21T13:17:59.355 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Cleaning up (deleting benchmark objects) 2024-03-21T13:17:59.373 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:59.411 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:59.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:59.512 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:17:59.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:59.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:17:59.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:17:59.652 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:59.682 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:59.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:59.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:59.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:59 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:17:59.754 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:59 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:17:59.754 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:59 smithi163 ceph-mon[29621]: osdmap e424: 8 total, 8 up, 6 in 2024-03-21T13:17:59.754 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:59 smithi163 ceph-mon[29621]: 3.a scrub starts 2024-03-21T13:17:59.754 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:17:59 smithi163 ceph-mon[29621]: 3.a scrub ok 2024-03-21T13:17:59.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:59 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:17:59.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:59 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:17:59.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:59 smithi163 ceph-mon[40675]: osdmap e424: 8 total, 8 up, 6 in 2024-03-21T13:17:59.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:59 smithi163 ceph-mon[40675]: 3.a scrub starts 2024-03-21T13:17:59.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:17:59 smithi163 ceph-mon[40675]: 3.a scrub ok 2024-03-21T13:17:59.784 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:59.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:59.827 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:59.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:17:59.929 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:17:59.963 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:17:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:17:59.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:59 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:18:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:59 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:18:00.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:59 smithi204 ceph-mon[33153]: osdmap e424: 8 total, 8 up, 6 in 2024-03-21T13:18:00.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:59 smithi204 ceph-mon[33153]: 3.a scrub starts 2024-03-21T13:18:00.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:17:59 smithi204 ceph-mon[33153]: 3.a scrub ok 2024-03-21T13:18:00.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:00.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:00.108 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:00.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:00.124 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:18:00.209 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:00.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:00.259 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:00.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:00.359 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:00.417 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:00.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:00.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:00.519 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:00.555 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:00.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:00.645 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:00.658 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:00.729 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:00.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:00.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:00.871 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:00.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:00.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:00.886 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:00 smithi163 ceph-mon[29621]: pgmap v908: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 7.2 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s; 30950/355542 objects misplaced (8.705%); 52 MiB/s, 825 objects/s recovering 2024-03-21T13:18:00.887 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:00 smithi163 ceph-mon[40675]: pgmap v908: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 7.2 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s; 30950/355542 objects misplaced (8.705%); 52 MiB/s, 825 objects/s recovering 2024-03-21T13:18:00.973 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:01.002 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:00 smithi204 ceph-mon[33153]: pgmap v908: 148 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 140 active+clean; 7.2 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s; 30950/355542 objects misplaced (8.705%); 52 MiB/s, 825 objects/s recovering 2024-03-21T13:18:01.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:01.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:01.104 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:01.165 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:01.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:01.267 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:01.302 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:01.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:01.404 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:01.445 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:01.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:01.547 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:01.586 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:01.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:01.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:01.687 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:01.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:01.727 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:01.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:01.828 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:01.864 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:01.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:01.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:02.000 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:02.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:02.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:02.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:02.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:02.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:02.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:02.238 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:02.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:02.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:02.392 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:02.435 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:02.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:02.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:02.537 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:02.575 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:02.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:02.676 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:02.715 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:02.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:02 smithi204 ceph-mon[33153]: pgmap v909: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 1.00k op/s; 28409/360948 objects misplaced (7.871%); 53 MiB/s, 855 objects/s recovering 2024-03-21T13:18:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:02 smithi204 ceph-mon[33153]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:18:02.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:02 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:18:02.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:18:02 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:18:02.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:02 smithi163 ceph-mon[29621]: pgmap v909: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 1.00k op/s; 28409/360948 objects misplaced (7.871%); 53 MiB/s, 855 objects/s recovering 2024-03-21T13:18:02.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:02 smithi163 ceph-mon[29621]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:18:02.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:02 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:18:02.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:02 smithi163 ceph-mon[40675]: pgmap v909: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 1.00k op/s; 28409/360948 objects misplaced (7.871%); 53 MiB/s, 855 objects/s recovering 2024-03-21T13:18:02.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:02 smithi163 ceph-mon[40675]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T13:18:02.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:02 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:18:02.817 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:02.865 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:02.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:02.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:02.967 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:03.017 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:03.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:03.118 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:03.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:03.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:03.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:03.263 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:03.299 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:03.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:03.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:03.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:03.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:03.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:03.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:03.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:03.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:03.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:03.720 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:03.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:03.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:03.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:03.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:03.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:03.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:03.860 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:03.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:03.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:03.997 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:03.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:03.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:04.099 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:04.139 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:04.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:04.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:04.241 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:04.287 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:04.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:04.389 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:04.425 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:04.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:04.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:04.527 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:04.567 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:04.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:04.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:04.705 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:04.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:04.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:04.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:04 smithi163 ceph-mon[29621]: pgmap v910: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.4 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 53 MiB/s wr, 998 op/s; 27328/361995 objects misplaced (7.549%); 53 MiB/s, 851 objects/s recovering 2024-03-21T13:18:04.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:04 smithi163 ceph-mon[40675]: pgmap v910: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.4 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 53 MiB/s wr, 998 op/s; 27328/361995 objects misplaced (7.549%); 53 MiB/s, 851 objects/s recovering 2024-03-21T13:18:04.807 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:04.860 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:04.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:04.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:04 smithi204 ceph-mon[33153]: pgmap v910: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.4 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 53 MiB/s wr, 998 op/s; 27328/361995 objects misplaced (7.549%); 53 MiB/s, 851 objects/s recovering 2024-03-21T13:18:05.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:05.007 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:05.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:05.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:05.142 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:05.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:05.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:05.244 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:05.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:05.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:05.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:05.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:05.434 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:05.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:05.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:05.513 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:18:05.513 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-03-21T13:18:05.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-03-21T13:18:05.513 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:18:05.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.514 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:54244 9.12 9.7c67c952 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-03-21T13:18:05.514 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:09:19.445638+0000", 2024-03-21T13:18:05.514 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 526.06024700800003, 2024-03-21T13:18:05.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.99024263899999998, 2024-03-21T13:18:05.514 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.514 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:18:05.514 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:18:05.514 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:18:05.514 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:18:05.514 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 54244 2024-03-21T13:18:05.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.514 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445638+0000", 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445638+0000", 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445639+0000", 2024-03-21T13:18:05.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0890000000000001e-06 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445668+0000", 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9726e-05 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445676+0000", 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0660000000000004e-06 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.516 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445683+0000", 2024-03-21T13:18:05.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3990000000000001e-06 2024-03-21T13:18:05.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445741+0000", 2024-03-21T13:18:05.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7972e-05 2024-03-21T13:18:05.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:18:05.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.445751+0000", 2024-03-21T13:18:05.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0227999999999999e-05 2024-03-21T13:18:05.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:20.435868+0000", 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.99011690600000002 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:20.435880+0000", 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2252999999999999e-05 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.518 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:54255 9.12 9.40534e52 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-03-21T13:18:05.519 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:09:19.449680+0000", 2024-03-21T13:18:05.519 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 526.05620450900005, 2024-03-21T13:18:05.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98623570800000004, 2024-03-21T13:18:05.519 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.519 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:18:05.519 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:18:05.519 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:18:05.519 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:18:05.519 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 54255 2024-03-21T13:18:05.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.519 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449680+0000", 2024-03-21T13:18:05.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.520 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449680+0000", 2024-03-21T13:18:05.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.520 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449681+0000", 2024-03-21T13:18:05.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2559999999999999e-06 2024-03-21T13:18:05.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449704+0000", 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2748e-05 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449719+0000", 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4584e-05 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449726+0000", 2024-03-21T13:18:05.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6990000000000003e-06 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449788+0000", 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1667000000000004e-05 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:19.449797+0000", 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.799e-06 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:20.435907+0000", 2024-03-21T13:18:05.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98611042400000004 2024-03-21T13:18:05.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:09:20.435916+0000", 2024-03-21T13:18:05.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5310000000000001e-06 2024-03-21T13:18:05.523 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.523 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.523 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.523 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:100377 9.12 9:4e5aa4e2:::benchmark_data_smithi163_131650_object100376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-03-21T13:18:05.523 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:02.557198+0000", 2024-03-21T13:18:05.523 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 482.94868660899999, 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1069594089999999, 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100377 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557198+0000", 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557198+0000", 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557199+0000", 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.344e-06 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557224+0000", 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888e-05 2024-03-21T13:18:05.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557238+0000", 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3759e-05 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557245+0000", 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3529999999999997e-06 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557311+0000", 2024-03-21T13:18:05.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5513999999999996e-05 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557434+0000", 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012332599999999999 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.557755+0000", 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000320718 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:18:05.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.560432+0000", 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026767940000000001 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.561289+0000", 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00085723799999999997 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.573020+0000", 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011730908999999999 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:18:05.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.573043+0000", 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3289999999999999e-05 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584602+0000", 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0115587029999999 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584621+0000", 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9125000000000001e-05 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.664079+0000", 2024-03-21T13:18:05.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0794580840000001 2024-03-21T13:18:05.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.530 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.530 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.664157+0000", 2024-03-21T13:18:05.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8363999999999997e-05 2024-03-21T13:18:05.530 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.530 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.530 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.530 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:100402 9.12 9:4d69482a:::benchmark_data_smithi163_131650_object100401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:18:05.530 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:02.595353+0000", 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 482.91053129800002, 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0785798529999999, 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100402 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595353+0000", 2024-03-21T13:18:05.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595353+0000", 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595354+0000", 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6400000000000002e-07 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595374+0000", 2024-03-21T13:18:05.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0288e-05 2024-03-21T13:18:05.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595381+0000", 2024-03-21T13:18:05.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3690000000000001e-06 2024-03-21T13:18:05.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595388+0000", 2024-03-21T13:18:05.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-03-21T13:18:05.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595445+0000", 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7123000000000002e-05 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.595461+0000", 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5758000000000001e-05 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584764+0000", 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98930377599999997 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584774+0000", 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7000000000000003e-06 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.664696+0000", 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.079921769 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.664847+0000", 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000151249 2024-03-21T13:18:05.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,3,7", 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.665217+0000", 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00037034999999999998 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673509+0000", 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082912909999999992 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673550+0000", 2024-03-21T13:18:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1844000000000001e-05 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673597+0000", 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.651e-05 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673749+0000", 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000152082 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673821+0000", 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2371000000000002e-05 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673933+0000", 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000111629 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:100408 9.12 9:4f7d3001:::benchmark_data_smithi163_131650_object100407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:02.605697+0000", 2024-03-21T13:18:05.538 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 482.90018776300002, 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0683809150000001, 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100408 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605697+0000", 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605697+0000", 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605701+0000", 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.985e-06 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605730+0000", 2024-03-21T13:18:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8864999999999999e-05 2024-03-21T13:18:05.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.541 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:05.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605741+0000", 2024-03-21T13:18:05.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.148e-05 2024-03-21T13:18:05.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605748+0000", 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3839999999999999e-06 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605818+0000", 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9108000000000003e-05 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.605829+0000", 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1874e-05 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584799+0000", 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.97896959400000005 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.584808+0000", 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2320000000000007e-06 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.667194+0000", 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.082386031 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.667343+0000", 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000148672 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,3,7", 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.667701+0000", 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035776999999999998 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673640+0000", 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.005939405 2024-03-21T13:18:05.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.546 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:18:05.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673684+0000", 2024-03-21T13:18:05.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4180000000000001e-05 2024-03-21T13:18:05.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.546 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:18:05.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673730+0000", 2024-03-21T13:18:05.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5911000000000001e-05 2024-03-21T13:18:05.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.546 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:18:05.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673945+0000", 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000214606 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.673989+0000", 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3665000000000003e-05 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:04.674078+0000", 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9153000000000005e-05 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25226.0:187739 9.12 9:48cfd7ca:::benchmark_data_smithi163_131650_object66209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-03-21T13:18:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:11:17.703816+0000", 2024-03-21T13:18:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 407.80206888999999, 2024-03-21T13:18:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98840552800000003, 2024-03-21T13:18:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:18:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25226", 2024-03-21T13:18:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/4066745164", 2024-03-21T13:18:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 187739 2024-03-21T13:18:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703816+0000", 2024-03-21T13:18:05.549 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703816+0000", 2024-03-21T13:18:05.549 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703817+0000", 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1850000000000001e-06 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703822+0000", 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1120000000000004e-06 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703836+0000", 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4517e-05 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.550 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703846+0000", 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1209999999999996e-06 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703908+0000", 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2137000000000005e-05 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:17.703918+0000", 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9569999999999997e-06 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.686943+0000", 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98302562299999996 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.687082+0000", 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000138381 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,7", 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.687230+0000", 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014813700000000001 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.689821+0000", 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0025908120000000001 2024-03-21T13:18:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.690005+0000", 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018441599999999999 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.692114+0000", 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002109016 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.692176+0000", 2024-03-21T13:18:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2414000000000002e-05 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:11:18.692221+0000", 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4700000000000002e-05 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93012 11.7 e417/416 11:e49b414a:::benchmark_data_smithi163_180069_object93011:head v 417'5689, mlcod=0'0)", 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:28.768967+0000", 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.736917630000001, 2024-03-21T13:18:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.42767649000000002, 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.768967+0000", 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.768967+0000", 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.768967+0000", 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6499999999999999e-07 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.771597+0000", 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026292149999999999 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.771608+0000", 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1127999999999999e-05 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.771615+0000", 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2529999999999998e-06 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.771690+0000", 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4859999999999998e-05 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.771712+0000", 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1937e-05 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.196589+0000", 2024-03-21T13:18:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.42487707600000002 2024-03-21T13:18:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.196643+0000", 2024-03-21T13:18:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4555999999999997e-05 2024-03-21T13:18:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93035 11.e e417/416 11:70c494e9:::benchmark_data_smithi163_180069_object93034:head v 417'5863, mlcod=414'5860)", 2024-03-21T13:18:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:28.772822+0000", 2024-03-21T13:18:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.733062046000001, 2024-03-21T13:18:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.20042238800000001, 2024-03-21T13:18:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.772822+0000", 2024-03-21T13:18:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.772822+0000", 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.772823+0000", 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8800000000000002e-07 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.772856+0000", 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2353e-05 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.772866+0000", 2024-03-21T13:18:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0051000000000001e-05 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.772873+0000", 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7249999999999997e-06 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.873785+0000", 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.100911638 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.873808+0000", 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3150999999999999e-05 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.973212+0000", 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.099403804999999998 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.973245+0000", 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2876999999999997e-05 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.562 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93019 11.7 e417/416 11:e54d94e6:::benchmark_data_smithi163_180069_object93018:head v 417'5690, mlcod=0'0)", 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:28.787293+0000", 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.718591320999998, 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.40938723500000002, 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.787293+0000", 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.787293+0000", 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.787294+0000", 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2799999999999995e-07 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.787357+0000", 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2792000000000005e-05 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.787370+0000", 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3477e-05 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.787378+0000", 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5399999999999998e-06 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.875354+0000", 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.087975721000000007 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.875378+0000", 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4723999999999999e-05 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.196650+0000", 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.32127159700000002 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.196680+0000", 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0555999999999997e-05 2024-03-21T13:18:05.566 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93045 11.7 e417/416 11:eb5e6aee:::benchmark_data_smithi163_180069_object93044:head v 417'5691, mlcod=0'0)", 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:28.812602+0000", 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.693282263, 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.38411144400000002, 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.812602+0000", 2024-03-21T13:18:05.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.812602+0000", 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.812604+0000", 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.279e-06 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.812640+0000", 2024-03-21T13:18:05.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6368999999999997e-05 2024-03-21T13:18:05.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.812661+0000", 2024-03-21T13:18:05.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0849999999999999e-05 2024-03-21T13:18:05.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.812671+0000", 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0587e-05 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.965574+0000", 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.15290237100000001 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.965604+0000", 2024-03-21T13:18:05.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9836999999999999e-05 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.196683+0000", 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.23107958100000001 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.196714+0000", 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.057e-05 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93059 11.7 e417/416 11:eafef986:::benchmark_data_smithi163_180069_object93058:head v 417'5692, mlcod=0'0)", 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:28.985035+0000", 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.520849867999999, 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.211763433, 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.985035+0000", 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.572 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.985035+0000", 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.985036+0000", 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1800000000000004e-07 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.985078+0000", 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2700000000000001e-05 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.985089+0000", 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0892e-05 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.985097+0000", 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5410000000000003e-06 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.085337+0000", 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.100240512 2024-03-21T13:18:05.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.085410+0000", 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2447000000000006e-05 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.196769+0000", 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.11135891000000001 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.196798+0000", 2024-03-21T13:18:05.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9513e-05 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93062 11.7 e417/416 11:e8be1a38:::benchmark_data_smithi163_180069_object93061:head v 417'5693, mlcod=0'0)", 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:28.993700+0000", 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.51218497, 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.20328547599999999, 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.576 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.993700+0000", 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.993700+0000", 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.993701+0000", 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0580000000000001e-06 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.993736+0000", 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5267e-05 2024-03-21T13:18:05.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.993752+0000", 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5902000000000001e-05 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.993760+0000", 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9869999999999999e-06 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.090576+0000", 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.096816546000000003 2024-03-21T13:18:05.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.090601+0000", 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5083999999999999e-05 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.196838+0000", 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.10623640600000001 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.196985+0000", 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147226 2024-03-21T13:18:05.579 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.580 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.580 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.580 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93183 11.2 e417/416 11:46d7a472:::benchmark_data_smithi163_180069_object93182:head v 417'5784, mlcod=417'5781)", 2024-03-21T13:18:05.580 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:29.290829+0000", 2024-03-21T13:18:05.580 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.215055327999998, 2024-03-21T13:18:05.580 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.31993612199999999, 2024-03-21T13:18:05.580 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.580 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.580 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.290829+0000", 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.290829+0000", 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.290831+0000", 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.486e-06 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.290862+0000", 2024-03-21T13:18:05.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0849000000000002e-05 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.290875+0000", 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3922e-05 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.290885+0000", 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6590000000000002e-06 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.582 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.381335+0000", 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090449963999999994 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.381382+0000", 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6723e-05 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.610677+0000", 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.229294845 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.610765+0000", 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8673999999999998e-05 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93204 11.2 e417/416 11:4af33cbc:::benchmark_data_smithi163_180069_object93203:head v 417'5785, mlcod=417'5782)", 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:29.307275+0000", 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.198609394000002, 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.48628104999999999, 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.584 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.307275+0000", 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.307275+0000", 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.307276+0000", 2024-03-21T13:18:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.291e-06 2024-03-21T13:18:05.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.307312+0000", 2024-03-21T13:18:05.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6028999999999997e-05 2024-03-21T13:18:05.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.307329+0000", 2024-03-21T13:18:05.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.63e-05 2024-03-21T13:18:05.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.307339+0000", 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.91e-06 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.430511+0000", 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.123172106 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.430538+0000", 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.764e-05 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.793493+0000", 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.36295439200000001 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.793556+0000", 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3381999999999997e-05 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.588 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93213 11.2 e417/416 11:4ac3c9c8:::benchmark_data_smithi163_180069_object93212:head v 417'5786, mlcod=417'5782)", 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:29.330300+0000", 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.175584364000002, 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.46329026499999998, 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.330300+0000", 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.330300+0000", 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.330301+0000", 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.51e-07 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.330326+0000", 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5579999999999999e-05 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.330333+0000", 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9460000000000001e-06 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.330339+0000", 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4369999999999998e-06 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.595452+0000", 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.26511385799999998 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.595472+0000", 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9697000000000001e-05 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.793560+0000", 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.19808764400000001 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.793590+0000", 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0651999999999997e-05 2024-03-21T13:18:05.592 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93223 11.f e417/416 11:f840e9eb:::benchmark_data_smithi163_180069_object93222:head v 417'5783, mlcod=417'5780)", 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:29.388487+0000", 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.117397660000002, 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.20676420100000001, 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.388487+0000", 2024-03-21T13:18:05.593 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.388487+0000", 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.388488+0000", 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.297e-06 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.594 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.388541+0000", 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.388558+0000", 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7356000000000001e-05 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.388568+0000", 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0046e-05 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.596 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.388647+0000", 2024-03-21T13:18:05.596 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8650000000000001e-05 2024-03-21T13:18:05.596 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.596 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.596 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.388672+0000", 2024-03-21T13:18:05.596 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5131e-05 2024-03-21T13:18:05.596 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.596 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.597 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.595191+0000", 2024-03-21T13:18:05.597 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.206518597 2024-03-21T13:18:05.597 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.598 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.598 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.595251+0000", 2024-03-21T13:18:05.598 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0537e-05 2024-03-21T13:18:05.598 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.598 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.598 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.598 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93230 11.1 e417/416 11:85cf3d60:::benchmark_data_smithi163_180069_object93229:head v 417'5765, mlcod=417'5760)", 2024-03-21T13:18:05.598 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:29.391843+0000", 2024-03-21T13:18:05.598 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.114041161999999, 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.20678419100000001, 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.391843+0000", 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.391843+0000", 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.391844+0000", 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1030000000000001e-06 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.391880+0000", 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5292e-05 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.391898+0000", 2024-03-21T13:18:05.600 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7910999999999998e-05 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.391908+0000", 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0339e-05 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.433545+0000", 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.041637334999999998 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.601 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.433569+0000", 2024-03-21T13:18:05.602 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3626999999999999e-05 2024-03-21T13:18:05.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.602 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.602 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.598524+0000", 2024-03-21T13:18:05.602 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.16495474900000001 2024-03-21T13:18:05.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.602 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.602 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.598628+0000", 2024-03-21T13:18:05.602 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000103835 2024-03-21T13:18:05.602 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.602 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93228 11.f e417/416 11:fb6bee62:::benchmark_data_smithi163_180069_object93227:head v 417'5784, mlcod=417'5781)", 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:29.392105+0000", 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.113779311999998, 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.20640081499999999, 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.392105+0000", 2024-03-21T13:18:05.603 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.392105+0000", 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.392106+0000", 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2399999999999997e-07 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.604 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.392136+0000", 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9937000000000002e-05 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.392147+0000", 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1303e-05 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.392154+0000", 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1919999999999999e-06 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.605 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.430881+0000", 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038726877 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.430904+0000", 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2430999999999999e-05 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.598457+0000", 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.16755345599999999 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.606 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.598506+0000", 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8795000000000003e-05 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93232 11.6 e417/416 11:603dbfce:::benchmark_data_smithi163_180069_object93231:head v 417'5844, mlcod=417'5842)", 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:29.395737+0000", 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.110147499, 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.20299217899999999, 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.607 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.395737+0000", 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.395737+0000", 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.395738+0000", 2024-03-21T13:18:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.302e-06 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.396031+0000", 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029316800000000001 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.396049+0000", 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7526000000000001e-05 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.396058+0000", 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.833e-06 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.396173+0000", 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011480300000000001 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.396197+0000", 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4515000000000002e-05 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.610 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.598484+0000", 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.20228701299999999 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.598729+0000", 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024501900000000002 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.15858.0:93238 11.2 e417/416 11:4c2af2cd:::benchmark_data_smithi163_180069_object93237:head v 417'5787, mlcod=417'5782)", 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:29.409585+0000", 2024-03-21T13:18:05.611 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.096299508999998, 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.38403290699999998, 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409585+0000", 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409585+0000", 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:18:05.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409587+0000", 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.629e-06 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409616+0000", 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9519e-05 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409632+0000", 2024-03-21T13:18:05.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6175999999999999e-05 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409643+0000", 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0244999999999999e-05 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.597992+0000", 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.18834920199999999 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:18:05.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.598017+0000", 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5618000000000001e-05 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.793593+0000", 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.19557590999999999 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.793618+0000", 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4607999999999999e-05 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:18:05.615 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:18:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:05.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:05.718 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:05.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:18:05.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:18:05.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:18:05.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:05 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1998575274' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:18:05.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:18:05.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:18:05.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:18:05.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:05 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1998575274' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:18:05.765 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:05.765 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:18:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:18:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:18:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:05.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:05.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:05.890 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:18:05.905 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:05.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:18:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:18:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:18:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:05 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1998575274' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:18:06.007 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:06.048 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:06.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:06.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:06.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:06.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:06.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:06.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:06.263 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:18:06.266 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:18:06.301 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:06.348 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:06.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:06.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:06.450 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:06.489 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:06.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:06.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:06.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:06.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:06.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:06.737 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:06.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:06.788 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:06.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:06.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[29621]: pgmap v911: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 999 op/s; 26677/361104 objects misplaced (7.388%); 53 MiB/s, 847 objects/s recovering 2024-03-21T13:18:06.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:18:06.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:06.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:18:06.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:18:06.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1998575274' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:18:06.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[29621]: osdmap e425: 8 total, 8 up, 6 in 2024-03-21T13:18:06.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[40675]: pgmap v911: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 999 op/s; 26677/361104 objects misplaced (7.388%); 53 MiB/s, 847 objects/s recovering 2024-03-21T13:18:06.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:18:06.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:06.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:18:06.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:18:06.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1998575274' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:18:06.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:06 smithi163 ceph-mon[40675]: osdmap e425: 8 total, 8 up, 6 in 2024-03-21T13:18:06.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:06.926 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:06.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:06.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:06.990 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-21T13:18:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:06 smithi204 ceph-mon[33153]: pgmap v911: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 999 op/s; 26677/361104 objects misplaced (7.388%); 53 MiB/s, 847 objects/s recovering 2024-03-21T13:18:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:06 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:18:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:06 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:06 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:18:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:06 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:18:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:06 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1998575274' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:18:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:06 smithi204 ceph-mon[33153]: osdmap e425: 8 total, 8 up, 6 in 2024-03-21T13:18:07.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:07.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:07.065 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:07.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:07.167 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:07.210 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:07.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:07.311 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:07.349 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:07.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:07.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:07.451 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:07.485 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:07.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:07.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:07.587 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:07.620 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:07.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:07.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:07.721 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:07.755 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:07.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:07.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:07.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:07 smithi163 ceph-mon[29621]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:18:07.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:07 smithi163 ceph-mon[40675]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:18:07.827 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":912,"stamp":"2024-03-21T13:18:05.884990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7672633792,"num_objects":117265,"num_object_clones":0,"num_object_copies":351795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":23130,"num_objects_unfound":0,"num_objects_dirty":117265,"num_whiteouts":0,"num_read":2495,"num_read_kb":2341,"num_write":127798,"num_write_kb":7809246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32353,"num_bytes_recovered":2094019381,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66748,"ondisk_log_size":66748,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":26306640,"kb_used_data":20990752,"kb_used_omap":9,"kb_used_meta":5315574,"kb_avail":536164272,"statfs":{"total":575970213888,"available":549032214528,"internally_reserved":0,"allocated":21494530048,"data_stored":21492205810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":5443148454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,5,7,16,8,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-81919976,"num_objects":-1249,"num_object_clones":0,"num_object_copies":-3747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-7820,"num_objects_unfound":0,"num_objects_dirty":-1249,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":8581,"num_write_kb":234561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7636,"num_bytes_recovered":500432896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.902763"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":206,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.579497+0000","last_change":"2024-03-21T13:14:59.311326+0000","last_active":"2024-03-21T13:17:58.579497+0000","last_peered":"2024-03-21T13:17:58.579497+0000","last_clean":"2024-03-21T13:17:58.579497+0000","last_became_active":"2024-03-21T13:12:29.525755+0000","last_became_peered":"2024-03-21T13:12:29.525755+0000","last_unstale":"2024-03-21T13:17:58.579497+0000","last_undegraded":"2024-03-21T13:17:58.579497+0000","last_fullsized":"2024-03-21T13:17:58.579497+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:59.311141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:14:59.311141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":214,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.578269+0000","last_change":"2024-03-21T13:16:21.204738+0000","last_active":"2024-03-21T13:17:58.578269+0000","last_peered":"2024-03-21T13:17:58.578269+0000","last_clean":"2024-03-21T13:17:58.578269+0000","last_became_active":"2024-03-21T13:12:29.510271+0000","last_became_peered":"2024-03-21T13:12:29.510271+0000","last_unstale":"2024-03-21T13:17:58.578269+0000","last_undegraded":"2024-03-21T13:17:58.578269+0000","last_fullsized":"2024-03-21T13:17:58.578269+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:21.204657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.204657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":145,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584926+0000","last_change":"2024-03-21T13:17:27.660661+0000","last_active":"2024-03-21T13:17:58.584926+0000","last_peered":"2024-03-21T13:17:58.584926+0000","last_clean":"2024-03-21T13:17:58.584926+0000","last_became_active":"2024-03-21T13:17:27.657450+0000","last_became_peered":"2024-03-21T13:17:27.657450+0000","last_unstale":"2024-03-21T13:17:58.584926+0000","last_undegraded":"2024-03-21T13:17:58.584926+0000","last_fullsized":"2024-03-21T13:17:58.584926+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:02.044853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:02.044853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"234'19","reported_seq":859,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.588600+0000","last_change":"2024-03-21T13:16:09.694036+0000","last_active":"2024-03-21T13:17:58.588600+0000","last_peered":"2024-03-21T13:17:58.588600+0000","last_clean":"2024-03-21T13:17:58.588600+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:17:58.588600+0000","last_undegraded":"2024-03-21T13:17:58.588600+0000","last_fullsized":"2024-03-21T13:17:58.588600+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:09.693933+0000","last_deep_scrub":"234'19","last_deep_scrub_stamp":"2024-03-21T13:16:09.693933+0000","last_clean_scrub_stamp":"2024-03-21T13:16:09.693933+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":757,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.581024+0000","last_change":"2024-03-21T13:15:02.457035+0000","last_active":"2024-03-21T13:17:58.581024+0000","last_peered":"2024-03-21T13:17:58.581024+0000","last_clean":"2024-03-21T13:17:58.581024+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:17:58.581024+0000","last_undegraded":"2024-03-21T13:17:58.581024+0000","last_fullsized":"2024-03-21T13:17:58.581024+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:02.456902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:15:02.456902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":715,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583217+0000","last_change":"2024-03-21T13:17:12.860708+0000","last_active":"2024-03-21T13:17:58.583217+0000","last_peered":"2024-03-21T13:17:58.583217+0000","last_clean":"2024-03-21T13:17:58.583217+0000","last_became_active":"2024-03-21T13:10:03.591543+0000","last_became_peered":"2024-03-21T13:10:03.591543+0000","last_unstale":"2024-03-21T13:17:58.583217+0000","last_undegraded":"2024-03-21T13:17:58.583217+0000","last_fullsized":"2024-03-21T13:17:58.583217+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:12.860612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:17:12.860612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:18.222457+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":798,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585192+0000","last_change":"2024-03-21T13:17:03.752054+0000","last_active":"2024-03-21T13:17:58.585192+0000","last_peered":"2024-03-21T13:17:58.585192+0000","last_clean":"2024-03-21T13:17:58.585192+0000","last_became_active":"2024-03-21T13:10:03.610318+0000","last_became_peered":"2024-03-21T13:10:03.610318+0000","last_unstale":"2024-03-21T13:17:58.585192+0000","last_undegraded":"2024-03-21T13:17:58.585192+0000","last_fullsized":"2024-03-21T13:17:58.585192+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:17:03.751920+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_clean_scrub_stamp":"2024-03-21T13:17:03.751920+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:15.990057+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":318,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585269+0000","last_change":"2024-03-21T13:17:27.668489+0000","last_active":"2024-03-21T13:17:58.585269+0000","last_peered":"2024-03-21T13:17:58.585269+0000","last_clean":"2024-03-21T13:17:58.585269+0000","last_became_active":"2024-03-21T13:17:27.652219+0000","last_became_peered":"2024-03-21T13:17:27.652219+0000","last_unstale":"2024-03-21T13:17:58.585269+0000","last_undegraded":"2024-03-21T13:17:58.585269+0000","last_fullsized":"2024-03-21T13:17:58.585269+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:17:01.724037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:01.724037+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:16.695375+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":590,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.582856+0000","last_change":"2024-03-21T13:17:27.645240+0000","last_active":"2024-03-21T13:17:58.582856+0000","last_peered":"2024-03-21T13:17:58.582856+0000","last_clean":"2024-03-21T13:17:58.582856+0000","last_became_active":"2024-03-21T13:17:27.641680+0000","last_became_peered":"2024-03-21T13:17:27.641680+0000","last_unstale":"2024-03-21T13:17:58.582856+0000","last_undegraded":"2024-03-21T13:17:58.582856+0000","last_fullsized":"2024-03-21T13:17:58.582856+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:20.951893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.951893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":212,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.581754+0000","last_change":"2024-03-21T13:17:27.659929+0000","last_active":"2024-03-21T13:17:58.581754+0000","last_peered":"2024-03-21T13:17:58.581754+0000","last_clean":"2024-03-21T13:17:58.581754+0000","last_became_active":"2024-03-21T13:17:27.659272+0000","last_became_peered":"2024-03-21T13:17:27.659272+0000","last_unstale":"2024-03-21T13:17:58.581754+0000","last_undegraded":"2024-03-21T13:17:58.581754+0000","last_fullsized":"2024-03-21T13:17:58.581754+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:12.302013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:15:12.302013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":206,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583790+0000","last_change":"2024-03-21T13:15:18.085333+0000","last_active":"2024-03-21T13:17:58.583790+0000","last_peered":"2024-03-21T13:17:58.583790+0000","last_clean":"2024-03-21T13:17:58.583790+0000","last_became_active":"2024-03-21T13:12:29.526129+0000","last_became_peered":"2024-03-21T13:12:29.526129+0000","last_unstale":"2024-03-21T13:17:58.583790+0000","last_undegraded":"2024-03-21T13:17:58.583790+0000","last_fullsized":"2024-03-21T13:17:58.583790+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:18.085215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:15:18.085215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":784,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.582423+0000","last_change":"2024-03-21T13:17:27.650123+0000","last_active":"2024-03-21T13:17:58.582423+0000","last_peered":"2024-03-21T13:17:58.582423+0000","last_clean":"2024-03-21T13:17:58.582423+0000","last_became_active":"2024-03-21T13:17:27.639498+0000","last_became_peered":"2024-03-21T13:17:27.639498+0000","last_unstale":"2024-03-21T13:17:58.582423+0000","last_undegraded":"2024-03-21T13:17:58.582423+0000","last_fullsized":"2024-03-21T13:17:58.582423+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:07.739235+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.739235+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":214,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.578344+0000","last_change":"2024-03-21T13:16:56.397703+0000","last_active":"2024-03-21T13:17:58.578344+0000","last_peered":"2024-03-21T13:17:58.578344+0000","last_clean":"2024-03-21T13:17:58.578344+0000","last_became_active":"2024-03-21T13:12:29.508520+0000","last_became_peered":"2024-03-21T13:12:29.508520+0000","last_unstale":"2024-03-21T13:17:58.578344+0000","last_undegraded":"2024-03-21T13:17:58.578344+0000","last_fullsized":"2024-03-21T13:17:58.578344+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:56.397585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:56.397585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:13.161134+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":794,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583280+0000","last_change":"2024-03-21T13:17:27.641476+0000","last_active":"2024-03-21T13:17:58.583280+0000","last_peered":"2024-03-21T13:17:58.583280+0000","last_clean":"2024-03-21T13:17:58.583280+0000","last_became_active":"2024-03-21T13:17:27.641161+0000","last_became_peered":"2024-03-21T13:17:27.641161+0000","last_unstale":"2024-03-21T13:17:58.583280+0000","last_undegraded":"2024-03-21T13:17:58.583280+0000","last_fullsized":"2024-03-21T13:17:58.583280+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:18.791316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:18.791316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:19.350759+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":206,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.581933+0000","last_change":"2024-03-21T13:16:04.184426+0000","last_active":"2024-03-21T13:17:58.581933+0000","last_peered":"2024-03-21T13:17:58.581933+0000","last_clean":"2024-03-21T13:17:58.581933+0000","last_became_active":"2024-03-21T13:12:29.524473+0000","last_became_peered":"2024-03-21T13:12:29.524473+0000","last_unstale":"2024-03-21T13:17:58.581933+0000","last_undegraded":"2024-03-21T13:17:58.581933+0000","last_fullsized":"2024-03-21T13:17:58.581933+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:04.184314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:04.184314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"234'9","reported_seq":796,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580667+0000","last_change":"2024-03-21T13:17:09.946959+0000","last_active":"2024-03-21T13:17:58.580667+0000","last_peered":"2024-03-21T13:17:58.580667+0000","last_clean":"2024-03-21T13:17:58.580667+0000","last_became_active":"2024-03-21T13:10:03.595745+0000","last_became_peered":"2024-03-21T13:10:03.595745+0000","last_unstale":"2024-03-21T13:17:58.580667+0000","last_undegraded":"2024-03-21T13:17:58.580667+0000","last_fullsized":"2024-03-21T13:17:58.580667+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'9","last_scrub_stamp":"2024-03-21T13:17:09.946815+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:17:09.946815+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:13.523384+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":811,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.591295+0000","last_change":"2024-03-21T13:16:06.811101+0000","last_active":"2024-03-21T13:17:58.591295+0000","last_peered":"2024-03-21T13:17:58.591295+0000","last_clean":"2024-03-21T13:17:58.591295+0000","last_became_active":"2024-03-21T13:12:29.520189+0000","last_became_peered":"2024-03-21T13:12:29.520189+0000","last_unstale":"2024-03-21T13:17:58.591295+0000","last_undegraded":"2024-03-21T13:17:58.591295+0000","last_fullsized":"2024-03-21T13:17:58.591295+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:06.810993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:23.865785+0000","last_clean_scrub_stamp":"2024-03-21T13:16:06.810993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":704,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.589622+0000","last_change":"2024-03-21T13:17:27.643575+0000","last_active":"2024-03-21T13:17:58.589622+0000","last_peered":"2024-03-21T13:17:58.589622+0000","last_clean":"2024-03-21T13:17:58.589622+0000","last_became_active":"2024-03-21T13:17:27.643258+0000","last_became_peered":"2024-03-21T13:17:27.643258+0000","last_unstale":"2024-03-21T13:17:58.589622+0000","last_undegraded":"2024-03-21T13:17:58.589622+0000","last_fullsized":"2024-03-21T13:17:58.589622+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:07.756896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.756896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":582,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.586559+0000","last_change":"2024-03-21T13:17:27.669084+0000","last_active":"2024-03-21T13:17:58.586559+0000","last_peered":"2024-03-21T13:17:58.586559+0000","last_clean":"2024-03-21T13:17:58.586559+0000","last_became_active":"2024-03-21T13:17:27.668691+0000","last_became_peered":"2024-03-21T13:17:27.668691+0000","last_unstale":"2024-03-21T13:17:58.586559+0000","last_undegraded":"2024-03-21T13:17:58.586559+0000","last_fullsized":"2024-03-21T13:17:58.586559+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:08.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:12.172304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":380,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583947+0000","last_change":"2024-03-21T13:17:27.662499+0000","last_active":"2024-03-21T13:17:58.583947+0000","last_peered":"2024-03-21T13:17:58.583947+0000","last_clean":"2024-03-21T13:17:58.583947+0000","last_became_active":"2024-03-21T13:17:27.639926+0000","last_became_peered":"2024-03-21T13:17:27.639926+0000","last_unstale":"2024-03-21T13:17:58.583947+0000","last_undegraded":"2024-03-21T13:17:58.583947+0000","last_fullsized":"2024-03-21T13:17:58.583947+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:16:54.409873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.409873+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:22.301273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":783,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.581847+0000","last_change":"2024-03-21T13:15:59.096623+0000","last_active":"2024-03-21T13:17:58.581847+0000","last_peered":"2024-03-21T13:17:58.581847+0000","last_clean":"2024-03-21T13:17:58.581847+0000","last_became_active":"2024-03-21T13:12:29.502808+0000","last_became_peered":"2024-03-21T13:12:29.502808+0000","last_unstale":"2024-03-21T13:17:58.581847+0000","last_undegraded":"2024-03-21T13:17:58.581847+0000","last_fullsized":"2024-03-21T13:17:58.581847+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:59.096520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:59.096520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":702,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580098+0000","last_change":"2024-03-21T13:17:15.990962+0000","last_active":"2024-03-21T13:17:58.580098+0000","last_peered":"2024-03-21T13:17:58.580098+0000","last_clean":"2024-03-21T13:17:58.580098+0000","last_became_active":"2024-03-21T13:10:03.612409+0000","last_became_peered":"2024-03-21T13:10:03.612409+0000","last_unstale":"2024-03-21T13:17:58.580098+0000","last_undegraded":"2024-03-21T13:17:58.580098+0000","last_fullsized":"2024-03-21T13:17:58.580098+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:15.990807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:15.990807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:35.320404+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":833,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.588764+0000","last_change":"2024-03-21T13:17:27.630867+0000","last_active":"2024-03-21T13:17:58.588764+0000","last_peered":"2024-03-21T13:17:58.588764+0000","last_clean":"2024-03-21T13:17:58.588764+0000","last_became_active":"2024-03-21T13:17:27.630488+0000","last_became_peered":"2024-03-21T13:17:27.630488+0000","last_unstale":"2024-03-21T13:17:58.588764+0000","last_undegraded":"2024-03-21T13:17:58.588764+0000","last_fullsized":"2024-03-21T13:17:58.588764+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:48.967258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:20.956778+0000","last_clean_scrub_stamp":"2024-03-21T13:16:48.967258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:08.637038+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":833,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.581495+0000","last_change":"2024-03-21T13:16:55.685055+0000","last_active":"2024-03-21T13:17:58.581495+0000","last_peered":"2024-03-21T13:17:58.581495+0000","last_clean":"2024-03-21T13:17:58.581495+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:17:58.581495+0000","last_undegraded":"2024-03-21T13:17:58.581495+0000","last_fullsized":"2024-03-21T13:17:58.581495+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-21T13:16:55.684896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:55.684896+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:05.652758+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":193,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583293+0000","last_change":"2024-03-21T13:17:05.780695+0000","last_active":"2024-03-21T13:17:58.583293+0000","last_peered":"2024-03-21T13:17:58.583293+0000","last_clean":"2024-03-21T13:17:58.583293+0000","last_became_active":"2024-03-21T13:13:06.420220+0000","last_became_peered":"2024-03-21T13:13:06.420220+0000","last_unstale":"2024-03-21T13:17:58.583293+0000","last_undegraded":"2024-03-21T13:17:58.583293+0000","last_fullsized":"2024-03-21T13:17:58.583293+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:17:05.780572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:05.780572+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:22.825319+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":1015,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580264+0000","last_change":"2024-03-21T13:16:12.489669+0000","last_active":"2024-03-21T13:17:58.580264+0000","last_peered":"2024-03-21T13:17:58.580264+0000","last_clean":"2024-03-21T13:17:58.580264+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:17:58.580264+0000","last_undegraded":"2024-03-21T13:17:58.580264+0000","last_fullsized":"2024-03-21T13:17:58.580264+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:16:12.489518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.489518+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"407'2","reported_seq":677,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.578896+0000","last_change":"2024-03-21T13:16:58.735925+0000","last_active":"2024-03-21T13:17:58.578896+0000","last_peered":"2024-03-21T13:17:58.578896+0000","last_clean":"2024-03-21T13:17:58.578896+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:17:58.578896+0000","last_undegraded":"2024-03-21T13:17:58.578896+0000","last_fullsized":"2024-03-21T13:17:58.578896+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'2","last_scrub_stamp":"2024-03-21T13:16:58.735421+0000","last_deep_scrub":"407'2","last_deep_scrub_stamp":"2024-03-21T13:16:58.735421+0000","last_clean_scrub_stamp":"2024-03-21T13:16:58.735421+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:03.350682+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"410'25","reported_seq":451,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584827+0000","last_change":"2024-03-21T13:17:27.682401+0000","last_active":"2024-03-21T13:17:58.584827+0000","last_peered":"2024-03-21T13:17:58.584827+0000","last_clean":"2024-03-21T13:17:58.584827+0000","last_became_active":"2024-03-21T13:17:27.655020+0000","last_became_peered":"2024-03-21T13:17:27.655020+0000","last_unstale":"2024-03-21T13:17:58.584827+0000","last_undegraded":"2024-03-21T13:17:58.584827+0000","last_fullsized":"2024-03-21T13:17:58.584827+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-03-21T13:17:09.876547+0000","last_deep_scrub":"410'25","last_deep_scrub_stamp":"2024-03-21T13:17:09.876547+0000","last_clean_scrub_stamp":"2024-03-21T13:17:09.876547+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:29.810842+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":206,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583703+0000","last_change":"2024-03-21T13:16:07.247321+0000","last_active":"2024-03-21T13:17:58.583703+0000","last_peered":"2024-03-21T13:17:58.583703+0000","last_clean":"2024-03-21T13:17:58.583703+0000","last_became_active":"2024-03-21T13:12:29.527129+0000","last_became_peered":"2024-03-21T13:12:29.527129+0000","last_unstale":"2024-03-21T13:17:58.583703+0000","last_undegraded":"2024-03-21T13:17:58.583703+0000","last_fullsized":"2024-03-21T13:17:58.583703+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:07.247194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.247194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":603,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584740+0000","last_change":"2024-03-21T13:17:27.650677+0000","last_active":"2024-03-21T13:17:58.584740+0000","last_peered":"2024-03-21T13:17:58.584740+0000","last_clean":"2024-03-21T13:17:58.584740+0000","last_became_active":"2024-03-21T13:17:27.649940+0000","last_became_peered":"2024-03-21T13:17:27.649940+0000","last_unstale":"2024-03-21T13:17:58.584740+0000","last_undegraded":"2024-03-21T13:17:58.584740+0000","last_fullsized":"2024-03-21T13:17:58.584740+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:12.814107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:12.814107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:17.128461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":801,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.576391+0000","last_change":"2024-03-21T13:17:27.662093+0000","last_active":"2024-03-21T13:17:58.576391+0000","last_peered":"2024-03-21T13:17:58.576391+0000","last_clean":"2024-03-21T13:17:58.576391+0000","last_became_active":"2024-03-21T13:17:27.661238+0000","last_became_peered":"2024-03-21T13:17:27.661238+0000","last_unstale":"2024-03-21T13:17:58.576391+0000","last_undegraded":"2024-03-21T13:17:58.576391+0000","last_fullsized":"2024-03-21T13:17:58.576391+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:04.838772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:04.838772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:13.829507+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"331'24","reported_seq":882,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580318+0000","last_change":"2024-03-21T13:17:27.715932+0000","last_active":"2024-03-21T13:17:58.580318+0000","last_peered":"2024-03-21T13:17:58.580318+0000","last_clean":"2024-03-21T13:17:58.580318+0000","last_became_active":"2024-03-21T13:17:27.652511+0000","last_became_peered":"2024-03-21T13:17:27.652511+0000","last_unstale":"2024-03-21T13:17:58.580318+0000","last_undegraded":"2024-03-21T13:17:58.580318+0000","last_fullsized":"2024-03-21T13:17:58.580318+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'24","last_scrub_stamp":"2024-03-21T13:15:09.670459+0000","last_deep_scrub":"331'24","last_deep_scrub_stamp":"2024-03-21T13:15:09.670459+0000","last_clean_scrub_stamp":"2024-03-21T13:15:09.670459+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":27,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.582899+0000","last_change":"2024-03-21T13:17:27.637231+0000","last_active":"2024-03-21T13:17:58.582899+0000","last_peered":"2024-03-21T13:17:58.582899+0000","last_clean":"2024-03-21T13:17:58.582899+0000","last_became_active":"2024-03-21T13:17:27.636917+0000","last_became_peered":"2024-03-21T13:17:27.636917+0000","last_unstale":"2024-03-21T13:17:58.582899+0000","last_undegraded":"2024-03-21T13:17:58.582899+0000","last_fullsized":"2024-03-21T13:17:58.582899+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:59.238664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:16:59.238664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:10.772088+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":777,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.579895+0000","last_change":"2024-03-21T13:16:21.557896+0000","last_active":"2024-03-21T13:17:58.579895+0000","last_peered":"2024-03-21T13:17:58.579895+0000","last_clean":"2024-03-21T13:17:58.579895+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:17:58.579895+0000","last_undegraded":"2024-03-21T13:17:58.579895+0000","last_fullsized":"2024-03-21T13:17:58.579895+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:21.557805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.557805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":137,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585359+0000","last_change":"2024-03-21T13:17:27.660095+0000","last_active":"2024-03-21T13:17:58.585359+0000","last_peered":"2024-03-21T13:17:58.585359+0000","last_clean":"2024-03-21T13:17:58.585359+0000","last_became_active":"2024-03-21T13:17:27.657429+0000","last_became_peered":"2024-03-21T13:17:27.657429+0000","last_unstale":"2024-03-21T13:17:58.585359+0000","last_undegraded":"2024-03-21T13:17:58.585359+0000","last_fullsized":"2024-03-21T13:17:58.585359+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:13.840785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:22.955737+0000","last_clean_scrub_stamp":"2024-03-21T13:17:13.840785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:26.460782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":790,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.582956+0000","last_change":"2024-03-21T13:16:20.734468+0000","last_active":"2024-03-21T13:17:58.582956+0000","last_peered":"2024-03-21T13:17:58.582956+0000","last_clean":"2024-03-21T13:17:58.582956+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:17:58.582956+0000","last_undegraded":"2024-03-21T13:17:58.582956+0000","last_fullsized":"2024-03-21T13:17:58.582956+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-21T13:16:20.734379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.734379+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":764,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580957+0000","last_change":"2024-03-21T13:17:27.637166+0000","last_active":"2024-03-21T13:17:58.580957+0000","last_peered":"2024-03-21T13:17:58.580957+0000","last_clean":"2024-03-21T13:17:58.580957+0000","last_became_active":"2024-03-21T13:17:27.636805+0000","last_became_peered":"2024-03-21T13:17:27.636805+0000","last_unstale":"2024-03-21T13:17:58.580957+0000","last_undegraded":"2024-03-21T13:17:58.580957+0000","last_fullsized":"2024-03-21T13:17:58.580957+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:14:59.413139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:14:59.413139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":27,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583517+0000","last_change":"2024-03-21T13:17:27.645475+0000","last_active":"2024-03-21T13:17:58.583517+0000","last_peered":"2024-03-21T13:17:58.583517+0000","last_clean":"2024-03-21T13:17:58.583517+0000","last_became_active":"2024-03-21T13:17:27.644954+0000","last_became_peered":"2024-03-21T13:17:27.644954+0000","last_unstale":"2024-03-21T13:17:58.583517+0000","last_undegraded":"2024-03-21T13:17:58.583517+0000","last_fullsized":"2024-03-21T13:17:58.583517+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:05.059443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.059443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":214,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584186+0000","last_change":"2024-03-21T13:16:23.266104+0000","last_active":"2024-03-21T13:17:58.584186+0000","last_peered":"2024-03-21T13:17:58.584186+0000","last_clean":"2024-03-21T13:17:58.584186+0000","last_became_active":"2024-03-21T13:12:29.507706+0000","last_became_peered":"2024-03-21T13:12:29.507706+0000","last_unstale":"2024-03-21T13:17:58.584186+0000","last_undegraded":"2024-03-21T13:17:58.584186+0000","last_fullsized":"2024-03-21T13:17:58.584186+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:23.266023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:23.266023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":511,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580986+0000","last_change":"2024-03-21T13:15:55.923610+0000","last_active":"2024-03-21T13:17:58.580986+0000","last_peered":"2024-03-21T13:17:58.580986+0000","last_clean":"2024-03-21T13:17:58.580986+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:17:58.580986+0000","last_undegraded":"2024-03-21T13:17:58.580986+0000","last_fullsized":"2024-03-21T13:17:58.580986+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:15:55.923514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:55.923514+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":129,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.578658+0000","last_change":"2024-03-21T13:17:27.646425+0000","last_active":"2024-03-21T13:17:58.578658+0000","last_peered":"2024-03-21T13:17:58.578658+0000","last_clean":"2024-03-21T13:17:58.578658+0000","last_became_active":"2024-03-21T13:17:27.645569+0000","last_became_peered":"2024-03-21T13:17:27.645569+0000","last_unstale":"2024-03-21T13:17:58.578658+0000","last_undegraded":"2024-03-21T13:17:58.578658+0000","last_fullsized":"2024-03-21T13:17:58.578658+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:50.025891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:50.025891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:59.335115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":786,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580763+0000","last_change":"2024-03-21T13:17:27.646045+0000","last_active":"2024-03-21T13:17:58.580763+0000","last_peered":"2024-03-21T13:17:58.580763+0000","last_clean":"2024-03-21T13:17:58.580763+0000","last_became_active":"2024-03-21T13:17:27.645710+0000","last_became_peered":"2024-03-21T13:17:27.645710+0000","last_unstale":"2024-03-21T13:17:58.580763+0000","last_undegraded":"2024-03-21T13:17:58.580763+0000","last_fullsized":"2024-03-21T13:17:58.580763+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:18.312139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:18.312139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":775,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583419+0000","last_change":"2024-03-21T13:16:08.584734+0000","last_active":"2024-03-21T13:17:58.583419+0000","last_peered":"2024-03-21T13:17:58.583419+0000","last_clean":"2024-03-21T13:17:58.583419+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:17:58.583419+0000","last_undegraded":"2024-03-21T13:17:58.583419+0000","last_fullsized":"2024-03-21T13:17:58.583419+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:08.584643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:19.383090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:08.584643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":800,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.579808+0000","last_change":"2024-03-21T13:17:27.688812+0000","last_active":"2024-03-21T13:17:58.579808+0000","last_peered":"2024-03-21T13:17:58.579808+0000","last_clean":"2024-03-21T13:17:58.579808+0000","last_became_active":"2024-03-21T13:17:27.653358+0000","last_became_peered":"2024-03-21T13:17:27.653358+0000","last_unstale":"2024-03-21T13:17:58.579808+0000","last_undegraded":"2024-03-21T13:17:58.579808+0000","last_fullsized":"2024-03-21T13:17:58.579808+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:53.867747+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.867747+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:07.058914+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":610,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583189+0000","last_change":"2024-03-21T13:17:08.823533+0000","last_active":"2024-03-21T13:17:58.583189+0000","last_peered":"2024-03-21T13:17:58.583189+0000","last_clean":"2024-03-21T13:17:58.583189+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:17:58.583189+0000","last_undegraded":"2024-03-21T13:17:58.583189+0000","last_fullsized":"2024-03-21T13:17:58.583189+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:08.823336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.823336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:24.082746+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":214,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.579697+0000","last_change":"2024-03-21T13:16:52.230103+0000","last_active":"2024-03-21T13:17:58.579697+0000","last_peered":"2024-03-21T13:17:58.579697+0000","last_clean":"2024-03-21T13:17:58.579697+0000","last_became_active":"2024-03-21T13:12:29.510100+0000","last_became_peered":"2024-03-21T13:12:29.510100+0000","last_unstale":"2024-03-21T13:17:58.579697+0000","last_undegraded":"2024-03-21T13:17:58.579697+0000","last_fullsized":"2024-03-21T13:17:58.579697+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.229670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:22.501055+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.229670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:08.462478+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":487,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583840+0000","last_change":"2024-03-21T13:15:59.651653+0000","last_active":"2024-03-21T13:17:58.583840+0000","last_peered":"2024-03-21T13:17:58.583840+0000","last_clean":"2024-03-21T13:17:58.583840+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:17:58.583840+0000","last_undegraded":"2024-03-21T13:17:58.583840+0000","last_fullsized":"2024-03-21T13:17:58.583840+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:59.651571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:15:59.651571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":39,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.588918+0000","last_change":"2024-03-21T13:17:27.675968+0000","last_active":"2024-03-21T13:17:58.588918+0000","last_peered":"2024-03-21T13:17:58.588918+0000","last_clean":"2024-03-21T13:17:58.588918+0000","last_became_active":"2024-03-21T13:17:27.653620+0000","last_became_peered":"2024-03-21T13:17:27.653620+0000","last_unstale":"2024-03-21T13:17:58.588918+0000","last_undegraded":"2024-03-21T13:17:58.588918+0000","last_fullsized":"2024-03-21T13:17:58.588918+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:17:11.846315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:11.846315+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:37.521390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":590,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584588+0000","last_change":"2024-03-21T13:17:27.651621+0000","last_active":"2024-03-21T13:17:58.584588+0000","last_peered":"2024-03-21T13:17:58.584588+0000","last_clean":"2024-03-21T13:17:58.584588+0000","last_became_active":"2024-03-21T13:17:27.651255+0000","last_became_peered":"2024-03-21T13:17:27.651255+0000","last_unstale":"2024-03-21T13:17:58.584588+0000","last_undegraded":"2024-03-21T13:17:58.584588+0000","last_fullsized":"2024-03-21T13:17:58.584588+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:53.026764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.026764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:14.701854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"416'13","reported_seq":297,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583986+0000","last_change":"2024-03-21T13:17:27.662539+0000","last_active":"2024-03-21T13:17:58.583986+0000","last_peered":"2024-03-21T13:17:58.583986+0000","last_clean":"2024-03-21T13:17:58.583986+0000","last_became_active":"2024-03-21T13:17:27.644197+0000","last_became_peered":"2024-03-21T13:17:27.644197+0000","last_unstale":"2024-03-21T13:17:58.583986+0000","last_undegraded":"2024-03-21T13:17:58.583986+0000","last_fullsized":"2024-03-21T13:17:58.583986+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'12","last_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_deep_scrub":"394'12","last_deep_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_clean_scrub_stamp":"2024-03-21T13:16:24.237800+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":795,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585944+0000","last_change":"2024-03-21T13:17:16.856970+0000","last_active":"2024-03-21T13:17:58.585944+0000","last_peered":"2024-03-21T13:17:58.585944+0000","last_clean":"2024-03-21T13:17:58.585944+0000","last_became_active":"2024-03-21T13:10:03.599265+0000","last_became_peered":"2024-03-21T13:10:03.599265+0000","last_unstale":"2024-03-21T13:17:58.585944+0000","last_undegraded":"2024-03-21T13:17:58.585944+0000","last_fullsized":"2024-03-21T13:17:58.585944+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:16.856845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:16.856845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:18.973771+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'20","reported_seq":850,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580369+0000","last_change":"2024-03-21T13:15:16.283969+0000","last_active":"2024-03-21T13:17:58.580369+0000","last_peered":"2024-03-21T13:17:58.580369+0000","last_clean":"2024-03-21T13:17:58.580369+0000","last_became_active":"2024-03-21T13:10:03.613119+0000","last_became_peered":"2024-03-21T13:10:03.613119+0000","last_unstale":"2024-03-21T13:17:58.580369+0000","last_undegraded":"2024-03-21T13:17:58.580369+0000","last_fullsized":"2024-03-21T13:17:58.580369+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:15:16.283831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:15:16.283831+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":348,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.587141+0000","last_change":"2024-03-21T13:16:17.497436+0000","last_active":"2024-03-21T13:17:58.587141+0000","last_peered":"2024-03-21T13:17:58.587141+0000","last_clean":"2024-03-21T13:17:58.587141+0000","last_became_active":"2024-03-21T13:12:29.507751+0000","last_became_peered":"2024-03-21T13:12:29.507751+0000","last_unstale":"2024-03-21T13:17:58.587141+0000","last_undegraded":"2024-03-21T13:17:58.587141+0000","last_fullsized":"2024-03-21T13:17:58.587141+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:17.497325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:17.497325+0000","last_clean_scrub_stamp":"2024-03-21T13:16:17.497325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"272'10","reported_seq":1024,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580501+0000","last_change":"2024-03-21T13:16:09.700298+0000","last_active":"2024-03-21T13:17:58.580501+0000","last_peered":"2024-03-21T13:17:58.580501+0000","last_clean":"2024-03-21T13:17:58.580501+0000","last_became_active":"2024-03-21T13:10:03.594651+0000","last_became_peered":"2024-03-21T13:10:03.594651+0000","last_unstale":"2024-03-21T13:17:58.580501+0000","last_undegraded":"2024-03-21T13:17:58.580501+0000","last_fullsized":"2024-03-21T13:17:58.580501+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'10","last_scrub_stamp":"2024-03-21T13:16:09.700215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:09.700215+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":590,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583126+0000","last_change":"2024-03-21T13:17:27.650659+0000","last_active":"2024-03-21T13:17:58.583126+0000","last_peered":"2024-03-21T13:17:58.583126+0000","last_clean":"2024-03-21T13:17:58.583126+0000","last_became_active":"2024-03-21T13:17:27.650261+0000","last_became_peered":"2024-03-21T13:17:27.650261+0000","last_unstale":"2024-03-21T13:17:58.583126+0000","last_undegraded":"2024-03-21T13:17:58.583126+0000","last_fullsized":"2024-03-21T13:17:58.583126+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:15.939231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:16:15.939231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"234'15","reported_seq":224,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583334+0000","last_change":"2024-03-21T13:16:22.227639+0000","last_active":"2024-03-21T13:17:58.583334+0000","last_peered":"2024-03-21T13:17:58.583334+0000","last_clean":"2024-03-21T13:17:58.583334+0000","last_became_active":"2024-03-21T13:12:29.510247+0000","last_became_peered":"2024-03-21T13:12:29.510247+0000","last_unstale":"2024-03-21T13:17:58.583334+0000","last_undegraded":"2024-03-21T13:17:58.583334+0000","last_fullsized":"2024-03-21T13:17:58.583334+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:16:22.227536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.227536+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":342,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583671+0000","last_change":"2024-03-21T13:17:07.670896+0000","last_active":"2024-03-21T13:17:58.583671+0000","last_peered":"2024-03-21T13:17:58.583671+0000","last_clean":"2024-03-21T13:17:58.583671+0000","last_became_active":"2024-03-21T13:10:03.613841+0000","last_became_peered":"2024-03-21T13:10:03.613841+0000","last_unstale":"2024-03-21T13:17:58.583671+0000","last_undegraded":"2024-03-21T13:17:58.583671+0000","last_fullsized":"2024-03-21T13:17:58.583671+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:07.670782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:17:07.670782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:29.821007+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":782,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583090+0000","last_change":"2024-03-21T13:17:27.659110+0000","last_active":"2024-03-21T13:17:58.583090+0000","last_peered":"2024-03-21T13:17:58.583090+0000","last_clean":"2024-03-21T13:17:58.583090+0000","last_became_active":"2024-03-21T13:17:27.658400+0000","last_became_peered":"2024-03-21T13:17:27.658400+0000","last_unstale":"2024-03-21T13:17:58.583090+0000","last_undegraded":"2024-03-21T13:17:58.583090+0000","last_fullsized":"2024-03-21T13:17:58.583090+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:13.018536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:13.018536+0000","last_clean_scrub_stamp":"2024-03-21T13:17:13.018536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:36.589427+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":789,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585041+0000","last_change":"2024-03-21T13:16:54.653009+0000","last_active":"2024-03-21T13:17:58.585041+0000","last_peered":"2024-03-21T13:17:58.585041+0000","last_clean":"2024-03-21T13:17:58.585041+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:17:58.585041+0000","last_undegraded":"2024-03-21T13:17:58.585041+0000","last_fullsized":"2024-03-21T13:17:58.585041+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:16:54.652821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.652821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'15","reported_seq":636,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585100+0000","last_change":"2024-03-21T13:17:27.719115+0000","last_active":"2024-03-21T13:17:58.585100+0000","last_peered":"2024-03-21T13:17:58.585100+0000","last_clean":"2024-03-21T13:17:58.585100+0000","last_became_active":"2024-03-21T13:17:27.659084+0000","last_became_peered":"2024-03-21T13:17:27.659084+0000","last_unstale":"2024-03-21T13:17:58.585100+0000","last_undegraded":"2024-03-21T13:17:58.585100+0000","last_fullsized":"2024-03-21T13:17:58.585100+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:16:58.080100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:58.080100+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:17.327062+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":763,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583658+0000","last_change":"2024-03-21T13:16:02.716355+0000","last_active":"2024-03-21T13:17:58.583658+0000","last_peered":"2024-03-21T13:17:58.583658+0000","last_clean":"2024-03-21T13:17:58.583658+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:17:58.583658+0000","last_undegraded":"2024-03-21T13:17:58.583658+0000","last_fullsized":"2024-03-21T13:17:58.583658+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:02.716255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:02.716255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":214,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584098+0000","last_change":"2024-03-21T13:16:18.220471+0000","last_active":"2024-03-21T13:17:58.584098+0000","last_peered":"2024-03-21T13:17:58.584098+0000","last_clean":"2024-03-21T13:17:58.584098+0000","last_became_active":"2024-03-21T13:12:29.514113+0000","last_became_peered":"2024-03-21T13:12:29.514113+0000","last_unstale":"2024-03-21T13:17:58.584098+0000","last_undegraded":"2024-03-21T13:17:58.584098+0000","last_fullsized":"2024-03-21T13:17:58.584098+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.220368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.220368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":803,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.581342+0000","last_change":"2024-03-21T13:17:27.656617+0000","last_active":"2024-03-21T13:17:58.581342+0000","last_peered":"2024-03-21T13:17:58.581342+0000","last_clean":"2024-03-21T13:17:58.581342+0000","last_became_active":"2024-03-21T13:17:27.654135+0000","last_became_peered":"2024-03-21T13:17:27.654135+0000","last_unstale":"2024-03-21T13:17:58.581342+0000","last_undegraded":"2024-03-21T13:17:58.581342+0000","last_fullsized":"2024-03-21T13:17:58.581342+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:16.777565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:16.777565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:32.661864+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":834,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.582930+0000","last_change":"2024-03-21T13:17:02.736704+0000","last_active":"2024-03-21T13:17:58.582930+0000","last_peered":"2024-03-21T13:17:58.582930+0000","last_clean":"2024-03-21T13:17:58.582930+0000","last_became_active":"2024-03-21T13:10:03.610810+0000","last_became_peered":"2024-03-21T13:10:03.610810+0000","last_unstale":"2024-03-21T13:17:58.582930+0000","last_undegraded":"2024-03-21T13:17:58.582930+0000","last_fullsized":"2024-03-21T13:17:58.582930+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-21T13:17:02.736579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:02.736579+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:21.749488+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"424'7756","reported_seq":16321,"reported_epoch":424,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:18:04.973596+0000","last_change":"2024-03-21T13:17:28.793959+0000","last_active":"2024-03-21T13:18:04.973596+0000","last_peered":"2024-03-21T13:18:04.973596+0000","last_clean":"2024-03-21T13:17:26.583717+0000","last_became_active":"2024-03-21T13:17:28.760772+0000","last_became_peered":"2024-03-21T13:17:28.760772+0000","last_unstale":"2024-03-21T13:18:04.973596+0000","last_undegraded":"2024-03-21T13:18:04.973596+0000","last_fullsized":"2024-03-21T13:18:04.973596+0000","mapping_epoch":416,"log_start":"408'3634","ondisk_log_start":"408'3634","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4122,"log_dups_size":0,"ondisk_log_size":4122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464650240,"num_objects":7090,"num_object_clones":0,"num_object_copies":21270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2431,"num_objects_unfound":0,"num_objects_dirty":7090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7756,"num_write_kb":475072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4242,"num_bytes_recovered":278003712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,0],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":7090}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":212,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.581362+0000","last_change":"2024-03-21T13:17:27.656889+0000","last_active":"2024-03-21T13:17:58.581362+0000","last_peered":"2024-03-21T13:17:58.581362+0000","last_clean":"2024-03-21T13:17:58.581362+0000","last_became_active":"2024-03-21T13:17:27.654442+0000","last_became_peered":"2024-03-21T13:17:27.654442+0000","last_unstale":"2024-03-21T13:17:58.581362+0000","last_undegraded":"2024-03-21T13:17:58.581362+0000","last_fullsized":"2024-03-21T13:17:58.581362+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:06.391193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:15:06.391193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":831,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.588093+0000","last_change":"2024-03-21T13:17:27.677784+0000","last_active":"2024-03-21T13:17:58.588093+0000","last_peered":"2024-03-21T13:17:58.588093+0000","last_clean":"2024-03-21T13:17:58.588093+0000","last_became_active":"2024-03-21T13:17:27.656174+0000","last_became_peered":"2024-03-21T13:17:27.656174+0000","last_unstale":"2024-03-21T13:17:58.588093+0000","last_undegraded":"2024-03-21T13:17:58.588093+0000","last_fullsized":"2024-03-21T13:17:58.588093+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:16:56.698173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:56.698173+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:20.080206+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":230,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.578741+0000","last_change":"2024-03-21T13:16:53.176991+0000","last_active":"2024-03-21T13:17:58.578741+0000","last_peered":"2024-03-21T13:17:58.578741+0000","last_clean":"2024-03-21T13:17:58.578741+0000","last_became_active":"2024-03-21T13:12:29.513019+0000","last_became_peered":"2024-03-21T13:12:29.513019+0000","last_unstale":"2024-03-21T13:17:58.578741+0000","last_undegraded":"2024-03-21T13:17:58.578741+0000","last_fullsized":"2024-03-21T13:17:58.578741+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:16:53.176836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.176836+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:13.256411+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":1013,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585393+0000","last_change":"2024-03-21T13:16:10.554480+0000","last_active":"2024-03-21T13:17:58.585393+0000","last_peered":"2024-03-21T13:17:58.585393+0000","last_clean":"2024-03-21T13:17:58.585393+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:17:58.585393+0000","last_undegraded":"2024-03-21T13:17:58.585393+0000","last_fullsized":"2024-03-21T13:17:58.585393+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:16:10.554378+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:16:10.554378+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"424'8035","reported_seq":18124,"reported_epoch":424,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:18:05.684083+0000","last_change":"2024-03-21T13:17:28.790269+0000","last_active":"2024-03-21T13:18:05.684083+0000","last_peered":"2024-03-21T13:18:05.684083+0000","last_clean":"2024-03-21T13:17:26.562189+0000","last_became_active":"2024-03-21T13:17:28.776285+0000","last_became_peered":"2024-03-21T13:17:28.776285+0000","last_unstale":"2024-03-21T13:18:05.684083+0000","last_undegraded":"2024-03-21T13:18:05.684083+0000","last_fullsized":"2024-03-21T13:18:05.684083+0000","mapping_epoch":416,"log_start":"406'3500","ondisk_log_start":"406'3500","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":2399,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473104384,"num_objects":7219,"num_object_clones":0,"num_object_copies":21657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1822,"num_objects_unfound":0,"num_objects_dirty":7219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8035,"num_write_kb":488128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4989,"num_bytes_recovered":326959104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,0],"avail_no_missing":["2","0","4"],"object_location_counts":[{"shards":"0,2,4","objects":7219}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":764,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580865+0000","last_change":"2024-03-21T13:15:58.046280+0000","last_active":"2024-03-21T13:17:58.580865+0000","last_peered":"2024-03-21T13:17:58.580865+0000","last_clean":"2024-03-21T13:17:58.580865+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:17:58.580865+0000","last_undegraded":"2024-03-21T13:17:58.580865+0000","last_fullsized":"2024-03-21T13:17:58.580865+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:58.046156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:05:43.596691+0000","last_clean_scrub_stamp":"2024-03-21T13:15:58.046156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":795,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584705+0000","last_change":"2024-03-21T13:17:27.653349+0000","last_active":"2024-03-21T13:17:58.584705+0000","last_peered":"2024-03-21T13:17:58.584705+0000","last_clean":"2024-03-21T13:17:58.584705+0000","last_became_active":"2024-03-21T13:17:27.652993+0000","last_became_peered":"2024-03-21T13:17:27.652993+0000","last_unstale":"2024-03-21T13:17:58.584705+0000","last_undegraded":"2024-03-21T13:17:58.584705+0000","last_fullsized":"2024-03-21T13:17:58.584705+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.653388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.653388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:01.570876+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"346'23","reported_seq":185,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584189+0000","last_change":"2024-03-21T13:16:10.260270+0000","last_active":"2024-03-21T13:17:58.584189+0000","last_peered":"2024-03-21T13:17:58.584189+0000","last_clean":"2024-03-21T13:17:58.584189+0000","last_became_active":"2024-03-21T13:13:21.819111+0000","last_became_peered":"2024-03-21T13:13:21.819111+0000","last_unstale":"2024-03-21T13:17:58.584189+0000","last_undegraded":"2024-03-21T13:17:58.584189+0000","last_fullsized":"2024-03-21T13:17:58.584189+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'23","last_scrub_stamp":"2024-03-21T13:16:10.260157+0000","last_deep_scrub":"346'23","last_deep_scrub_stamp":"2024-03-21T13:13:20.232807+0000","last_clean_scrub_stamp":"2024-03-21T13:16:10.260157+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":354,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583756+0000","last_change":"2024-03-21T13:17:27.633122+0000","last_active":"2024-03-21T13:17:58.583756+0000","last_peered":"2024-03-21T13:17:58.583756+0000","last_clean":"2024-03-21T13:17:58.583756+0000","last_became_active":"2024-03-21T13:17:27.632559+0000","last_became_peered":"2024-03-21T13:17:27.632559+0000","last_unstale":"2024-03-21T13:17:58.583756+0000","last_undegraded":"2024-03-21T13:17:58.583756+0000","last_fullsized":"2024-03-21T13:17:58.583756+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:55.431004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:55.431004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:00.882865+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"424'8247","reported_seq":8324,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:18:05.448406+0000","last_change":"2024-03-21T13:16:13.903159+0000","last_active":"2024-03-21T13:18:05.448406+0000","last_peered":"2024-03-21T13:18:05.448406+0000","last_clean":"2024-03-21T13:18:05.448406+0000","last_became_active":"2024-03-21T13:16:13.902868+0000","last_became_peered":"2024-03-21T13:16:13.902868+0000","last_unstale":"2024-03-21T13:18:05.448406+0000","last_undegraded":"2024-03-21T13:18:05.448406+0000","last_fullsized":"2024-03-21T13:18:05.448406+0000","mapping_epoch":390,"log_start":"410'4412","ondisk_log_start":"410'4412","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3835,"log_dups_size":0,"ondisk_log_size":3835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488046592,"num_objects":7447,"num_object_clones":0,"num_object_copies":22341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8247,"num_write_kb":502208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":783,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.582968+0000","last_change":"2024-03-21T13:16:06.611026+0000","last_active":"2024-03-21T13:17:58.582968+0000","last_peered":"2024-03-21T13:17:58.582968+0000","last_clean":"2024-03-21T13:17:58.582968+0000","last_became_active":"2024-03-21T13:10:03.598366+0000","last_became_peered":"2024-03-21T13:10:03.598366+0000","last_unstale":"2024-03-21T13:17:58.582968+0000","last_undegraded":"2024-03-21T13:17:58.582968+0000","last_fullsized":"2024-03-21T13:17:58.582968+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:06.610947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:16:06.610947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":362,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584273+0000","last_change":"2024-03-21T13:17:27.650325+0000","last_active":"2024-03-21T13:17:58.584273+0000","last_peered":"2024-03-21T13:17:58.584273+0000","last_clean":"2024-03-21T13:17:58.584273+0000","last_became_active":"2024-03-21T13:17:27.649975+0000","last_became_peered":"2024-03-21T13:17:27.649975+0000","last_unstale":"2024-03-21T13:17:58.584273+0000","last_undegraded":"2024-03-21T13:17:58.584273+0000","last_fullsized":"2024-03-21T13:17:58.584273+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:49.448804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:49.448804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":795,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583710+0000","last_change":"2024-03-21T13:16:47.661139+0000","last_active":"2024-03-21T13:17:58.583710+0000","last_peered":"2024-03-21T13:17:58.583710+0000","last_clean":"2024-03-21T13:17:58.583710+0000","last_became_active":"2024-03-21T13:12:29.508984+0000","last_became_peered":"2024-03-21T13:12:29.508984+0000","last_unstale":"2024-03-21T13:17:58.583710+0000","last_undegraded":"2024-03-21T13:17:58.583710+0000","last_fullsized":"2024-03-21T13:17:58.583710+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:47.661063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:47.661063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:15.875522+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'35","reported_seq":914,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.579365+0000","last_change":"2024-03-21T13:17:08.873256+0000","last_active":"2024-03-21T13:17:58.579365+0000","last_peered":"2024-03-21T13:17:58.579365+0000","last_clean":"2024-03-21T13:17:58.579365+0000","last_became_active":"2024-03-21T13:12:29.523359+0000","last_became_peered":"2024-03-21T13:12:29.523359+0000","last_unstale":"2024-03-21T13:17:58.579365+0000","last_undegraded":"2024-03-21T13:17:58.579365+0000","last_fullsized":"2024-03-21T13:17:58.579365+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:17:08.873105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.873105+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:09.856433+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"424'8075","reported_seq":8152,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:18:05.479940+0000","last_change":"2024-03-21T13:16:13.904507+0000","last_active":"2024-03-21T13:18:05.479940+0000","last_peered":"2024-03-21T13:18:05.479940+0000","last_clean":"2024-03-21T13:18:05.479940+0000","last_became_active":"2024-03-21T13:16:13.904210+0000","last_became_peered":"2024-03-21T13:16:13.904210+0000","last_unstale":"2024-03-21T13:18:05.479940+0000","last_undegraded":"2024-03-21T13:18:05.479940+0000","last_fullsized":"2024-03-21T13:18:05.479940+0000","mapping_epoch":390,"log_start":"410'4304","ondisk_log_start":"410'4304","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3771,"log_dups_size":0,"ondisk_log_size":3771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474284032,"num_objects":7237,"num_object_clones":0,"num_object_copies":21711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8075,"num_write_kb":489984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":595,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583376+0000","last_change":"2024-03-21T13:17:27.649278+0000","last_active":"2024-03-21T13:17:58.583376+0000","last_peered":"2024-03-21T13:17:58.583376+0000","last_clean":"2024-03-21T13:17:58.583376+0000","last_became_active":"2024-03-21T13:17:27.648954+0000","last_became_peered":"2024-03-21T13:17:27.648954+0000","last_unstale":"2024-03-21T13:17:58.583376+0000","last_undegraded":"2024-03-21T13:17:58.583376+0000","last_fullsized":"2024-03-21T13:17:58.583376+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:07.141148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:07.141148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:30.292818+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":342,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584363+0000","last_change":"2024-03-21T13:16:56.458265+0000","last_active":"2024-03-21T13:17:58.584363+0000","last_peered":"2024-03-21T13:17:58.584363+0000","last_clean":"2024-03-21T13:17:58.584363+0000","last_became_active":"2024-03-21T13:10:03.603646+0000","last_became_peered":"2024-03-21T13:10:03.603646+0000","last_unstale":"2024-03-21T13:17:58.584363+0000","last_undegraded":"2024-03-21T13:17:58.584363+0000","last_fullsized":"2024-03-21T13:17:58.584363+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:56.458050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:56.458050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:09.246703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":214,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.581841+0000","last_change":"2024-03-21T13:16:54.243549+0000","last_active":"2024-03-21T13:17:58.581841+0000","last_peered":"2024-03-21T13:17:58.581841+0000","last_clean":"2024-03-21T13:17:58.581841+0000","last_became_active":"2024-03-21T13:12:29.512741+0000","last_became_peered":"2024-03-21T13:12:29.512741+0000","last_unstale":"2024-03-21T13:17:58.581841+0000","last_undegraded":"2024-03-21T13:17:58.581841+0000","last_fullsized":"2024-03-21T13:17:58.581841+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:54.243395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.243395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:03.270582+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"234'19","reported_seq":689,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.589040+0000","last_change":"2024-03-21T13:17:27.709292+0000","last_active":"2024-03-21T13:17:58.589040+0000","last_peered":"2024-03-21T13:17:58.589040+0000","last_clean":"2024-03-21T13:17:58.589040+0000","last_became_active":"2024-03-21T13:17:27.653652+0000","last_became_peered":"2024-03-21T13:17:27.653652+0000","last_unstale":"2024-03-21T13:17:58.589040+0000","last_undegraded":"2024-03-21T13:17:58.589040+0000","last_fullsized":"2024-03-21T13:17:58.589040+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:16:23.840950+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:16:23.840950+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.b","version":"424'7548","reported_seq":15650,"reported_epoch":424,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:18:01.313962+0000","last_change":"2024-03-21T13:17:28.802858+0000","last_active":"2024-03-21T13:18:01.313962+0000","last_peered":"2024-03-21T13:18:01.313962+0000","last_clean":"2024-03-21T13:17:26.621292+0000","last_became_active":"2024-03-21T13:17:28.783364+0000","last_became_peered":"2024-03-21T13:17:28.783364+0000","last_unstale":"2024-03-21T13:18:01.313962+0000","last_undegraded":"2024-03-21T13:18:01.313962+0000","last_fullsized":"2024-03-21T13:18:01.313962+0000","mapping_epoch":416,"log_start":"408'3655","ondisk_log_start":"408'3655","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3893,"log_dups_size":0,"ondisk_log_size":3893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479461376,"num_objects":7316,"num_object_clones":0,"num_object_copies":21948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2782,"num_objects_unfound":0,"num_objects_dirty":7316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":475648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4010,"num_bytes_recovered":262799360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,0],"avail_no_missing":["3","0","4"],"object_location_counts":[{"shards":"0,3,4","objects":7316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":758,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585373+0000","last_change":"2024-03-21T13:17:06.844511+0000","last_active":"2024-03-21T13:17:58.585373+0000","last_peered":"2024-03-21T13:17:58.585373+0000","last_clean":"2024-03-21T13:17:58.585373+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:17:58.585373+0000","last_undegraded":"2024-03-21T13:17:58.585373+0000","last_fullsized":"2024-03-21T13:17:58.585373+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:06.844387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:17:06.844387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:18.955021+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":220,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.579601+0000","last_change":"2024-03-21T13:17:27.702648+0000","last_active":"2024-03-21T13:17:58.579601+0000","last_peered":"2024-03-21T13:17:58.579601+0000","last_clean":"2024-03-21T13:17:58.579601+0000","last_became_active":"2024-03-21T13:17:27.652655+0000","last_became_peered":"2024-03-21T13:17:27.652655+0000","last_unstale":"2024-03-21T13:17:58.579601+0000","last_undegraded":"2024-03-21T13:17:58.579601+0000","last_fullsized":"2024-03-21T13:17:58.579601+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:16:08.210967+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:16:08.210967+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":402,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.578848+0000","last_change":"2024-03-21T13:17:27.655022+0000","last_active":"2024-03-21T13:17:58.578848+0000","last_peered":"2024-03-21T13:17:58.578848+0000","last_clean":"2024-03-21T13:17:58.578848+0000","last_became_active":"2024-03-21T13:17:27.652986+0000","last_became_peered":"2024-03-21T13:17:27.652986+0000","last_unstale":"2024-03-21T13:17:58.578848+0000","last_undegraded":"2024-03-21T13:17:58.578848+0000","last_fullsized":"2024-03-21T13:17:58.578848+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:22.861136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:22.861136+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.861136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"272'43","reported_seq":461,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583550+0000","last_change":"2024-03-21T13:16:16.292178+0000","last_active":"2024-03-21T13:17:58.583550+0000","last_peered":"2024-03-21T13:17:58.583550+0000","last_clean":"2024-03-21T13:17:58.583550+0000","last_became_active":"2024-03-21T13:10:03.603889+0000","last_became_peered":"2024-03-21T13:10:03.603889+0000","last_unstale":"2024-03-21T13:17:58.583550+0000","last_undegraded":"2024-03-21T13:17:58.583550+0000","last_fullsized":"2024-03-21T13:17:58.583550+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'43","last_scrub_stamp":"2024-03-21T13:16:16.292073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:16.292073+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":406,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580152+0000","last_change":"2024-03-21T13:17:27.643833+0000","last_active":"2024-03-21T13:17:58.580152+0000","last_peered":"2024-03-21T13:17:58.580152+0000","last_clean":"2024-03-21T13:17:58.580152+0000","last_became_active":"2024-03-21T13:17:27.643330+0000","last_became_peered":"2024-03-21T13:17:27.643330+0000","last_unstale":"2024-03-21T13:17:58.580152+0000","last_undegraded":"2024-03-21T13:17:58.580152+0000","last_fullsized":"2024-03-21T13:17:58.580152+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:10.968402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:15.968926+0000","last_clean_scrub_stamp":"2024-03-21T13:16:10.968402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"424'7614","reported_seq":7690,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:18:01.299164+0000","last_change":"2024-03-21T13:16:13.900592+0000","last_active":"2024-03-21T13:18:01.299164+0000","last_peered":"2024-03-21T13:18:01.299164+0000","last_clean":"2024-03-21T13:18:01.299164+0000","last_became_active":"2024-03-21T13:16:13.900119+0000","last_became_peered":"2024-03-21T13:16:13.900119+0000","last_unstale":"2024-03-21T13:18:01.299164+0000","last_undegraded":"2024-03-21T13:18:01.299164+0000","last_fullsized":"2024-03-21T13:18:01.299164+0000","mapping_epoch":390,"log_start":"407'3695","ondisk_log_start":"407'3695","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481165312,"num_objects":7342,"num_object_clones":0,"num_object_copies":22026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7614,"num_write_kb":478592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":772,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583889+0000","last_change":"2024-03-21T13:16:18.683140+0000","last_active":"2024-03-21T13:17:58.583889+0000","last_peered":"2024-03-21T13:17:58.583889+0000","last_clean":"2024-03-21T13:17:58.583889+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:17:58.583889+0000","last_undegraded":"2024-03-21T13:17:58.583889+0000","last_fullsized":"2024-03-21T13:17:58.583889+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.683041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":349,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583894+0000","last_change":"2024-03-21T13:17:27.634210+0000","last_active":"2024-03-21T13:17:58.583894+0000","last_peered":"2024-03-21T13:17:58.583894+0000","last_clean":"2024-03-21T13:17:58.583894+0000","last_became_active":"2024-03-21T13:17:27.633871+0000","last_became_peered":"2024-03-21T13:17:27.633871+0000","last_unstale":"2024-03-21T13:17:58.583894+0000","last_undegraded":"2024-03-21T13:17:58.583894+0000","last_fullsized":"2024-03-21T13:17:58.583894+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:12.859249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.231115+0000","last_clean_scrub_stamp":"2024-03-21T13:17:12.859249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:33.420595+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":348,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584676+0000","last_change":"2024-03-21T13:17:27.640952+0000","last_active":"2024-03-21T13:17:58.584676+0000","last_peered":"2024-03-21T13:17:58.584676+0000","last_clean":"2024-03-21T13:17:58.584676+0000","last_became_active":"2024-03-21T13:17:27.640621+0000","last_became_peered":"2024-03-21T13:17:27.640621+0000","last_unstale":"2024-03-21T13:17:58.584676+0000","last_undegraded":"2024-03-21T13:17:58.584676+0000","last_fullsized":"2024-03-21T13:17:58.584676+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.465427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.465427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:05.271230+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"243'20","reported_seq":244,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585448+0000","last_change":"2024-03-21T13:17:27.722357+0000","last_active":"2024-03-21T13:17:58.585448+0000","last_peered":"2024-03-21T13:17:58.585448+0000","last_clean":"2024-03-21T13:17:58.585448+0000","last_became_active":"2024-03-21T13:17:27.660008+0000","last_became_peered":"2024-03-21T13:17:27.660008+0000","last_unstale":"2024-03-21T13:17:58.585448+0000","last_undegraded":"2024-03-21T13:17:58.585448+0000","last_fullsized":"2024-03-21T13:17:58.585448+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'20","last_scrub_stamp":"2024-03-21T13:17:18.348476+0000","last_deep_scrub":"243'20","last_deep_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_clean_scrub_stamp":"2024-03-21T13:17:18.348476+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:35.254471+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":340,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585154+0000","last_change":"2024-03-21T13:16:15.247743+0000","last_active":"2024-03-21T13:17:58.585154+0000","last_peered":"2024-03-21T13:17:58.585154+0000","last_clean":"2024-03-21T13:17:58.585154+0000","last_became_active":"2024-03-21T13:12:29.506751+0000","last_became_peered":"2024-03-21T13:12:29.506751+0000","last_unstale":"2024-03-21T13:17:58.585154+0000","last_undegraded":"2024-03-21T13:17:58.585154+0000","last_fullsized":"2024-03-21T13:17:58.585154+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:15.247661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:15.247661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"424'7799","reported_seq":7876,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:18:01.298660+0000","last_change":"2024-03-21T13:16:13.900783+0000","last_active":"2024-03-21T13:18:01.298660+0000","last_peered":"2024-03-21T13:18:01.298660+0000","last_clean":"2024-03-21T13:18:01.298660+0000","last_became_active":"2024-03-21T13:16:13.900477+0000","last_became_peered":"2024-03-21T13:16:13.900477+0000","last_unstale":"2024-03-21T13:18:01.298660+0000","last_undegraded":"2024-03-21T13:18:01.298660+0000","last_fullsized":"2024-03-21T13:18:01.298660+0000","mapping_epoch":390,"log_start":"408'3922","ondisk_log_start":"408'3922","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3877,"log_dups_size":0,"ondisk_log_size":3877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495255552,"num_objects":7557,"num_object_clones":0,"num_object_copies":22671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7799,"num_write_kb":491392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":206,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585267+0000","last_change":"2024-03-21T13:15:04.365662+0000","last_active":"2024-03-21T13:17:58.585267+0000","last_peered":"2024-03-21T13:17:58.585267+0000","last_clean":"2024-03-21T13:17:58.585267+0000","last_became_active":"2024-03-21T13:12:29.527083+0000","last_became_peered":"2024-03-21T13:12:29.527083+0000","last_unstale":"2024-03-21T13:17:58.585267+0000","last_undegraded":"2024-03-21T13:17:58.585267+0000","last_fullsized":"2024-03-21T13:17:58.585267+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:04.365534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:15:04.365534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":1049,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583187+0000","last_change":"2024-03-21T13:17:30.675051+0000","last_active":"2024-03-21T13:17:58.583187+0000","last_peered":"2024-03-21T13:17:58.583187+0000","last_clean":"2024-03-21T13:17:58.583187+0000","last_became_active":"2024-03-21T13:17:30.671981+0000","last_became_peered":"2024-03-21T13:17:30.671981+0000","last_unstale":"2024-03-21T13:17:58.583187+0000","last_undegraded":"2024-03-21T13:17:58.583187+0000","last_fullsized":"2024-03-21T13:17:58.583187+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:16:14.628384+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:16:14.628384+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":280,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584081+0000","last_change":"2024-03-21T13:16:02.247782+0000","last_active":"2024-03-21T13:17:58.584081+0000","last_peered":"2024-03-21T13:17:58.584081+0000","last_clean":"2024-03-21T13:17:58.584081+0000","last_became_active":"2024-03-21T13:12:29.499989+0000","last_became_peered":"2024-03-21T13:12:29.499989+0000","last_unstale":"2024-03-21T13:17:58.584081+0000","last_undegraded":"2024-03-21T13:17:58.584081+0000","last_fullsized":"2024-03-21T13:17:58.584081+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:02.247693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:02.247693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"234'36","reported_seq":1056,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580421+0000","last_change":"2024-03-21T13:16:05.528913+0000","last_active":"2024-03-21T13:17:58.580421+0000","last_peered":"2024-03-21T13:17:58.580421+0000","last_clean":"2024-03-21T13:17:58.580421+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:17:58.580421+0000","last_undegraded":"2024-03-21T13:17:58.580421+0000","last_fullsized":"2024-03-21T13:17:58.580421+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'36","last_scrub_stamp":"2024-03-21T13:16:05.528782+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.528782+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":800,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583991+0000","last_change":"2024-03-21T13:17:02.759517+0000","last_active":"2024-03-21T13:17:58.583991+0000","last_peered":"2024-03-21T13:17:58.583991+0000","last_clean":"2024-03-21T13:17:58.583991+0000","last_became_active":"2024-03-21T13:12:29.507753+0000","last_became_peered":"2024-03-21T13:12:29.507753+0000","last_unstale":"2024-03-21T13:17:58.583991+0000","last_undegraded":"2024-03-21T13:17:58.583991+0000","last_fullsized":"2024-03-21T13:17:58.583991+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:02.759379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:17:02.759379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:12.075990+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"424'7879","reported_seq":15635,"reported_epoch":424,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:18:01.310468+0000","last_change":"2024-03-21T13:17:29.018908+0000","last_active":"2024-03-21T13:18:01.310468+0000","last_peered":"2024-03-21T13:18:01.310468+0000","last_clean":"2024-03-21T13:17:26.566589+0000","last_became_active":"2024-03-21T13:17:28.767399+0000","last_became_peered":"2024-03-21T13:17:28.767399+0000","last_unstale":"2024-03-21T13:18:01.310468+0000","last_undegraded":"2024-03-21T13:18:01.310468+0000","last_fullsized":"2024-03-21T13:18:01.310468+0000","mapping_epoch":416,"log_start":"408'3948","ondisk_log_start":"408'3948","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3931,"log_dups_size":0,"ondisk_log_size":3931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499318784,"num_objects":7619,"num_object_clones":0,"num_object_copies":22857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3292,"num_objects_unfound":0,"num_objects_dirty":7619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7879,"num_write_kb":495936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3836,"num_bytes_recovered":251396096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,0],"avail_no_missing":["3","0","6"],"object_location_counts":[{"shards":"0,3,6","objects":7619}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":206,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.582019+0000","last_change":"2024-03-21T13:16:01.195705+0000","last_active":"2024-03-21T13:17:58.582019+0000","last_peered":"2024-03-21T13:17:58.582019+0000","last_clean":"2024-03-21T13:17:58.582019+0000","last_became_active":"2024-03-21T13:12:29.514542+0000","last_became_peered":"2024-03-21T13:12:29.514542+0000","last_unstale":"2024-03-21T13:17:58.582019+0000","last_undegraded":"2024-03-21T13:17:58.582019+0000","last_fullsized":"2024-03-21T13:17:58.582019+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:01.195618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:01.195618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":615,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.588422+0000","last_change":"2024-03-21T13:17:27.639488+0000","last_active":"2024-03-21T13:17:58.588422+0000","last_peered":"2024-03-21T13:17:58.588422+0000","last_clean":"2024-03-21T13:17:58.588422+0000","last_became_active":"2024-03-21T13:17:27.639147+0000","last_became_peered":"2024-03-21T13:17:27.639147+0000","last_unstale":"2024-03-21T13:17:58.588422+0000","last_undegraded":"2024-03-21T13:17:58.588422+0000","last_fullsized":"2024-03-21T13:17:58.588422+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:00.086941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:00.086941+0000","last_clean_scrub_stamp":"2024-03-21T13:17:00.086941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:08.929613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":1964,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:18:03.779780+0000","last_change":"2024-03-21T13:17:27.705431+0000","last_active":"2024-03-21T13:18:03.779780+0000","last_peered":"2024-03-21T13:18:03.779780+0000","last_clean":"2024-03-21T13:18:03.779780+0000","last_became_active":"2024-03-21T13:17:27.656453+0000","last_became_peered":"2024-03-21T13:17:27.656453+0000","last_unstale":"2024-03-21T13:18:03.779780+0000","last_undegraded":"2024-03-21T13:18:03.779780+0000","last_fullsized":"2024-03-21T13:18:03.779780+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:16:09.547783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:16:09.547783+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1156,"num_read_kb":1151,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":589,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584862+0000","last_change":"2024-03-21T13:17:27.674914+0000","last_active":"2024-03-21T13:17:58.584862+0000","last_peered":"2024-03-21T13:17:58.584862+0000","last_clean":"2024-03-21T13:17:58.584862+0000","last_became_active":"2024-03-21T13:17:27.645689+0000","last_became_peered":"2024-03-21T13:17:27.645689+0000","last_unstale":"2024-03-21T13:17:58.584862+0000","last_undegraded":"2024-03-21T13:17:58.584862+0000","last_fullsized":"2024-03-21T13:17:58.584862+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-21T13:16:18.882994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.882994+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":220,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585178+0000","last_change":"2024-03-21T13:17:27.662358+0000","last_active":"2024-03-21T13:17:58.585178+0000","last_peered":"2024-03-21T13:17:58.585178+0000","last_clean":"2024-03-21T13:17:58.585178+0000","last_became_active":"2024-03-21T13:17:27.658711+0000","last_became_peered":"2024-03-21T13:17:27.658711+0000","last_unstale":"2024-03-21T13:17:58.585178+0000","last_undegraded":"2024-03-21T13:17:58.585178+0000","last_fullsized":"2024-03-21T13:17:58.585178+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:08.237986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:08.237986+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.237986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:26.719524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"424'8021","reported_seq":8099,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:18:05.670220+0000","last_change":"2024-03-21T13:16:13.899271+0000","last_active":"2024-03-21T13:18:05.670220+0000","last_peered":"2024-03-21T13:18:05.670220+0000","last_clean":"2024-03-21T13:18:05.670220+0000","last_became_active":"2024-03-21T13:16:13.898890+0000","last_became_peered":"2024-03-21T13:16:13.898890+0000","last_unstale":"2024-03-21T13:18:05.670220+0000","last_undegraded":"2024-03-21T13:18:05.670220+0000","last_fullsized":"2024-03-21T13:18:05.670220+0000","mapping_epoch":390,"log_start":"406'3521","ondisk_log_start":"406'3521","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471007232,"num_objects":7187,"num_object_clones":0,"num_object_copies":21561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8021,"num_write_kb":486656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":674,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.579739+0000","last_change":"2024-03-21T13:16:15.593032+0000","last_active":"2024-03-21T13:17:58.579739+0000","last_peered":"2024-03-21T13:17:58.579739+0000","last_clean":"2024-03-21T13:17:58.579739+0000","last_became_active":"2024-03-21T13:10:03.591088+0000","last_became_peered":"2024-03-21T13:10:03.591088+0000","last_unstale":"2024-03-21T13:17:58.579739+0000","last_undegraded":"2024-03-21T13:17:58.579739+0000","last_fullsized":"2024-03-21T13:17:58.579739+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:15.592931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:15.592931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":354,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583882+0000","last_change":"2024-03-21T13:17:27.639455+0000","last_active":"2024-03-21T13:17:58.583882+0000","last_peered":"2024-03-21T13:17:58.583882+0000","last_clean":"2024-03-21T13:17:58.583882+0000","last_became_active":"2024-03-21T13:17:27.639129+0000","last_became_peered":"2024-03-21T13:17:27.639129+0000","last_unstale":"2024-03-21T13:17:58.583882+0000","last_undegraded":"2024-03-21T13:17:58.583882+0000","last_fullsized":"2024-03-21T13:17:58.583882+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:22.293625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.293625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":256,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.582040+0000","last_change":"2024-03-21T13:17:27.724123+0000","last_active":"2024-03-21T13:17:58.582040+0000","last_peered":"2024-03-21T13:17:58.582040+0000","last_clean":"2024-03-21T13:17:58.582040+0000","last_became_active":"2024-03-21T13:17:27.658954+0000","last_became_peered":"2024-03-21T13:17:27.658954+0000","last_unstale":"2024-03-21T13:17:58.582040+0000","last_undegraded":"2024-03-21T13:17:58.582040+0000","last_fullsized":"2024-03-21T13:17:58.582040+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-21T13:17:03.309259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:03.309259+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:22.375380+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":808,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.579479+0000","last_change":"2024-03-21T13:17:27.653252+0000","last_active":"2024-03-21T13:17:58.579479+0000","last_peered":"2024-03-21T13:17:58.579479+0000","last_clean":"2024-03-21T13:17:58.579479+0000","last_became_active":"2024-03-21T13:17:27.652408+0000","last_became_peered":"2024-03-21T13:17:27.652408+0000","last_unstale":"2024-03-21T13:17:58.579479+0000","last_undegraded":"2024-03-21T13:17:58.579479+0000","last_fullsized":"2024-03-21T13:17:58.579479+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:02.830721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:02.830721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:29.443885+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"424'8010","reported_seq":8089,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:18:05.660304+0000","last_change":"2024-03-21T13:16:13.899935+0000","last_active":"2024-03-21T13:18:05.660304+0000","last_peered":"2024-03-21T13:18:05.660304+0000","last_clean":"2024-03-21T13:18:05.660304+0000","last_became_active":"2024-03-21T13:16:13.899614+0000","last_became_peered":"2024-03-21T13:16:13.899614+0000","last_unstale":"2024-03-21T13:18:05.660304+0000","last_undegraded":"2024-03-21T13:18:05.660304+0000","last_fullsized":"2024-03-21T13:18:05.660304+0000","mapping_epoch":390,"log_start":"406'3524","ondisk_log_start":"406'3524","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4486,"log_dups_size":0,"ondisk_log_size":4486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477560856,"num_objects":7288,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7288,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8010,"num_write_kb":489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":595,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.576287+0000","last_change":"2024-03-21T13:17:27.646840+0000","last_active":"2024-03-21T13:17:58.576287+0000","last_peered":"2024-03-21T13:17:58.576287+0000","last_clean":"2024-03-21T13:17:58.576287+0000","last_became_active":"2024-03-21T13:17:27.646513+0000","last_became_peered":"2024-03-21T13:17:27.646513+0000","last_unstale":"2024-03-21T13:17:58.576287+0000","last_undegraded":"2024-03-21T13:17:58.576287+0000","last_fullsized":"2024-03-21T13:17:58.576287+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:51.069883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:51.069883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:08.036043+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":206,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583498+0000","last_change":"2024-03-21T13:16:17.169677+0000","last_active":"2024-03-21T13:17:58.583498+0000","last_peered":"2024-03-21T13:17:58.583498+0000","last_clean":"2024-03-21T13:17:58.583498+0000","last_became_active":"2024-03-21T13:12:29.512291+0000","last_became_peered":"2024-03-21T13:12:29.512291+0000","last_unstale":"2024-03-21T13:17:58.583498+0000","last_undegraded":"2024-03-21T13:17:58.583498+0000","last_fullsized":"2024-03-21T13:17:58.583498+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:17.169572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:28.030369+0000","last_clean_scrub_stamp":"2024-03-21T13:16:17.169572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"234'25","reported_seq":236,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.579811+0000","last_change":"2024-03-21T13:17:27.684131+0000","last_active":"2024-03-21T13:17:58.579811+0000","last_peered":"2024-03-21T13:17:58.579811+0000","last_clean":"2024-03-21T13:17:58.579811+0000","last_became_active":"2024-03-21T13:17:27.646744+0000","last_became_peered":"2024-03-21T13:17:27.646744+0000","last_unstale":"2024-03-21T13:17:58.579811+0000","last_undegraded":"2024-03-21T13:17:58.579811+0000","last_fullsized":"2024-03-21T13:17:58.579811+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-21T13:15:57.053797+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:15:57.053797+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"416'15","reported_seq":811,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.579472+0000","last_change":"2024-03-21T13:17:27.685140+0000","last_active":"2024-03-21T13:17:58.579472+0000","last_peered":"2024-03-21T13:17:58.579472+0000","last_clean":"2024-03-21T13:17:58.579472+0000","last_became_active":"2024-03-21T13:17:27.653797+0000","last_became_peered":"2024-03-21T13:17:27.653797+0000","last_unstale":"2024-03-21T13:17:58.579472+0000","last_undegraded":"2024-03-21T13:17:58.579472+0000","last_fullsized":"2024-03-21T13:17:58.579472+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'14","last_scrub_stamp":"2024-03-21T13:16:07.043880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:07.043880+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.e","version":"424'8073","reported_seq":17252,"reported_epoch":424,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:18:05.322819+0000","last_change":"2024-03-21T13:17:28.759540+0000","last_active":"2024-03-21T13:18:05.322819+0000","last_peered":"2024-03-21T13:18:05.322819+0000","last_clean":"2024-03-21T13:17:26.618296+0000","last_became_active":"2024-03-21T13:17:28.741875+0000","last_became_peered":"2024-03-21T13:17:28.741875+0000","last_unstale":"2024-03-21T13:18:05.322819+0000","last_undegraded":"2024-03-21T13:18:05.322819+0000","last_fullsized":"2024-03-21T13:18:05.322819+0000","mapping_epoch":416,"log_start":"408'4060","ondisk_log_start":"408'4060","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481361920,"num_objects":7345,"num_object_clones":0,"num_object_copies":22035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2418,"num_objects_unfound":0,"num_objects_dirty":7345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8073,"num_write_kb":493376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":298057728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[{"shards":"0,4,7","objects":7345}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":784,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.586649+0000","last_change":"2024-03-21T13:17:09.938651+0000","last_active":"2024-03-21T13:17:58.586649+0000","last_peered":"2024-03-21T13:17:58.586649+0000","last_clean":"2024-03-21T13:17:58.586649+0000","last_became_active":"2024-03-21T13:10:03.606557+0000","last_became_peered":"2024-03-21T13:10:03.606557+0000","last_unstale":"2024-03-21T13:17:58.586649+0000","last_undegraded":"2024-03-21T13:17:58.586649+0000","last_fullsized":"2024-03-21T13:17:58.586649+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:09.938560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:09.938560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:28.686201+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":40,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.591047+0000","last_change":"2024-03-21T13:17:27.661843+0000","last_active":"2024-03-21T13:17:58.591047+0000","last_peered":"2024-03-21T13:17:58.591047+0000","last_clean":"2024-03-21T13:17:58.591047+0000","last_became_active":"2024-03-21T13:17:27.638397+0000","last_became_peered":"2024-03-21T13:17:27.638397+0000","last_unstale":"2024-03-21T13:17:58.591047+0000","last_undegraded":"2024-03-21T13:17:58.591047+0000","last_fullsized":"2024-03-21T13:17:58.591047+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:16:57.064293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:57.064293+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:00.642501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"234'35","reported_seq":230,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.578174+0000","last_change":"2024-03-21T13:17:01.334745+0000","last_active":"2024-03-21T13:17:58.578174+0000","last_peered":"2024-03-21T13:17:58.578174+0000","last_clean":"2024-03-21T13:17:58.578174+0000","last_became_active":"2024-03-21T13:12:29.525154+0000","last_became_peered":"2024-03-21T13:12:29.525154+0000","last_unstale":"2024-03-21T13:17:58.578174+0000","last_undegraded":"2024-03-21T13:17:58.578174+0000","last_fullsized":"2024-03-21T13:17:58.578174+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:17:01.334611+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:17:01.334611+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:06.225704+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"416'11","reported_seq":1026,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583575+0000","last_change":"2024-03-21T13:17:27.644678+0000","last_active":"2024-03-21T13:17:58.583575+0000","last_peered":"2024-03-21T13:17:58.583575+0000","last_clean":"2024-03-21T13:17:58.583575+0000","last_became_active":"2024-03-21T13:17:27.639409+0000","last_became_peered":"2024-03-21T13:17:27.639409+0000","last_unstale":"2024-03-21T13:17:58.583575+0000","last_undegraded":"2024-03-21T13:17:58.583575+0000","last_fullsized":"2024-03-21T13:17:58.583575+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T13:16:18.672392+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.672392+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"424'8006","reported_seq":18146,"reported_epoch":424,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:18:05.495332+0000","last_change":"2024-03-21T13:17:28.735905+0000","last_active":"2024-03-21T13:18:05.495332+0000","last_peered":"2024-03-21T13:18:05.495332+0000","last_clean":"2024-03-21T13:17:26.592561+0000","last_became_active":"2024-03-21T13:17:28.707228+0000","last_became_peered":"2024-03-21T13:17:28.707228+0000","last_unstale":"2024-03-21T13:18:05.495332+0000","last_undegraded":"2024-03-21T13:18:05.495332+0000","last_fullsized":"2024-03-21T13:18:05.495332+0000","mapping_epoch":416,"log_start":"409'4219","ondisk_log_start":"409'4219","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3787,"log_dups_size":0,"ondisk_log_size":3787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475529216,"num_objects":7256,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1735,"num_objects_unfound":0,"num_objects_dirty":7256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":488384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5030,"num_bytes_recovered":329646080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,0],"avail_no_missing":["7","0","2"],"object_location_counts":[{"shards":"0,2,7","objects":7256}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":695,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.582581+0000","last_change":"2024-03-21T13:15:10.158338+0000","last_active":"2024-03-21T13:17:58.582581+0000","last_peered":"2024-03-21T13:17:58.582581+0000","last_clean":"2024-03-21T13:17:58.582581+0000","last_became_active":"2024-03-21T13:10:03.610471+0000","last_became_peered":"2024-03-21T13:10:03.610471+0000","last_unstale":"2024-03-21T13:17:58.582581+0000","last_undegraded":"2024-03-21T13:17:58.582581+0000","last_fullsized":"2024-03-21T13:17:58.582581+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:10.157920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:15:10.157920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":784,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.582556+0000","last_change":"2024-03-21T13:17:27.659245+0000","last_active":"2024-03-21T13:17:58.582556+0000","last_peered":"2024-03-21T13:17:58.582556+0000","last_clean":"2024-03-21T13:17:58.582556+0000","last_became_active":"2024-03-21T13:17:27.658806+0000","last_became_peered":"2024-03-21T13:17:27.658806+0000","last_unstale":"2024-03-21T13:17:58.582556+0000","last_undegraded":"2024-03-21T13:17:58.582556+0000","last_fullsized":"2024-03-21T13:17:58.582556+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:05.843337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:17:05.843337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:35.047634+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"234'30","reported_seq":849,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583390+0000","last_change":"2024-03-21T13:17:27.711943+0000","last_active":"2024-03-21T13:17:58.583390+0000","last_peered":"2024-03-21T13:17:58.583390+0000","last_clean":"2024-03-21T13:17:58.583390+0000","last_became_active":"2024-03-21T13:17:27.661667+0000","last_became_peered":"2024-03-21T13:17:27.661667+0000","last_unstale":"2024-03-21T13:17:58.583390+0000","last_undegraded":"2024-03-21T13:17:58.583390+0000","last_fullsized":"2024-03-21T13:17:58.583390+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'30","last_scrub_stamp":"2024-03-21T13:17:18.984086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:18.984086+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:26.391749+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":787,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.588185+0000","last_change":"2024-03-21T13:16:53.866824+0000","last_active":"2024-03-21T13:17:58.588185+0000","last_peered":"2024-03-21T13:17:58.588185+0000","last_clean":"2024-03-21T13:17:58.588185+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:17:58.588185+0000","last_undegraded":"2024-03-21T13:17:58.588185+0000","last_fullsized":"2024-03-21T13:17:58.588185+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:53.866692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.866692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:12.696345+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"424'7872","reported_seq":7950,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:18:01.176961+0000","last_change":"2024-03-21T13:16:13.900272+0000","last_active":"2024-03-21T13:18:01.176961+0000","last_peered":"2024-03-21T13:18:01.176961+0000","last_clean":"2024-03-21T13:18:01.176961+0000","last_became_active":"2024-03-21T13:16:13.899979+0000","last_became_peered":"2024-03-21T13:16:13.899979+0000","last_unstale":"2024-03-21T13:18:01.176961+0000","last_undegraded":"2024-03-21T13:18:01.176961+0000","last_fullsized":"2024-03-21T13:18:01.176961+0000","mapping_epoch":390,"log_start":"408'3992","ondisk_log_start":"408'3992","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":3880,"log_dups_size":0,"ondisk_log_size":3880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500826112,"num_objects":7642,"num_object_clones":0,"num_object_copies":22926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7872,"num_write_kb":496448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":212,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583613+0000","last_change":"2024-03-21T13:17:27.654000+0000","last_active":"2024-03-21T13:17:58.583613+0000","last_peered":"2024-03-21T13:17:58.583613+0000","last_clean":"2024-03-21T13:17:58.583613+0000","last_became_active":"2024-03-21T13:17:27.653461+0000","last_became_peered":"2024-03-21T13:17:27.653461+0000","last_unstale":"2024-03-21T13:17:58.583613+0000","last_undegraded":"2024-03-21T13:17:58.583613+0000","last_fullsized":"2024-03-21T13:17:58.583613+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:14.280515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:14.280515+0000","last_clean_scrub_stamp":"2024-03-21T13:16:14.280515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"416'14","reported_seq":288,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584330+0000","last_change":"2024-03-21T13:17:27.672436+0000","last_active":"2024-03-21T13:17:58.584330+0000","last_peered":"2024-03-21T13:17:58.584330+0000","last_clean":"2024-03-21T13:17:58.584330+0000","last_became_active":"2024-03-21T13:17:27.650953+0000","last_became_peered":"2024-03-21T13:17:27.650953+0000","last_unstale":"2024-03-21T13:17:58.584330+0000","last_undegraded":"2024-03-21T13:17:58.584330+0000","last_fullsized":"2024-03-21T13:17:58.584330+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:15:15.202826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:15.202826+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":784,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.583298+0000","last_change":"2024-03-21T13:17:00.746992+0000","last_active":"2024-03-21T13:17:58.583298+0000","last_peered":"2024-03-21T13:17:58.583298+0000","last_clean":"2024-03-21T13:17:58.583298+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:17:58.583298+0000","last_undegraded":"2024-03-21T13:17:58.583298+0000","last_fullsized":"2024-03-21T13:17:58.583298+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:00.746886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:00.746886+0000","last_clean_scrub_stamp":"2024-03-21T13:17:00.746886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:22.824064+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"234'34","reported_seq":743,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.582836+0000","last_change":"2024-03-21T13:16:53.621472+0000","last_active":"2024-03-21T13:17:58.582836+0000","last_peered":"2024-03-21T13:17:58.582836+0000","last_clean":"2024-03-21T13:17:58.582836+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:17:58.582836+0000","last_undegraded":"2024-03-21T13:17:58.582836+0000","last_fullsized":"2024-03-21T13:17:58.582836+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'34","last_scrub_stamp":"2024-03-21T13:16:53.621335+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.621335+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:09.903027+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"424'7914","reported_seq":9434,"reported_epoch":424,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:18:05.589352+0000","last_change":"2024-03-21T13:17:58.727898+0000","last_active":"2024-03-21T13:18:05.589352+0000","last_peered":"2024-03-21T13:18:05.589352+0000","last_clean":"2024-03-21T13:17:26.622957+0000","last_became_active":"2024-03-21T13:17:28.692584+0000","last_became_peered":"2024-03-21T13:17:28.692584+0000","last_unstale":"2024-03-21T13:18:05.589352+0000","last_undegraded":"2024-03-21T13:18:05.589352+0000","last_fullsized":"2024-03-21T13:18:05.589352+0000","mapping_epoch":416,"log_start":"407'3564","ondisk_log_start":"407'3564","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465567744,"num_objects":7104,"num_object_clones":0,"num_object_copies":21312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6406,"num_objects_unfound":0,"num_objects_dirty":7104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7914,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":717,"num_bytes_recovered":46989312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,0],"avail_no_missing":["6","0","3"],"object_location_counts":[{"shards":"0,3,6","objects":7104}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"375'2","reported_seq":780,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.579987+0000","last_change":"2024-03-21T13:15:14.313700+0000","last_active":"2024-03-21T13:17:58.579987+0000","last_peered":"2024-03-21T13:17:58.579987+0000","last_clean":"2024-03-21T13:17:58.579987+0000","last_became_active":"2024-03-21T13:12:29.502816+0000","last_became_peered":"2024-03-21T13:12:29.502816+0000","last_unstale":"2024-03-21T13:17:58.579987+0000","last_undegraded":"2024-03-21T13:17:58.579987+0000","last_fullsized":"2024-03-21T13:17:58.579987+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'2","last_scrub_stamp":"2024-03-21T13:15:14.313376+0000","last_deep_scrub":"375'2","last_deep_scrub_stamp":"2024-03-21T13:15:14.313376+0000","last_clean_scrub_stamp":"2024-03-21T13:15:14.313376+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":206,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585093+0000","last_change":"2024-03-21T13:15:07.382672+0000","last_active":"2024-03-21T13:17:58.585093+0000","last_peered":"2024-03-21T13:17:58.585093+0000","last_clean":"2024-03-21T13:17:58.585093+0000","last_became_active":"2024-03-21T13:12:29.524087+0000","last_became_peered":"2024-03-21T13:12:29.524087+0000","last_unstale":"2024-03-21T13:17:58.585093+0000","last_undegraded":"2024-03-21T13:17:58.585093+0000","last_fullsized":"2024-03-21T13:17:58.585093+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:07.382536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:15:07.382536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":814,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.588254+0000","last_change":"2024-03-21T13:16:04.825360+0000","last_active":"2024-03-21T13:17:58.588254+0000","last_peered":"2024-03-21T13:17:58.588254+0000","last_clean":"2024-03-21T13:17:58.588254+0000","last_became_active":"2024-03-21T13:10:03.586593+0000","last_became_peered":"2024-03-21T13:10:03.586593+0000","last_unstale":"2024-03-21T13:17:58.588254+0000","last_undegraded":"2024-03-21T13:17:58.588254+0000","last_fullsized":"2024-03-21T13:17:58.588254+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:04.825280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:04.825280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":538,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.579958+0000","last_change":"2024-03-21T13:17:48.700635+0000","last_active":"2024-03-21T13:17:58.579958+0000","last_peered":"2024-03-21T13:17:58.579958+0000","last_clean":"2024-03-21T13:17:58.579958+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:17:58.579958+0000","last_undegraded":"2024-03-21T13:17:58.579958+0000","last_fullsized":"2024-03-21T13:17:58.579958+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:17:48.700488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:48.700488+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:56.678680+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"424'7962","reported_seq":17132,"reported_epoch":424,"state":"active+remapped+backfilling","last_fresh":"2024-03-21T13:18:05.688565+0000","last_change":"2024-03-21T13:17:28.817298+0000","last_active":"2024-03-21T13:18:05.688565+0000","last_peered":"2024-03-21T13:18:05.688565+0000","last_clean":"2024-03-21T13:17:26.609592+0000","last_became_active":"2024-03-21T13:17:28.794973+0000","last_became_peered":"2024-03-21T13:17:28.794973+0000","last_unstale":"2024-03-21T13:18:05.688565+0000","last_undegraded":"2024-03-21T13:18:05.688565+0000","last_fullsized":"2024-03-21T13:18:05.688565+0000","mapping_epoch":416,"log_start":"406'3395","ondisk_log_start":"406'3395","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466616320,"num_objects":7120,"num_object_clones":0,"num_object_copies":21360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2244,"num_objects_unfound":0,"num_objects_dirty":7120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7962,"num_write_kb":482624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":297795584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,0],"avail_no_missing":["2","0","3"],"object_location_counts":[{"shards":"0,2,3","objects":7120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":350,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.588241+0000","last_change":"2024-03-21T13:16:20.483084+0000","last_active":"2024-03-21T13:17:58.588241+0000","last_peered":"2024-03-21T13:17:58.588241+0000","last_clean":"2024-03-21T13:17:58.588241+0000","last_became_active":"2024-03-21T13:10:03.602752+0000","last_became_peered":"2024-03-21T13:10:03.602752+0000","last_unstale":"2024-03-21T13:17:58.588241+0000","last_undegraded":"2024-03-21T13:17:58.588241+0000","last_fullsized":"2024-03-21T13:17:58.588241+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:20.482943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.482943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":701,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.588987+0000","last_change":"2024-03-21T13:17:27.640568+0000","last_active":"2024-03-21T13:17:58.588987+0000","last_peered":"2024-03-21T13:17:58.588987+0000","last_clean":"2024-03-21T13:17:58.588987+0000","last_became_active":"2024-03-21T13:17:27.640173+0000","last_became_peered":"2024-03-21T13:17:27.640173+0000","last_unstale":"2024-03-21T13:17:58.588987+0000","last_undegraded":"2024-03-21T13:17:58.588987+0000","last_fullsized":"2024-03-21T13:17:58.588987+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:11.936426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:11.936426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:39.664715+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":799,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.720855+0000","last_change":"2024-03-21T13:17:58.720904+0000","last_active":"2024-03-21T13:17:58.720855+0000","last_peered":"2024-03-21T13:17:58.720855+0000","last_clean":"2024-03-21T13:17:58.720855+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:17:58.720855+0000","last_undegraded":"2024-03-21T13:17:58.720855+0000","last_fullsized":"2024-03-21T13:17:58.720855+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:58.720766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:58.720766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:19:27.150053+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"391'39","reported_seq":360,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.593066+0000","last_change":"2024-03-21T13:16:14.266738+0000","last_active":"2024-03-21T13:17:58.593066+0000","last_peered":"2024-03-21T13:17:58.593066+0000","last_clean":"2024-03-21T13:17:58.593066+0000","last_became_active":"2024-03-21T13:10:03.591832+0000","last_became_peered":"2024-03-21T13:10:03.591832+0000","last_unstale":"2024-03-21T13:17:58.593066+0000","last_undegraded":"2024-03-21T13:17:58.593066+0000","last_fullsized":"2024-03-21T13:17:58.593066+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'39","last_scrub_stamp":"2024-03-21T13:16:14.266573+0000","last_deep_scrub":"391'39","last_deep_scrub_stamp":"2024-03-21T13:16:14.266573+0000","last_clean_scrub_stamp":"2024-03-21T13:16:14.266573+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"424'8086","reported_seq":8163,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:18:05.586704+0000","last_change":"2024-03-21T13:16:13.904234+0000","last_active":"2024-03-21T13:18:05.586704+0000","last_peered":"2024-03-21T13:18:05.586704+0000","last_clean":"2024-03-21T13:18:05.586704+0000","last_became_active":"2024-03-21T13:16:13.903407+0000","last_became_peered":"2024-03-21T13:16:13.903407+0000","last_unstale":"2024-03-21T13:18:05.586704+0000","last_undegraded":"2024-03-21T13:18:05.586704+0000","last_fullsized":"2024-03-21T13:18:05.586704+0000","mapping_epoch":390,"log_start":"407'3799","ondisk_log_start":"407'3799","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.879090+0000","last_clean_scrub_stamp":"2024-03-21T13:16:12.879090+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478281728,"num_objects":7298,"num_object_clones":0,"num_object_copies":21894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":492288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":348,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.585057+0000","last_change":"2024-03-21T13:16:18.322285+0000","last_active":"2024-03-21T13:17:58.585057+0000","last_peered":"2024-03-21T13:17:58.585057+0000","last_clean":"2024-03-21T13:17:58.585057+0000","last_became_active":"2024-03-21T13:12:29.499442+0000","last_became_peered":"2024-03-21T13:12:29.499442+0000","last_unstale":"2024-03-21T13:17:58.585057+0000","last_undegraded":"2024-03-21T13:17:58.585057+0000","last_fullsized":"2024-03-21T13:17:58.585057+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.322198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.322198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"331'13","reported_seq":454,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.584549+0000","last_change":"2024-03-21T13:16:21.250482+0000","last_active":"2024-03-21T13:17:58.584549+0000","last_peered":"2024-03-21T13:17:58.584549+0000","last_clean":"2024-03-21T13:17:58.584549+0000","last_became_active":"2024-03-21T13:12:29.507925+0000","last_became_peered":"2024-03-21T13:12:29.507925+0000","last_unstale":"2024-03-21T13:17:58.584549+0000","last_undegraded":"2024-03-21T13:17:58.584549+0000","last_fullsized":"2024-03-21T13:17:58.584549+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:16:21.250398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.250398+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":776,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580722+0000","last_change":"2024-03-21T13:17:27.660137+0000","last_active":"2024-03-21T13:17:58.580722+0000","last_peered":"2024-03-21T13:17:58.580722+0000","last_clean":"2024-03-21T13:17:58.580722+0000","last_became_active":"2024-03-21T13:17:27.659812+0000","last_became_peered":"2024-03-21T13:17:27.659812+0000","last_unstale":"2024-03-21T13:17:58.580722+0000","last_undegraded":"2024-03-21T13:17:58.580722+0000","last_fullsized":"2024-03-21T13:17:58.580722+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:05.813657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.813657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"234'20","reported_seq":740,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-03-21T13:17:58.580182+0000","last_change":"2024-03-21T13:17:16.981637+0000","last_active":"2024-03-21T13:17:58.580182+0000","last_peered":"2024-03-21T13:17:58.580182+0000","last_clean":"2024-03-21T13:17:58.580182+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:17:58.580182+0000","last_undegraded":"2024-03-21T13:17:58.580182+0000","last_fullsized":"2024-03-21T13:17:58.580182+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:17:16.981490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:16.981490+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:18:45.770824+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":7672037400,"num_objects":117067,"num_object_clones":0,"num_object_copies":351201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":23130,"num_objects_unfound":0,"num_objects_dirty":117067,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126897,"num_write_kb":7806785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31916,"num_bytes_recovered":2091646976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25181433856,"data_stored":25181421640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65753,"ondisk_log_size":65753,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1156,"num_read_kb":1151,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":4823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1102,"num_read_kb":859,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":393,"num_bytes_recovered":8416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":14874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":753,"ondisk_log_size":753,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168814,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5193724,"kb_used_data":4192052,"kb_used_omap":1,"kb_used_meta":1001598,"kb_avail":88551428,"statfs":{"total":95995035648,"available":90676662272,"internally_reserved":0,"allocated":4292661248,"data_stored":4292270685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1025636810},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463156,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5169216,"kb_used_data":4217592,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":88575936,"statfs":{"total":95995035648,"available":90701758464,"internally_reserved":0,"allocated":4318814208,"data_stored":4318419282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":974453190},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575116,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738642,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411592,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3692588,"kb_used_data":2826916,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":90052564,"statfs":{"total":95995035648,"available":92213825536,"internally_reserved":0,"allocated":2894761984,"data_stored":2894386655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,4,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182658,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4760264,"kb_used_data":3647424,"kb_used_omap":1,"kb_used_meta":1112830,"kb_avail":88984888,"statfs":{"total":95995035648,"available":91120525312,"internally_reserved":0,"allocated":3734962176,"data_stored":3734558364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1139538375},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019133,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4189416,"kb_used_data":3125344,"kb_used_omap":1,"kb_used_meta":1064062,"kb_avail":89555736,"statfs":{"total":95995035648,"available":91705073664,"internally_reserved":0,"allocated":3200352256,"data_stored":3199980628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1089599941},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724792,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3301432,"kb_used_data":2981424,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":90443720,"statfs":{"total":95995035648,"available":92614369280,"internally_reserved":0,"allocated":3052978176,"data_stored":3052590196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3764453376,"data_stored":3764453376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2882736128,"data_stored":2882732056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3727425536,"data_stored":3727425536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3195146240,"data_stored":3195142168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3044016128,"data_stored":3044016128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4286451712,"data_stored":4286447640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4281204736,"data_stored":4281204736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:18:07.833 INFO:tasks.thrashosds.thrasher:Setting 6.6 to [1, 5, 7, 3, 4, 0] 2024-03-21T13:18:07.833 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.6', '1', '5', '7', '3', '4', '0'] 2024-03-21T13:18:07.833 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pg-upmap-items 6.6 1 5 7 3 4 0 2024-03-21T13:18:07.857 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:07.919 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:07.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:07 smithi204 ceph-mon[33153]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:18:08.020 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:08.060 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:08.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:08.112 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:08.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:08.243 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:08.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:08.345 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:08.382 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:08.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:08.484 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:08.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:08.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:08.622 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:08.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:08.667 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:08.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:08.768 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:08.803 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:08.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:08.837 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:08 smithi163 ceph-mon[29621]: pgmap v913: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.0 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.3 KiB/s rd, 27 MiB/s wr, 1.21k op/s; 21854/347088 objects misplaced (6.296%); 64 MiB/s, 1.03k objects/s recovering 2024-03-21T13:18:08.838 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:08 smithi163 ceph-mon[40675]: pgmap v913: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.0 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.3 KiB/s rd, 27 MiB/s wr, 1.21k op/s; 21854/347088 objects misplaced (6.296%); 64 MiB/s, 1.03k objects/s recovering 2024-03-21T13:18:08.904 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:08.946 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:08.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:08 smithi204 ceph-mon[33153]: pgmap v913: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.0 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.3 KiB/s rd, 27 MiB/s wr, 1.21k op/s; 21854/347088 objects misplaced (6.296%); 64 MiB/s, 1.03k objects/s recovering 2024-03-21T13:18:09.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:09.093 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:09.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:09.195 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:09.231 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:09.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:09.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:09.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:09.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:09.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:09.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:09.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:09.512 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:09.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:09.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:09.614 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:09.655 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:09.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:09.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:09.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:09.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:09.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:09.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:09.943 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:09.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:09.966 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:10.044 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:10.080 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:10.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:10.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:10.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:10.207 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:10.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:10.308 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:10.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:10.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:10.452 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:10.495 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:10.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:10.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:10.597 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:10.634 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:10.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:10.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:10.683 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:18:10.703 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:10 smithi163 ceph-mon[40675]: pgmap v914: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.0 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 23 MiB/s wr, 1.01k op/s; 21854/347088 objects misplaced (6.296%); 54 MiB/s, 867 objects/s recovering 2024-03-21T13:18:10.703 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:10 smithi163 ceph-mon[29621]: pgmap v914: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.0 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 23 MiB/s wr, 1.01k op/s; 21854/347088 objects misplaced (6.296%); 54 MiB/s, 867 objects/s recovering 2024-03-21T13:18:10.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:10.771 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:10.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:10.873 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:10.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:10.916 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:10.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:10 smithi204 ceph-mon[33153]: pgmap v914: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 7.0 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 23 MiB/s wr, 1.01k op/s; 21854/347088 objects misplaced (6.296%); 54 MiB/s, 867 objects/s recovering 2024-03-21T13:18:11.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:11.055 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:11.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:11.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:11.157 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:11.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:11.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:11.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:11.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:11.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:11.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:11.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:11.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:11.480 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:11.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:11.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:11.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:11.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:11.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:11.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:11.763 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:11.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:11.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:11.864 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:11.901 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:11.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:12.002 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:12.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:12.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:12.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:12.147 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:12.183 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:12.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:12.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:12.284 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:12.343 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:12.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:12.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:12.445 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:12.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:12.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:12.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:12.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:12.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:12.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:12.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:12.724 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:12.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:12 smithi204 ceph-mon[33153]: pgmap v915: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 6.0 MiB/s wr, 1.02k op/s; 17724/336141 objects misplaced (5.273%); 56 MiB/s, 897 objects/s recovering 2024-03-21T13:18:12.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:18:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:18:12.766 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:12.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:12.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:12.838 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:12 smithi163 ceph-mon[29621]: pgmap v915: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 6.0 MiB/s wr, 1.02k op/s; 17724/336141 objects misplaced (5.273%); 56 MiB/s, 897 objects/s recovering 2024-03-21T13:18:12.838 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:12 smithi163 ceph-mon[40675]: pgmap v915: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.8 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 6.0 MiB/s wr, 1.02k op/s; 17724/336141 objects misplaced (5.273%); 56 MiB/s, 897 objects/s recovering 2024-03-21T13:18:12.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:12.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:12.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:12.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:13.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:13.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:13.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:13.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:13.063 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:18:13.103 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:18:13.140 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:18:13.144 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:18:13.144 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:18:13.144 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-03-21T13:18:13.144 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:18:13.158 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:13.189 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:18:13.197 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:13.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:13.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:13.231 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:18:13.267 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:18:13.299 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:13.307 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:18:13.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:13.338 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:13.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:13.345 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:18:13.347 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:18:13.347 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:18:13.348 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:18:13.348 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:18:13.348 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:18:13.348 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:18:13.348 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:18:13.348 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:18:13.439 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:13.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:13.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:13.576 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:13.610 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:13.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:13.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:13.635 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:13.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:13.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:13 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:13.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:13 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4249722150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:18:13.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:13 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:13.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:13 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4249722150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:18:13.754 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:13.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:13.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:13.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:13.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:13.895 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:13.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:13.996 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:14.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:13 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:14.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:13 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4249722150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:18:14.036 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:14.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:14.133 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-21T13:18:14.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:14.177 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:14.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:14.278 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:14.324 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:14.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:14.425 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:14.465 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:14.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:14.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:14.567 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:14.615 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:14.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:14.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:14.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:14.757 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:14.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:14 smithi163 ceph-mon[29621]: pgmap v916: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.7 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 16346/331116 objects misplaced (4.937%); 56 MiB/s, 901 objects/s recovering 2024-03-21T13:18:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:14 smithi163 ceph-mon[40675]: pgmap v916: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.7 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 16346/331116 objects misplaced (4.937%); 56 MiB/s, 901 objects/s recovering 2024-03-21T13:18:14.859 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:14.894 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:14.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:14.996 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:14 smithi204 ceph-mon[33153]: pgmap v916: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.7 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 16346/331116 objects misplaced (4.937%); 56 MiB/s, 901 objects/s recovering 2024-03-21T13:18:15.031 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:15.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:15.133 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:15.169 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:15.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:15.271 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:15.306 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:15.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:15.408 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:15.445 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:15.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:15.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:15.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:15.583 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:15.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:15.684 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:15.719 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:15.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:15.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:15.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:15.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:15.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:15.960 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:18:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:18:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:18:15.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:16.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:16.010 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:16.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:16.138 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:16.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:16.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:16.240 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:16.274 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:16.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:16.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:16.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:16.411 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:16.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:16.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:16.508 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:16.513 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:16.566 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:16.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:16.667 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:16.707 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:16.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:16.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:16 smithi163 ceph-mon[29621]: pgmap v917: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.7 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1k op/s; 16346/331116 objects misplaced (4.937%); 53 MiB/s, 848 objects/s recovering 2024-03-21T13:18:16.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:16 smithi163 ceph-mon[40675]: pgmap v917: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.7 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1k op/s; 16346/331116 objects misplaced (4.937%); 53 MiB/s, 848 objects/s recovering 2024-03-21T13:18:16.808 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:16.840 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:16.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:16.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:16.942 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:16.976 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:16.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:17.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:16 smithi204 ceph-mon[33153]: pgmap v917: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.7 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1k op/s; 16346/331116 objects misplaced (4.937%); 53 MiB/s, 848 objects/s recovering 2024-03-21T13:18:17.079 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:17.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:17.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:17.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:17.217 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:17.249 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:17.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:17.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:17.351 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:17.386 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:17.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:17.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:17.488 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:17.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:17.518 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:17.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:17.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:17.657 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:17.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:17.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:17.759 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:17.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:17.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:17.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:17.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:17.942 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:17.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:17.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:18.043 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:18.084 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:18.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:18.186 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:18.227 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:18.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:18.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:18.371 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:18.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:18.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:18.473 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:18.486 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:18:18.500 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:18.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:18.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:18.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:18.638 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:18.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:18.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:18.778 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:18.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:18.880 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:18.912 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:18.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:19.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:18 smithi204 ceph-mon[33153]: pgmap v918: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 922 B/s rd, 0 B/s wr, 1.10k op/s; 11233/315030 objects misplaced (3.566%); 57 MiB/s, 909 objects/s recovering 2024-03-21T13:18:19.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:18 smithi163 ceph-mon[29621]: pgmap v918: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 922 B/s rd, 0 B/s wr, 1.10k op/s; 11233/315030 objects misplaced (3.566%); 57 MiB/s, 909 objects/s recovering 2024-03-21T13:18:19.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:18 smithi163 ceph-mon[40675]: pgmap v918: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 922 B/s rd, 0 B/s wr, 1.10k op/s; 11233/315030 objects misplaced (3.566%); 57 MiB/s, 909 objects/s recovering 2024-03-21T13:18:19.014 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:19.047 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:19.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:19.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:19.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:19.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:19.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:19.289 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:19.327 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:19.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:19.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:19.429 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:19.463 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:19.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:19.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:19.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:19.601 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:19.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:19.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:19.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:19.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:19.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:19.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:19.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:19.863 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:19.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:19.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:19.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:20.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:20.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:20.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:20.130 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:20.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:20.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:20.231 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:20.267 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:20.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:20.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:20.369 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:20.393 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:20.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:20.494 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:20.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:20.531 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:20.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:20.632 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:20.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:20.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:20.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:20.759 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:20.793 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:20.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:20.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:20.895 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:20.919 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:20.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:21.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:20 smithi204 ceph-mon[33153]: pgmap v919: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 890 op/s; 11233/315030 objects misplaced (3.566%); 47 MiB/s, 754 objects/s recovering 2024-03-21T13:18:21.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:18:21.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:21.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:20 smithi163 ceph-mon[29621]: pgmap v919: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 890 op/s; 11233/315030 objects misplaced (3.566%); 47 MiB/s, 754 objects/s recovering 2024-03-21T13:18:21.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:18:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:20 smithi163 ceph-mon[40675]: pgmap v919: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 890 op/s; 11233/315030 objects misplaced (3.566%); 47 MiB/s, 754 objects/s recovering 2024-03-21T13:18:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:18:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:21.021 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:21.051 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:21.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:21.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:21.153 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:21.179 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:21.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:21.280 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:21.312 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:21.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:21.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:21.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:21.445 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:21.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:21.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:21.576 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:21.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:21.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:21.677 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:21.709 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:21.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:21.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:21 smithi163 ceph-mon[29621]: osdmap e426: 8 total, 8 up, 6 in 2024-03-21T13:18:21.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:21 smithi163 ceph-mon[29621]: pgmap v921: 148 pgs: 1 peering, 1 active+clean+remapped, 6 active+remapped+backfilling, 140 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 7671/305016 objects misplaced (2.515%); 55 MiB/s, 885 objects/s recovering 2024-03-21T13:18:21.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:21 smithi163 ceph-mon[40675]: osdmap e426: 8 total, 8 up, 6 in 2024-03-21T13:18:21.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:21 smithi163 ceph-mon[40675]: pgmap v921: 148 pgs: 1 peering, 1 active+clean+remapped, 6 active+remapped+backfilling, 140 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 7671/305016 objects misplaced (2.515%); 55 MiB/s, 885 objects/s recovering 2024-03-21T13:18:21.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:21.837 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:21.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:21.939 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:21.964 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:21.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:21.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:22.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:21 smithi204 ceph-mon[33153]: osdmap e426: 8 total, 8 up, 6 in 2024-03-21T13:18:22.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:21 smithi204 ceph-mon[33153]: pgmap v921: 148 pgs: 1 peering, 1 active+clean+remapped, 6 active+remapped+backfilling, 140 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 7671/305016 objects misplaced (2.515%); 55 MiB/s, 885 objects/s recovering 2024-03-21T13:18:22.066 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:22.094 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:22.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:22.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:22.221 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:22.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:22.323 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:22.347 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:22.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:22.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:22.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:22.473 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:22.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:22.575 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:22.599 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:22.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:22.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:22.701 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:22.726 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:22.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:22.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:22 smithi204 ceph-mon[33153]: osdmap e427: 8 total, 8 up, 6 in 2024-03-21T13:18:22.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:22 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:22.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:18:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:18:22.827 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:22.851 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:22.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:22.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:22.953 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:22.977 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:22.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:22.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:23.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:22 smithi163 ceph-mon[29621]: osdmap e427: 8 total, 8 up, 6 in 2024-03-21T13:18:23.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:22 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:23.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:22 smithi163 ceph-mon[40675]: osdmap e427: 8 total, 8 up, 6 in 2024-03-21T13:18:23.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:22 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:23.078 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:23.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:23.103 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:23.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:23.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:23.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:23.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:23.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:23.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:23.354 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:23.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:23.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:23.456 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:23.479 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:23.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:23.581 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:23.606 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:23.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:23.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:23.708 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:23.737 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:23.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:23.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:23.839 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:23.872 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:23.877 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:23.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:23.978 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:24.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:23 smithi163 ceph-mon[29621]: osdmap e428: 8 total, 8 up, 6 in 2024-03-21T13:18:24.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:23 smithi163 ceph-mon[29621]: pgmap v924: 148 pgs: 1 peering, 1 active+clean+remapped, 6 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 771 op/s; 6195/301155 objects misplaced (2.057%); 48 MiB/s, 768 objects/s recovering 2024-03-21T13:18:24.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:23 smithi163 ceph-mon[40675]: osdmap e428: 8 total, 8 up, 6 in 2024-03-21T13:18:24.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:23 smithi163 ceph-mon[40675]: pgmap v924: 148 pgs: 1 peering, 1 active+clean+remapped, 6 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 771 op/s; 6195/301155 objects misplaced (2.057%); 48 MiB/s, 768 objects/s recovering 2024-03-21T13:18:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:23 smithi204 ceph-mon[33153]: osdmap e428: 8 total, 8 up, 6 in 2024-03-21T13:18:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:23 smithi204 ceph-mon[33153]: pgmap v924: 148 pgs: 1 peering, 1 active+clean+remapped, 6 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 771 op/s; 6195/301155 objects misplaced (2.057%); 48 MiB/s, 768 objects/s recovering 2024-03-21T13:18:24.015 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:24.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:24.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:24.167 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:24.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:24.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:24.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:24.305 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:24.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:24.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:24.407 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:24.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:24.441 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:24.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:24.542 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:24.576 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:24.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:24.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:24.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:24.718 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:24.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:24.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:24.819 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:24.854 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:24.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:24.941 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[29621]: 6.13 scrub starts 2024-03-21T13:18:24.942 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[29621]: 6.13 scrub ok 2024-03-21T13:18:24.942 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[29621]: 5.f scrub starts 2024-03-21T13:18:24.942 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[29621]: 5.f scrub ok 2024-03-21T13:18:24.942 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[29621]: osdmap e429: 8 total, 8 up, 6 in 2024-03-21T13:18:24.942 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:18:24.942 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:18:24.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[40675]: 6.13 scrub starts 2024-03-21T13:18:24.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[40675]: 6.13 scrub ok 2024-03-21T13:18:24.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[40675]: 5.f scrub starts 2024-03-21T13:18:24.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[40675]: 5.f scrub ok 2024-03-21T13:18:24.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[40675]: osdmap e429: 8 total, 8 up, 6 in 2024-03-21T13:18:24.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:18:24.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:24 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:18:24.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:24.986 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:24.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:24 smithi204 ceph-mon[33153]: 6.13 scrub starts 2024-03-21T13:18:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:24 smithi204 ceph-mon[33153]: 6.13 scrub ok 2024-03-21T13:18:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:24 smithi204 ceph-mon[33153]: 5.f scrub starts 2024-03-21T13:18:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:24 smithi204 ceph-mon[33153]: 5.f scrub ok 2024-03-21T13:18:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:24 smithi204 ceph-mon[33153]: osdmap e429: 8 total, 8 up, 6 in 2024-03-21T13:18:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:24 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:18:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:24 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:18:25.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:25.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:25.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:25.229 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:25.262 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:25.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:25.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:25.364 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:25.400 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:25.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:25.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:25.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:25.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:25.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:25.643 INFO:teuthology.orchestra.run.smithi163.stderr:set 6.6 pg_upmap_items mapping to [1->5,7->3,4->0] 2024-03-21T13:18:25.644 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:18:25.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:25.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:25.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:25.813 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:25.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:25.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:25.914 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:25.941 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:25.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:26.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:25 smithi204 ceph-mon[33153]: 6.2 scrub starts 2024-03-21T13:18:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:25 smithi204 ceph-mon[33153]: 6.2 scrub ok 2024-03-21T13:18:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:25 smithi204 ceph-mon[33153]: 6.a scrub starts 2024-03-21T13:18:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:25 smithi204 ceph-mon[33153]: 6.a scrub ok 2024-03-21T13:18:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:25 smithi204 ceph-mon[33153]: osdmap e430: 8 total, 8 up, 6 in 2024-03-21T13:18:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:25 smithi204 ceph-mon[33153]: pgmap v927: 148 pgs: 1 peering, 1 active+clean+remapped, 6 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 321 op/s; 6195/301155 objects misplaced (2.057%); 22 MiB/s, 351 objects/s recovering 2024-03-21T13:18:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:25 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1455651670' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [1, 5, 7, 3, 4, 0]}]: dispatch 2024-03-21T13:18:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:25 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [1, 5, 7, 3, 4, 0]}]: dispatch 2024-03-21T13:18:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:25 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:18:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:25 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:18:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:25 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2671834211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:18:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:25 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:18:26.010 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:18:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:18:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[29621]: 6.2 scrub starts 2024-03-21T13:18:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[29621]: 6.2 scrub ok 2024-03-21T13:18:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[29621]: 6.a scrub starts 2024-03-21T13:18:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[29621]: 6.a scrub ok 2024-03-21T13:18:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[29621]: osdmap e430: 8 total, 8 up, 6 in 2024-03-21T13:18:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[29621]: pgmap v927: 148 pgs: 1 peering, 1 active+clean+remapped, 6 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 321 op/s; 6195/301155 objects misplaced (2.057%); 22 MiB/s, 351 objects/s recovering 2024-03-21T13:18:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1455651670' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [1, 5, 7, 3, 4, 0]}]: dispatch 2024-03-21T13:18:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [1, 5, 7, 3, 4, 0]}]: dispatch 2024-03-21T13:18:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:18:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:18:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2671834211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:18:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:18:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[40675]: 6.2 scrub starts 2024-03-21T13:18:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[40675]: 6.2 scrub ok 2024-03-21T13:18:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[40675]: 6.a scrub starts 2024-03-21T13:18:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[40675]: 6.a scrub ok 2024-03-21T13:18:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[40675]: osdmap e430: 8 total, 8 up, 6 in 2024-03-21T13:18:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[40675]: pgmap v927: 148 pgs: 1 peering, 1 active+clean+remapped, 6 active+remapped+backfilling, 140 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 321 op/s; 6195/301155 objects misplaced (2.057%); 22 MiB/s, 351 objects/s recovering 2024-03-21T13:18:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1455651670' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [1, 5, 7, 3, 4, 0]}]: dispatch 2024-03-21T13:18:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [1, 5, 7, 3, 4, 0]}]: dispatch 2024-03-21T13:18:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:18:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:18:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2671834211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:18:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:25 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:18:26.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:26.072 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:26.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:26.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:26.174 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:26.210 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:26.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:26.311 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:26.354 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:26.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:26.456 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:26.489 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:26.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:26.590 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:26.624 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:26.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:26.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:26.725 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:26.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:26.757 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:26.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:26.858 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:26.890 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:26.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:26.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:26.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:26 smithi204 ceph-mon[33153]: 4.14 scrub starts 2024-03-21T13:18:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:26 smithi204 ceph-mon[33153]: 4.14 scrub ok 2024-03-21T13:18:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:26 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:18:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:26 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [1, 5, 7, 3, 4, 0]}]': finished 2024-03-21T13:18:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:26 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:18:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:26 smithi204 ceph-mon[33153]: osdmap e431: 8 total, 8 up, 6 in 2024-03-21T13:18:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:26 smithi163 ceph-mon[29621]: 4.14 scrub starts 2024-03-21T13:18:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:26 smithi163 ceph-mon[29621]: 4.14 scrub ok 2024-03-21T13:18:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:26 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:18:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:26 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [1, 5, 7, 3, 4, 0]}]': finished 2024-03-21T13:18:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:26 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:18:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:26 smithi163 ceph-mon[29621]: osdmap e431: 8 total, 8 up, 6 in 2024-03-21T13:18:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:26 smithi163 ceph-mon[40675]: 4.14 scrub starts 2024-03-21T13:18:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:26 smithi163 ceph-mon[40675]: 4.14 scrub ok 2024-03-21T13:18:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:26 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:18:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:26 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [1, 5, 7, 3, 4, 0]}]': finished 2024-03-21T13:18:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:26 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:18:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:26 smithi163 ceph-mon[40675]: osdmap e431: 8 total, 8 up, 6 in 2024-03-21T13:18:27.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:27.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:27.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:27.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:27.177 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:27.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:27.278 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:27.308 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:27.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:27.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:27.410 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:27.440 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:27.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:27.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:27.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:27.589 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:27.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:27.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:27.691 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:27.720 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:27.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:27.822 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:27.860 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:27.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:27.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:27.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:27.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:27.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:28.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:27 smithi204 ceph-mon[33153]: pgmap v929: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 710 op/s; 2308/291810 objects misplaced (0.791%); 54 MiB/s, 863 objects/s recovering 2024-03-21T13:18:28.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:27 smithi163 ceph-mon[29621]: pgmap v929: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 710 op/s; 2308/291810 objects misplaced (0.791%); 54 MiB/s, 863 objects/s recovering 2024-03-21T13:18:28.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:27 smithi163 ceph-mon[40675]: pgmap v929: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 710 op/s; 2308/291810 objects misplaced (0.791%); 54 MiB/s, 863 objects/s recovering 2024-03-21T13:18:28.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:28.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:28.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:28.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:28.234 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:28.262 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:28.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:28.364 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:28.392 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:28.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:28.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:28.494 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:28.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:28.627 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:28.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:28.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:28.768 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:28.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:28.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:28.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:28.905 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:28.934 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:28.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:28.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:29.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:28 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:18:29.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:28 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:18:29.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:28 smithi204 ceph-mon[33153]: osdmap e432: 8 total, 8 up, 6 in 2024-03-21T13:18:29.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:28 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:18:29.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:28 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:18:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:28 smithi163 ceph-mon[29621]: osdmap e432: 8 total, 8 up, 6 in 2024-03-21T13:18:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:28 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:18:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:28 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:18:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:28 smithi163 ceph-mon[40675]: osdmap e432: 8 total, 8 up, 6 in 2024-03-21T13:18:29.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:29.073 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:29.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:29.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:29.174 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:29.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:29.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:29.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:29.341 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:29.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:29.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:29.443 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:29.474 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:29.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:29.575 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:29.606 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:29.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:29.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:29.741 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:29.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:29.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:29.878 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:29.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:29.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:29.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:29 smithi204 ceph-mon[33153]: pgmap v931: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 951 B/s rd, 0 B/s wr, 579 op/s; 2308/291810 objects misplaced (0.791%); 44 MiB/s, 704 objects/s recovering 2024-03-21T13:18:30.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:30.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:30.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:30.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:29 smithi163 ceph-mon[29621]: pgmap v931: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 951 B/s rd, 0 B/s wr, 579 op/s; 2308/291810 objects misplaced (0.791%); 44 MiB/s, 704 objects/s recovering 2024-03-21T13:18:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:29 smithi163 ceph-mon[40675]: pgmap v931: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 951 B/s rd, 0 B/s wr, 579 op/s; 2308/291810 objects misplaced (0.791%); 44 MiB/s, 704 objects/s recovering 2024-03-21T13:18:30.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:30.134 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:30.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:30.236 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:30.260 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:30.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:30.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:30.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:30.386 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:30.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:30.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:30.487 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:30.524 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:30.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:30.625 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:30.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:30.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:30.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:30.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:30.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:30.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:30.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:30.904 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:30.928 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:30.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:31.030 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:31.059 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:31.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:31.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:31.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:31.197 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:31.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:31.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:31.299 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:31.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:31.335 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:31.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:31.436 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:31.466 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:31.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:31.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:31.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:31.604 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:31.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:31.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:31.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:31.742 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:31.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:31.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:31.844 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:31.873 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:31.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:31.975 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:32.004 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:32.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:32.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:32.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:32.136 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:32.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:32.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:32.238 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:32.268 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:32.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:32.369 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:32.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:32.402 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:32.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:32.503 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:32.536 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:32.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:32.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:32.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:32.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:32.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:32.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:32 smithi204 ceph-mon[33153]: pgmap v932: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.13k op/s; 1556/279558 objects misplaced (0.557%); 43 MiB/s, 695 objects/s recovering 2024-03-21T13:18:32.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:18:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:18:32.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:32.813 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:32.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:32.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:32.865 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:32 smithi163 ceph-mon[29621]: pgmap v932: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.13k op/s; 1556/279558 objects misplaced (0.557%); 43 MiB/s, 695 objects/s recovering 2024-03-21T13:18:32.866 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:32 smithi163 ceph-mon[40675]: pgmap v932: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.13k op/s; 1556/279558 objects misplaced (0.557%); 43 MiB/s, 695 objects/s recovering 2024-03-21T13:18:32.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:32.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:32.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:32.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:33.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:33.089 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:33.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:33.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:33.227 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:33.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:33.329 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:33.360 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:33.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:33.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:33.462 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:33.501 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:33.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:33.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:33.641 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:33.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:33.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:33.706 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:33 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:33.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:33 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:33.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:33.784 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:33.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:33.878 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:18:33.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:33.910 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:18:33.922 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:33.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:33.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:33.938 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:18:33.969 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:18:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:33 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:34.007 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:18:34.024 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:34.052 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:18:34.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:34.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:34.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:34.086 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:18:34.128 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:18:34.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:18:34.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:18:34.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:18:34.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:18:34.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:18:34.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:18:34.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:18:34.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:18:34.161 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:34.194 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:34.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:34.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:18:34.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:34.334 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:34.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:34.435 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:34.464 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:18:34.464 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:18:34.464 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-03-21T13:18:34.464 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-03-21T13:18:34.464 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:18:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:34.478 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:34.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:34.578 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:34.614 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:34.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:34.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:34.716 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:34.748 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:34.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:34.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:34.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:34.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:34.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:34.985 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:35.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:34 smithi204 ceph-mon[33153]: pgmap v933: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 1556/273807 objects misplaced (0.568%); 35 MiB/s, 553 objects/s recovering 2024-03-21T13:18:35.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:34 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1600808715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:18:35.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:34 smithi163 ceph-mon[29621]: pgmap v933: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 1556/273807 objects misplaced (0.568%); 35 MiB/s, 553 objects/s recovering 2024-03-21T13:18:35.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:34 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1600808715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:18:35.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:34 smithi163 ceph-mon[40675]: pgmap v933: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 1556/273807 objects misplaced (0.568%); 35 MiB/s, 553 objects/s recovering 2024-03-21T13:18:35.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:34 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1600808715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:18:35.022 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:35.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:35.124 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:35.153 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:35.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:35.255 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:35.284 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:35.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:35.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:35.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:35.421 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:35.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:35.522 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:35.563 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:35.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:35.664 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:35.694 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:35.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:35.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:35.795 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:35.829 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:35.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:35.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:35.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:35.970 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:35.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:35.973 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:18:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:18:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:18:36.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:36.104 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:36.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:36.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:36.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:36.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:36.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:36.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:36.343 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:36.366 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:36.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:36.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:36.467 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:36.498 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:36.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:36.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:36.536 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 6, 7, 2, 5] out_osds: [1, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T13:18:36.536 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-21T13:18:36.600 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:36.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:36.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:36.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:36.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:36.778 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:36.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:36.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:36.880 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:36.912 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:36.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:36 smithi204 ceph-mon[33153]: pgmap v934: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 546 B/s rd, 0 B/s wr, 878 op/s; 1556/273807 objects misplaced (0.568%); 22 MiB/s, 350 objects/s recovering 2024-03-21T13:18:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:36 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:18:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:36 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:37.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:36 smithi163 ceph-mon[29621]: pgmap v934: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 546 B/s rd, 0 B/s wr, 878 op/s; 1556/273807 objects misplaced (0.568%); 22 MiB/s, 350 objects/s recovering 2024-03-21T13:18:37.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:36 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:18:37.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:36 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:37.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:36 smithi163 ceph-mon[40675]: pgmap v934: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 546 B/s rd, 0 B/s wr, 878 op/s; 1556/273807 objects misplaced (0.568%); 22 MiB/s, 350 objects/s recovering 2024-03-21T13:18:37.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:36 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:18:37.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:36 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:37.014 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:37.046 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:37.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:37.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:37.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:37.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:37.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:37.288 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:37.321 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:37.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:37.392 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-21T13:18:37.422 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:37.505 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:37.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:37.519 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:37.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:37 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:18:37.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:37 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:18:37.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:37 smithi163 ceph-mon[29621]: osdmap e433: 8 total, 8 up, 6 in 2024-03-21T13:18:37.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:37.644 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:37.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:37.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:37.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:37.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:37.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:37.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:37.851 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:18:37.882 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:37 smithi163 ceph-mon[29621]: pgmap v936: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 877/254634 objects misplaced (0.344%); 8.3 MiB/s, 132 objects/s recovering 2024-03-21T13:18:37.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:37 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:18:37.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:37 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:18:37.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:37 smithi163 ceph-mon[40675]: osdmap e433: 8 total, 8 up, 6 in 2024-03-21T13:18:37.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:37 smithi163 ceph-mon[40675]: pgmap v936: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 877/254634 objects misplaced (0.344%); 8.3 MiB/s, 132 objects/s recovering 2024-03-21T13:18:37.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:37.926 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:37.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:37.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:38.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:37 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:18:38.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:37 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:18:38.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:37 smithi204 ceph-mon[33153]: osdmap e433: 8 total, 8 up, 6 in 2024-03-21T13:18:38.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:37 smithi204 ceph-mon[33153]: pgmap v936: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 877/254634 objects misplaced (0.344%); 8.3 MiB/s, 132 objects/s recovering 2024-03-21T13:18:38.028 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:38.064 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:38.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:38.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:38.166 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:38.199 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:38.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:38.301 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:38.330 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:38.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:38.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:38.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:38.466 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:38.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:38.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:38.604 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:38.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:38.706 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:38.739 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:38.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:38.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:38.840 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:38.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:38.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:38.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:39.032 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:39.067 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:39.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:39.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:39.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:39.207 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:39.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:39.308 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:39.339 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:39.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:39.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:39.474 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:39.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:39.576 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:39.605 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:39.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:39.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:39.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:39.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:39.746 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:39.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:39.847 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:39.895 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:39.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:39.898 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:39.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:39.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:40.038 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:40.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:40.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:40.140 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:40.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:40.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:40.301 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:40.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:40.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:40.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:40.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:40.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:40 smithi204 ceph-mon[33153]: pgmap v937: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 877/254634 objects misplaced (0.344%); 7.7 MiB/s, 123 objects/s recovering 2024-03-21T13:18:40.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:40 smithi163 ceph-mon[29621]: pgmap v937: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 877/254634 objects misplaced (0.344%); 7.7 MiB/s, 123 objects/s recovering 2024-03-21T13:18:40.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:40 smithi163 ceph-mon[40675]: pgmap v937: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 877/254634 objects misplaced (0.344%); 7.7 MiB/s, 123 objects/s recovering 2024-03-21T13:18:40.537 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:40.575 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:40.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:40.676 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:40.721 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:40.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:40.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:40.823 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:40.855 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:40.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:40.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:40.956 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:40.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:40.986 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:40.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:41.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:41.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:41.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:41.226 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:41.256 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:41.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:41.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:41.358 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:41.388 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:41.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:41.490 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:41.527 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:41.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:41.628 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:41.661 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:41.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:41.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:41.804 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:41.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:41.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:41.906 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:41.940 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:41.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:41.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:42.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:42.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:42.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:42.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:42.220 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:42.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:42.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:42 smithi163 ceph-mon[29621]: pgmap v938: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 234/241704 objects misplaced (0.097%); 7.5 MiB/s, 120 objects/s recovering 2024-03-21T13:18:42.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:42 smithi163 ceph-mon[40675]: pgmap v938: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 234/241704 objects misplaced (0.097%); 7.5 MiB/s, 120 objects/s recovering 2024-03-21T13:18:42.322 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:42.356 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:42.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:42.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:42.391 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:42 smithi204 ceph-mon[33153]: pgmap v938: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 234/241704 objects misplaced (0.097%); 7.5 MiB/s, 120 objects/s recovering 2024-03-21T13:18:42.457 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:42.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:42.486 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:42.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:42.587 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:42.623 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:42.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:42.725 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:42.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:18:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:18:42.756 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:42.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:42.858 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:42.889 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:42.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:42.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:43.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:43.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:43.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:43.121 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.3 to 1 (10000) 2024-03-21T13:18:43.125 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:18:43.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:43.156 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:43.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:43.258 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:43.286 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:43.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:43.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:43.365 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:43.365 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:43 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/576089925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:18:43.365 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:43 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1641784578' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:18:43.365 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:43 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:18:43.366 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:43.366 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:43 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/576089925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:18:43.366 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:43 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1641784578' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:18:43.366 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:43 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:18:43.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:43.412 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:43.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:43.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:43.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:43 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/576089925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:18:43.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:43 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1641784578' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:18:43.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:43 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:18:43.513 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:43.538 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:43.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:43.639 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:43.663 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:43.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:43.764 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:43.788 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:43.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:43.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:43.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:43.914 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:43.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:43.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:44.015 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:44.040 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:44.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:44.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:44.167 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:44.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:44.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:44.269 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:44.298 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:44.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:44.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:44.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:44.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:44.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:44.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:44.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:44 smithi204 ceph-mon[33153]: pgmap v939: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 234/235686 objects misplaced (0.099%); 7.5 MiB/s, 120 objects/s recovering 2024-03-21T13:18:44.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:44 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:18:44.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/576089925' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:18:44.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:44 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:18:44.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:44 smithi204 ceph-mon[33153]: osdmap e434: 8 total, 8 up, 6 in 2024-03-21T13:18:44.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:44 smithi163 ceph-mon[29621]: pgmap v939: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 234/235686 objects misplaced (0.099%); 7.5 MiB/s, 120 objects/s recovering 2024-03-21T13:18:44.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:44 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:18:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/576089925' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:18:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:44 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:18:44.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:44 smithi163 ceph-mon[29621]: osdmap e434: 8 total, 8 up, 6 in 2024-03-21T13:18:44.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:44 smithi163 ceph-mon[40675]: pgmap v939: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 234/235686 objects misplaced (0.099%); 7.5 MiB/s, 120 objects/s recovering 2024-03-21T13:18:44.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:44 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:18:44.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/576089925' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:18:44.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:44 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:18:44.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:44 smithi163 ceph-mon[40675]: osdmap e434: 8 total, 8 up, 6 in 2024-03-21T13:18:44.552 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:44.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:44.576 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:44.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:44.677 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:44.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:44.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:44.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:44.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:44.851 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:44.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:44.952 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:44.983 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:44.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:45.084 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:45.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:45.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:45.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:45.225 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:45.256 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:45.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:45.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:45.358 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:45.387 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:45.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:45.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:45 smithi204 ceph-mon[33153]: osdmap e435: 8 total, 8 up, 6 in 2024-03-21T13:18:45.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:45 smithi163 ceph-mon[29621]: osdmap e435: 8 total, 8 up, 6 in 2024-03-21T13:18:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:45 smithi163 ceph-mon[40675]: osdmap e435: 8 total, 8 up, 6 in 2024-03-21T13:18:45.524 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:45.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:45.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:45.626 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:45.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:45.656 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:45.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:45.756 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:45.788 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:45.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:45.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:45.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:45.920 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:45.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:45.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:46.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:18:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:18:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:18:46.022 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:46.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:46.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:46.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:46.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:46.195 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:46.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:46.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:46.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:46.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:46.439 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:46.470 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:46.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:46.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:46.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:46 smithi204 ceph-mon[33153]: pgmap v942: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 789 op/s; 234/235686 objects misplaced (0.099%); 4.8 MiB/s, 76 objects/s recovering 2024-03-21T13:18:46.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:46 smithi163 ceph-mon[29621]: pgmap v942: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 789 op/s; 234/235686 objects misplaced (0.099%); 4.8 MiB/s, 76 objects/s recovering 2024-03-21T13:18:46.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:46 smithi163 ceph-mon[40675]: pgmap v942: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 789 op/s; 234/235686 objects misplaced (0.099%); 4.8 MiB/s, 76 objects/s recovering 2024-03-21T13:18:46.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:46.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:46.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:46.714 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:46.741 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:46.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:46.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:46.843 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:46.871 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:46.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:46.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:46.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:47.009 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:47.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:47.111 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:47.140 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:47.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:47.241 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:47.279 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:47.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:47.374 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:47.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:47.417 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:47.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:47.519 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:47.550 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:47.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:47.651 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:47.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:47.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:47.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:47.793 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:47.822 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:47.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:47.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:47.924 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:47.963 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:47.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:47.981 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-21T13:18:48.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:48.101 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:48.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:48.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:48.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:48.240 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:48.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:48.341 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:48.374 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:48.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:48.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:48.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:48.512 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:48.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:48.614 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:48.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:48.648 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:48.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:48.749 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:48.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:48 smithi163 ceph-mon[29621]: pgmap v943: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 26 KiB/s rd, 0 B/s wr, 1.46k op/s; 6.5 MiB/s, 104 objects/s recovering 2024-03-21T13:18:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:48 smithi163 ceph-mon[40675]: pgmap v943: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 26 KiB/s rd, 0 B/s wr, 1.46k op/s; 6.5 MiB/s, 104 objects/s recovering 2024-03-21T13:18:48.781 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:48.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:48.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:48.883 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:48.913 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:48.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:48.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:48 smithi204 ceph-mon[33153]: pgmap v943: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 26 KiB/s rd, 0 B/s wr, 1.46k op/s; 6.5 MiB/s, 104 objects/s recovering 2024-03-21T13:18:49.014 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:49.050 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:49.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:49.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:49.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:49.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:49.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:49.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:49.332 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:49.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:49.346 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:18:49.432 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:49.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:49.566 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:49.604 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:49.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:49.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:49.744 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:49.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:49.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:49.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:49.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:49.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:49.989 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:50.016 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:50.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:50.118 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:50.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:50.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:50.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:50.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:50.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:50.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:50.401 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:50.430 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:50.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:50.532 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:50.563 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:50.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:50.627 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:50.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:50.704 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:50.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:50.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:50.806 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:50.836 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:50.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:50.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:50.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:50 smithi163 ceph-mon[29621]: pgmap v944: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 25 KiB/s rd, 0 B/s wr, 918 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-03-21T13:18:50.903 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:50 smithi163 ceph-mon[40675]: pgmap v944: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 25 KiB/s rd, 0 B/s wr, 918 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-03-21T13:18:50.939 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:50.969 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:50.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:50 smithi204 ceph-mon[33153]: pgmap v944: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 25 KiB/s rd, 0 B/s wr, 918 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-03-21T13:18:51.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:51.102 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:51.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:51.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:51.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:51.245 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:51.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:51.346 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:51.377 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:51.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:51.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:51.520 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:51.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:51.599 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:18:51.622 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:51.626 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:18:51.654 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:51.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:51.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:51.661 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:18:51.693 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:18:51.731 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:18:51.756 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:51.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:51.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:51.765 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:18:51.768 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:18:51.769 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-03-21T13:18:51.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-03-21T13:18:51.769 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [] 2024-03-21T13:18:51.769 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:18:51.788 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:51.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:51.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:51.802 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:18:51.830 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:18:51.831 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:18:51.832 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:18:51.832 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:18:51.832 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:18:51.832 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:18:51.832 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:18:51.832 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:18:51.832 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:18:51.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:51.927 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:51.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:52.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:52.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:52.068 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:52.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:52.170 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:52.200 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:52.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:52.303 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:52.332 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:52.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:52.434 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:52.462 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:52.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:52.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:52.523 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-21T13:18:52.564 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:52.600 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:52.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:52.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:52.703 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:52.736 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:52.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:52 smithi204 ceph-mon[33153]: pgmap v945: 148 pgs: 148 active+clean; 4.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 50 KiB/s rd, 0 B/s wr, 1.26k op/s; 1.8 MiB/s, 28 objects/s recovering 2024-03-21T13:18:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:52 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3763171074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:18:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:52 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:18:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:52 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:18:52.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:18:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:18:52.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:52 smithi163 ceph-mon[29621]: pgmap v945: 148 pgs: 148 active+clean; 4.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 50 KiB/s rd, 0 B/s wr, 1.26k op/s; 1.8 MiB/s, 28 objects/s recovering 2024-03-21T13:18:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:52 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3763171074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:18:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:52 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:18:52.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:52 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:18:52.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:52 smithi163 ceph-mon[40675]: pgmap v945: 148 pgs: 148 active+clean; 4.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 50 KiB/s rd, 0 B/s wr, 1.26k op/s; 1.8 MiB/s, 28 objects/s recovering 2024-03-21T13:18:52.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:52 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3763171074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:18:52.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:52 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:18:52.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:52 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:18:52.839 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:52.875 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:52.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:52.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:52.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:53.014 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:53.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:53.016 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:53.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:53.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:53.145 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:53.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:53.247 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:53.277 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:53.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:53.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:53.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:53.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:53.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:53.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:53.521 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:53.553 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:53.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:53.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:53.655 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:53.686 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:53.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:53.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:53.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:53.825 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:53.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:53.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:53.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:53.908 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:53 smithi163 ceph-mon[29621]: osdmap e436: 8 total, 8 up, 6 in 2024-03-21T13:18:53.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:53.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:53 smithi163 ceph-mon[40675]: osdmap e436: 8 total, 8 up, 6 in 2024-03-21T13:18:53.928 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:53.958 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:53.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:53.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:54.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:54.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:53 smithi204 ceph-mon[33153]: osdmap e436: 8 total, 8 up, 6 in 2024-03-21T13:18:54.060 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:54.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:54.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:54.200 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:54.233 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:54.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:54.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:54.372 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:54.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:54.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:54.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:54.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:54.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:54.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:54.630 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:54.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:54.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:54.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:54.769 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:54.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:54.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:54.909 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:54.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:54.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:54.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:54 smithi163 ceph-mon[40675]: pgmap v947: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 49 KiB/s rd, 0 B/s wr, 1.37k op/s; 1.6 MiB/s, 25 objects/s recovering 2024-03-21T13:18:54.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:54 smithi163 ceph-mon[29621]: pgmap v947: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 49 KiB/s rd, 0 B/s wr, 1.37k op/s; 1.6 MiB/s, 25 objects/s recovering 2024-03-21T13:18:55.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:54 smithi204 ceph-mon[33153]: pgmap v947: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 49 KiB/s rd, 0 B/s wr, 1.37k op/s; 1.6 MiB/s, 25 objects/s recovering 2024-03-21T13:18:55.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:55.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:55.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:55.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:55.189 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:55.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:55.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:55.291 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:55.321 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:55.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:55.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:55.423 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:55.460 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:55.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:55.562 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:55.594 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:55.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:55.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:55.696 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:55.699 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:18:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:18:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:18:55.730 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:55.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:55.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:55.863 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:55.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:55.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:55.967 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:55.995 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:55.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:56.096 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:56.124 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:56.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:56.225 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:56.256 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:56.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:56.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:56.396 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:56.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:56.497 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:56.527 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:56.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:56.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:56.610 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:56 smithi163 ceph-mon[29621]: pgmap v948: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 44 KiB/s rd, 0 B/s wr, 1.22k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-03-21T13:18:56.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:56 smithi163 ceph-mon[40675]: pgmap v948: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 44 KiB/s rd, 0 B/s wr, 1.22k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-03-21T13:18:56.629 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:56.661 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:56.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:56.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:56.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:56.801 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:56.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:56.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:56.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:18:56.934 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:56.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:56.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:18:57.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:56 smithi204 ceph-mon[33153]: pgmap v948: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 44 KiB/s rd, 0 B/s wr, 1.22k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-03-21T13:18:57.037 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:57.065 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:57.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:57.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:57.205 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:57.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:57.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:57.232 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:18:57.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:18:57.345 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:57.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:57.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:18:57.447 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:57.451 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:18:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:57.484 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:57.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:57.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:57.603 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:18:57.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:57.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:57.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:57.727 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:57.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:57 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/969968235' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:18:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:57 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/969968235' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:18:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:57.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:57.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:57.863 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:57.900 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:57.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:58.001 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:18:58.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:57 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/969968235' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:18:58.031 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:58.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:18:58.133 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:58.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:58.160 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:58.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:58.261 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:58.292 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:58.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:58.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:58.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:58.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:58.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:58.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:58.567 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:58.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:58.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:58.669 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:58.705 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:58.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:58.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:58.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:18:58.843 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:58.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:58.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:18:58.945 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:58.981 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:58.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:58 smithi204 ceph-mon[33153]: pgmap v949: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-21T13:18:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:58 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:18:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:58 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:18:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:58 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:18:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:58 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/969968235' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:18:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:58 smithi204 ceph-mon[33153]: osdmap e437: 8 total, 8 up, 6 in 2024-03-21T13:18:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:58 smithi204 ceph-mon[33153]: 6.c scrub starts 2024-03-21T13:18:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:58 smithi204 ceph-mon[33153]: 6.c scrub ok 2024-03-21T13:18:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:58 smithi204 ceph-mon[33153]: 5.1d scrub starts 2024-03-21T13:18:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:58 smithi204 ceph-mon[33153]: 5.1d scrub ok 2024-03-21T13:18:59.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[29621]: pgmap v949: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-21T13:18:59.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:18:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:18:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:18:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/969968235' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:18:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[29621]: osdmap e437: 8 total, 8 up, 6 in 2024-03-21T13:18:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[29621]: 6.c scrub starts 2024-03-21T13:18:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[29621]: 6.c scrub ok 2024-03-21T13:18:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[29621]: 5.1d scrub starts 2024-03-21T13:18:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[29621]: 5.1d scrub ok 2024-03-21T13:18:59.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[40675]: pgmap v949: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-21T13:18:59.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:18:59.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:18:59.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:18:59.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/969968235' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:18:59.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[40675]: osdmap e437: 8 total, 8 up, 6 in 2024-03-21T13:18:59.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[40675]: 6.c scrub starts 2024-03-21T13:18:59.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[40675]: 6.c scrub ok 2024-03-21T13:18:59.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[40675]: 5.1d scrub starts 2024-03-21T13:18:59.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:58 smithi163 ceph-mon[40675]: 5.1d scrub ok 2024-03-21T13:18:59.081 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:59.104 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:59.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:59.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:59.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:59.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:59.244 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:59.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:18:59.345 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:59.376 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:59.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:59.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:59.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:18:59.517 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:59.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:59.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:18:59.619 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:18:59.620 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.4 to 1 (10000) 2024-03-21T13:18:59.651 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:59.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:18:59.752 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:18:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:59 smithi163 ceph-mon[29621]: 11.2 scrub starts 2024-03-21T13:18:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:59 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/979010886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:18:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:59 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:18:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:18:59 smithi163 ceph-mon[29621]: pgmap v951: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-03-21T13:18:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:59 smithi163 ceph-mon[40675]: 11.2 scrub starts 2024-03-21T13:18:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:59 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/979010886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:18:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:59 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:18:59.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:18:59 smithi163 ceph-mon[40675]: pgmap v951: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-03-21T13:18:59.784 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:18:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:59.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:18:59.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:18:59.924 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:18:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:18:59.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:59 smithi204 ceph-mon[33153]: 11.2 scrub starts 2024-03-21T13:19:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:59 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/979010886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:19:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:59 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:19:00.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:18:59 smithi204 ceph-mon[33153]: pgmap v951: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-03-21T13:19:00.027 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:00.070 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:00.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:00.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:00.172 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:00.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:00.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:00.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:00.311 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:00.340 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:00.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:00.442 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:00.478 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:00.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:00.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:00.579 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:00.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:00.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:00.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:00.717 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:00.748 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:00.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:00.850 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:00.878 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:00.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:00.980 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:00 smithi204 ceph-mon[33153]: 11.2 scrub ok 2024-03-21T13:19:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:00 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:19:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:00 smithi204 ceph-mon[33153]: osdmap e438: 8 total, 8 up, 6 in 2024-03-21T13:19:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:00 smithi204 ceph-mon[33153]: 4.1f scrub starts 2024-03-21T13:19:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:00 smithi204 ceph-mon[33153]: 4.1f scrub ok 2024-03-21T13:19:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:00 smithi204 ceph-mon[33153]: 3.9 scrub starts 2024-03-21T13:19:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:00 smithi204 ceph-mon[33153]: 3.9 scrub ok 2024-03-21T13:19:01.008 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:01.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:01.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[29621]: 11.2 scrub ok 2024-03-21T13:19:01.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:19:01.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[29621]: osdmap e438: 8 total, 8 up, 6 in 2024-03-21T13:19:01.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[29621]: 4.1f scrub starts 2024-03-21T13:19:01.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[29621]: 4.1f scrub ok 2024-03-21T13:19:01.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[29621]: 3.9 scrub starts 2024-03-21T13:19:01.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[29621]: 3.9 scrub ok 2024-03-21T13:19:01.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[40675]: 11.2 scrub ok 2024-03-21T13:19:01.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:19:01.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[40675]: osdmap e438: 8 total, 8 up, 6 in 2024-03-21T13:19:01.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[40675]: 4.1f scrub starts 2024-03-21T13:19:01.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[40675]: 4.1f scrub ok 2024-03-21T13:19:01.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[40675]: 3.9 scrub starts 2024-03-21T13:19:01.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:00 smithi163 ceph-mon[40675]: 3.9 scrub ok 2024-03-21T13:19:01.110 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:01.136 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:01.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:01.238 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:01.274 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:01.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:01.375 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:01.405 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:01.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:01.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:01.549 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:01.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:01.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:01.651 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:01.680 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:01.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:01.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:01.782 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:01.813 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:01.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:01.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:01.951 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:01.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:01 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:19:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:01 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:19:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:01 smithi204 ceph-mon[33153]: 5.1f scrub starts 2024-03-21T13:19:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:01 smithi204 ceph-mon[33153]: 5.1f scrub ok 2024-03-21T13:19:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:01 smithi204 ceph-mon[33153]: pgmap v953: 148 pgs: 148 active+clean; 3.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-21T13:19:02.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:01 smithi204 ceph-mon[33153]: 6.1e scrub starts 2024-03-21T13:19:02.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:01 smithi204 ceph-mon[33153]: 6.1e scrub ok 2024-03-21T13:19:02.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:19:02.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:19:02.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[29621]: 5.1f scrub starts 2024-03-21T13:19:02.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[29621]: 5.1f scrub ok 2024-03-21T13:19:02.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[29621]: pgmap v953: 148 pgs: 148 active+clean; 3.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-21T13:19:02.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[29621]: 6.1e scrub starts 2024-03-21T13:19:02.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[29621]: 6.1e scrub ok 2024-03-21T13:19:02.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:19:02.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:19:02.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[40675]: 5.1f scrub starts 2024-03-21T13:19:02.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[40675]: 5.1f scrub ok 2024-03-21T13:19:02.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[40675]: pgmap v953: 148 pgs: 148 active+clean; 3.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-21T13:19:02.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[40675]: 6.1e scrub starts 2024-03-21T13:19:02.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:01 smithi163 ceph-mon[40675]: 6.1e scrub ok 2024-03-21T13:19:02.053 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:02.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:19:02.090 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:02.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:02.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:02.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:02.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:02.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:02.368 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:02.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:02.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:02.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:02.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:02.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:02.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:02.645 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:02.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:02.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:02.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:02 smithi204 ceph-mon[33153]: 6.15 scrub starts 2024-03-21T13:19:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:02 smithi204 ceph-mon[33153]: 6.15 scrub ok 2024-03-21T13:19:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:02 smithi204 ceph-mon[33153]: osdmap e439: 8 total, 8 up, 6 in 2024-03-21T13:19:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:02 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:02.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:19:02 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:19:02.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:02 smithi163 ceph-mon[40675]: 6.15 scrub starts 2024-03-21T13:19:02.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:02 smithi163 ceph-mon[40675]: 6.15 scrub ok 2024-03-21T13:19:02.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:02 smithi163 ceph-mon[40675]: osdmap e439: 8 total, 8 up, 6 in 2024-03-21T13:19:02.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:02 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:02 smithi163 ceph-mon[29621]: 6.15 scrub starts 2024-03-21T13:19:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:02 smithi163 ceph-mon[29621]: 6.15 scrub ok 2024-03-21T13:19:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:02 smithi163 ceph-mon[29621]: osdmap e439: 8 total, 8 up, 6 in 2024-03-21T13:19:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:02 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:02.782 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:02.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:02.884 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:02.902 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:02.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:02.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:02.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:02.944 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-21T13:19:02.957 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:03.027 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:03.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:03.059 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:03.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:03.160 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:03.199 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:03.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:03.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:03.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:03.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:03.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:03.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:03.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:03.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:03.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:03.578 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:03.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:03.614 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:03.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:03.715 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:03.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:03.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:03.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:03.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:03 smithi163 ceph-mon[29621]: 6.1a scrub starts 2024-03-21T13:19:03.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:03 smithi163 ceph-mon[29621]: 6.1a scrub ok 2024-03-21T13:19:03.826 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:03 smithi163 ceph-mon[29621]: pgmap v955: 148 pgs: 148 active+clean; 3.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T13:19:03.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:03 smithi163 ceph-mon[40675]: 6.1a scrub starts 2024-03-21T13:19:03.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:03 smithi163 ceph-mon[40675]: 6.1a scrub ok 2024-03-21T13:19:03.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:03 smithi163 ceph-mon[40675]: pgmap v955: 148 pgs: 148 active+clean; 3.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T13:19:03.850 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:03.878 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:03.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:03.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:03 smithi204 ceph-mon[33153]: 6.1a scrub starts 2024-03-21T13:19:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:03 smithi204 ceph-mon[33153]: 6.1a scrub ok 2024-03-21T13:19:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:03 smithi204 ceph-mon[33153]: pgmap v955: 148 pgs: 148 active+clean; 3.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T13:19:04.017 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:04.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:04.119 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:04.154 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:04.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:04.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:04.256 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:04.281 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:19:04.281 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:19:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:04.292 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:04.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:04.393 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:04.424 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:04.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:04.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:04.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:04.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:04.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:04.707 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:04.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:04.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:04.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:04.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:04.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:04.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:04.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[29621]: 11.d scrub starts 2024-03-21T13:19:04.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[29621]: 6.7 scrub starts 2024-03-21T13:19:04.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:19:04.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:19:04.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[29621]: osdmap e440: 8 total, 8 up, 6 in 2024-03-21T13:19:04.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[29621]: 6.1b scrub starts 2024-03-21T13:19:04.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[29621]: 6.7 scrub ok 2024-03-21T13:19:04.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[29621]: 11.d scrub ok 2024-03-21T13:19:04.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[29621]: 6.8 scrub starts 2024-03-21T13:19:04.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[29621]: 6.8 scrub ok 2024-03-21T13:19:04.917 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[40675]: 11.d scrub starts 2024-03-21T13:19:04.917 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[40675]: 6.7 scrub starts 2024-03-21T13:19:04.917 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:19:04.917 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:19:04.917 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[40675]: osdmap e440: 8 total, 8 up, 6 in 2024-03-21T13:19:04.917 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[40675]: 6.1b scrub starts 2024-03-21T13:19:04.917 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[40675]: 6.7 scrub ok 2024-03-21T13:19:04.917 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[40675]: 11.d scrub ok 2024-03-21T13:19:04.917 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[40675]: 6.8 scrub starts 2024-03-21T13:19:04.917 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:04 smithi163 ceph-mon[40675]: 6.8 scrub ok 2024-03-21T13:19:04.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:04.985 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:04.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:04 smithi204 ceph-mon[33153]: 11.d scrub starts 2024-03-21T13:19:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:04 smithi204 ceph-mon[33153]: 6.7 scrub starts 2024-03-21T13:19:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:04 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:19:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:04 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:19:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:04 smithi204 ceph-mon[33153]: osdmap e440: 8 total, 8 up, 6 in 2024-03-21T13:19:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:04 smithi204 ceph-mon[33153]: 6.1b scrub starts 2024-03-21T13:19:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:04 smithi204 ceph-mon[33153]: 6.7 scrub ok 2024-03-21T13:19:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:04 smithi204 ceph-mon[33153]: 11.d scrub ok 2024-03-21T13:19:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:04 smithi204 ceph-mon[33153]: 6.8 scrub starts 2024-03-21T13:19:05.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:04 smithi204 ceph-mon[33153]: 6.8 scrub ok 2024-03-21T13:19:05.086 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:05.124 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:05.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:05.225 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:05.259 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:05.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:05.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:05.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:05.394 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:05.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:05.496 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:05.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:05.532 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:05.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:05.560 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-21T13:19:05.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:05.681 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:05.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:05.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:05.784 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:19:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:19:05.785 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[29621]: 6.1b scrub ok 2024-03-21T13:19:05.785 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[29621]: 6.f scrub starts 2024-03-21T13:19:05.785 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[29621]: 6.f scrub ok 2024-03-21T13:19:05.785 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[29621]: 5.e scrub starts 2024-03-21T13:19:05.785 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[29621]: 5.e scrub ok 2024-03-21T13:19:05.785 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[29621]: pgmap v957: 148 pgs: 148 active+clean; 3.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T13:19:05.785 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:19:05.785 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:05.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[40675]: 6.1b scrub ok 2024-03-21T13:19:05.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[40675]: 6.f scrub starts 2024-03-21T13:19:05.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[40675]: 6.f scrub ok 2024-03-21T13:19:05.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[40675]: 5.e scrub starts 2024-03-21T13:19:05.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[40675]: 5.e scrub ok 2024-03-21T13:19:05.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[40675]: pgmap v957: 148 pgs: 148 active+clean; 3.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T13:19:05.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:19:05.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:05 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:05.851 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:05.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:05.952 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:05.991 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:05.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:05 smithi204 ceph-mon[33153]: 6.1b scrub ok 2024-03-21T13:19:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:05 smithi204 ceph-mon[33153]: 6.f scrub starts 2024-03-21T13:19:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:05 smithi204 ceph-mon[33153]: 6.f scrub ok 2024-03-21T13:19:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:05 smithi204 ceph-mon[33153]: 5.e scrub starts 2024-03-21T13:19:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:05 smithi204 ceph-mon[33153]: 5.e scrub ok 2024-03-21T13:19:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:05 smithi204 ceph-mon[33153]: pgmap v957: 148 pgs: 148 active+clean; 3.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T13:19:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:19:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:05 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:06.092 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:06.126 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:06.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:06.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:06.267 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:06.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:06.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:06.369 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:06.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:06.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:06.508 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:06.545 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:06.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:06.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:06.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:06.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:06.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:06.793 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:06.839 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:06.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:06.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:06.937 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:06 smithi163 ceph-mon[29621]: 4.18 scrub starts 2024-03-21T13:19:06.937 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:06 smithi163 ceph-mon[29621]: 4.18 scrub ok 2024-03-21T13:19:06.937 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:06 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:19:06.937 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:06 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:19:06.937 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:06 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:19:06.938 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:06 smithi163 ceph-mon[40675]: 4.18 scrub starts 2024-03-21T13:19:06.938 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:06 smithi163 ceph-mon[40675]: 4.18 scrub ok 2024-03-21T13:19:06.938 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:06 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:19:06.938 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:06 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:19:06.938 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:06 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:19:06.941 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:06.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:06.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:06.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:06 smithi204 ceph-mon[33153]: 4.18 scrub starts 2024-03-21T13:19:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:06 smithi204 ceph-mon[33153]: 4.18 scrub ok 2024-03-21T13:19:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:06 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:19:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:06 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:19:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:06 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:19:07.076 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:07.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:07.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:07.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:07.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:07.254 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:07.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:07.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:07.280 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:07.343 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:07.356 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:07.393 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:07.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:07.495 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:07.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:07.525 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:07.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:07.626 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:07.657 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:07.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:07.759 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:07.788 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:07.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:07.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:07.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:07.927 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:07.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:07.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:07 smithi204 ceph-mon[33153]: 5.1b scrub starts 2024-03-21T13:19:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:07 smithi204 ceph-mon[33153]: 5.1b scrub ok 2024-03-21T13:19:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:07 smithi204 ceph-mon[33153]: 5.5 scrub starts 2024-03-21T13:19:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:07 smithi204 ceph-mon[33153]: 5.5 scrub ok 2024-03-21T13:19:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:07 smithi204 ceph-mon[33153]: pgmap v958: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-03-21T13:19:08.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:07 smithi204 ceph-mon[33153]: 11.e scrub starts 2024-03-21T13:19:08.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:07 smithi163 ceph-mon[29621]: 5.1b scrub starts 2024-03-21T13:19:08.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:07 smithi163 ceph-mon[29621]: 5.1b scrub ok 2024-03-21T13:19:08.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:07 smithi163 ceph-mon[29621]: 5.5 scrub starts 2024-03-21T13:19:08.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:07 smithi163 ceph-mon[29621]: 5.5 scrub ok 2024-03-21T13:19:08.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:07 smithi163 ceph-mon[29621]: pgmap v958: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-03-21T13:19:08.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:07 smithi163 ceph-mon[29621]: 11.e scrub starts 2024-03-21T13:19:08.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:07 smithi163 ceph-mon[40675]: 5.1b scrub starts 2024-03-21T13:19:08.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:07 smithi163 ceph-mon[40675]: 5.1b scrub ok 2024-03-21T13:19:08.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:07 smithi163 ceph-mon[40675]: 5.5 scrub starts 2024-03-21T13:19:08.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:07 smithi163 ceph-mon[40675]: 5.5 scrub ok 2024-03-21T13:19:08.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:07 smithi163 ceph-mon[40675]: pgmap v958: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-03-21T13:19:08.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:07 smithi163 ceph-mon[40675]: 11.e scrub starts 2024-03-21T13:19:08.029 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:08.056 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:08.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:08.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:08.158 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:08.188 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:08.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:08.289 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:08.322 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:08.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:08.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:08.424 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:08.454 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:08.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:08.556 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:08.589 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:08.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:08.690 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:08.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:08 smithi163 ceph-mon[29621]: 5.13 scrub starts 2024-03-21T13:19:08.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:08 smithi163 ceph-mon[29621]: 5.13 scrub ok 2024-03-21T13:19:08.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:08 smithi163 ceph-mon[29621]: 11.e scrub ok 2024-03-21T13:19:08.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:08 smithi163 ceph-mon[40675]: 5.13 scrub starts 2024-03-21T13:19:08.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:08 smithi163 ceph-mon[40675]: 5.13 scrub ok 2024-03-21T13:19:08.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:08 smithi163 ceph-mon[40675]: 11.e scrub ok 2024-03-21T13:19:08.727 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:08.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:08.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:08.829 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:08.867 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:08.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:08.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:08.969 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:08 smithi204 ceph-mon[33153]: 5.13 scrub starts 2024-03-21T13:19:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:08 smithi204 ceph-mon[33153]: 5.13 scrub ok 2024-03-21T13:19:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:08 smithi204 ceph-mon[33153]: 11.e scrub ok 2024-03-21T13:19:09.003 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:09.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:09.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:09.138 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:09.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:09.240 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:09.272 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:09.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:09.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:09.374 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:09.403 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:09.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:09.505 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:09.534 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:09.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:09.636 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:09.671 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:09.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:09.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:09.679 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:19:09.727 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:19:09.727 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:19:09.727 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:19:09.727 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:19:09.727 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:19:09.727 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:19:09.728 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:19:09.728 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:19:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:19:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:19:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:19:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:19:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:19:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:19:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:19:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:19:09.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:09.812 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:09.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:09.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:09.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:09.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:09.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:09.960 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[29621]: 3.1d scrub starts 2024-03-21T13:19:09.960 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[29621]: 3.1d scrub ok 2024-03-21T13:19:09.960 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[29621]: 11.1 scrub starts 2024-03-21T13:19:09.960 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[29621]: pgmap v959: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-21T13:19:09.960 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/720702842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:19:09.960 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[29621]: 4.b scrub starts 2024-03-21T13:19:09.960 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[29621]: 4.b scrub ok 2024-03-21T13:19:09.960 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[29621]: 1.0 scrub starts 2024-03-21T13:19:09.961 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[29621]: 1.0 scrub ok 2024-03-21T13:19:09.961 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1441797345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:19:09.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[40675]: 3.1d scrub starts 2024-03-21T13:19:09.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[40675]: 3.1d scrub ok 2024-03-21T13:19:09.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[40675]: 11.1 scrub starts 2024-03-21T13:19:09.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[40675]: pgmap v959: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-21T13:19:09.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/720702842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:19:09.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[40675]: 4.b scrub starts 2024-03-21T13:19:09.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[40675]: 4.b scrub ok 2024-03-21T13:19:09.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[40675]: 1.0 scrub starts 2024-03-21T13:19:09.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[40675]: 1.0 scrub ok 2024-03-21T13:19:09.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:09 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1441797345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:19:10.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:09 smithi204 ceph-mon[33153]: 3.1d scrub starts 2024-03-21T13:19:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:09 smithi204 ceph-mon[33153]: 3.1d scrub ok 2024-03-21T13:19:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:09 smithi204 ceph-mon[33153]: 11.1 scrub starts 2024-03-21T13:19:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:09 smithi204 ceph-mon[33153]: pgmap v959: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-21T13:19:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:09 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/720702842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:19:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:09 smithi204 ceph-mon[33153]: 4.b scrub starts 2024-03-21T13:19:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:09 smithi204 ceph-mon[33153]: 4.b scrub ok 2024-03-21T13:19:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:09 smithi204 ceph-mon[33153]: 1.0 scrub starts 2024-03-21T13:19:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:09 smithi204 ceph-mon[33153]: 1.0 scrub ok 2024-03-21T13:19:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:09 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1441797345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:19:10.053 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:10.082 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:10.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:10.184 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:10.214 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:10.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:10.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:10.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:10.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:10.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:10.490 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:10.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:10.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:10.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:10.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:10.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:10.763 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:10.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:10.865 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:10.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:10 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:19:10.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:10 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:19:10.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:10 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1441797345' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:19:10.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:10 smithi163 ceph-mon[29621]: osdmap e441: 8 total, 8 up, 6 in 2024-03-21T13:19:10.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:10 smithi163 ceph-mon[29621]: 11.1 scrub ok 2024-03-21T13:19:10.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:10 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:19:10.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:10 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:19:10.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:10 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1441797345' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:19:10.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:10 smithi163 ceph-mon[40675]: osdmap e441: 8 total, 8 up, 6 in 2024-03-21T13:19:10.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:10 smithi163 ceph-mon[40675]: 11.1 scrub ok 2024-03-21T13:19:10.896 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:10.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:10.998 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:10 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:19:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:10 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:19:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:10 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1441797345' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:19:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:10 smithi204 ceph-mon[33153]: osdmap e441: 8 total, 8 up, 6 in 2024-03-21T13:19:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:10 smithi204 ceph-mon[33153]: 11.1 scrub ok 2024-03-21T13:19:11.049 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:11.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:11.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:11.151 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:11.180 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:11.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:11.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:11.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:11.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:11.423 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:11.460 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:11.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:11.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:11.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:11.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:11.700 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:11.732 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:11.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:11.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:11.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:11.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:11.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:11 smithi204 ceph-mon[33153]: 3.4 scrub starts 2024-03-21T13:19:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:11 smithi204 ceph-mon[33153]: 3.4 scrub ok 2024-03-21T13:19:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:11 smithi204 ceph-mon[33153]: 3.1f scrub starts 2024-03-21T13:19:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:11 smithi204 ceph-mon[33153]: 3.1f scrub ok 2024-03-21T13:19:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:11 smithi204 ceph-mon[33153]: 4.11 scrub starts 2024-03-21T13:19:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:11 smithi204 ceph-mon[33153]: 4.11 scrub ok 2024-03-21T13:19:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:11 smithi204 ceph-mon[33153]: pgmap v961: 148 pgs: 148 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-21T13:19:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:11 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:19:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:11 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:19:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:11 smithi204 ceph-mon[33153]: 3.15 scrub starts 2024-03-21T13:19:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:11 smithi204 ceph-mon[33153]: 3.15 scrub ok 2024-03-21T13:19:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[29621]: 3.4 scrub starts 2024-03-21T13:19:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[29621]: 3.4 scrub ok 2024-03-21T13:19:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[29621]: 3.1f scrub starts 2024-03-21T13:19:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[29621]: 3.1f scrub ok 2024-03-21T13:19:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[29621]: 4.11 scrub starts 2024-03-21T13:19:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[29621]: 4.11 scrub ok 2024-03-21T13:19:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[29621]: pgmap v961: 148 pgs: 148 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-21T13:19:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:19:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:19:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[29621]: 3.15 scrub starts 2024-03-21T13:19:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[29621]: 3.15 scrub ok 2024-03-21T13:19:12.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[40675]: 3.4 scrub starts 2024-03-21T13:19:12.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[40675]: 3.4 scrub ok 2024-03-21T13:19:12.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[40675]: 3.1f scrub starts 2024-03-21T13:19:12.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[40675]: 3.1f scrub ok 2024-03-21T13:19:12.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[40675]: 4.11 scrub starts 2024-03-21T13:19:12.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[40675]: 4.11 scrub ok 2024-03-21T13:19:12.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[40675]: pgmap v961: 148 pgs: 148 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-21T13:19:12.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:19:12.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:19:12.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[40675]: 3.15 scrub starts 2024-03-21T13:19:12.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:11 smithi163 ceph-mon[40675]: 3.15 scrub ok 2024-03-21T13:19:12.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:12.013 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:12.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:12.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:12.158 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:12.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:12.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:12.294 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:12.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:12.397 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:12.423 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:12.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:12.524 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:12.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:12.554 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:12.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:12.655 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:12.686 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:12.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:12.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:12.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:19:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:19:12.694 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.6 to 1 (10000) 2024-03-21T13:19:12.744 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:19:12.744 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:19:12.745 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-03-21T13:19:12.745 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-03-21T13:19:12.745 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:19:12.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:12.830 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:12.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:12.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:12.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:12.972 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:12.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:13.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:12 smithi204 ceph-mon[33153]: 3.2 scrub starts 2024-03-21T13:19:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:12 smithi204 ceph-mon[33153]: 3.2 scrub ok 2024-03-21T13:19:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:12 smithi204 ceph-mon[33153]: 6.b deep-scrub starts 2024-03-21T13:19:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:12 smithi204 ceph-mon[33153]: 11.3 scrub starts 2024-03-21T13:19:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:12 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:12 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1655550638' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:19:13.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:12 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:19:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[29621]: 3.2 scrub starts 2024-03-21T13:19:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[29621]: 3.2 scrub ok 2024-03-21T13:19:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[29621]: 6.b deep-scrub starts 2024-03-21T13:19:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[29621]: 11.3 scrub starts 2024-03-21T13:19:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1655550638' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:19:13.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:19:13.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[40675]: 3.2 scrub starts 2024-03-21T13:19:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[40675]: 3.2 scrub ok 2024-03-21T13:19:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[40675]: 6.b deep-scrub starts 2024-03-21T13:19:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[40675]: 11.3 scrub starts 2024-03-21T13:19:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1655550638' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:19:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:12 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:19:13.073 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:13.103 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:13.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:13.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:13.237 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:13.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:13.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:13.339 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:13.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:13.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:13.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:13.470 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:13.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:13.500 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:13.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:13.599 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-21T13:19:13.601 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:13.630 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:13.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:13.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:13.733 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:13.766 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:13.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:13.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:13.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:13.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[29621]: 6.b deep-scrub ok 2024-03-21T13:19:13.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[29621]: 4.6 scrub starts 2024-03-21T13:19:13.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[29621]: 4.6 scrub ok 2024-03-21T13:19:13.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[29621]: 4.2 scrub starts 2024-03-21T13:19:13.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[29621]: 4.2 scrub ok 2024-03-21T13:19:13.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:19:13.910 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[29621]: osdmap e442: 8 total, 8 up, 6 in 2024-03-21T13:19:13.910 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[29621]: 11.3 scrub ok 2024-03-21T13:19:13.910 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[29621]: pgmap v963: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-03-21T13:19:13.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[40675]: 6.b deep-scrub ok 2024-03-21T13:19:13.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[40675]: 4.6 scrub starts 2024-03-21T13:19:13.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[40675]: 4.6 scrub ok 2024-03-21T13:19:13.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[40675]: 4.2 scrub starts 2024-03-21T13:19:13.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[40675]: 4.2 scrub ok 2024-03-21T13:19:13.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:19:13.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[40675]: osdmap e442: 8 total, 8 up, 6 in 2024-03-21T13:19:13.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[40675]: 11.3 scrub ok 2024-03-21T13:19:13.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:13 smithi163 ceph-mon[40675]: pgmap v963: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-03-21T13:19:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:13.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:13 smithi204 ceph-mon[33153]: 6.b deep-scrub ok 2024-03-21T13:19:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:13 smithi204 ceph-mon[33153]: 4.6 scrub starts 2024-03-21T13:19:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:13 smithi204 ceph-mon[33153]: 4.6 scrub ok 2024-03-21T13:19:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:13 smithi204 ceph-mon[33153]: 4.2 scrub starts 2024-03-21T13:19:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:13 smithi204 ceph-mon[33153]: 4.2 scrub ok 2024-03-21T13:19:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:13 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:19:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:13 smithi204 ceph-mon[33153]: osdmap e442: 8 total, 8 up, 6 in 2024-03-21T13:19:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:13 smithi204 ceph-mon[33153]: 11.3 scrub ok 2024-03-21T13:19:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:13 smithi204 ceph-mon[33153]: pgmap v963: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-03-21T13:19:14.012 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:14.046 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:14.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:14.148 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:14.177 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:14.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:14.273 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-21T13:19:14.279 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:14.356 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:14.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:14.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:14.457 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:14.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:14.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:14.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:14.627 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:14.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:14.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:14.729 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:14.760 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:14.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:14.861 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:14.897 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:14.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:14.967 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:14 smithi163 ceph-mon[29621]: 6.17 scrub starts 2024-03-21T13:19:14.967 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:14 smithi163 ceph-mon[29621]: 6.17 scrub ok 2024-03-21T13:19:14.967 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:14 smithi163 ceph-mon[29621]: 6.1d scrub starts 2024-03-21T13:19:14.967 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:14 smithi163 ceph-mon[29621]: 6.1d scrub ok 2024-03-21T13:19:14.968 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:14 smithi163 ceph-mon[29621]: 3.1a deep-scrub starts 2024-03-21T13:19:14.968 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:14 smithi163 ceph-mon[29621]: 3.1a deep-scrub ok 2024-03-21T13:19:14.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:14 smithi163 ceph-mon[40675]: 6.17 scrub starts 2024-03-21T13:19:14.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:14 smithi163 ceph-mon[40675]: 6.17 scrub ok 2024-03-21T13:19:14.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:14 smithi163 ceph-mon[40675]: 6.1d scrub starts 2024-03-21T13:19:14.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:14 smithi163 ceph-mon[40675]: 6.1d scrub ok 2024-03-21T13:19:14.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:14 smithi163 ceph-mon[40675]: 3.1a deep-scrub starts 2024-03-21T13:19:14.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:14 smithi163 ceph-mon[40675]: 3.1a deep-scrub ok 2024-03-21T13:19:15.000 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:14 smithi204 ceph-mon[33153]: 6.17 scrub starts 2024-03-21T13:19:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:14 smithi204 ceph-mon[33153]: 6.17 scrub ok 2024-03-21T13:19:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:14 smithi204 ceph-mon[33153]: 6.1d scrub starts 2024-03-21T13:19:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:14 smithi204 ceph-mon[33153]: 6.1d scrub ok 2024-03-21T13:19:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:14 smithi204 ceph-mon[33153]: 3.1a deep-scrub starts 2024-03-21T13:19:15.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:14 smithi204 ceph-mon[33153]: 3.1a deep-scrub ok 2024-03-21T13:19:15.037 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:15.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:15.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:15.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:15.182 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:15.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:15.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:15.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:15.319 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:15.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:15.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:15.421 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:15.428 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:15.458 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:15.481 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:15.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:15.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:15.583 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:15.616 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:15.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:15.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:15.757 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:15.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:15.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:15.856 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:15 smithi163 ceph-mon[29621]: 3.1c deep-scrub starts 2024-03-21T13:19:15.856 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:15 smithi163 ceph-mon[29621]: 3.1c deep-scrub ok 2024-03-21T13:19:15.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:15 smithi163 ceph-mon[29621]: pgmap v964: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 803 op/s 2024-03-21T13:19:15.857 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:15 smithi163 ceph-mon[40675]: 3.1c deep-scrub starts 2024-03-21T13:19:15.857 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:15 smithi163 ceph-mon[40675]: 3.1c deep-scrub ok 2024-03-21T13:19:15.857 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:15 smithi163 ceph-mon[40675]: pgmap v964: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 803 op/s 2024-03-21T13:19:15.857 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:19:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:19:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:19:15.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:15.896 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:15.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:15.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:15.998 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:16.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:15 smithi204 ceph-mon[33153]: 3.1c deep-scrub starts 2024-03-21T13:19:16.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:15 smithi204 ceph-mon[33153]: 3.1c deep-scrub ok 2024-03-21T13:19:16.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:15 smithi204 ceph-mon[33153]: pgmap v964: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 803 op/s 2024-03-21T13:19:16.031 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:16.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:16.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:16.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:16.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:16.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:16.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:16.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:16.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:16.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:16.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:16.456 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:16.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:16.557 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:16.600 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:16.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:16.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:16.702 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:16.726 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:16.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:16.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:16.828 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:16.856 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:16.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:16.870 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T13:19:16.958 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:16.992 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:16.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:16.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:17.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:16 smithi204 ceph-mon[33153]: 11.6 scrub starts 2024-03-21T13:19:17.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:16 smithi204 ceph-mon[33153]: 2.2 scrub starts 2024-03-21T13:19:17.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:16 smithi204 ceph-mon[33153]: 2.2 scrub ok 2024-03-21T13:19:17.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:16 smithi204 ceph-mon[33153]: 11.6 scrub ok 2024-03-21T13:19:17.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:16 smithi204 ceph-mon[33153]: 11.4 scrub starts 2024-03-21T13:19:17.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:16 smithi163 ceph-mon[29621]: 11.6 scrub starts 2024-03-21T13:19:17.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:16 smithi163 ceph-mon[29621]: 2.2 scrub starts 2024-03-21T13:19:17.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:16 smithi163 ceph-mon[29621]: 2.2 scrub ok 2024-03-21T13:19:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:16 smithi163 ceph-mon[29621]: 11.6 scrub ok 2024-03-21T13:19:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:16 smithi163 ceph-mon[29621]: 11.4 scrub starts 2024-03-21T13:19:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:16 smithi163 ceph-mon[40675]: 11.6 scrub starts 2024-03-21T13:19:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:16 smithi163 ceph-mon[40675]: 2.2 scrub starts 2024-03-21T13:19:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:16 smithi163 ceph-mon[40675]: 2.2 scrub ok 2024-03-21T13:19:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:16 smithi163 ceph-mon[40675]: 11.6 scrub ok 2024-03-21T13:19:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:16 smithi163 ceph-mon[40675]: 11.4 scrub starts 2024-03-21T13:19:17.095 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:17.128 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:17.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:17.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:17.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:17.267 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:17.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:17.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:17.360 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:19:17.369 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:17.438 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:17.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:17.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:17.540 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:17.574 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:17.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:17.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:17.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:17.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:17.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:17.840 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:17.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:17.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:17.941 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:17.980 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:17.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:18.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:17 smithi204 ceph-mon[33153]: 6.d scrub starts 2024-03-21T13:19:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:17 smithi204 ceph-mon[33153]: 6.d scrub ok 2024-03-21T13:19:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:17 smithi204 ceph-mon[33153]: 4.5 scrub starts 2024-03-21T13:19:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:17 smithi204 ceph-mon[33153]: 4.5 scrub ok 2024-03-21T13:19:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:17 smithi204 ceph-mon[33153]: 3.1 scrub starts 2024-03-21T13:19:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:17 smithi204 ceph-mon[33153]: 3.1 scrub ok 2024-03-21T13:19:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:17 smithi204 ceph-mon[33153]: pgmap v965: 148 pgs: 148 active+clean; 2.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-03-21T13:19:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:17 smithi204 ceph-mon[33153]: 11.4 scrub ok 2024-03-21T13:19:18.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[29621]: 6.d scrub starts 2024-03-21T13:19:18.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[29621]: 6.d scrub ok 2024-03-21T13:19:18.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[29621]: 4.5 scrub starts 2024-03-21T13:19:18.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[29621]: 4.5 scrub ok 2024-03-21T13:19:18.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[29621]: 3.1 scrub starts 2024-03-21T13:19:18.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[29621]: 3.1 scrub ok 2024-03-21T13:19:18.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[29621]: pgmap v965: 148 pgs: 148 active+clean; 2.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-03-21T13:19:18.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[29621]: 11.4 scrub ok 2024-03-21T13:19:18.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[40675]: 6.d scrub starts 2024-03-21T13:19:18.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[40675]: 6.d scrub ok 2024-03-21T13:19:18.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[40675]: 4.5 scrub starts 2024-03-21T13:19:18.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[40675]: 4.5 scrub ok 2024-03-21T13:19:18.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[40675]: 3.1 scrub starts 2024-03-21T13:19:18.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[40675]: 3.1 scrub ok 2024-03-21T13:19:18.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[40675]: pgmap v965: 148 pgs: 148 active+clean; 2.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-03-21T13:19:18.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:17 smithi163 ceph-mon[40675]: 11.4 scrub ok 2024-03-21T13:19:18.082 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:18.118 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:18.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:18.219 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:18.259 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:18.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:18.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:18.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:18.390 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:18.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:18.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:18.492 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:18.528 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:18.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:18.630 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:18.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:18.659 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:18.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:18.760 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:18.793 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:18.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:18.895 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:18.929 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:18.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:18.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:19.030 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:19.067 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:19.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:19.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:19.099 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:18 smithi163 ceph-mon[29621]: 11.b scrub starts 2024-03-21T13:19:19.100 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:18 smithi163 ceph-mon[40675]: 11.b scrub starts 2024-03-21T13:19:19.169 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:19.198 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:19.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:19.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:18 smithi204 ceph-mon[33153]: 11.b scrub starts 2024-03-21T13:19:19.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:19.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:19.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:19.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:19.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:19.452 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:19:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-03-21T13:19:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-03-21T13:19:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-03-21T13:19:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24325.0:2726 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94483560961536 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:19:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:02.908756+0000", 2024-03-21T13:19:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 556.53768171199999, 2024-03-21T13:19:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.72060023299999998, 2024-03-21T13:19:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:19:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24325", 2024-03-21T13:19:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/1345925145", 2024-03-21T13:19:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2726 2024-03-21T13:19:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908756+0000", 2024-03-21T13:19:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908756+0000", 2024-03-21T13:19:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908758+0000", 2024-03-21T13:19:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4410000000000001e-06 2024-03-21T13:19:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908765+0000", 2024-03-21T13:19:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0600000000000002e-06 2024-03-21T13:19:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908782+0000", 2024-03-21T13:19:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7317999999999999e-05 2024-03-21T13:19:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908793+0000", 2024-03-21T13:19:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0492000000000001e-05 2024-03-21T13:19:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908855+0000", 2024-03-21T13:19:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2321000000000002e-05 2024-03-21T13:19:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:19:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908863+0000", 2024-03-21T13:19:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6690000000000006e-06 2024-03-21T13:19:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.609271+0000", 2024-03-21T13:19:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.70040877999999995 2024-03-21T13:19:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for missing object", 2024-03-21T13:19:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.609413+0000", 2024-03-21T13:19:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141738 2024-03-21T13:19:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.627023+0000", 2024-03-21T13:19:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017609763 2024-03-21T13:19:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-03-21T13:19:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.627177+0000", 2024-03-21T13:19:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015424699999999999 2024-03-21T13:19:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.627685+0000", 2024-03-21T13:19:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050788099999999996 2024-03-21T13:19:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.627782+0000", 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6812999999999996e-05 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.629357+0000", 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00157471 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24325.0:2727 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94483560959232 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:10:02.908815+0000", 2024-03-21T13:19:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 556.53762305400005, 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.71802334999999995, 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24325", 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/1345925145", 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2727 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908815+0000", 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908815+0000", 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908816+0000", 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.89e-07 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908819+0000", 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7129999999999999e-06 2024-03-21T13:19:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908826+0000", 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3790000000000003e-06 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908831+0000", 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8239999999999996e-06 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908877+0000", 2024-03-21T13:19:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5116e-05 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:02.908883+0000", 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1090000000000003e-06 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.610133+0000", 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.701249868 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for missing object", 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.610247+0000", 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114774 2024-03-21T13:19:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.622278+0000", 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012030394999999999 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.622383+0000", 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105176 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.623065+0000", 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00068221899999999997 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.623139+0000", 2024-03-21T13:19:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3801999999999996e-05 2024-03-21T13:19:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:10:03.626838+0000", 2024-03-21T13:19:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0036994860000000001 2024-03-21T13:19:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25481.0:78778 10.3 10:cdfc1d43:::benchmark_data_smithi163_154138_object78777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-21T13:19:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:13:15.133042+0000", 2024-03-21T13:19:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 364.31339632300001, 2024-03-21T13:19:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.196393335, 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25481", 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 78778 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.133042+0000", 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.133042+0000", 2024-03-21T13:19:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.133043+0000", 2024-03-21T13:19:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6700000000000002e-07 2024-03-21T13:19:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.133091+0000", 2024-03-21T13:19:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8238999999999997e-05 2024-03-21T13:19:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.133101+0000", 2024-03-21T13:19:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0074e-05 2024-03-21T13:19:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.133108+0000", 2024-03-21T13:19:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3780000000000003e-06 2024-03-21T13:19:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.133171+0000", 2024-03-21T13:19:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2817999999999994e-05 2024-03-21T13:19:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.133314+0000", 2024-03-21T13:19:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014239799999999999 2024-03-21T13:19:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,5", 2024-03-21T13:19:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.133534+0000", 2024-03-21T13:19:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00022081299999999999 2024-03-21T13:19:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.136850+0000", 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033150609999999998 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.142109+0000", 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052595410000000004 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.329238+0000", 2024-03-21T13:19:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.18712920899999999 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.329300+0000", 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1407000000000003e-05 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:13:15.329435+0000", 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013543 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25481.0:191611 10.8 10:13088c9a:::benchmark_data_smithi163_154138_object59812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-21T13:19:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:14:49.854328+0000", 2024-03-21T13:19:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 269.592110225, 2024-03-21T13:19:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.22767456999999999, 2024-03-21T13:19:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25481", 2024-03-21T13:19:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:19:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 191611 2024-03-21T13:19:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.854328+0000", 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.854328+0000", 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.854329+0000", 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3990000000000001e-06 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.854336+0000", 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3709999999999996e-06 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.854347+0000", 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1705000000000001e-05 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.854356+0000", 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4100000000000008e-06 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.854425+0000", 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9442000000000004e-05 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.854544+0000", 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119047 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,7", 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.854657+0000", 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112339 2024-03-21T13:19:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081682+0000", 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.22702566800000001 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081729+0000", 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7055999999999998e-05 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081895+0000", 2024-03-21T13:19:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000165866 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081949+0000", 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3797e-05 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.082002+0000", 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3470000000000001e-05 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25481.0:191614 10.5 10:a00b29b8:::benchmark_data_smithi163_154138_object59815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:14:49.856999+0000", 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 269.58943863600001, 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.24713754499999999, 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25481", 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 191614 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.856999+0000", 2024-03-21T13:19:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.856999+0000", 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.857001+0000", 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4300000000000001e-06 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.857008+0000", 2024-03-21T13:19:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1420000000000004e-06 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.857022+0000", 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4173e-05 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.857039+0000", 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6719999999999999e-05 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081680+0000", 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.22464155199999999 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081833+0000", 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000152526 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081960+0000", 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012676300000000001 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.092678+0000", 2024-03-21T13:19:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010718581 2024-03-21T13:19:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.097196+0000", 2024-03-21T13:19:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045174739999999996 2024-03-21T13:19:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.103984+0000", 2024-03-21T13:19:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0067882139999999999 2024-03-21T13:19:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.104067+0000", 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3312999999999993e-05 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.104137+0000", 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9657000000000001e-05 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25481.0:191615 10.3 10:c1e3fb4d:::benchmark_data_smithi163_154138_object59816:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-21T13:19:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:14:49.857168+0000", 2024-03-21T13:19:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 269.589269785, 2024-03-21T13:19:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.23633910699999999, 2024-03-21T13:19:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25481", 2024-03-21T13:19:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/974724800", 2024-03-21T13:19:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 191615 2024-03-21T13:19:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.857168+0000", 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.857168+0000", 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.857169+0000", 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2999999999999999e-07 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.857174+0000", 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9280000000000001e-06 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081640+0000", 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.22446626 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081657+0000", 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6694999999999999e-05 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081730+0000", 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2815e-05 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081842+0000", 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011214600000000001 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,5", 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081968+0000", 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012590999999999999 2024-03-21T13:19:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.093303+0000", 2024-03-21T13:19:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011334815 2024-03-21T13:19:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.093358+0000", 2024-03-21T13:19:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5575999999999998e-05 2024-03-21T13:19:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.093396+0000", 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7595999999999997e-05 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.093456+0000", 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0591000000000003e-05 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.093507+0000", 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0945000000000001e-05 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.25481.0:191611 10.8 e369/324 ondisk, result = 0)", 2024-03-21T13:19:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:14:49.857932+0000", 2024-03-21T13:19:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 269.58850567399998, 2024-03-21T13:19:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.223805897, 2024-03-21T13:19:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-03-21T13:19:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.857932+0000", 2024-03-21T13:19:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.857932+0000", 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.857933+0000", 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3799999999999996e-07 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.857939+0000", 2024-03-21T13:19:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5820000000000001e-06 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081651+0000", 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.22371253199999999 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081662+0000", 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0373999999999999e-05 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081713+0000", 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1573999999999998e-05 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081723+0000", 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8190000000000003e-06 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081738+0000", 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5078e-05 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.25481.0:191632 10.9 e369/330 10:98911861:::benchmark_data_smithi163_154138_object59833:head v 369'11906, mlcod=369'11904)", 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:14:49.880351+0000", 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 269.566086724, 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.213780795, 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.880351+0000", 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.880351+0000", 2024-03-21T13:19:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.880353+0000", 2024-03-21T13:19:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.22e-06 2024-03-21T13:19:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:49.880363+0000", 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0129000000000001e-05 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081643+0000", 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.201280403 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081657+0000", 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3899e-05 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081721+0000", 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4104999999999997e-05 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.081746+0000", 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5347999999999999e-05 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.094094+0000", 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012347639000000001 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:50.094132+0000", 2024-03-21T13:19:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8052000000000001e-05 2024-03-21T13:19:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.25481.0:196234 10.f e370/324 10:f6491fc0:::benchmark_data_smithi163_154138_object64435:head v 370'12199, mlcod=370'12197)", 2024-03-21T13:19:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:14:53.597866+0000", 2024-03-21T13:19:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 265.848572241, 2024-03-21T13:19:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.17444104099999999, 2024-03-21T13:19:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:53.597866+0000", 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:53.597866+0000", 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:53.597866+0000", 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8599999999999998e-07 2024-03-21T13:19:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:53.597876+0000", 2024-03-21T13:19:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2089999999999994e-06 2024-03-21T13:19:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:53.597883+0000", 2024-03-21T13:19:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2239999999999998e-06 2024-03-21T13:19:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:53.597889+0000", 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8590000000000001e-06 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:53.598008+0000", 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0001189 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:53.598035+0000", 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7277999999999999e-05 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:53.772276+0000", 2024-03-21T13:19:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.174241022 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:14:53.772307+0000", 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1062999999999997e-05 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:93001 11.2 11:45e9c1ef:::benchmark_data_smithi163_180069_object93000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:26.616232+0000", 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 112.83020634499999, 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1867763340000002, 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-03-21T13:19:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:19:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:19:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 93001 2024-03-21T13:19:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.616232+0000", 2024-03-21T13:19:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.616232+0000", 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.616272+0000", 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9993000000000001e-05 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.616295+0000", 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3122e-05 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.616305+0000", 2024-03-21T13:19:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0502999999999999e-05 2024-03-21T13:19:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.616331+0000", 2024-03-21T13:19:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6024e-05 2024-03-21T13:19:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.616491+0000", 2024-03-21T13:19:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015983100000000001 2024-03-21T13:19:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.616652+0000", 2024-03-21T13:19:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000160459 2024-03-21T13:19:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-03-21T13:19:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.616814+0000", 2024-03-21T13:19:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016221000000000001 2024-03-21T13:19:19.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:19.510 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:19.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.625488+0000", 2024-03-21T13:19:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0086745989999999999 2024-03-21T13:19:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.643989+0000", 2024-03-21T13:19:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018500909999999999 2024-03-21T13:19:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:19:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.644004+0000", 2024-03-21T13:19:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4463e-05 2024-03-21T13:19:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:27.623937+0000", 2024-03-21T13:19:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.97993298500000003 2024-03-21T13:19:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:27.623954+0000", 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6719e-05 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.802918+0000", 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.178964063 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.803008+0000", 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0452999999999996e-05 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.513 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:93015 11.6 11:66256075:::benchmark_data_smithi163_180069_object93014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:26.653613+0000", 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 112.792824739, 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3217963560000001, 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 93015 2024-03-21T13:19:19.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.653613+0000", 2024-03-21T13:19:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.653613+0000", 2024-03-21T13:19:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.653615+0000", 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8530000000000001e-06 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.653658+0000", 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2564000000000001e-05 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.653673+0000", 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4912999999999999e-05 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.653683+0000", 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0180000000000001e-05 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.653760+0000", 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7620999999999994e-05 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:26.653771+0000", 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0723999999999999e-05 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:27.624495+0000", 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.97072357099999995 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:27.624505+0000", 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0154e-05 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.780944+0000", 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1564389930000001 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.783073+0000", 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021287770000000001 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,4,5", 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.783275+0000", 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020206900000000001 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.787121+0000", 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0038458239999999999 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.790414+0000", 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032929510000000001 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.790480+0000", 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6169999999999998e-05 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.975177+0000", 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.18469718700000001 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.975291+0000", 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000113881 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.975410+0000", 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118924 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:93024 11.2 11:4dd9b9da:::benchmark_data_smithi163_180069_object93023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:28.723242+0000", 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 110.72319630600001, 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.253725916, 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 93024 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.723242+0000", 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.723242+0000", 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.723244+0000", 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.035e-06 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.723273+0000", 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9039000000000001e-05 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.723288+0000", 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5115e-05 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.723301+0000", 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2694000000000001e-05 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.723384+0000", 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3540000000000003e-05 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.723397+0000", 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2537999999999999e-05 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.803386+0000", 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.079989242000000002 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.803533+0000", 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014713500000000001 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,7", 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.804051+0000", 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00051825700000000005 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.814169+0000", 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010117625 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.817381+0000", 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032117740000000001 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.817443+0000", 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2141e-05 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.976785+0000", 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.15934215099999999 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.976850+0000", 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5048000000000005e-05 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.976968+0000", 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000117582 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:93032 11.2 11:46f20db8:::benchmark_data_smithi163_180069_object93031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:28.760700+0000", 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 110.685738055, 2024-03-21T13:19:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.216499045, 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 93032 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.760700+0000", 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.760700+0000", 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.760701+0000", 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3209999999999999e-06 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.760729+0000", 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7319999999999999e-05 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.760744+0000", 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5370999999999999e-05 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.760754+0000", 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9799999999999993e-06 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.760826+0000", 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2276999999999999e-05 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.760836+0000", 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9199999999999999e-06 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.814229+0000", 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.053393193999999998 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.814401+0000", 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017169200000000001 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,7", 2024-03-21T13:19:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.814664+0000", 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000263272 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.817507+0000", 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002843043 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.817525+0000", 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8000999999999999e-05 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.824475+0000", 2024-03-21T13:19:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0069496050000000002 2024-03-21T13:19:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.977014+0000", 2024-03-21T13:19:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.15253862100000001 2024-03-21T13:19:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.977089+0000", 2024-03-21T13:19:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5054000000000003e-05 2024-03-21T13:19:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.977199+0000", 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110374 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:93041 11.6 11:68a59478:::benchmark_data_smithi163_180069_object93040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:28.790151+0000", 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 110.65628716499999, 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.18611889300000001, 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 93041 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.790151+0000", 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.790151+0000", 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.790151+0000", 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7899999999999999e-07 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.790172+0000", 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0509999999999998e-05 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.790179+0000", 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0779999999999998e-06 2024-03-21T13:19:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.790185+0000", 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6130000000000003e-06 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.790515+0000", 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000330513 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.790667+0000", 2024-03-21T13:19:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000151464 2024-03-21T13:19:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,4,5", 2024-03-21T13:19:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.790842+0000", 2024-03-21T13:19:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017587500000000001 2024-03-21T13:19:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.800876+0000", 2024-03-21T13:19:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010033231 2024-03-21T13:19:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.800925+0000", 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9165000000000001e-05 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.800941+0000", 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6602000000000001e-05 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.976050+0000", 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.17510816200000001 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.976147+0000", 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6927000000000002e-05 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.976270+0000", 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012327400000000001 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:93044 11.6 11:62791aec:::benchmark_data_smithi163_180069_object93043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:28.803424+0000", 2024-03-21T13:19:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 110.64301364799999, 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.17312730400000001, 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 93044 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.803424+0000", 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.803424+0000", 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.803426+0000", 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4929999999999999e-06 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.803451+0000", 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4995e-05 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.803460+0000", 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2920000000000001e-06 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.803467+0000", 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9929999999999998e-06 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.803593+0000", 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000126215 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.803736+0000", 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014304000000000001 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,4,5", 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.804000+0000", 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00026401799999999998 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.807053+0000", 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003052679 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.545 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.807100+0000", 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6876000000000003e-05 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.810599+0000", 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034986449999999999 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.976315+0000", 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.16571676900000001 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.546 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.976420+0000", 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010408000000000001 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:28.976552+0000", 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013220899999999999 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:93183 11.2 11:46d7a472:::benchmark_data_smithi163_180069_object93182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-21T13:19:19.547 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:29.261734+0000", 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 110.184704252, 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.35491524000000002, 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 93183 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.261734+0000", 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.548 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.261734+0000", 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.261735+0000", 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.652e-06 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.261756+0000", 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-05 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.261767+0000", 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0516999999999999e-05 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.261774+0000", 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.024e-06 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.276580+0000", 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014806038000000001 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.276761+0000", 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018138599999999999 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,7", 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.276946+0000", 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018437999999999999 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.298675+0000", 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.021728860999999999 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.316582+0000", 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017907134000000002 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.335590+0000", 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019008516999999999 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.616344+0000", 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.28075355299999999 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.552 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.616495+0000", 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015136500000000001 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.616649+0000", 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015401299999999999 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:93204 11.2 11:4af33cbc:::benchmark_data_smithi163_180069_object93203:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-21T13:19:19.553 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:29.296281+0000", 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 110.15015679, 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.49992733099999997, 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 93204 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.296281+0000", 2024-03-21T13:19:19.554 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.296281+0000", 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.296283+0000", 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5880000000000001e-06 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.296312+0000", 2024-03-21T13:19:19.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8657999999999999e-05 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.296324+0000", 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2517000000000001e-05 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.296333+0000", 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7390000000000005e-06 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.298713+0000", 2024-03-21T13:19:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002380227 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.298868+0000", 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000155165 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,7", 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.299060+0000", 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000191907 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.316591+0000", 2024-03-21T13:19:19.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017530713 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.316665+0000", 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3955999999999995e-05 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.335644+0000", 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018979474999999999 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.795990+0000", 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.46034602099999999 2024-03-21T13:19:19.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.796073+0000", 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3131000000000001e-05 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.796209+0000", 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000135234 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:93213 11.2 11:4ac3c9c8:::benchmark_data_smithi163_180069_object93212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:29.319947+0000", 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 110.12649137299999, 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.476497632, 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 93213 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.560 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.319947+0000", 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.319947+0000", 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.319949+0000", 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9039999999999999e-06 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.320030+0000", 2024-03-21T13:19:19.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1306999999999999e-05 2024-03-21T13:19:19.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.562 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.320043+0000", 2024-03-21T13:19:19.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3293e-05 2024-03-21T13:19:19.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.562 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.320053+0000", 2024-03-21T13:19:19.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6970000000000005e-06 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.320342+0000", 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00028872700000000002 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.320482+0000", 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000140703 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,7", 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.320686+0000", 2024-03-21T13:19:19.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020335399999999999 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.335505+0000", 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014819087 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.335662+0000", 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015709 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.339061+0000", 2024-03-21T13:19:19.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033992240000000002 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.796262+0000", 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.45720115500000003 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.796324+0000", 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1655000000000005e-05 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.796444+0000", 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120436 2024-03-21T13:19:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:93232 11.6 11:603dbfce:::benchmark_data_smithi163_180069_object93231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:29.394386+0000", 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 110.052052012, 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.20487492400000001, 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 93232 2024-03-21T13:19:19.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.394386+0000", 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.394386+0000", 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.394388+0000", 2024-03-21T13:19:19.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.486e-06 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.394413+0000", 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5442000000000001e-05 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.394427+0000", 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4283e-05 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.394437+0000", 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0036e-05 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.394573+0000", 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013612699999999999 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.394832+0000", 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025885600000000002 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,4,5", 2024-03-21T13:19:19.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.395006+0000", 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017375299999999999 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.400021+0000", 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050148559999999998 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.400087+0000", 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6322000000000006e-05 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.400101+0000", 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3747999999999999e-05 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.598982+0000", 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.19888074 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.599124+0000", 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141977 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.599261+0000", 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000137298 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.15858.0:93238 11.2 11:4c2af2cd:::benchmark_data_smithi163_180069_object93237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-03-21T13:17:29.409010+0000", 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 110.037428379, 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.38770579300000002, 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-03-21T13:19:19.572 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.15858", 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/915669083", 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 93238 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409010+0000", 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409010+0000", 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-03-21T13:19:19.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-03-21T13:19:19.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409011+0000", 2024-03-21T13:19:19.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.339e-06 2024-03-21T13:19:19.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-03-21T13:19:19.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409045+0000", 2024-03-21T13:19:19.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4326999999999998e-05 2024-03-21T13:19:19.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-03-21T13:19:19.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409055+0000", 2024-03-21T13:19:19.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0003999999999999e-05 2024-03-21T13:19:19.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-03-21T13:19:19.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409063+0000", 2024-03-21T13:19:19.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8650000000000001e-06 2024-03-21T13:19:19.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-03-21T13:19:19.576 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409119+0000", 2024-03-21T13:19:19.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5609000000000002e-05 2024-03-21T13:19:19.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-03-21T13:19:19.576 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409271+0000", 2024-03-21T13:19:19.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015218600000000001 2024-03-21T13:19:19.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3,7", 2024-03-21T13:19:19.576 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.409461+0000", 2024-03-21T13:19:19.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018974700000000001 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.412757+0000", 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032959690000000002 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.412810+0000", 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3430000000000002e-05 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.412824+0000", 2024-03-21T13:19:19.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-03-21T13:19:19.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-03-21T13:19:19.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.796511+0000", 2024-03-21T13:19:19.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.38368714900000001 2024-03-21T13:19:19.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-03-21T13:19:19.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.796588+0000", 2024-03-21T13:19:19.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7737000000000005e-05 2024-03-21T13:19:19.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-03-21T13:19:19.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-03-21T13:19:19.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-03-21T13:19:19.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-03-21T13:17:29.796716+0000", 2024-03-21T13:19:19.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012709400000000001 2024-03-21T13:19:19.579 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.579 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.579 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.579 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-03-21T13:19:19.579 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-21T13:19:19.579 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:19:19.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:19.653 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:19.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:19.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:19.755 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:19.790 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:19.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:19.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:19.929 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:19.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:19.988 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:19.989 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:20.030 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:20.070 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:20.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:20.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:19 smithi163 ceph-mon[29621]: pgmap v966: 148 pgs: 148 active+clean; 2.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-21T13:19:20.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:19 smithi163 ceph-mon[29621]: 2.0 scrub starts 2024-03-21T13:19:20.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:19 smithi163 ceph-mon[29621]: 2.0 scrub ok 2024-03-21T13:19:20.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:19 smithi163 ceph-mon[29621]: 11.b scrub ok 2024-03-21T13:19:20.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:19 smithi163 ceph-mon[40675]: pgmap v966: 148 pgs: 148 active+clean; 2.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-21T13:19:20.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:19 smithi163 ceph-mon[40675]: 2.0 scrub starts 2024-03-21T13:19:20.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:19 smithi163 ceph-mon[40675]: 2.0 scrub ok 2024-03-21T13:19:20.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:19 smithi163 ceph-mon[40675]: 11.b scrub ok 2024-03-21T13:19:20.172 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:20.202 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:20.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:20.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:20.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:19 smithi204 ceph-mon[33153]: pgmap v966: 148 pgs: 148 active+clean; 2.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-21T13:19:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:19 smithi204 ceph-mon[33153]: 2.0 scrub starts 2024-03-21T13:19:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:19 smithi204 ceph-mon[33153]: 2.0 scrub ok 2024-03-21T13:19:20.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:19 smithi204 ceph-mon[33153]: 11.b scrub ok 2024-03-21T13:19:20.304 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:20.335 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:20.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:20.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:20.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:20.472 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:20.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:20.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:20.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:20.607 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:20.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:20.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:20.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:20.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:20.811 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.7 to 1 (10000) 2024-03-21T13:19:20.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:20.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:20.884 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:20.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:20.985 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:21.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[29621]: 4.9 scrub starts 2024-03-21T13:19:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[29621]: 4.9 scrub ok 2024-03-21T13:19:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[29621]: 5.16 scrub starts 2024-03-21T13:19:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[29621]: 5.16 scrub ok 2024-03-21T13:19:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4096726265' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:19:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:19:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[40675]: 4.9 scrub starts 2024-03-21T13:19:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[40675]: 4.9 scrub ok 2024-03-21T13:19:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[40675]: 5.16 scrub starts 2024-03-21T13:19:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[40675]: 5.16 scrub ok 2024-03-21T13:19:21.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4096726265' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:19:21.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:19:21.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:20 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:21.020 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:21.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:21.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:21.121 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:21.151 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:21.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:21.252 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:21.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:20 smithi204 ceph-mon[33153]: 4.9 scrub starts 2024-03-21T13:19:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:20 smithi204 ceph-mon[33153]: 4.9 scrub ok 2024-03-21T13:19:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:20 smithi204 ceph-mon[33153]: 5.16 scrub starts 2024-03-21T13:19:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:20 smithi204 ceph-mon[33153]: 5.16 scrub ok 2024-03-21T13:19:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:20 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4096726265' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:19:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:19:21.254 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:20 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:21.282 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:21.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:21.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:21.384 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:21.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:21.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:21.521 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:21.563 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-21T13:19:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:21.566 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:21.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:21.667 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:21.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:21.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:21.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:21.802 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:21.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:21.838 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:21.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:21.939 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:21.973 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:21.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:22.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:22.112 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:22.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:22.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:22.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:21 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4096726265' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:19:22.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:21 smithi163 ceph-mon[29621]: osdmap e443: 8 total, 8 up, 6 in 2024-03-21T13:19:22.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:21 smithi163 ceph-mon[29621]: pgmap v968: 148 pgs: 148 active+clean; 1.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:19:22.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:21 smithi163 ceph-mon[29621]: 5.9 scrub starts 2024-03-21T13:19:22.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:21 smithi163 ceph-mon[29621]: 5.9 scrub ok 2024-03-21T13:19:22.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:21 smithi163 ceph-mon[29621]: 11.8 scrub starts 2024-03-21T13:19:22.140 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:21 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4096726265' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:19:22.140 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:21 smithi163 ceph-mon[40675]: osdmap e443: 8 total, 8 up, 6 in 2024-03-21T13:19:22.140 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:21 smithi163 ceph-mon[40675]: pgmap v968: 148 pgs: 148 active+clean; 1.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:19:22.140 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:21 smithi163 ceph-mon[40675]: 5.9 scrub starts 2024-03-21T13:19:22.140 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:21 smithi163 ceph-mon[40675]: 5.9 scrub ok 2024-03-21T13:19:22.140 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:21 smithi163 ceph-mon[40675]: 11.8 scrub starts 2024-03-21T13:19:22.214 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:22.242 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:22.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:22.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:22.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:21 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4096726265' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:19:22.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:21 smithi204 ceph-mon[33153]: osdmap e443: 8 total, 8 up, 6 in 2024-03-21T13:19:22.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:21 smithi204 ceph-mon[33153]: pgmap v968: 148 pgs: 148 active+clean; 1.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:19:22.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:21 smithi204 ceph-mon[33153]: 5.9 scrub starts 2024-03-21T13:19:22.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:21 smithi204 ceph-mon[33153]: 5.9 scrub ok 2024-03-21T13:19:22.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:21 smithi204 ceph-mon[33153]: 11.8 scrub starts 2024-03-21T13:19:22.344 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:22.373 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:22.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:22.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:22.475 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:22.504 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:22.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:22.605 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:22.643 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:22.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:22.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:22.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:19:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:19:22.784 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:22.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:22.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:22.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:22.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:22.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:23.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:23.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:23.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:23.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:23.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:23.188 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:23.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:22 smithi204 ceph-mon[33153]: 4.d scrub starts 2024-03-21T13:19:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:22 smithi204 ceph-mon[33153]: 4.d scrub ok 2024-03-21T13:19:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:22 smithi204 ceph-mon[33153]: 6.4 scrub starts 2024-03-21T13:19:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:22 smithi204 ceph-mon[33153]: 6.4 scrub ok 2024-03-21T13:19:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:22 smithi204 ceph-mon[33153]: 5.1 scrub starts 2024-03-21T13:19:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:22 smithi204 ceph-mon[33153]: 5.1 scrub ok 2024-03-21T13:19:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:22 smithi204 ceph-mon[33153]: 11.8 scrub ok 2024-03-21T13:19:23.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:22 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[29621]: 4.d scrub starts 2024-03-21T13:19:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[29621]: 4.d scrub ok 2024-03-21T13:19:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[29621]: 6.4 scrub starts 2024-03-21T13:19:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[29621]: 6.4 scrub ok 2024-03-21T13:19:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[29621]: 5.1 scrub starts 2024-03-21T13:19:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[29621]: 5.1 scrub ok 2024-03-21T13:19:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[29621]: 11.8 scrub ok 2024-03-21T13:19:23.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:23.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[40675]: 4.d scrub starts 2024-03-21T13:19:23.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[40675]: 4.d scrub ok 2024-03-21T13:19:23.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[40675]: 6.4 scrub starts 2024-03-21T13:19:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[40675]: 6.4 scrub ok 2024-03-21T13:19:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[40675]: 5.1 scrub starts 2024-03-21T13:19:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[40675]: 5.1 scrub ok 2024-03-21T13:19:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[40675]: 11.8 scrub ok 2024-03-21T13:19:23.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:22 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:23.289 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:23.326 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:23.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:23.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:23.428 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:23.457 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:23.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:23.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:23.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:23.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:23.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:23.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:23.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:23.727 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:23.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:23.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:23.829 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:23.861 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:23.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:23.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:23.997 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:23.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:24.099 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:24.129 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:24.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:24.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:23 smithi163 ceph-mon[29621]: 5.6 scrub starts 2024-03-21T13:19:24.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:23 smithi163 ceph-mon[29621]: 5.6 scrub ok 2024-03-21T13:19:24.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:23 smithi163 ceph-mon[29621]: 5.3 scrub starts 2024-03-21T13:19:24.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:23 smithi163 ceph-mon[29621]: 5.3 scrub ok 2024-03-21T13:19:24.180 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:23 smithi163 ceph-mon[29621]: pgmap v969: 148 pgs: 148 active+clean; 1.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-21T13:19:24.180 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:23 smithi163 ceph-mon[40675]: 5.6 scrub starts 2024-03-21T13:19:24.180 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:23 smithi163 ceph-mon[40675]: 5.6 scrub ok 2024-03-21T13:19:24.180 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:23 smithi163 ceph-mon[40675]: 5.3 scrub starts 2024-03-21T13:19:24.180 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:23 smithi163 ceph-mon[40675]: 5.3 scrub ok 2024-03-21T13:19:24.180 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:23 smithi163 ceph-mon[40675]: pgmap v969: 148 pgs: 148 active+clean; 1.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-21T13:19:24.231 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:24.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:23 smithi204 ceph-mon[33153]: 5.6 scrub starts 2024-03-21T13:19:24.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:23 smithi204 ceph-mon[33153]: 5.6 scrub ok 2024-03-21T13:19:24.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:23 smithi204 ceph-mon[33153]: 5.3 scrub starts 2024-03-21T13:19:24.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:23 smithi204 ceph-mon[33153]: 5.3 scrub ok 2024-03-21T13:19:24.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:23 smithi204 ceph-mon[33153]: pgmap v969: 148 pgs: 148 active+clean; 1.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-21T13:19:24.260 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:24.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:24.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:24.403 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:24.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:24.505 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:24.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:24.542 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:24.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:24.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:24.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:24.677 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:24.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:24.778 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:24.807 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:24.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:24.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:24.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:24.946 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:24.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:25.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:25.083 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:25.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:25.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:25.217 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:25.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:25.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:25.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:24 smithi163 ceph-mon[29621]: 4.13 scrub starts 2024-03-21T13:19:25.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:24 smithi163 ceph-mon[29621]: 4.13 scrub ok 2024-03-21T13:19:25.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:24 smithi163 ceph-mon[40675]: 4.13 scrub starts 2024-03-21T13:19:25.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:24 smithi163 ceph-mon[40675]: 4.13 scrub ok 2024-03-21T13:19:25.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:24 smithi204 ceph-mon[33153]: 4.13 scrub starts 2024-03-21T13:19:25.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:24 smithi204 ceph-mon[33153]: 4.13 scrub ok 2024-03-21T13:19:25.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:25.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:25.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:25.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:25.460 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:25.488 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:25.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:25.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:25.590 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:25.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:25.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:25.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:25.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:25.769 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:25.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:25.871 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:25.906 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:25.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:26.007 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:26.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[29621]: 3.5 deep-scrub starts 2024-03-21T13:19:26.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[29621]: 3.5 deep-scrub ok 2024-03-21T13:19:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[29621]: 5.0 scrub starts 2024-03-21T13:19:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[29621]: 5.0 scrub ok 2024-03-21T13:19:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[29621]: 3.3 scrub starts 2024-03-21T13:19:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[29621]: 3.3 scrub ok 2024-03-21T13:19:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[29621]: 5.c scrub starts 2024-03-21T13:19:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[29621]: 5.c scrub ok 2024-03-21T13:19:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[29621]: pgmap v970: 148 pgs: 148 active+clean; 1.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-21T13:19:26.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:19:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:19:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[40675]: 3.5 deep-scrub starts 2024-03-21T13:19:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[40675]: 3.5 deep-scrub ok 2024-03-21T13:19:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[40675]: 5.0 scrub starts 2024-03-21T13:19:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[40675]: 5.0 scrub ok 2024-03-21T13:19:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[40675]: 3.3 scrub starts 2024-03-21T13:19:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[40675]: 3.3 scrub ok 2024-03-21T13:19:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[40675]: 5.c scrub starts 2024-03-21T13:19:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[40675]: 5.c scrub ok 2024-03-21T13:19:26.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:25 smithi163 ceph-mon[40675]: pgmap v970: 148 pgs: 148 active+clean; 1.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-21T13:19:26.039 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:26.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:26.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:26.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:26.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:26.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:26.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:25 smithi204 ceph-mon[33153]: 3.5 deep-scrub starts 2024-03-21T13:19:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:25 smithi204 ceph-mon[33153]: 3.5 deep-scrub ok 2024-03-21T13:19:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:25 smithi204 ceph-mon[33153]: 5.0 scrub starts 2024-03-21T13:19:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:25 smithi204 ceph-mon[33153]: 5.0 scrub ok 2024-03-21T13:19:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:25 smithi204 ceph-mon[33153]: 3.3 scrub starts 2024-03-21T13:19:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:25 smithi204 ceph-mon[33153]: 3.3 scrub ok 2024-03-21T13:19:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:25 smithi204 ceph-mon[33153]: 5.c scrub starts 2024-03-21T13:19:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:25 smithi204 ceph-mon[33153]: 5.c scrub ok 2024-03-21T13:19:26.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:25 smithi204 ceph-mon[33153]: pgmap v970: 148 pgs: 148 active+clean; 1.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-21T13:19:26.281 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:26.311 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:26.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:26.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:26.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:26.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:26.551 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:26.581 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:26.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:26.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:26.683 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:26.713 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:26.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:26.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:26.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:26.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:26.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:26.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:26.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:26.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:26 smithi163 ceph-mon[29621]: 5.19 deep-scrub starts 2024-03-21T13:19:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:26 smithi163 ceph-mon[29621]: 5.19 deep-scrub ok 2024-03-21T13:19:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:26 smithi163 ceph-mon[29621]: 5.11 scrub starts 2024-03-21T13:19:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:26 smithi163 ceph-mon[29621]: 5.11 scrub ok 2024-03-21T13:19:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:26 smithi163 ceph-mon[29621]: 11.0 scrub starts 2024-03-21T13:19:27.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:26 smithi163 ceph-mon[29621]: 11.0 scrub ok 2024-03-21T13:19:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:26 smithi163 ceph-mon[40675]: 5.19 deep-scrub starts 2024-03-21T13:19:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:26 smithi163 ceph-mon[40675]: 5.19 deep-scrub ok 2024-03-21T13:19:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:26 smithi163 ceph-mon[40675]: 5.11 scrub starts 2024-03-21T13:19:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:26 smithi163 ceph-mon[40675]: 5.11 scrub ok 2024-03-21T13:19:27.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:26 smithi163 ceph-mon[40675]: 11.0 scrub starts 2024-03-21T13:19:27.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:26 smithi163 ceph-mon[40675]: 11.0 scrub ok 2024-03-21T13:19:27.098 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:27.132 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:27.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:27.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:27.234 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:27.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:26 smithi204 ceph-mon[33153]: 5.19 deep-scrub starts 2024-03-21T13:19:27.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:26 smithi204 ceph-mon[33153]: 5.19 deep-scrub ok 2024-03-21T13:19:27.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:26 smithi204 ceph-mon[33153]: 5.11 scrub starts 2024-03-21T13:19:27.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:26 smithi204 ceph-mon[33153]: 5.11 scrub ok 2024-03-21T13:19:27.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:26 smithi204 ceph-mon[33153]: 11.0 scrub starts 2024-03-21T13:19:27.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:26 smithi204 ceph-mon[33153]: 11.0 scrub ok 2024-03-21T13:19:27.263 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:27.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:27.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:27.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:27.389 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:27.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:27.490 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:27.520 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:27.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:27.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:27.622 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:27.649 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:27.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:27.751 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:27.777 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:27.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:27.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:27.879 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:27.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:27.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:27.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:28.019 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:28.050 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:28.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:28.152 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:28.184 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:28.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:28.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:28.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:27 smithi204 ceph-mon[33153]: 6.1f scrub starts 2024-03-21T13:19:28.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:27 smithi204 ceph-mon[33153]: pgmap v971: 148 pgs: 148 active+clean; 1.4 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:19:28.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:27 smithi204 ceph-mon[33153]: 6.1f scrub ok 2024-03-21T13:19:28.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:27 smithi163 ceph-mon[29621]: 6.1f scrub starts 2024-03-21T13:19:28.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:27 smithi163 ceph-mon[29621]: pgmap v971: 148 pgs: 148 active+clean; 1.4 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:19:28.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:27 smithi163 ceph-mon[29621]: 6.1f scrub ok 2024-03-21T13:19:28.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:27 smithi163 ceph-mon[40675]: 6.1f scrub starts 2024-03-21T13:19:28.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:27 smithi163 ceph-mon[40675]: pgmap v971: 148 pgs: 148 active+clean; 1.4 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T13:19:28.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:27 smithi163 ceph-mon[40675]: 6.1f scrub ok 2024-03-21T13:19:28.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:28.326 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:28.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:28.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:28.428 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:28.459 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:28.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:28.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:28.561 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:28.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:28.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:28.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:28.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:28.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:28.840 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:28.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:28.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:28.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:28.986 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:28 smithi163 ceph-mon[40675]: 3.16 scrub starts 2024-03-21T13:19:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:28 smithi163 ceph-mon[40675]: 3.16 scrub ok 2024-03-21T13:19:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:28 smithi163 ceph-mon[40675]: 3.b scrub starts 2024-03-21T13:19:29.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:28 smithi163 ceph-mon[40675]: 3.b scrub ok 2024-03-21T13:19:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:28 smithi163 ceph-mon[29621]: 3.16 scrub starts 2024-03-21T13:19:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:28 smithi163 ceph-mon[29621]: 3.16 scrub ok 2024-03-21T13:19:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:28 smithi163 ceph-mon[29621]: 3.b scrub starts 2024-03-21T13:19:29.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:28 smithi163 ceph-mon[29621]: 3.b scrub ok 2024-03-21T13:19:29.016 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:29.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:29.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:29.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:29.151 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:29.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:29.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:29.176 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-21T13:19:29.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:28 smithi204 ceph-mon[33153]: 3.16 scrub starts 2024-03-21T13:19:29.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:28 smithi204 ceph-mon[33153]: 3.16 scrub ok 2024-03-21T13:19:29.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:28 smithi204 ceph-mon[33153]: 3.b scrub starts 2024-03-21T13:19:29.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:28 smithi204 ceph-mon[33153]: 3.b scrub ok 2024-03-21T13:19:29.253 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:29.293 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:29.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:29.394 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:29.427 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:29.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:29.529 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:29.558 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:29.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:29.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:29.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:29.698 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:29.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:29.800 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:29.839 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:29.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:29.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:29.973 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:29.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:29.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:30.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:29 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:19:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:29 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:19:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:29 smithi163 ceph-mon[29621]: osdmap e444: 8 total, 8 up, 6 in 2024-03-21T13:19:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:29 smithi163 ceph-mon[29621]: pgmap v973: 148 pgs: 148 active+clean; 1.4 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 623 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-21T13:19:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:29 smithi163 ceph-mon[29621]: 3.12 scrub starts 2024-03-21T13:19:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:29 smithi163 ceph-mon[29621]: 3.12 scrub ok 2024-03-21T13:19:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:29 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:19:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:29 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:19:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:29 smithi163 ceph-mon[40675]: osdmap e444: 8 total, 8 up, 6 in 2024-03-21T13:19:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:29 smithi163 ceph-mon[40675]: pgmap v973: 148 pgs: 148 active+clean; 1.4 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 623 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-21T13:19:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:29 smithi163 ceph-mon[40675]: 3.12 scrub starts 2024-03-21T13:19:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:29 smithi163 ceph-mon[40675]: 3.12 scrub ok 2024-03-21T13:19:30.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:30.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:30.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:30.217 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:30.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:29 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:19:30.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:29 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:19:30.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:29 smithi204 ceph-mon[33153]: osdmap e444: 8 total, 8 up, 6 in 2024-03-21T13:19:30.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:29 smithi204 ceph-mon[33153]: pgmap v973: 148 pgs: 148 active+clean; 1.4 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 623 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-21T13:19:30.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:29 smithi204 ceph-mon[33153]: 3.12 scrub starts 2024-03-21T13:19:30.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:29 smithi204 ceph-mon[33153]: 3.12 scrub ok 2024-03-21T13:19:30.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:30.269 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:30.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:30.370 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:30.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:30.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:30.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:30.507 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:30.537 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:30.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:30.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:30.605 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:30.638 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:30.670 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:30.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:30.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:30.771 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:30.804 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:30.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:30.906 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:30.940 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:30.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:31.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:30 smithi163 ceph-mon[29621]: 11.c scrub starts 2024-03-21T13:19:31.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:30 smithi163 ceph-mon[29621]: 11.c scrub ok 2024-03-21T13:19:31.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:30 smithi163 ceph-mon[29621]: 6.1 deep-scrub starts 2024-03-21T13:19:31.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:30 smithi163 ceph-mon[29621]: 6.1 deep-scrub ok 2024-03-21T13:19:31.006 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:30 smithi163 ceph-mon[40675]: 11.c scrub starts 2024-03-21T13:19:31.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:30 smithi163 ceph-mon[40675]: 11.c scrub ok 2024-03-21T13:19:31.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:30 smithi163 ceph-mon[40675]: 6.1 deep-scrub starts 2024-03-21T13:19:31.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:30 smithi163 ceph-mon[40675]: 6.1 deep-scrub ok 2024-03-21T13:19:31.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:31.075 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:31.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:31.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:31.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:31.212 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:31.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:31.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:31.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:30 smithi204 ceph-mon[33153]: 11.c scrub starts 2024-03-21T13:19:31.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:30 smithi204 ceph-mon[33153]: 11.c scrub ok 2024-03-21T13:19:31.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:30 smithi204 ceph-mon[33153]: 6.1 deep-scrub starts 2024-03-21T13:19:31.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:30 smithi204 ceph-mon[33153]: 6.1 deep-scrub ok 2024-03-21T13:19:31.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:31.352 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:31.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:31.453 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:31.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:31.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:31.501 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:19:31.530 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:19:31.560 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:19:31.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:31.592 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:19:31.620 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:19:31.625 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:31.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:31.653 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:19:31.687 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:19:31.726 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:19:31.727 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:31.734 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:31.735 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:19:31.735 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:19:31.735 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:19:31.735 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:19:31.735 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:19:31.735 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:19:31.735 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:19:31.735 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:19:31.778 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:31.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:31.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:31.879 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:31.892 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-03-21T13:19:31.915 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:31.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:32.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:32.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:32.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:32.157 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:32.187 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:32.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:32.216 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:31 smithi163 ceph-mon[29621]: 4.12 scrub starts 2024-03-21T13:19:32.216 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:31 smithi163 ceph-mon[29621]: 4.12 scrub ok 2024-03-21T13:19:32.216 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:31 smithi163 ceph-mon[29621]: pgmap v974: 148 pgs: 148 active+clean; 1.1 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-03-21T13:19:32.216 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:31 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3238720805' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:19:32.216 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:31 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3307513846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:19:32.216 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:31 smithi163 ceph-mon[40675]: 4.12 scrub starts 2024-03-21T13:19:32.216 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:31 smithi163 ceph-mon[40675]: 4.12 scrub ok 2024-03-21T13:19:32.216 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:31 smithi163 ceph-mon[40675]: pgmap v974: 148 pgs: 148 active+clean; 1.1 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-03-21T13:19:32.216 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:31 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3238720805' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:19:32.216 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:31 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3307513846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:19:32.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:31 smithi204 ceph-mon[33153]: 4.12 scrub starts 2024-03-21T13:19:32.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:31 smithi204 ceph-mon[33153]: 4.12 scrub ok 2024-03-21T13:19:32.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:31 smithi204 ceph-mon[33153]: pgmap v974: 148 pgs: 148 active+clean; 1.1 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-03-21T13:19:32.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:31 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3238720805' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T13:19:32.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:31 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3307513846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:19:32.288 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:32.316 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:32.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:32.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:32.417 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:32.448 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:32.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:32.549 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:32.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:32.579 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:32.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:32.680 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:32.709 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:32.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:32.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:19:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:19:32.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:32.842 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:32.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:32.944 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:32.978 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:32.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:32.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:33.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:33.116 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:33.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:33.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:33.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[29621]: 6.e scrub starts 2024-03-21T13:19:33.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[29621]: 6.e scrub ok 2024-03-21T13:19:33.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[29621]: 4.1a scrub starts 2024-03-21T13:19:33.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[29621]: 4.1a scrub ok 2024-03-21T13:19:33.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[29621]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:19:33.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[29621]: 4.a scrub starts 2024-03-21T13:19:33.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[29621]: 4.a scrub ok 2024-03-21T13:19:33.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3238720805' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:19:33.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[29621]: osdmap e445: 8 total, 8 up, 6 in 2024-03-21T13:19:33.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:33.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[40675]: 6.e scrub starts 2024-03-21T13:19:33.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[40675]: 6.e scrub ok 2024-03-21T13:19:33.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[40675]: 4.1a scrub starts 2024-03-21T13:19:33.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[40675]: 4.1a scrub ok 2024-03-21T13:19:33.216 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[40675]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:19:33.216 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[40675]: 4.a scrub starts 2024-03-21T13:19:33.216 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[40675]: 4.a scrub ok 2024-03-21T13:19:33.216 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3238720805' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:19:33.216 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[40675]: osdmap e445: 8 total, 8 up, 6 in 2024-03-21T13:19:33.216 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:32 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:33.217 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:33.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:32 smithi204 ceph-mon[33153]: 6.e scrub starts 2024-03-21T13:19:33.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:32 smithi204 ceph-mon[33153]: 6.e scrub ok 2024-03-21T13:19:33.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:32 smithi204 ceph-mon[33153]: 4.1a scrub starts 2024-03-21T13:19:33.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:32 smithi204 ceph-mon[33153]: 4.1a scrub ok 2024-03-21T13:19:33.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:32 smithi204 ceph-mon[33153]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:19:33.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:32 smithi204 ceph-mon[33153]: 4.a scrub starts 2024-03-21T13:19:33.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:32 smithi204 ceph-mon[33153]: 4.a scrub ok 2024-03-21T13:19:33.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:32 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3238720805' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T13:19:33.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:32 smithi204 ceph-mon[33153]: osdmap e445: 8 total, 8 up, 6 in 2024-03-21T13:19:33.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:32 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:33.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:33.254 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:33.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:33.355 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:33.394 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:33.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:33.495 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:33.521 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:33.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:33.623 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:33.649 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:33.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:33.751 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:33.779 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:33.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:33.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:33.881 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:33.914 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:33.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:34.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:34.053 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:34.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:34.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:34.155 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:34.183 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:34.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:34.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:33 smithi204 ceph-mon[33153]: pgmap v976: 148 pgs: 148 active+clean; 1006 MiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-03-21T13:19:34.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:33 smithi163 ceph-mon[40675]: pgmap v976: 148 pgs: 148 active+clean; 1006 MiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-03-21T13:19:34.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:33 smithi163 ceph-mon[29621]: pgmap v976: 148 pgs: 148 active+clean; 1006 MiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-03-21T13:19:34.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:34.322 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:34.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:34.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:34.423 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:34.453 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:34.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:34.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:34.555 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:34.584 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:34.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:34.686 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:34.718 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:34.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:34.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:34.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:34.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:34.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:34.963 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:34.993 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:34.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:34.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:35.094 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:35.124 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:35.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:35.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:35.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:35.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T13:19:35.371 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:35.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:35.473 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:35.503 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:35.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:35.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:35.605 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:35.633 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:35.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:35.734 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:35.771 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:35.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:35.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:35.873 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:35.903 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:35.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:35.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:35.913 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:19:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:19:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:19:36.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:36.038 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:36.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:36.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:36.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:36.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:36.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:36.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:36.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:36 smithi163 ceph-mon[29621]: pgmap v977: 148 pgs: 148 active+clean; 1006 MiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 807 op/s 2024-03-21T13:19:36.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:36 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:19:36.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:36 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:36.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:36 smithi163 ceph-mon[40675]: pgmap v977: 148 pgs: 148 active+clean; 1006 MiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 807 op/s 2024-03-21T13:19:36.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:36 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:19:36.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:36 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:36.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:36.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:36.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:36.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:36.416 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:36.443 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:36.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:36.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:36.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:36 smithi204 ceph-mon[33153]: pgmap v977: 148 pgs: 148 active+clean; 1006 MiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 807 op/s 2024-03-21T13:19:36.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:36 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:19:36.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:36 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:36.506 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:19:36.506 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:19:36.508 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T13:19:36.545 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:36.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:36.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:36.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:36.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:36.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:36.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:36.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:36.821 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:36.851 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:36.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:36.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:36.953 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:36.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:36.982 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:36.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:37.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:37.117 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:37.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:37.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:37.219 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:37.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:37.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:37.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:37.359 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:37.400 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:37.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:37.502 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:37.532 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:37.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:37.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:37.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:37.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:37.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:37.672 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:37.770 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:37.800 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:37.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:37.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:37.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:37.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:37.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:37.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:38.039 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:38.075 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:38.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:38.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:38.215 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:38.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:38 smithi163 ceph-mon[29621]: pgmap v978: 148 pgs: 148 active+clean; 606 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-03-21T13:19:38.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:38 smithi163 ceph-mon[40675]: pgmap v978: 148 pgs: 148 active+clean; 606 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-03-21T13:19:38.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:38.347 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:38.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:38.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:38.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:38.486 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:38.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:38.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:38 smithi204 ceph-mon[33153]: pgmap v978: 148 pgs: 148 active+clean; 606 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-03-21T13:19:38.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:38.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:38.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:38.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:38.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:38.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:38.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:38.775 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-21T13:19:38.792 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:38.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:38.904 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:38.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:39.005 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:39.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:39.087 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.2 to 1 (10000) 2024-03-21T13:19:39.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:39.171 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:39.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:39.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:39 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1854657219' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:19:39.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:39 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:19:39.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:39 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1854657219' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:19:39.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:39 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:19:39.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:39.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:39.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:39.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:39.411 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:39.439 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:39.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:39.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:39.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:39 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1854657219' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:19:39.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:39 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:19:39.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:39.576 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:39.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:39.678 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:39.709 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:39.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:39.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:39.847 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:39.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:39.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:39.949 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:39.978 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:39.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:40.080 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:40.115 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:40.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:40.216 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:40.256 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:40.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:40.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:40.396 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:40.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:40.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:40.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:40 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:19:40.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:40 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:19:40.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:40 smithi204 ceph-mon[33153]: pgmap v979: 148 pgs: 148 active+clean; 606 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-21T13:19:40.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:40 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:19:40.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:40 smithi204 ceph-mon[33153]: osdmap e446: 8 total, 8 up, 6 in 2024-03-21T13:19:40.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:40 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:19:40.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:40 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:19:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:19:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:19:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[29621]: pgmap v979: 148 pgs: 148 active+clean; 606 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-21T13:19:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:19:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[29621]: osdmap e446: 8 total, 8 up, 6 in 2024-03-21T13:19:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:19:40.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:19:40.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:19:40.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:19:40.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[40675]: pgmap v979: 148 pgs: 148 active+clean; 606 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-21T13:19:40.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:19:40.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[40675]: osdmap e446: 8 total, 8 up, 6 in 2024-03-21T13:19:40.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:19:40.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:40 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:19:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:40.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:40.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:40.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:40.680 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:40.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:40.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:40.820 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:40.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:40.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:40.922 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:40.947 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:40.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:40.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:41.049 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:41.078 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:41.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:41.181 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:41.213 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:41.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:41.315 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:41.353 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:41.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:41.444 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:41 smithi163 ceph-mon[29621]: osdmap e447: 8 total, 8 up, 6 in 2024-03-21T13:19:41.444 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:41 smithi163 ceph-mon[40675]: osdmap e447: 8 total, 8 up, 6 in 2024-03-21T13:19:41.455 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:41.495 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:41.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:41.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:41.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:41 smithi204 ceph-mon[33153]: osdmap e447: 8 total, 8 up, 6 in 2024-03-21T13:19:41.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:41.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:41.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:41.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:41.761 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:41.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:41.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:41.903 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:41.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:42.005 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:42.041 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:42.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:42.143 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:42.175 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:42.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:42.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:42.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:42 smithi163 ceph-mon[40675]: pgmap v982: 148 pgs: 148 active+clean; 327 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-21T13:19:42.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:42 smithi163 ceph-mon[29621]: pgmap v982: 148 pgs: 148 active+clean; 327 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-21T13:19:42.277 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:42.308 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:42.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:42.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:42.403 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:42 smithi204 ceph-mon[33153]: pgmap v982: 148 pgs: 148 active+clean; 327 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-21T13:19:42.410 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:42.440 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:42.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:42.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:42.542 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:42.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:42.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:42.682 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:42.711 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:42.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:42.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:42.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:19:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:19:42.813 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:42.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:42.846 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:42.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:42.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:42.985 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:42.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:43.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:43.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:43.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:43.224 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:43.252 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:43.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:43.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:43.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:43.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:43.355 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:43.362 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Removed 121981 objects 2024-03-21T13:19:43.363 INFO:tasks.radosbench.radosbench.0.smithi163.stdout:Clean up completed and total clean up time :104.008 2024-03-21T13:19:43.381 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:43.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:43.483 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:43.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:43.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:43.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:43.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:43.608 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:43.631 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:43.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:43.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:43.757 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:43.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:43.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:43.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:43.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:43.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:43.984 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:44.009 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:44.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:44.080 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:19:44.100 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:19:44.111 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:44.120 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-03-21T13:19:44.124 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:19:44.135 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:44.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:44.146 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:19:44.168 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:19:44.188 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:19:44.209 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:19:44.209 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:44 smithi163 ceph-mon[29621]: pgmap v983: 148 pgs: 148 active+clean; 202 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-21T13:19:44.209 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/554721777' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:19:44.209 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:44 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:19:44.209 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1443703610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:19:44.209 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:44 smithi163 ceph-mon[40675]: pgmap v983: 148 pgs: 148 active+clean; 202 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-21T13:19:44.210 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/554721777' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:19:44.210 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:44 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:19:44.210 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1443703610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:19:44.228 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:19:44.229 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:19:44.229 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:19:44.229 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:19:44.229 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:19:44.229 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:19:44.229 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:19:44.230 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:19:44.230 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:19:44.237 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:44.262 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:44.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:44.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:44.387 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:44.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:44.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:44.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:44 smithi204 ceph-mon[33153]: pgmap v983: 148 pgs: 148 active+clean; 202 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-21T13:19:44.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/554721777' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:19:44.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:44 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:19:44.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1443703610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:19:44.513 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:44.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:44.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:44.528 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-21T13:19:44.528 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-21T13:19:44.614 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:44.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:44.639 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:44.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:44.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:44.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:44.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:44.870 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:44.894 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:44.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:44.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:45.022 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:45.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:45.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:45.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:45.131 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-21T13:19:45.141 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:45.150 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:45.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:45.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:45.252 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:45.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:45.277 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:45.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:45.378 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:45.402 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:45.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:45.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:45 smithi204 ceph-mon[33153]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:19:45.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:45 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:19:45.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:45 smithi204 ceph-mon[33153]: osdmap e448: 8 total, 8 up, 6 in 2024-03-21T13:19:45.504 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:45.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:45 smithi163 ceph-mon[29621]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:19:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:45 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:19:45.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:45 smithi163 ceph-mon[29621]: osdmap e448: 8 total, 8 up, 6 in 2024-03-21T13:19:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:45 smithi163 ceph-mon[40675]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:19:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:45 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T13:19:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:45 smithi163 ceph-mon[40675]: osdmap e448: 8 total, 8 up, 6 in 2024-03-21T13:19:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:45.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:45.628 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:45.657 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:45.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:45.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:45.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:45.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:45.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:45.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:45.912 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:45.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:45.921 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:19:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:19:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:19:46.014 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:46.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:46.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:46.140 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:46.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:46.164 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:46.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:46.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:46 smithi163 ceph-mon[29621]: pgmap v985: 148 pgs: 148 active+clean; 202 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-21T13:19:46.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:46 smithi163 ceph-mon[40675]: pgmap v985: 148 pgs: 148 active+clean; 202 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-21T13:19:46.265 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:46.294 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:46.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:46.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:46.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:46.419 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:46.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:46.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:46 smithi204 ceph-mon[33153]: pgmap v985: 148 pgs: 148 active+clean; 202 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-21T13:19:46.521 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:46.546 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:46.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:46.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:46.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:46.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:46.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:46.777 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:46.800 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:46.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:46.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:46.902 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:46.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:46.927 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:46.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:47.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:47.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:47.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:47.158 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:47.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:47.183 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:47.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:47.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:47.312 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:47.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:47.414 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:47.438 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:47.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:47.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:47.540 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:47.565 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:47.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:47.667 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:47.692 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:47.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:47.709 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:19:47.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:47.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:47.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:47.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:47.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:47.952 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:47.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:48.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:48.081 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:48.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:48.183 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:48.207 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:48.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:48.309 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:48.333 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:48.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:48.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:48.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:48.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:48.564 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:48.589 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:48.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:48.690 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:48.716 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:48.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:48.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:48.818 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:48.842 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:48.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:48.943 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:48.967 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:48.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:48.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:48 smithi204 ceph-mon[33153]: pgmap v986: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-21T13:19:49.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:48 smithi163 ceph-mon[29621]: pgmap v986: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-21T13:19:49.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:48 smithi163 ceph-mon[40675]: pgmap v986: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-21T13:19:49.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:49.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:49.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:49.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:49.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:49.231 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:49.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:49.333 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:49.358 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:49.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:49.459 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:49.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:49.487 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:49.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:49.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:49.613 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:49.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:49.714 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:49.738 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:49.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:49.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:49.840 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:49.864 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:49.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:49.965 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:49.977 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:49.977 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:49.997 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:49.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:50.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:50.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:50.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:50.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:50.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:50.256 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:50.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:50.358 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:50.384 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:50.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:50.486 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:50.514 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:50.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:50.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:50.548 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-21T13:19:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-03-21T13:19:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-03-21T13:19:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-03-21T13:19:50.549 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-21T13:19:50.615 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:50.640 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:50.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:50.742 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:50.771 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:50.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:50.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:50.873 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:50.897 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:50.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:50.999 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:50 smithi204 ceph-mon[33153]: pgmap v987: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-21T13:19:51.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:50 smithi163 ceph-mon[29621]: pgmap v987: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-21T13:19:51.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:50 smithi163 ceph-mon[40675]: pgmap v987: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-21T13:19:51.023 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:51.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:51.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:51.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:51.154 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:51.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:51.255 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:51.281 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:51.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:51.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:51.383 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:51.407 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:51.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:51.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:51.480 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T13:19:51.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:51.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:51.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:51.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:51.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:51.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:51.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:51.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:51.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:51.768 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:51.791 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:51.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:51.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:51.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:51.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:52.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:52.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:52.049 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:52.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:52.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:52.178 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:52.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:52.279 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:52.304 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:52.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:52.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:52.406 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:52.431 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:52.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:52.532 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:52.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:52.557 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:52.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:52.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:52.688 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:52.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:52.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:52 smithi204 ceph-mon[33153]: pgmap v988: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 523 op/s 2024-03-21T13:19:52.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:19:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:19:52.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:52 smithi163 ceph-mon[29621]: pgmap v988: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 523 op/s 2024-03-21T13:19:52.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:52 smithi163 ceph-mon[40675]: pgmap v988: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 523 op/s 2024-03-21T13:19:52.789 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:52.814 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:52.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:52.896 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-21T13:19:52.916 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:52.917 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:52.945 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:52.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:53.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:53.075 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:53.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:53.177 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:53.202 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:53.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:53.303 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:53.327 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:53.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:53.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:53.457 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:53.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:53.558 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:53.585 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:53.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:53.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:53.688 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:53.712 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:53.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:53.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:53.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:53.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:53.814 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:53.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:53.840 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:53.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:53.941 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:53.970 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:53.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:53.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:54.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:54.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:54.096 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:54.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:54.197 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:54.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:54.223 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:54.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:54.324 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:54.352 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:54.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:54.454 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:54.479 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:54.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:54.554 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.5 to 1 (10000) 2024-03-21T13:19:54.581 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:54.604 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:54.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:54.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:54.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:54.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:54.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:54.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:54 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:19:54.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:54 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:19:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:54 smithi163 ceph-mon[29621]: pgmap v989: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 323 op/s 2024-03-21T13:19:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:54 smithi163 ceph-mon[29621]: osdmap e449: 8 total, 8 up, 6 in 2024-03-21T13:19:54.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:54 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/263088553' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:19:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:54 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:19:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:54 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:19:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:54 smithi163 ceph-mon[40675]: pgmap v989: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 323 op/s 2024-03-21T13:19:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:54 smithi163 ceph-mon[40675]: osdmap e449: 8 total, 8 up, 6 in 2024-03-21T13:19:54.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:54 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/263088553' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:19:54.791 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:54.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:54.863 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:54.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:54.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:54.965 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:54.988 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:54.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:55.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:54 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:19:55.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:54 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:19:55.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:54 smithi204 ceph-mon[33153]: pgmap v989: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 323 op/s 2024-03-21T13:19:55.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:54 smithi204 ceph-mon[33153]: osdmap e449: 8 total, 8 up, 6 in 2024-03-21T13:19:55.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:54 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/263088553' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:19:55.091 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:55.115 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:55.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:55.216 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:55.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:55.241 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:55.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:55.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:55.370 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:55.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:55.471 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:55.496 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:55.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:55.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:55.623 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:55.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:55.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:55.724 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:55.748 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:55.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:55.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:55.850 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:55.873 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:55.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:55.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:19:56.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:55 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/263088553' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:19:56.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:55 smithi204 ceph-mon[33153]: osdmap e450: 8 total, 8 up, 6 in 2024-03-21T13:19:56.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:56.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:56.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:19:56.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:55 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/263088553' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:19:56.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:55 smithi163 ceph-mon[29621]: osdmap e450: 8 total, 8 up, 6 in 2024-03-21T13:19:56.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:55 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/263088553' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:19:56.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:55 smithi163 ceph-mon[40675]: osdmap e450: 8 total, 8 up, 6 in 2024-03-21T13:19:56.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:19:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:19:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:19:56.104 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:56.132 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:56.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:56.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:56.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:56.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:56.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:56.362 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:56.389 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:56.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:56.428 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-21T13:19:56.429 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:19:56.489 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:56.536 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:56.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:56.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:56.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:56 smithi163 ceph-mon[29621]: pgmap v992: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:19:56.662 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:56 smithi163 ceph-mon[40675]: pgmap v992: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:19:56.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:56.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:56.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:56.767 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:56.791 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:56.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:56.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:56.921 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:56.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:57.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:56 smithi204 ceph-mon[33153]: pgmap v992: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:19:57.023 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:57.047 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:57.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:57.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:57.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:57.176 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:57.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:57.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:57.302 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:57.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:57.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:57.429 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:57.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:57.531 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:57.557 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:57.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:57.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:57.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:57.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:57.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:57.785 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:57.810 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:57.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:57.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:57.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:57.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:57.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:57.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:58.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:58.063 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:58.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:58.164 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:58.188 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:58.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:58.251 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:19:58.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:58.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:58.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:58.417 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:58.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:58.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:58.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:58.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:19:58.567 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:58.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:58.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:19:58.573 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:19:58.585 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'unique_pool_4' removed 2024-03-21T13:19:58.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:58.692 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:58.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:58.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:58 smithi163 ceph-mon[29621]: pgmap v993: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:58.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:58 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/739650240' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:19:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:58 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:19:58.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:58 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3600867014' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:19:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:58 smithi163 ceph-mon[40675]: pgmap v993: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:58 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/739650240' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:19:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:58 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:19:58.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:58 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3600867014' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:19:58.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:58.818 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:58.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:58.919 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:58.945 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:58.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:58 smithi204 ceph-mon[33153]: pgmap v993: 148 pgs: 148 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:58 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/739650240' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:19:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:58 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:19:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:58 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3600867014' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:19:59.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:19:59.069 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:59.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:59.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:19:59.171 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:59.196 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:59.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:59.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:59.297 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:19:59.321 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:59.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:19:59.423 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:19:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:59.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:59.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:19:59.549 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:59.579 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:59.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:59.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:19:59.680 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:19:59.704 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:59.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:59.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:19:59.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:59 smithi163 ceph-mon[29621]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:19:59.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:59 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:19:59.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:59 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3600867014' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:19:59.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:19:59 smithi163 ceph-mon[29621]: osdmap e451: 8 total, 8 up, 6 in 2024-03-21T13:19:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:59 smithi163 ceph-mon[40675]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:19:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:59 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:19:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:59 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3600867014' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:19:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:19:59 smithi163 ceph-mon[40675]: osdmap e451: 8 total, 8 up, 6 in 2024-03-21T13:19:59.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:19:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:59.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:19:59.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:19:59.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:19:59.954 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:19:59.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:19:59.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:20:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:59 smithi204 ceph-mon[33153]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:20:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:59 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:20:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:59 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3600867014' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:20:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:19:59 smithi204 ceph-mon[33153]: osdmap e451: 8 total, 8 up, 6 in 2024-03-21T13:20:00.056 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:20:00.079 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:20:00.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:00.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:20:00.181 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:20:00.205 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:20:00.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:00.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:20:00.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:20:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:00.333 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:20:00.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:20:00.434 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:20:00.458 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:20:00.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:00.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:20:00.560 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:20:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:00.585 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:20:00.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:20:00.686 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:20:00.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:20:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:00.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:20:00.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:00 smithi163 ceph-mon[29621]: pgmap v995: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:20:00.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:00 smithi163 ceph-mon[29621]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-21T13:20:00.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:00 smithi163 ceph-mon[29621]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-21T13:20:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:00 smithi163 ceph-mon[40675]: pgmap v995: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:20:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:00 smithi163 ceph-mon[40675]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-21T13:20:00.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:00 smithi163 ceph-mon[40675]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-21T13:20:00.813 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:20:00.837 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:20:00.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:00.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:20:00.938 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:20:00.964 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:20:00.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:00.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:20:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:00 smithi204 ceph-mon[33153]: pgmap v995: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:20:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:00 smithi204 ceph-mon[33153]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-21T13:20:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:00 smithi204 ceph-mon[33153]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-21T13:20:01.066 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:20:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:01.090 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:20:01.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:20:01.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:20:01.218 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:20:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:01.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:20:01.319 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6 2024-03-21T13:20:01.344 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:20:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:01.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.6' 2024-03-21T13:20:01.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7 2024-03-21T13:20:01.471 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:20:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:01.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.7' 2024-03-21T13:20:01.573 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3 2024-03-21T13:20:01.601 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:20:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:01.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.3' 2024-03-21T13:20:01.693 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:01 smithi163 ceph-mon[29621]: 5.14 scrub starts 2024-03-21T13:20:01.693 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:01 smithi163 ceph-mon[29621]: 5.14 scrub ok 2024-03-21T13:20:01.693 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:01 smithi163 ceph-mon[40675]: 5.14 scrub starts 2024-03-21T13:20:01.693 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:01 smithi163 ceph-mon[40675]: 5.14 scrub ok 2024-03-21T13:20:01.703 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:20:01.728 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:20:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:01.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:20:01.829 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0 2024-03-21T13:20:01.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:01.854 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:20:01.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.0' 2024-03-21T13:20:01.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1 2024-03-21T13:20:01.977 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:20:01.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:01.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.1' 2024-03-21T13:20:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:01 smithi204 ceph-mon[33153]: 5.14 scrub starts 2024-03-21T13:20:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:01 smithi204 ceph-mon[33153]: 5.14 scrub ok 2024-03-21T13:20:02.080 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2 2024-03-21T13:20:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:02.104 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: docker: command not found 2024-03-21T13:20:02.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.2' 2024-03-21T13:20:02.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:20:02.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:20:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:02.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:20:02.237 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-03-21T13:20:02.256 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-03-21T13:20:02.275 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-03-21T13:20:02.294 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-03-21T13:20:02.314 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-03-21T13:20:02.334 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-03-21T13:20:02.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5 2024-03-21T13:20:02.354 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-03-21T13:20:02.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:20:02.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:02.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.5' 2024-03-21T13:20:02.374 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-03-21T13:20:02.375 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T13:20:02.375 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T13:20:02.375 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T13:20:02.375 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T13:20:02.375 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T13:20:02.375 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T13:20:02.376 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T13:20:02.376 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T13:20:02.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4 2024-03-21T13:20:02.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-03-21T13:20:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:20:02.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd.4' 2024-03-21T13:20:02.503 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:02.534 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-21T13:20:02.544 INFO:tasks.thrashosds:joining thrashosds 2024-03-21T13:20:02.545 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-03-21T13:20:02.545 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:20:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:02 smithi204 ceph-mon[33153]: pgmap v996: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:02.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:02 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1793385768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:20:02.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:20:02 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:20:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:02 smithi163 ceph-mon[29621]: pgmap v996: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:02.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:02 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1793385768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:20:02.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:02 smithi163 ceph-mon[40675]: pgmap v996: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:02.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:02 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1793385768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T13:20:02.898 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T13:20:03.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:03.979 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:04.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-21T13:20:04.860 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:04 smithi204 ceph-mon[33153]: pgmap v997: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:20:04.860 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:04 smithi204 ceph-mon[33153]: 4.4 scrub starts 2024-03-21T13:20:04.860 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:04 smithi204 ceph-mon[33153]: 4.4 scrub ok 2024-03-21T13:20:04.993 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:20:05.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:04 smithi163 ceph-mon[29621]: pgmap v997: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:20:05.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:04 smithi163 ceph-mon[29621]: 4.4 scrub starts 2024-03-21T13:20:05.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:04 smithi163 ceph-mon[29621]: 4.4 scrub ok 2024-03-21T13:20:05.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:04 smithi163 ceph-mon[40675]: pgmap v997: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:20:05.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:04 smithi163 ceph-mon[40675]: 4.4 scrub starts 2024-03-21T13:20:05.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:04 smithi163 ceph-mon[40675]: 4.4 scrub ok 2024-03-21T13:20:05.216 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:05.675 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:20:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:20:05.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[29621]: 4.10 scrub starts 2024-03-21T13:20:05.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[29621]: 4.10 scrub ok 2024-03-21T13:20:05.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[29621]: 3.19 scrub starts 2024-03-21T13:20:05.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[29621]: 3.19 scrub ok 2024-03-21T13:20:05.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[29621]: 5.18 scrub starts 2024-03-21T13:20:05.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[29621]: 5.18 scrub ok 2024-03-21T13:20:05.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[29621]: 6.1e scrub starts 2024-03-21T13:20:05.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[29621]: 6.1e scrub ok 2024-03-21T13:20:05.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[40675]: 4.10 scrub starts 2024-03-21T13:20:05.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[40675]: 4.10 scrub ok 2024-03-21T13:20:05.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[40675]: 3.19 scrub starts 2024-03-21T13:20:05.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[40675]: 3.19 scrub ok 2024-03-21T13:20:05.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[40675]: 5.18 scrub starts 2024-03-21T13:20:05.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[40675]: 5.18 scrub ok 2024-03-21T13:20:05.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[40675]: 6.1e scrub starts 2024-03-21T13:20:05.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:05 smithi163 ceph-mon[40675]: 6.1e scrub ok 2024-03-21T13:20:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:05 smithi204 ceph-mon[33153]: 4.10 scrub starts 2024-03-21T13:20:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:05 smithi204 ceph-mon[33153]: 4.10 scrub ok 2024-03-21T13:20:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:05 smithi204 ceph-mon[33153]: 3.19 scrub starts 2024-03-21T13:20:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:05 smithi204 ceph-mon[33153]: 3.19 scrub ok 2024-03-21T13:20:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:05 smithi204 ceph-mon[33153]: 5.18 scrub starts 2024-03-21T13:20:05.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:05 smithi204 ceph-mon[33153]: 5.18 scrub ok 2024-03-21T13:20:05.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:05 smithi204 ceph-mon[33153]: 6.1e scrub starts 2024-03-21T13:20:05.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:05 smithi204 ceph-mon[33153]: 6.1e scrub ok 2024-03-21T13:20:06.157 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:20:06.159 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:20:06.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:06 smithi204 ceph-mon[33153]: pgmap v998: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T13:20:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:06 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:06 smithi204 ceph-mon[33153]: 4.15 scrub starts 2024-03-21T13:20:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:06 smithi204 ceph-mon[33153]: 4.15 scrub ok 2024-03-21T13:20:06.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:06 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:20:06.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:06 smithi163 ceph-mon[29621]: pgmap v998: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T13:20:06.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:06 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:06.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:06 smithi163 ceph-mon[29621]: 4.15 scrub starts 2024-03-21T13:20:06.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:06 smithi163 ceph-mon[29621]: 4.15 scrub ok 2024-03-21T13:20:06.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:06 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:20:06.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:06 smithi163 ceph-mon[40675]: pgmap v998: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T13:20:06.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:06 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:06.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:06 smithi163 ceph-mon[40675]: 4.15 scrub starts 2024-03-21T13:20:06.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:06 smithi163 ceph-mon[40675]: 4.15 scrub ok 2024-03-21T13:20:06.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:06 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:20:07.127 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":998,"stamp":"2024-03-21T13:20:05.020419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3038,"num_read_kb":2884,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":437,"num_bytes_recovered":2372405,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1281,"ondisk_log_size":1281,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":420,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":5920492,"kb_used_data":6756,"kb_used_omap":9,"kb_used_meta":5913654,"kb_avail":556550420,"statfs":{"total":575970213888,"available":569907630080,"internally_reserved":0,"allocated":6918144,"data_stored":4364988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":6055582374},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":103,"apply_latency_ms":103,"commit_latency_ns":103000000,"apply_latency_ns":103000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":-243964,"num_write_kb":-7806785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-52495,"num_bytes_recovered":-3440246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002430"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564202+0000","last_change":"2024-03-21T13:19:02.498730+0000","last_active":"2024-03-21T13:19:59.564202+0000","last_peered":"2024-03-21T13:19:59.564202+0000","last_clean":"2024-03-21T13:19:59.564202+0000","last_became_active":"2024-03-21T13:12:29.525755+0000","last_became_peered":"2024-03-21T13:12:29.525755+0000","last_unstale":"2024-03-21T13:19:59.564202+0000","last_undegraded":"2024-03-21T13:19:59.564202+0000","last_fullsized":"2024-03-21T13:19:59.564202+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:02.498361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:19:02.498361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:05.047035+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":277,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564037+0000","last_change":"2024-03-21T13:19:25.553886+0000","last_active":"2024-03-21T13:19:59.564037+0000","last_peered":"2024-03-21T13:19:59.564037+0000","last_clean":"2024-03-21T13:19:59.564037+0000","last_became_active":"2024-03-21T13:12:29.510271+0000","last_became_peered":"2024-03-21T13:12:29.510271+0000","last_unstale":"2024-03-21T13:19:59.564037+0000","last_undegraded":"2024-03-21T13:19:59.564037+0000","last_fullsized":"2024-03-21T13:19:59.564037+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:25.553760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:25.553760+0000","last_clean_scrub_stamp":"2024-03-21T13:19:25.553760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:32.675838+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":208,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.584068+0000","last_change":"2024-03-21T13:19:10.481899+0000","last_active":"2024-03-21T13:19:59.584068+0000","last_peered":"2024-03-21T13:19:59.584068+0000","last_clean":"2024-03-21T13:19:59.584068+0000","last_became_active":"2024-03-21T13:17:27.657450+0000","last_became_peered":"2024-03-21T13:17:27.657450+0000","last_unstale":"2024-03-21T13:19:59.584068+0000","last_undegraded":"2024-03-21T13:19:59.584068+0000","last_fullsized":"2024-03-21T13:19:59.584068+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:10.481779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:10.481779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:37.511613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"435'25","reported_seq":930,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.571824+0000","last_change":"2024-03-21T13:19:06.028666+0000","last_active":"2024-03-21T13:19:58.571824+0000","last_peered":"2024-03-21T13:19:58.571824+0000","last_clean":"2024-03-21T13:19:58.571824+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:19:58.571824+0000","last_undegraded":"2024-03-21T13:19:58.571824+0000","last_fullsized":"2024-03-21T13:19:58.571824+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:19:06.028535+0000","last_deep_scrub":"234'19","last_deep_scrub_stamp":"2024-03-21T13:16:09.693933+0000","last_clean_scrub_stamp":"2024-03-21T13:19:06.028535+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:28.738583+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":819,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561121+0000","last_change":"2024-03-21T13:19:03.937707+0000","last_active":"2024-03-21T13:19:59.561121+0000","last_peered":"2024-03-21T13:19:59.561121+0000","last_clean":"2024-03-21T13:19:59.561121+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:19:59.561121+0000","last_undegraded":"2024-03-21T13:19:59.561121+0000","last_fullsized":"2024-03-21T13:19:59.561121+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:03.937453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:19:03.937453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:25.885052+0000","scrub_duration":275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":770,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559455+0000","last_change":"2024-03-21T13:17:12.860708+0000","last_active":"2024-03-21T13:19:59.559455+0000","last_peered":"2024-03-21T13:19:59.559455+0000","last_clean":"2024-03-21T13:19:59.559455+0000","last_became_active":"2024-03-21T13:10:03.591543+0000","last_became_peered":"2024-03-21T13:10:03.591543+0000","last_unstale":"2024-03-21T13:19:59.559455+0000","last_undegraded":"2024-03-21T13:19:59.559455+0000","last_fullsized":"2024-03-21T13:19:59.559455+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:12.860612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:17:12.860612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.781009+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":852,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563023+0000","last_change":"2024-03-21T13:17:03.752054+0000","last_active":"2024-03-21T13:19:59.563023+0000","last_peered":"2024-03-21T13:19:59.563023+0000","last_clean":"2024-03-21T13:19:59.563023+0000","last_became_active":"2024-03-21T13:10:03.610318+0000","last_became_peered":"2024-03-21T13:10:03.610318+0000","last_unstale":"2024-03-21T13:19:59.563023+0000","last_undegraded":"2024-03-21T13:19:59.563023+0000","last_fullsized":"2024-03-21T13:19:59.563023+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:17:03.751920+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_clean_scrub_stamp":"2024-03-21T13:17:03.751920+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"435'39","reported_seq":391,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563415+0000","last_change":"2024-03-21T13:17:27.668489+0000","last_active":"2024-03-21T13:19:59.563415+0000","last_peered":"2024-03-21T13:19:59.563415+0000","last_clean":"2024-03-21T13:19:59.563415+0000","last_became_active":"2024-03-21T13:17:27.652219+0000","last_became_peered":"2024-03-21T13:17:27.652219+0000","last_unstale":"2024-03-21T13:19:59.563415+0000","last_undegraded":"2024-03-21T13:19:59.563415+0000","last_fullsized":"2024-03-21T13:19:59.563415+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:17:01.724037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:01.724037+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265824+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":645,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560484+0000","last_change":"2024-03-21T13:17:27.645240+0000","last_active":"2024-03-21T13:19:59.560484+0000","last_peered":"2024-03-21T13:19:59.560484+0000","last_clean":"2024-03-21T13:19:59.560484+0000","last_became_active":"2024-03-21T13:17:27.641680+0000","last_became_peered":"2024-03-21T13:17:27.641680+0000","last_unstale":"2024-03-21T13:19:59.560484+0000","last_undegraded":"2024-03-21T13:19:59.560484+0000","last_fullsized":"2024-03-21T13:19:59.560484+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:20.951893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.951893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":275,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.574319+0000","last_change":"2024-03-21T13:19:06.558479+0000","last_active":"2024-03-21T13:19:59.574319+0000","last_peered":"2024-03-21T13:19:59.574319+0000","last_clean":"2024-03-21T13:19:59.574319+0000","last_became_active":"2024-03-21T13:17:27.659272+0000","last_became_peered":"2024-03-21T13:17:27.659272+0000","last_unstale":"2024-03-21T13:19:59.574319+0000","last_undegraded":"2024-03-21T13:19:59.574319+0000","last_fullsized":"2024-03-21T13:19:59.574319+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:06.558247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:19:06.558247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:20.044230+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.593391+0000","last_change":"2024-03-21T13:19:08.551736+0000","last_active":"2024-03-21T13:19:59.593391+0000","last_peered":"2024-03-21T13:19:59.593391+0000","last_clean":"2024-03-21T13:19:59.593391+0000","last_became_active":"2024-03-21T13:12:29.526129+0000","last_became_peered":"2024-03-21T13:12:29.526129+0000","last_unstale":"2024-03-21T13:19:59.593391+0000","last_undegraded":"2024-03-21T13:19:59.593391+0000","last_fullsized":"2024-03-21T13:19:59.593391+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:08.551622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:08.551622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:29.479156+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"435'25","reported_seq":855,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560664+0000","last_change":"2024-03-21T13:19:31.802513+0000","last_active":"2024-03-21T13:19:59.560664+0000","last_peered":"2024-03-21T13:19:59.560664+0000","last_clean":"2024-03-21T13:19:59.560664+0000","last_became_active":"2024-03-21T13:17:27.639498+0000","last_became_peered":"2024-03-21T13:17:27.639498+0000","last_unstale":"2024-03-21T13:19:59.560664+0000","last_undegraded":"2024-03-21T13:19:59.560664+0000","last_fullsized":"2024-03-21T13:19:59.560664+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:19:31.802397+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:19:31.802397+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:57.229185+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564144+0000","last_change":"2024-03-21T13:16:56.397703+0000","last_active":"2024-03-21T13:19:59.564144+0000","last_peered":"2024-03-21T13:19:59.564144+0000","last_clean":"2024-03-21T13:19:59.564144+0000","last_became_active":"2024-03-21T13:12:29.508520+0000","last_became_peered":"2024-03-21T13:12:29.508520+0000","last_unstale":"2024-03-21T13:19:59.564144+0000","last_undegraded":"2024-03-21T13:19:59.564144+0000","last_fullsized":"2024-03-21T13:19:59.564144+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:56.397585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:56.397585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681496+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":849,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562025+0000","last_change":"2024-03-21T13:17:27.641476+0000","last_active":"2024-03-21T13:19:59.562025+0000","last_peered":"2024-03-21T13:19:59.562025+0000","last_clean":"2024-03-21T13:19:59.562025+0000","last_became_active":"2024-03-21T13:17:27.641161+0000","last_became_peered":"2024-03-21T13:17:27.641161+0000","last_unstale":"2024-03-21T13:19:59.562025+0000","last_undegraded":"2024-03-21T13:19:59.562025+0000","last_fullsized":"2024-03-21T13:19:59.562025+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:18.791316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:18.791316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.565867+0000","last_change":"2024-03-21T13:19:14.479563+0000","last_active":"2024-03-21T13:19:59.565867+0000","last_peered":"2024-03-21T13:19:59.565867+0000","last_clean":"2024-03-21T13:19:59.565867+0000","last_became_active":"2024-03-21T13:12:29.524473+0000","last_became_peered":"2024-03-21T13:12:29.524473+0000","last_unstale":"2024-03-21T13:19:59.565867+0000","last_undegraded":"2024-03-21T13:19:59.565867+0000","last_fullsized":"2024-03-21T13:19:59.565867+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:14.479454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:14.479454+0000","last_clean_scrub_stamp":"2024-03-21T13:19:14.479454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:42.111248+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"435'11","reported_seq":853,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560832+0000","last_change":"2024-03-21T13:17:09.946959+0000","last_active":"2024-03-21T13:19:59.560832+0000","last_peered":"2024-03-21T13:19:59.560832+0000","last_clean":"2024-03-21T13:19:59.560832+0000","last_became_active":"2024-03-21T13:10:03.595745+0000","last_became_peered":"2024-03-21T13:10:03.595745+0000","last_unstale":"2024-03-21T13:19:59.560832+0000","last_undegraded":"2024-03-21T13:19:59.560832+0000","last_fullsized":"2024-03-21T13:19:59.560832+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'9","last_scrub_stamp":"2024-03-21T13:17:09.946815+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:17:09.946815+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":874,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.573792+0000","last_change":"2024-03-21T13:19:01.198041+0000","last_active":"2024-03-21T13:19:58.573792+0000","last_peered":"2024-03-21T13:19:58.573792+0000","last_clean":"2024-03-21T13:19:58.573792+0000","last_became_active":"2024-03-21T13:12:29.520189+0000","last_became_peered":"2024-03-21T13:12:29.520189+0000","last_unstale":"2024-03-21T13:19:58.573792+0000","last_undegraded":"2024-03-21T13:19:58.573792+0000","last_fullsized":"2024-03-21T13:19:58.573792+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:01.197700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:23.865785+0000","last_clean_scrub_stamp":"2024-03-21T13:19:01.197700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:05.021700+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":767,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.572853+0000","last_change":"2024-03-21T13:18:58.081858+0000","last_active":"2024-03-21T13:19:58.572853+0000","last_peered":"2024-03-21T13:19:58.572853+0000","last_clean":"2024-03-21T13:19:58.572853+0000","last_became_active":"2024-03-21T13:17:27.643258+0000","last_became_peered":"2024-03-21T13:17:27.643258+0000","last_unstale":"2024-03-21T13:19:58.572853+0000","last_undegraded":"2024-03-21T13:19:58.572853+0000","last_fullsized":"2024-03-21T13:19:58.572853+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:18:58.081747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:18:58.081747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:15.605977+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":637,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563965+0000","last_change":"2024-03-21T13:17:27.669084+0000","last_active":"2024-03-21T13:19:59.563965+0000","last_peered":"2024-03-21T13:19:59.563965+0000","last_clean":"2024-03-21T13:19:59.563965+0000","last_became_active":"2024-03-21T13:17:27.668691+0000","last_became_peered":"2024-03-21T13:17:27.668691+0000","last_unstale":"2024-03-21T13:19:59.563965+0000","last_undegraded":"2024-03-21T13:19:59.563965+0000","last_fullsized":"2024-03-21T13:19:59.563965+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:08.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"435'39","reported_seq":452,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.578264+0000","last_change":"2024-03-21T13:17:27.662499+0000","last_active":"2024-03-21T13:19:58.578264+0000","last_peered":"2024-03-21T13:19:58.578264+0000","last_clean":"2024-03-21T13:19:58.578264+0000","last_became_active":"2024-03-21T13:17:27.639926+0000","last_became_peered":"2024-03-21T13:17:27.639926+0000","last_unstale":"2024-03-21T13:19:58.578264+0000","last_undegraded":"2024-03-21T13:19:58.578264+0000","last_fullsized":"2024-03-21T13:19:58.578264+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:16:54.409873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.409873+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":845,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562709+0000","last_change":"2024-03-21T13:19:27.206524+0000","last_active":"2024-03-21T13:19:59.562709+0000","last_peered":"2024-03-21T13:19:59.562709+0000","last_clean":"2024-03-21T13:19:59.562709+0000","last_became_active":"2024-03-21T13:12:29.502808+0000","last_became_peered":"2024-03-21T13:12:29.502808+0000","last_unstale":"2024-03-21T13:19:59.562709+0000","last_undegraded":"2024-03-21T13:19:59.562709+0000","last_fullsized":"2024-03-21T13:19:59.562709+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:27.206409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:19:27.206409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:28.406687+0000","scrub_duration":354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":756,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559609+0000","last_change":"2024-03-21T13:17:15.990962+0000","last_active":"2024-03-21T13:19:59.559609+0000","last_peered":"2024-03-21T13:19:59.559609+0000","last_clean":"2024-03-21T13:19:59.559609+0000","last_became_active":"2024-03-21T13:10:03.612409+0000","last_became_peered":"2024-03-21T13:10:03.612409+0000","last_unstale":"2024-03-21T13:19:59.559609+0000","last_undegraded":"2024-03-21T13:19:59.559609+0000","last_fullsized":"2024-03-21T13:19:59.559609+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:15.990807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:15.990807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":896,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.572027+0000","last_change":"2024-03-21T13:19:13.983514+0000","last_active":"2024-03-21T13:19:58.572027+0000","last_peered":"2024-03-21T13:19:58.572027+0000","last_clean":"2024-03-21T13:19:58.572027+0000","last_became_active":"2024-03-21T13:17:27.630488+0000","last_became_peered":"2024-03-21T13:17:27.630488+0000","last_unstale":"2024-03-21T13:19:58.572027+0000","last_undegraded":"2024-03-21T13:19:58.572027+0000","last_fullsized":"2024-03-21T13:19:58.572027+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:13.983407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:13.983407+0000","last_clean_scrub_stamp":"2024-03-21T13:19:13.983407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:24.623704+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"435'24","reported_seq":896,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562321+0000","last_change":"2024-03-21T13:16:55.685055+0000","last_active":"2024-03-21T13:19:59.562321+0000","last_peered":"2024-03-21T13:19:59.562321+0000","last_clean":"2024-03-21T13:19:59.562321+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:19:59.562321+0000","last_undegraded":"2024-03-21T13:19:59.562321+0000","last_fullsized":"2024-03-21T13:19:59.562321+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-21T13:16:55.684896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:55.684896+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":248,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.597952+0000","last_change":"2024-03-21T13:17:05.780695+0000","last_active":"2024-03-21T13:19:59.597952+0000","last_peered":"2024-03-21T13:19:59.597952+0000","last_clean":"2024-03-21T13:19:59.597952+0000","last_became_active":"2024-03-21T13:13:06.420220+0000","last_became_peered":"2024-03-21T13:13:06.420220+0000","last_unstale":"2024-03-21T13:19:59.597952+0000","last_undegraded":"2024-03-21T13:19:59.597952+0000","last_fullsized":"2024-03-21T13:19:59.597952+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:17:05.780572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:05.780572+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265944+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":1101,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560316+0000","last_change":"2024-03-21T13:19:00.935061+0000","last_active":"2024-03-21T13:19:59.560316+0000","last_peered":"2024-03-21T13:19:59.560316+0000","last_clean":"2024-03-21T13:19:59.560316+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:19:59.560316+0000","last_undegraded":"2024-03-21T13:19:59.560316+0000","last_fullsized":"2024-03-21T13:19:59.560316+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:19:00.934946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:00.934946+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:16.680758+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"407'2","reported_seq":731,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559602+0000","last_change":"2024-03-21T13:16:58.735925+0000","last_active":"2024-03-21T13:19:59.559602+0000","last_peered":"2024-03-21T13:19:59.559602+0000","last_clean":"2024-03-21T13:19:59.559602+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:19:59.559602+0000","last_undegraded":"2024-03-21T13:19:59.559602+0000","last_fullsized":"2024-03-21T13:19:59.559602+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'2","last_scrub_stamp":"2024-03-21T13:16:58.735421+0000","last_deep_scrub":"407'2","last_deep_scrub_stamp":"2024-03-21T13:16:58.735421+0000","last_clean_scrub_stamp":"2024-03-21T13:16:58.735421+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"435'35","reported_seq":521,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560974+0000","last_change":"2024-03-21T13:17:27.682401+0000","last_active":"2024-03-21T13:19:59.560974+0000","last_peered":"2024-03-21T13:19:59.560974+0000","last_clean":"2024-03-21T13:19:59.560974+0000","last_became_active":"2024-03-21T13:17:27.655020+0000","last_became_peered":"2024-03-21T13:17:27.655020+0000","last_unstale":"2024-03-21T13:19:59.560974+0000","last_undegraded":"2024-03-21T13:19:59.560974+0000","last_fullsized":"2024-03-21T13:19:59.560974+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-03-21T13:17:09.876547+0000","last_deep_scrub":"410'25","last_deep_scrub_stamp":"2024-03-21T13:17:09.876547+0000","last_clean_scrub_stamp":"2024-03-21T13:17:09.876547+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265994+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.593320+0000","last_change":"2024-03-21T13:19:13.594539+0000","last_active":"2024-03-21T13:19:59.593320+0000","last_peered":"2024-03-21T13:19:59.593320+0000","last_clean":"2024-03-21T13:19:59.593320+0000","last_became_active":"2024-03-21T13:12:29.527129+0000","last_became_peered":"2024-03-21T13:12:29.527129+0000","last_unstale":"2024-03-21T13:19:59.593320+0000","last_undegraded":"2024-03-21T13:19:59.593320+0000","last_fullsized":"2024-03-21T13:19:59.593320+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:13.594428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:19:13.594428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:19.950884+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":658,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560819+0000","last_change":"2024-03-21T13:17:27.650677+0000","last_active":"2024-03-21T13:19:59.560819+0000","last_peered":"2024-03-21T13:19:59.560819+0000","last_clean":"2024-03-21T13:19:59.560819+0000","last_became_active":"2024-03-21T13:17:27.649940+0000","last_became_peered":"2024-03-21T13:17:27.649940+0000","last_unstale":"2024-03-21T13:19:59.560819+0000","last_undegraded":"2024-03-21T13:19:59.560819+0000","last_fullsized":"2024-03-21T13:19:59.560819+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:12.814107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:12.814107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":855,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559252+0000","last_change":"2024-03-21T13:17:27.662093+0000","last_active":"2024-03-21T13:19:59.559252+0000","last_peered":"2024-03-21T13:19:59.559252+0000","last_clean":"2024-03-21T13:19:59.559252+0000","last_became_active":"2024-03-21T13:17:27.661238+0000","last_became_peered":"2024-03-21T13:17:27.661238+0000","last_unstale":"2024-03-21T13:19:59.559252+0000","last_undegraded":"2024-03-21T13:19:59.559252+0000","last_fullsized":"2024-03-21T13:19:59.559252+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:04.838772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:04.838772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"435'32","reported_seq":957,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560428+0000","last_change":"2024-03-21T13:18:59.931958+0000","last_active":"2024-03-21T13:19:59.560428+0000","last_peered":"2024-03-21T13:19:59.560428+0000","last_clean":"2024-03-21T13:19:59.560428+0000","last_became_active":"2024-03-21T13:17:27.652511+0000","last_became_peered":"2024-03-21T13:17:27.652511+0000","last_unstale":"2024-03-21T13:19:59.560428+0000","last_undegraded":"2024-03-21T13:19:59.560428+0000","last_fullsized":"2024-03-21T13:19:59.560428+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'32","last_scrub_stamp":"2024-03-21T13:18:59.931818+0000","last_deep_scrub":"331'24","last_deep_scrub_stamp":"2024-03-21T13:15:09.670459+0000","last_clean_scrub_stamp":"2024-03-21T13:18:59.931818+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:22.289931+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":82,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561843+0000","last_change":"2024-03-21T13:17:27.637231+0000","last_active":"2024-03-21T13:19:59.561843+0000","last_peered":"2024-03-21T13:19:59.561843+0000","last_clean":"2024-03-21T13:19:59.561843+0000","last_became_active":"2024-03-21T13:17:27.636917+0000","last_became_peered":"2024-03-21T13:17:27.636917+0000","last_unstale":"2024-03-21T13:19:59.561843+0000","last_undegraded":"2024-03-21T13:19:59.561843+0000","last_fullsized":"2024-03-21T13:19:59.561843+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:59.238664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:16:59.238664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265745+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":840,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563596+0000","last_change":"2024-03-21T13:19:25.802629+0000","last_active":"2024-03-21T13:19:59.563596+0000","last_peered":"2024-03-21T13:19:59.563596+0000","last_clean":"2024-03-21T13:19:59.563596+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:19:59.563596+0000","last_undegraded":"2024-03-21T13:19:59.563596+0000","last_fullsized":"2024-03-21T13:19:59.563596+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:25.802407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:25.802407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:47.811294+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":192,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.584484+0000","last_change":"2024-03-21T13:17:27.660095+0000","last_active":"2024-03-21T13:19:59.584484+0000","last_peered":"2024-03-21T13:19:59.584484+0000","last_clean":"2024-03-21T13:19:59.584484+0000","last_became_active":"2024-03-21T13:17:27.657429+0000","last_became_peered":"2024-03-21T13:17:27.657429+0000","last_unstale":"2024-03-21T13:19:59.584484+0000","last_undegraded":"2024-03-21T13:19:59.584484+0000","last_fullsized":"2024-03-21T13:19:59.584484+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:13.840785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:22.955737+0000","last_clean_scrub_stamp":"2024-03-21T13:17:13.840785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"435'12","reported_seq":850,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559380+0000","last_change":"2024-03-21T13:16:20.734468+0000","last_active":"2024-03-21T13:19:59.559380+0000","last_peered":"2024-03-21T13:19:59.559380+0000","last_clean":"2024-03-21T13:19:59.559380+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:19:59.559380+0000","last_undegraded":"2024-03-21T13:19:59.559380+0000","last_fullsized":"2024-03-21T13:19:59.559380+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-21T13:16:20.734379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.734379+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":826,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561315+0000","last_change":"2024-03-21T13:18:23.062341+0000","last_active":"2024-03-21T13:19:59.561315+0000","last_peered":"2024-03-21T13:19:59.561315+0000","last_clean":"2024-03-21T13:19:59.561315+0000","last_became_active":"2024-03-21T13:17:27.636805+0000","last_became_peered":"2024-03-21T13:17:27.636805+0000","last_unstale":"2024-03-21T13:19:59.561315+0000","last_undegraded":"2024-03-21T13:19:59.561315+0000","last_fullsized":"2024-03-21T13:19:59.561315+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:18:23.062249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:18:23.062249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":82,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561474+0000","last_change":"2024-03-21T13:17:27.645475+0000","last_active":"2024-03-21T13:19:59.561474+0000","last_peered":"2024-03-21T13:19:59.561474+0000","last_clean":"2024-03-21T13:19:59.561474+0000","last_became_active":"2024-03-21T13:17:27.644954+0000","last_became_peered":"2024-03-21T13:17:27.644954+0000","last_unstale":"2024-03-21T13:19:59.561474+0000","last_undegraded":"2024-03-21T13:19:59.561474+0000","last_fullsized":"2024-03-21T13:19:59.561474+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:05.059443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.059443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":277,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566771+0000","last_change":"2024-03-21T13:19:27.585259+0000","last_active":"2024-03-21T13:19:59.566771+0000","last_peered":"2024-03-21T13:19:59.566771+0000","last_clean":"2024-03-21T13:19:59.566771+0000","last_became_active":"2024-03-21T13:12:29.507706+0000","last_became_peered":"2024-03-21T13:12:29.507706+0000","last_unstale":"2024-03-21T13:19:59.566771+0000","last_undegraded":"2024-03-21T13:19:59.566771+0000","last_fullsized":"2024-03-21T13:19:59.566771+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:27.585034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:27.585034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:44.850926+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"435'27","reported_seq":585,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559892+0000","last_change":"2024-03-21T13:19:10.678826+0000","last_active":"2024-03-21T13:19:59.559892+0000","last_peered":"2024-03-21T13:19:59.559892+0000","last_clean":"2024-03-21T13:19:59.559892+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:19:59.559892+0000","last_undegraded":"2024-03-21T13:19:59.559892+0000","last_fullsized":"2024-03-21T13:19:59.559892+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'27","last_scrub_stamp":"2024-03-21T13:19:10.678656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:19:10.678656+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:15.956533+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":184,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559428+0000","last_change":"2024-03-21T13:17:27.646425+0000","last_active":"2024-03-21T13:19:59.559428+0000","last_peered":"2024-03-21T13:19:59.559428+0000","last_clean":"2024-03-21T13:19:59.559428+0000","last_became_active":"2024-03-21T13:17:27.645569+0000","last_became_peered":"2024-03-21T13:17:27.645569+0000","last_unstale":"2024-03-21T13:19:59.559428+0000","last_undegraded":"2024-03-21T13:19:59.559428+0000","last_fullsized":"2024-03-21T13:19:59.559428+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:50.025891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:50.025891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":848,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561025+0000","last_change":"2024-03-21T13:19:07.646720+0000","last_active":"2024-03-21T13:19:59.561025+0000","last_peered":"2024-03-21T13:19:59.561025+0000","last_clean":"2024-03-21T13:19:59.561025+0000","last_became_active":"2024-03-21T13:17:27.645710+0000","last_became_peered":"2024-03-21T13:17:27.645710+0000","last_unstale":"2024-03-21T13:19:59.561025+0000","last_undegraded":"2024-03-21T13:19:59.561025+0000","last_fullsized":"2024-03-21T13:19:59.561025+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:07.646564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:07.646564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:17.935079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":837,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562621+0000","last_change":"2024-03-21T13:19:11.213286+0000","last_active":"2024-03-21T13:19:59.562621+0000","last_peered":"2024-03-21T13:19:59.562621+0000","last_clean":"2024-03-21T13:19:59.562621+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:19:59.562621+0000","last_undegraded":"2024-03-21T13:19:59.562621+0000","last_fullsized":"2024-03-21T13:19:59.562621+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:11.213182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:19.383090+0000","last_clean_scrub_stamp":"2024-03-21T13:19:11.213182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:33.803636+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"435'25","reported_seq":872,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561769+0000","last_change":"2024-03-21T13:19:30.825646+0000","last_active":"2024-03-21T13:19:59.561769+0000","last_peered":"2024-03-21T13:19:59.561769+0000","last_clean":"2024-03-21T13:19:59.561769+0000","last_became_active":"2024-03-21T13:17:27.653358+0000","last_became_peered":"2024-03-21T13:17:27.653358+0000","last_unstale":"2024-03-21T13:19:59.561769+0000","last_undegraded":"2024-03-21T13:19:59.561769+0000","last_fullsized":"2024-03-21T13:19:59.561769+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:19:30.825510+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:19:30.825510+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:59.841946+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":665,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562832+0000","last_change":"2024-03-21T13:17:08.823533+0000","last_active":"2024-03-21T13:19:59.562832+0000","last_peered":"2024-03-21T13:19:59.562832+0000","last_clean":"2024-03-21T13:19:59.562832+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:19:59.562832+0000","last_undegraded":"2024-03-21T13:19:59.562832+0000","last_fullsized":"2024-03-21T13:19:59.562832+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:08.823336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.823336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265960+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564432+0000","last_change":"2024-03-21T13:16:52.230103+0000","last_active":"2024-03-21T13:19:59.564432+0000","last_peered":"2024-03-21T13:19:59.564432+0000","last_clean":"2024-03-21T13:19:59.564432+0000","last_became_active":"2024-03-21T13:12:29.510100+0000","last_became_peered":"2024-03-21T13:12:29.510100+0000","last_unstale":"2024-03-21T13:19:59.564432+0000","last_undegraded":"2024-03-21T13:19:59.564432+0000","last_fullsized":"2024-03-21T13:19:59.564432+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.229670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:22.501055+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.229670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681468+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":542,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.598139+0000","last_change":"2024-03-21T13:15:59.651653+0000","last_active":"2024-03-21T13:19:59.598139+0000","last_peered":"2024-03-21T13:19:59.598139+0000","last_clean":"2024-03-21T13:19:59.598139+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:19:59.598139+0000","last_undegraded":"2024-03-21T13:19:59.598139+0000","last_fullsized":"2024-03-21T13:19:59.598139+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:59.651571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:15:59.651571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"435'27","reported_seq":114,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.572775+0000","last_change":"2024-03-21T13:19:23.826510+0000","last_active":"2024-03-21T13:19:58.572775+0000","last_peered":"2024-03-21T13:19:58.572775+0000","last_clean":"2024-03-21T13:19:58.572775+0000","last_became_active":"2024-03-21T13:17:27.653620+0000","last_became_peered":"2024-03-21T13:17:27.653620+0000","last_unstale":"2024-03-21T13:19:58.572775+0000","last_undegraded":"2024-03-21T13:19:58.572775+0000","last_fullsized":"2024-03-21T13:19:58.572775+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'27","last_scrub_stamp":"2024-03-21T13:19:23.826370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:19:23.826370+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:49.759356+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":645,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560549+0000","last_change":"2024-03-21T13:17:27.651621+0000","last_active":"2024-03-21T13:19:59.560549+0000","last_peered":"2024-03-21T13:19:59.560549+0000","last_clean":"2024-03-21T13:19:59.560549+0000","last_became_active":"2024-03-21T13:17:27.651255+0000","last_became_peered":"2024-03-21T13:17:27.651255+0000","last_unstale":"2024-03-21T13:19:59.560549+0000","last_undegraded":"2024-03-21T13:19:59.560549+0000","last_fullsized":"2024-03-21T13:19:59.560549+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:53.026764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.026764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"416'13","reported_seq":375,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.593437+0000","last_change":"2024-03-21T13:17:27.662539+0000","last_active":"2024-03-21T13:19:59.593437+0000","last_peered":"2024-03-21T13:19:59.593437+0000","last_clean":"2024-03-21T13:19:59.593437+0000","last_became_active":"2024-03-21T13:17:27.644197+0000","last_became_peered":"2024-03-21T13:17:27.644197+0000","last_unstale":"2024-03-21T13:19:59.593437+0000","last_undegraded":"2024-03-21T13:19:59.593437+0000","last_fullsized":"2024-03-21T13:19:59.593437+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'12","last_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_deep_scrub":"394'12","last_deep_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_clean_scrub_stamp":"2024-03-21T13:16:24.237800+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":850,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562793+0000","last_change":"2024-03-21T13:17:16.856970+0000","last_active":"2024-03-21T13:19:59.562793+0000","last_peered":"2024-03-21T13:19:59.562793+0000","last_clean":"2024-03-21T13:19:59.562793+0000","last_became_active":"2024-03-21T13:10:03.599265+0000","last_became_peered":"2024-03-21T13:10:03.599265+0000","last_unstale":"2024-03-21T13:19:59.562793+0000","last_undegraded":"2024-03-21T13:19:59.562793+0000","last_fullsized":"2024-03-21T13:19:59.562793+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:16.856845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:16.856845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265855+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"435'30","reported_seq":927,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562127+0000","last_change":"2024-03-21T13:18:24.919602+0000","last_active":"2024-03-21T13:19:59.562127+0000","last_peered":"2024-03-21T13:19:59.562127+0000","last_clean":"2024-03-21T13:19:59.562127+0000","last_became_active":"2024-03-21T13:10:03.613119+0000","last_became_peered":"2024-03-21T13:10:03.613119+0000","last_unstale":"2024-03-21T13:19:59.562127+0000","last_undegraded":"2024-03-21T13:19:59.562127+0000","last_fullsized":"2024-03-21T13:19:59.562127+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:18:24.919459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:18:24.919459+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":411,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.577016+0000","last_change":"2024-03-21T13:19:13.371982+0000","last_active":"2024-03-21T13:19:58.577016+0000","last_peered":"2024-03-21T13:19:58.577016+0000","last_clean":"2024-03-21T13:19:58.577016+0000","last_became_active":"2024-03-21T13:12:29.507751+0000","last_became_peered":"2024-03-21T13:12:29.507751+0000","last_unstale":"2024-03-21T13:19:58.577016+0000","last_undegraded":"2024-03-21T13:19:58.577016+0000","last_fullsized":"2024-03-21T13:19:58.577016+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:13.371894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:17.497325+0000","last_clean_scrub_stamp":"2024-03-21T13:19:13.371894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:15.436618+0000","scrub_duration":256,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"272'10","reported_seq":1110,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.773459+0000","last_change":"2024-03-21T13:19:59.773503+0000","last_active":"2024-03-21T13:19:59.773459+0000","last_peered":"2024-03-21T13:19:59.773459+0000","last_clean":"2024-03-21T13:19:59.773459+0000","last_became_active":"2024-03-21T13:10:03.594651+0000","last_became_peered":"2024-03-21T13:10:03.594651+0000","last_unstale":"2024-03-21T13:19:59.773459+0000","last_undegraded":"2024-03-21T13:19:59.773459+0000","last_fullsized":"2024-03-21T13:19:59.773459+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'10","last_scrub_stamp":"2024-03-21T13:19:59.773373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:59.773373+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:21:18.537222+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":653,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561941+0000","last_change":"2024-03-21T13:19:29.152268+0000","last_active":"2024-03-21T13:19:59.561941+0000","last_peered":"2024-03-21T13:19:59.561941+0000","last_clean":"2024-03-21T13:19:59.561941+0000","last_became_active":"2024-03-21T13:17:27.650261+0000","last_became_peered":"2024-03-21T13:17:27.650261+0000","last_unstale":"2024-03-21T13:19:59.561941+0000","last_undegraded":"2024-03-21T13:19:59.561941+0000","last_fullsized":"2024-03-21T13:19:59.561941+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:29.152059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:19:29.152059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:57.020571+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"435'21","reported_seq":288,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.593029+0000","last_change":"2024-03-21T13:16:22.227639+0000","last_active":"2024-03-21T13:19:59.593029+0000","last_peered":"2024-03-21T13:19:59.593029+0000","last_clean":"2024-03-21T13:19:59.593029+0000","last_became_active":"2024-03-21T13:12:29.510247+0000","last_became_peered":"2024-03-21T13:12:29.510247+0000","last_unstale":"2024-03-21T13:19:59.593029+0000","last_undegraded":"2024-03-21T13:19:59.593029+0000","last_fullsized":"2024-03-21T13:19:59.593029+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:16:22.227536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.227536+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681256+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":397,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.577042+0000","last_change":"2024-03-21T13:17:07.670896+0000","last_active":"2024-03-21T13:19:58.577042+0000","last_peered":"2024-03-21T13:19:58.577042+0000","last_clean":"2024-03-21T13:19:58.577042+0000","last_became_active":"2024-03-21T13:10:03.613841+0000","last_became_peered":"2024-03-21T13:10:03.613841+0000","last_unstale":"2024-03-21T13:19:58.577042+0000","last_undegraded":"2024-03-21T13:19:58.577042+0000","last_fullsized":"2024-03-21T13:19:58.577042+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:07.670782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:17:07.670782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169702+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":835,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563811+0000","last_change":"2024-03-21T13:17:27.659110+0000","last_active":"2024-03-21T13:19:59.563811+0000","last_peered":"2024-03-21T13:19:59.563811+0000","last_clean":"2024-03-21T13:19:59.563811+0000","last_became_active":"2024-03-21T13:17:27.658400+0000","last_became_peered":"2024-03-21T13:17:27.658400+0000","last_unstale":"2024-03-21T13:19:59.563811+0000","last_undegraded":"2024-03-21T13:19:59.563811+0000","last_fullsized":"2024-03-21T13:19:59.563811+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:13.018536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:13.018536+0000","last_clean_scrub_stamp":"2024-03-21T13:17:13.018536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":844,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563168+0000","last_change":"2024-03-21T13:16:54.653009+0000","last_active":"2024-03-21T13:19:59.563168+0000","last_peered":"2024-03-21T13:19:59.563168+0000","last_clean":"2024-03-21T13:19:59.563168+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:19:59.563168+0000","last_undegraded":"2024-03-21T13:19:59.563168+0000","last_fullsized":"2024-03-21T13:19:59.563168+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:16:54.652821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.652821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265712+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"435'21","reported_seq":699,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563355+0000","last_change":"2024-03-21T13:17:27.719115+0000","last_active":"2024-03-21T13:19:59.563355+0000","last_peered":"2024-03-21T13:19:59.563355+0000","last_clean":"2024-03-21T13:19:59.563355+0000","last_became_active":"2024-03-21T13:17:27.659084+0000","last_became_peered":"2024-03-21T13:17:27.659084+0000","last_unstale":"2024-03-21T13:19:59.563355+0000","last_undegraded":"2024-03-21T13:19:59.563355+0000","last_fullsized":"2024-03-21T13:19:59.563355+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:16:58.080100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:58.080100+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":825,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562466+0000","last_change":"2024-03-21T13:19:01.216170+0000","last_active":"2024-03-21T13:19:59.562466+0000","last_peered":"2024-03-21T13:19:59.562466+0000","last_clean":"2024-03-21T13:19:59.562466+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:19:59.562466+0000","last_undegraded":"2024-03-21T13:19:59.562466+0000","last_fullsized":"2024-03-21T13:19:59.562466+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:01.216060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:19:01.216060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:13.540476+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":277,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566693+0000","last_change":"2024-03-21T13:19:19.541389+0000","last_active":"2024-03-21T13:19:59.566693+0000","last_peered":"2024-03-21T13:19:59.566693+0000","last_clean":"2024-03-21T13:19:59.566693+0000","last_became_active":"2024-03-21T13:12:29.514113+0000","last_became_peered":"2024-03-21T13:12:29.514113+0000","last_unstale":"2024-03-21T13:19:59.566693+0000","last_undegraded":"2024-03-21T13:19:59.566693+0000","last_fullsized":"2024-03-21T13:19:59.566693+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:19.541273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:19.541273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:22.878402+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":858,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559385+0000","last_change":"2024-03-21T13:17:27.656617+0000","last_active":"2024-03-21T13:19:59.559385+0000","last_peered":"2024-03-21T13:19:59.559385+0000","last_clean":"2024-03-21T13:19:59.559385+0000","last_became_active":"2024-03-21T13:17:27.654135+0000","last_became_peered":"2024-03-21T13:17:27.654135+0000","last_unstale":"2024-03-21T13:19:59.559385+0000","last_undegraded":"2024-03-21T13:19:59.559385+0000","last_fullsized":"2024-03-21T13:19:59.559385+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:16.777565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:16.777565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.781051+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"435'18","reported_seq":896,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563469+0000","last_change":"2024-03-21T13:17:02.736704+0000","last_active":"2024-03-21T13:19:59.563469+0000","last_peered":"2024-03-21T13:19:59.563469+0000","last_clean":"2024-03-21T13:19:59.563469+0000","last_became_active":"2024-03-21T13:10:03.610810+0000","last_became_peered":"2024-03-21T13:10:03.610810+0000","last_unstale":"2024-03-21T13:19:59.563469+0000","last_undegraded":"2024-03-21T13:19:59.563469+0000","last_fullsized":"2024-03-21T13:19:59.563469+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-21T13:17:02.736579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:02.736579+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":275,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564769+0000","last_change":"2024-03-21T13:18:24.415828+0000","last_active":"2024-03-21T13:19:59.564769+0000","last_peered":"2024-03-21T13:19:59.564769+0000","last_clean":"2024-03-21T13:19:59.564769+0000","last_became_active":"2024-03-21T13:17:27.654442+0000","last_became_peered":"2024-03-21T13:17:27.654442+0000","last_unstale":"2024-03-21T13:19:59.564769+0000","last_undegraded":"2024-03-21T13:19:59.564769+0000","last_fullsized":"2024-03-21T13:19:59.564769+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:18:24.415712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:18:24.415712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681728+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":886,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.565342+0000","last_change":"2024-03-21T13:17:27.677784+0000","last_active":"2024-03-21T13:19:59.565342+0000","last_peered":"2024-03-21T13:19:59.565342+0000","last_clean":"2024-03-21T13:19:59.565342+0000","last_became_active":"2024-03-21T13:17:27.656174+0000","last_became_peered":"2024-03-21T13:17:27.656174+0000","last_unstale":"2024-03-21T13:19:59.565342+0000","last_undegraded":"2024-03-21T13:19:59.565342+0000","last_fullsized":"2024-03-21T13:19:59.565342+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:16:56.698173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:56.698173+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265925+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"435'39","reported_seq":303,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559456+0000","last_change":"2024-03-21T13:16:53.176991+0000","last_active":"2024-03-21T13:19:59.559456+0000","last_peered":"2024-03-21T13:19:59.559456+0000","last_clean":"2024-03-21T13:19:59.559456+0000","last_became_active":"2024-03-21T13:12:29.513019+0000","last_became_peered":"2024-03-21T13:12:29.513019+0000","last_unstale":"2024-03-21T13:19:59.559456+0000","last_undegraded":"2024-03-21T13:19:59.559456+0000","last_fullsized":"2024-03-21T13:19:59.559456+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:16:53.176836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.176836+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681534+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":1099,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563502+0000","last_change":"2024-03-21T13:19:21.182012+0000","last_active":"2024-03-21T13:19:59.563502+0000","last_peered":"2024-03-21T13:19:59.563502+0000","last_clean":"2024-03-21T13:19:59.563502+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:19:59.563502+0000","last_undegraded":"2024-03-21T13:19:59.563502+0000","last_fullsized":"2024-03-21T13:19:59.563502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:19:21.181898+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:19:21.181898+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:46.906471+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":826,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561212+0000","last_change":"2024-03-21T13:19:11.950569+0000","last_active":"2024-03-21T13:19:59.561212+0000","last_peered":"2024-03-21T13:19:59.561212+0000","last_clean":"2024-03-21T13:19:59.561212+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:19:59.561212+0000","last_undegraded":"2024-03-21T13:19:59.561212+0000","last_fullsized":"2024-03-21T13:19:59.561212+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:11.950446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:11.950446+0000","last_clean_scrub_stamp":"2024-03-21T13:19:11.950446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:17.756677+0000","scrub_duration":296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":850,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560704+0000","last_change":"2024-03-21T13:17:27.653349+0000","last_active":"2024-03-21T13:19:59.560704+0000","last_peered":"2024-03-21T13:19:59.560704+0000","last_clean":"2024-03-21T13:19:59.560704+0000","last_became_active":"2024-03-21T13:17:27.652993+0000","last_became_peered":"2024-03-21T13:17:27.652993+0000","last_unstale":"2024-03-21T13:19:59.560704+0000","last_undegraded":"2024-03-21T13:19:59.560704+0000","last_fullsized":"2024-03-21T13:19:59.560704+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.653388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.653388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265729+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"435'29","reported_seq":257,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.576332+0000","last_change":"2024-03-21T13:19:19.130198+0000","last_active":"2024-03-21T13:19:58.576332+0000","last_peered":"2024-03-21T13:19:58.576332+0000","last_clean":"2024-03-21T13:19:58.576332+0000","last_became_active":"2024-03-21T13:13:21.819111+0000","last_became_peered":"2024-03-21T13:13:21.819111+0000","last_unstale":"2024-03-21T13:19:58.576332+0000","last_undegraded":"2024-03-21T13:19:58.576332+0000","last_fullsized":"2024-03-21T13:19:58.576332+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'29","last_scrub_stamp":"2024-03-21T13:19:19.130049+0000","last_deep_scrub":"346'23","last_deep_scrub_stamp":"2024-03-21T13:13:20.232807+0000","last_clean_scrub_stamp":"2024-03-21T13:19:19.130049+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:30.549031+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":30,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":409,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.576099+0000","last_change":"2024-03-21T13:17:27.633122+0000","last_active":"2024-03-21T13:19:58.576099+0000","last_peered":"2024-03-21T13:19:58.576099+0000","last_clean":"2024-03-21T13:19:58.576099+0000","last_became_active":"2024-03-21T13:17:27.632559+0000","last_became_peered":"2024-03-21T13:17:27.632559+0000","last_unstale":"2024-03-21T13:19:58.576099+0000","last_undegraded":"2024-03-21T13:19:58.576099+0000","last_fullsized":"2024-03-21T13:19:58.576099+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:55.431004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:55.431004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169583+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":846,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560592+0000","last_change":"2024-03-21T13:19:04.089379+0000","last_active":"2024-03-21T13:19:59.560592+0000","last_peered":"2024-03-21T13:19:59.560592+0000","last_clean":"2024-03-21T13:19:59.560592+0000","last_became_active":"2024-03-21T13:10:03.598366+0000","last_became_peered":"2024-03-21T13:10:03.598366+0000","last_unstale":"2024-03-21T13:19:59.560592+0000","last_undegraded":"2024-03-21T13:19:59.560592+0000","last_fullsized":"2024-03-21T13:19:59.560592+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:04.089161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:19:04.089161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:10.879320+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":417,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.576592+0000","last_change":"2024-03-21T13:17:27.650325+0000","last_active":"2024-03-21T13:19:58.576592+0000","last_peered":"2024-03-21T13:19:58.576592+0000","last_clean":"2024-03-21T13:19:58.576592+0000","last_became_active":"2024-03-21T13:17:27.649975+0000","last_became_peered":"2024-03-21T13:17:27.649975+0000","last_unstale":"2024-03-21T13:19:58.576592+0000","last_undegraded":"2024-03-21T13:19:58.576592+0000","last_fullsized":"2024-03-21T13:19:58.576592+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:49.448804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:49.448804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169554+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":850,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.578942+0000","last_change":"2024-03-21T13:16:47.661139+0000","last_active":"2024-03-21T13:19:59.578942+0000","last_peered":"2024-03-21T13:19:59.578942+0000","last_clean":"2024-03-21T13:19:59.578942+0000","last_became_active":"2024-03-21T13:12:29.508984+0000","last_became_peered":"2024-03-21T13:12:29.508984+0000","last_unstale":"2024-03-21T13:19:59.578942+0000","last_undegraded":"2024-03-21T13:19:59.578942+0000","last_fullsized":"2024-03-21T13:19:59.578942+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:47.661063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:47.661063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265806+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"435'51","reported_seq":1001,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560830+0000","last_change":"2024-03-21T13:19:31.884114+0000","last_active":"2024-03-21T13:19:59.560830+0000","last_peered":"2024-03-21T13:19:59.560830+0000","last_clean":"2024-03-21T13:19:59.560830+0000","last_became_active":"2024-03-21T13:12:29.523359+0000","last_became_peered":"2024-03-21T13:12:29.523359+0000","last_unstale":"2024-03-21T13:19:59.560830+0000","last_undegraded":"2024-03-21T13:19:59.560830+0000","last_fullsized":"2024-03-21T13:19:59.560830+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'51","last_scrub_stamp":"2024-03-21T13:19:31.883967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:19:31.883967+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:56.608448+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":650,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562978+0000","last_change":"2024-03-21T13:17:27.649278+0000","last_active":"2024-03-21T13:19:59.562978+0000","last_peered":"2024-03-21T13:19:59.562978+0000","last_clean":"2024-03-21T13:19:59.562978+0000","last_became_active":"2024-03-21T13:17:27.648954+0000","last_became_peered":"2024-03-21T13:17:27.648954+0000","last_unstale":"2024-03-21T13:19:59.562978+0000","last_undegraded":"2024-03-21T13:19:59.562978+0000","last_fullsized":"2024-03-21T13:19:59.562978+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:07.141148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:07.141148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.266010+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":397,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579914+0000","last_change":"2024-03-21T13:16:56.458265+0000","last_active":"2024-03-21T13:19:58.579914+0000","last_peered":"2024-03-21T13:19:58.579914+0000","last_clean":"2024-03-21T13:19:58.579914+0000","last_became_active":"2024-03-21T13:10:03.603646+0000","last_became_peered":"2024-03-21T13:10:03.603646+0000","last_unstale":"2024-03-21T13:19:58.579914+0000","last_undegraded":"2024-03-21T13:19:58.579914+0000","last_fullsized":"2024-03-21T13:19:58.579914+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:56.458050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:56.458050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169640+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.565756+0000","last_change":"2024-03-21T13:16:54.243549+0000","last_active":"2024-03-21T13:19:59.565756+0000","last_peered":"2024-03-21T13:19:59.565756+0000","last_clean":"2024-03-21T13:19:59.565756+0000","last_became_active":"2024-03-21T13:12:29.512741+0000","last_became_peered":"2024-03-21T13:12:29.512741+0000","last_unstale":"2024-03-21T13:19:59.565756+0000","last_undegraded":"2024-03-21T13:19:59.565756+0000","last_fullsized":"2024-03-21T13:19:59.565756+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:54.243395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.243395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681394+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"435'25","reported_seq":761,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.572333+0000","last_change":"2024-03-21T13:19:09.020946+0000","last_active":"2024-03-21T13:19:58.572333+0000","last_peered":"2024-03-21T13:19:58.572333+0000","last_clean":"2024-03-21T13:19:58.572333+0000","last_became_active":"2024-03-21T13:17:27.653652+0000","last_became_peered":"2024-03-21T13:17:27.653652+0000","last_unstale":"2024-03-21T13:19:58.572333+0000","last_undegraded":"2024-03-21T13:19:58.572333+0000","last_fullsized":"2024-03-21T13:19:58.572333+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:19:09.020814+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:19:09.020814+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:34.076510+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":812,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563244+0000","last_change":"2024-03-21T13:17:06.844511+0000","last_active":"2024-03-21T13:19:59.563244+0000","last_peered":"2024-03-21T13:19:59.563244+0000","last_clean":"2024-03-21T13:19:59.563244+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:19:59.563244+0000","last_undegraded":"2024-03-21T13:19:59.563244+0000","last_fullsized":"2024-03-21T13:19:59.563244+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:06.844387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:17:06.844387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":283,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564255+0000","last_change":"2024-03-21T13:19:15.494560+0000","last_active":"2024-03-21T13:19:59.564255+0000","last_peered":"2024-03-21T13:19:59.564255+0000","last_clean":"2024-03-21T13:19:59.564255+0000","last_became_active":"2024-03-21T13:17:27.652655+0000","last_became_peered":"2024-03-21T13:17:27.652655+0000","last_unstale":"2024-03-21T13:19:59.564255+0000","last_undegraded":"2024-03-21T13:19:59.564255+0000","last_fullsized":"2024-03-21T13:19:59.564255+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:19:15.494435+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:19:15.494435+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:38.244906+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":465,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566021+0000","last_change":"2024-03-21T13:19:24.818885+0000","last_active":"2024-03-21T13:19:59.566021+0000","last_peered":"2024-03-21T13:19:59.566021+0000","last_clean":"2024-03-21T13:19:59.566021+0000","last_became_active":"2024-03-21T13:17:27.652986+0000","last_became_peered":"2024-03-21T13:17:27.652986+0000","last_unstale":"2024-03-21T13:19:59.566021+0000","last_undegraded":"2024-03-21T13:19:59.566021+0000","last_fullsized":"2024-03-21T13:19:59.566021+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:24.818628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:22.861136+0000","last_clean_scrub_stamp":"2024-03-21T13:19:24.818628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:53.242996+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"435'55","reported_seq":557,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.576765+0000","last_change":"2024-03-21T13:16:16.292178+0000","last_active":"2024-03-21T13:19:58.576765+0000","last_peered":"2024-03-21T13:19:58.576765+0000","last_clean":"2024-03-21T13:19:58.576765+0000","last_became_active":"2024-03-21T13:10:03.603889+0000","last_became_peered":"2024-03-21T13:10:03.603889+0000","last_unstale":"2024-03-21T13:19:58.576765+0000","last_undegraded":"2024-03-21T13:19:58.576765+0000","last_fullsized":"2024-03-21T13:19:58.576765+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'43","last_scrub_stamp":"2024-03-21T13:16:16.292073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:16.292073+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169486+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":469,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.569688+0000","last_change":"2024-03-21T13:19:06.882018+0000","last_active":"2024-03-21T13:19:59.569688+0000","last_peered":"2024-03-21T13:19:59.569688+0000","last_clean":"2024-03-21T13:19:59.569688+0000","last_became_active":"2024-03-21T13:17:27.643330+0000","last_became_peered":"2024-03-21T13:17:27.643330+0000","last_unstale":"2024-03-21T13:19:59.569688+0000","last_undegraded":"2024-03-21T13:19:59.569688+0000","last_fullsized":"2024-03-21T13:19:59.569688+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:06.881903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:15.968926+0000","last_clean_scrub_stamp":"2024-03-21T13:19:06.881903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:08.464724+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":827,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.579167+0000","last_change":"2024-03-21T13:16:18.683140+0000","last_active":"2024-03-21T13:19:59.579167+0000","last_peered":"2024-03-21T13:19:59.579167+0000","last_clean":"2024-03-21T13:19:59.579167+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:19:59.579167+0000","last_undegraded":"2024-03-21T13:19:59.579167+0000","last_fullsized":"2024-03-21T13:19:59.579167+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.683041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265694+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":403,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.576110+0000","last_change":"2024-03-21T13:17:27.634210+0000","last_active":"2024-03-21T13:19:58.576110+0000","last_peered":"2024-03-21T13:19:58.576110+0000","last_clean":"2024-03-21T13:19:58.576110+0000","last_became_active":"2024-03-21T13:17:27.633871+0000","last_became_peered":"2024-03-21T13:17:27.633871+0000","last_unstale":"2024-03-21T13:19:58.576110+0000","last_undegraded":"2024-03-21T13:19:58.576110+0000","last_fullsized":"2024-03-21T13:19:58.576110+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:12.859249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.231115+0000","last_clean_scrub_stamp":"2024-03-21T13:17:12.859249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169733+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":403,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.576500+0000","last_change":"2024-03-21T13:17:27.640952+0000","last_active":"2024-03-21T13:19:58.576500+0000","last_peered":"2024-03-21T13:19:58.576500+0000","last_clean":"2024-03-21T13:19:58.576500+0000","last_became_active":"2024-03-21T13:17:27.640621+0000","last_became_peered":"2024-03-21T13:17:27.640621+0000","last_unstale":"2024-03-21T13:19:58.576500+0000","last_undegraded":"2024-03-21T13:19:58.576500+0000","last_fullsized":"2024-03-21T13:19:58.576500+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.465427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.465427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"435'24","reported_seq":305,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.584572+0000","last_change":"2024-03-21T13:17:27.722357+0000","last_active":"2024-03-21T13:19:59.584572+0000","last_peered":"2024-03-21T13:19:59.584572+0000","last_clean":"2024-03-21T13:19:59.584572+0000","last_became_active":"2024-03-21T13:17:27.660008+0000","last_became_peered":"2024-03-21T13:17:27.660008+0000","last_unstale":"2024-03-21T13:19:59.584572+0000","last_undegraded":"2024-03-21T13:19:59.584572+0000","last_fullsized":"2024-03-21T13:19:59.584572+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'20","last_scrub_stamp":"2024-03-21T13:17:18.348476+0000","last_deep_scrub":"243'20","last_deep_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_clean_scrub_stamp":"2024-03-21T13:17:18.348476+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681662+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":403,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579644+0000","last_change":"2024-03-21T13:19:22.107531+0000","last_active":"2024-03-21T13:19:58.579644+0000","last_peered":"2024-03-21T13:19:58.579644+0000","last_clean":"2024-03-21T13:19:58.579644+0000","last_became_active":"2024-03-21T13:12:29.506751+0000","last_became_peered":"2024-03-21T13:12:29.506751+0000","last_unstale":"2024-03-21T13:19:58.579644+0000","last_undegraded":"2024-03-21T13:19:58.579644+0000","last_fullsized":"2024-03-21T13:19:58.579644+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:22.107289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:22.107289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:45.912848+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.584398+0000","last_change":"2024-03-21T13:19:03.668928+0000","last_active":"2024-03-21T13:19:59.584398+0000","last_peered":"2024-03-21T13:19:59.584398+0000","last_clean":"2024-03-21T13:19:59.584398+0000","last_became_active":"2024-03-21T13:12:29.527083+0000","last_became_peered":"2024-03-21T13:12:29.527083+0000","last_unstale":"2024-03-21T13:19:59.584398+0000","last_undegraded":"2024-03-21T13:19:59.584398+0000","last_fullsized":"2024-03-21T13:19:59.584398+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:03.668710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:19:03.668710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:21.438873+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":1112,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560813+0000","last_change":"2024-03-21T13:19:09.154578+0000","last_active":"2024-03-21T13:19:59.560813+0000","last_peered":"2024-03-21T13:19:59.560813+0000","last_clean":"2024-03-21T13:19:59.560813+0000","last_became_active":"2024-03-21T13:17:30.671981+0000","last_became_peered":"2024-03-21T13:17:30.671981+0000","last_unstale":"2024-03-21T13:19:59.560813+0000","last_undegraded":"2024-03-21T13:19:59.560813+0000","last_fullsized":"2024-03-21T13:19:59.560813+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:19:09.154442+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:19:09.154442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:30.186517+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":343,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579656+0000","last_change":"2024-03-21T13:19:11.122953+0000","last_active":"2024-03-21T13:19:58.579656+0000","last_peered":"2024-03-21T13:19:58.579656+0000","last_clean":"2024-03-21T13:19:58.579656+0000","last_became_active":"2024-03-21T13:12:29.499989+0000","last_became_peered":"2024-03-21T13:12:29.499989+0000","last_unstale":"2024-03-21T13:19:58.579656+0000","last_undegraded":"2024-03-21T13:19:58.579656+0000","last_fullsized":"2024-03-21T13:19:58.579656+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:11.122754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:11.122754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:37.216401+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"435'48","reported_seq":1160,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.574702+0000","last_change":"2024-03-21T13:19:16.865894+0000","last_active":"2024-03-21T13:19:59.574702+0000","last_peered":"2024-03-21T13:19:59.574702+0000","last_clean":"2024-03-21T13:19:59.574702+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:19:59.574702+0000","last_undegraded":"2024-03-21T13:19:59.574702+0000","last_fullsized":"2024-03-21T13:19:59.574702+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'48","last_scrub_stamp":"2024-03-21T13:19:16.865744+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:19:16.865744+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:25.169108+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":855,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.598278+0000","last_change":"2024-03-21T13:17:02.759517+0000","last_active":"2024-03-21T13:19:59.598278+0000","last_peered":"2024-03-21T13:19:59.598278+0000","last_clean":"2024-03-21T13:19:59.598278+0000","last_became_active":"2024-03-21T13:12:29.507753+0000","last_became_peered":"2024-03-21T13:12:29.507753+0000","last_unstale":"2024-03-21T13:19:59.598278+0000","last_undegraded":"2024-03-21T13:19:59.598278+0000","last_fullsized":"2024-03-21T13:19:59.598278+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:02.759379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:17:02.759379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265761+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.565927+0000","last_change":"2024-03-21T13:19:21.577055+0000","last_active":"2024-03-21T13:19:59.565927+0000","last_peered":"2024-03-21T13:19:59.565927+0000","last_clean":"2024-03-21T13:19:59.565927+0000","last_became_active":"2024-03-21T13:12:29.514542+0000","last_became_peered":"2024-03-21T13:12:29.514542+0000","last_unstale":"2024-03-21T13:19:59.565927+0000","last_undegraded":"2024-03-21T13:19:59.565927+0000","last_fullsized":"2024-03-21T13:19:59.565927+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:21.576900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:19:21.576900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:22.445820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":678,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.571996+0000","last_change":"2024-03-21T13:19:16.959435+0000","last_active":"2024-03-21T13:19:58.571996+0000","last_peered":"2024-03-21T13:19:58.571996+0000","last_clean":"2024-03-21T13:19:58.571996+0000","last_became_active":"2024-03-21T13:17:27.639147+0000","last_became_peered":"2024-03-21T13:17:27.639147+0000","last_unstale":"2024-03-21T13:19:58.571996+0000","last_undegraded":"2024-03-21T13:19:58.571996+0000","last_fullsized":"2024-03-21T13:19:58.571996+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:16.959312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:00.086941+0000","last_clean_scrub_stamp":"2024-03-21T13:19:16.959312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:39.128234+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":2142,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560743+0000","last_change":"2024-03-21T13:19:19.184255+0000","last_active":"2024-03-21T13:19:59.560743+0000","last_peered":"2024-03-21T13:19:59.560743+0000","last_clean":"2024-03-21T13:19:59.560743+0000","last_became_active":"2024-03-21T13:17:27.656453+0000","last_became_peered":"2024-03-21T13:17:27.656453+0000","last_unstale":"2024-03-21T13:19:59.560743+0000","last_undegraded":"2024-03-21T13:19:59.560743+0000","last_fullsized":"2024-03-21T13:19:59.560743+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:19:19.184135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:19:19.184135+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:24.075770+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1271,"num_read_kb":1266,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"435'36","reported_seq":670,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579400+0000","last_change":"2024-03-21T13:19:12.131227+0000","last_active":"2024-03-21T13:19:58.579400+0000","last_peered":"2024-03-21T13:19:58.579400+0000","last_clean":"2024-03-21T13:19:58.579400+0000","last_became_active":"2024-03-21T13:17:27.645689+0000","last_became_peered":"2024-03-21T13:17:27.645689+0000","last_unstale":"2024-03-21T13:19:58.579400+0000","last_undegraded":"2024-03-21T13:19:58.579400+0000","last_fullsized":"2024-03-21T13:19:58.579400+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'36","last_scrub_stamp":"2024-03-21T13:19:12.131071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:19:12.131071+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:35.728431+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":275,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.584319+0000","last_change":"2024-03-21T13:17:27.662358+0000","last_active":"2024-03-21T13:19:59.584319+0000","last_peered":"2024-03-21T13:19:59.584319+0000","last_clean":"2024-03-21T13:19:59.584319+0000","last_became_active":"2024-03-21T13:17:27.658711+0000","last_became_peered":"2024-03-21T13:17:27.658711+0000","last_unstale":"2024-03-21T13:19:59.584319+0000","last_undegraded":"2024-03-21T13:19:59.584319+0000","last_fullsized":"2024-03-21T13:19:59.584319+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:08.237986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:08.237986+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.237986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":737,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563466+0000","last_change":"2024-03-21T13:19:29.809130+0000","last_active":"2024-03-21T13:19:59.563466+0000","last_peered":"2024-03-21T13:19:59.563466+0000","last_clean":"2024-03-21T13:19:59.563466+0000","last_became_active":"2024-03-21T13:10:03.591088+0000","last_became_peered":"2024-03-21T13:10:03.591088+0000","last_unstale":"2024-03-21T13:19:59.563466+0000","last_undegraded":"2024-03-21T13:19:59.563466+0000","last_fullsized":"2024-03-21T13:19:59.563466+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:29.809009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:29.809009+0000","last_clean_scrub_stamp":"2024-03-21T13:19:29.809009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:52.868665+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":417,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.578120+0000","last_change":"2024-03-21T13:19:10.154315+0000","last_active":"2024-03-21T13:19:58.578120+0000","last_peered":"2024-03-21T13:19:58.578120+0000","last_clean":"2024-03-21T13:19:58.578120+0000","last_became_active":"2024-03-21T13:17:27.639129+0000","last_became_peered":"2024-03-21T13:17:27.639129+0000","last_unstale":"2024-03-21T13:19:58.578120+0000","last_undegraded":"2024-03-21T13:19:58.578120+0000","last_fullsized":"2024-03-21T13:19:58.578120+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:10.154113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:10.154113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:26.879527+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"435'39","reported_seq":326,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.574510+0000","last_change":"2024-03-21T13:17:27.724123+0000","last_active":"2024-03-21T13:19:59.574510+0000","last_peered":"2024-03-21T13:19:59.574510+0000","last_clean":"2024-03-21T13:19:59.574510+0000","last_became_active":"2024-03-21T13:17:27.658954+0000","last_became_peered":"2024-03-21T13:17:27.658954+0000","last_unstale":"2024-03-21T13:19:59.574510+0000","last_undegraded":"2024-03-21T13:19:59.574510+0000","last_fullsized":"2024-03-21T13:19:59.574510+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-21T13:17:03.309259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:03.309259+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681567+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":863,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560968+0000","last_change":"2024-03-21T13:17:27.653252+0000","last_active":"2024-03-21T13:19:59.560968+0000","last_peered":"2024-03-21T13:19:59.560968+0000","last_clean":"2024-03-21T13:19:59.560968+0000","last_became_active":"2024-03-21T13:17:27.652408+0000","last_became_peered":"2024-03-21T13:17:27.652408+0000","last_unstale":"2024-03-21T13:19:59.560968+0000","last_undegraded":"2024-03-21T13:19:59.560968+0000","last_fullsized":"2024-03-21T13:19:59.560968+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:02.830721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:02.830721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.781030+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":649,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559245+0000","last_change":"2024-03-21T13:17:27.646840+0000","last_active":"2024-03-21T13:19:59.559245+0000","last_peered":"2024-03-21T13:19:59.559245+0000","last_clean":"2024-03-21T13:19:59.559245+0000","last_became_active":"2024-03-21T13:17:27.646513+0000","last_became_peered":"2024-03-21T13:17:27.646513+0000","last_unstale":"2024-03-21T13:19:59.559245+0000","last_undegraded":"2024-03-21T13:19:59.559245+0000","last_fullsized":"2024-03-21T13:19:59.559245+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:51.069883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:51.069883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.593125+0000","last_change":"2024-03-21T13:19:24.587323+0000","last_active":"2024-03-21T13:19:59.593125+0000","last_peered":"2024-03-21T13:19:59.593125+0000","last_clean":"2024-03-21T13:19:59.593125+0000","last_became_active":"2024-03-21T13:12:29.512291+0000","last_became_peered":"2024-03-21T13:12:29.512291+0000","last_unstale":"2024-03-21T13:19:59.593125+0000","last_undegraded":"2024-03-21T13:19:59.593125+0000","last_fullsized":"2024-03-21T13:19:59.593125+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:24.587197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:24.587197+0000","last_clean_scrub_stamp":"2024-03-21T13:19:24.587197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:49.477367+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"435'35","reported_seq":314,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564664+0000","last_change":"2024-03-21T13:19:12.474259+0000","last_active":"2024-03-21T13:19:59.564664+0000","last_peered":"2024-03-21T13:19:59.564664+0000","last_clean":"2024-03-21T13:19:59.564664+0000","last_became_active":"2024-03-21T13:17:27.646744+0000","last_became_peered":"2024-03-21T13:17:27.646744+0000","last_unstale":"2024-03-21T13:19:59.564664+0000","last_undegraded":"2024-03-21T13:19:59.564664+0000","last_fullsized":"2024-03-21T13:19:59.564664+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'35","last_scrub_stamp":"2024-03-21T13:19:12.473978+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:19:12.473978+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:39.342421+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"416'15","reported_seq":897,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566209+0000","last_change":"2024-03-21T13:19:22.820712+0000","last_active":"2024-03-21T13:19:59.566209+0000","last_peered":"2024-03-21T13:19:59.566209+0000","last_clean":"2024-03-21T13:19:59.566209+0000","last_became_active":"2024-03-21T13:17:27.653797+0000","last_became_peered":"2024-03-21T13:17:27.653797+0000","last_unstale":"2024-03-21T13:19:59.566209+0000","last_undegraded":"2024-03-21T13:19:59.566209+0000","last_fullsized":"2024-03-21T13:19:59.566209+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'15","last_scrub_stamp":"2024-03-21T13:19:22.820593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:22.820593+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:50.485875+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":839,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564066+0000","last_change":"2024-03-21T13:17:09.938651+0000","last_active":"2024-03-21T13:19:59.564066+0000","last_peered":"2024-03-21T13:19:59.564066+0000","last_clean":"2024-03-21T13:19:59.564066+0000","last_became_active":"2024-03-21T13:10:03.606557+0000","last_became_peered":"2024-03-21T13:10:03.606557+0000","last_unstale":"2024-03-21T13:19:59.564066+0000","last_undegraded":"2024-03-21T13:19:59.564066+0000","last_fullsized":"2024-03-21T13:19:59.564066+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:09.938560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:09.938560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265976+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":103,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.573128+0000","last_change":"2024-03-21T13:19:11.039395+0000","last_active":"2024-03-21T13:19:58.573128+0000","last_peered":"2024-03-21T13:19:58.573128+0000","last_clean":"2024-03-21T13:19:58.573128+0000","last_became_active":"2024-03-21T13:17:27.638397+0000","last_became_peered":"2024-03-21T13:17:27.638397+0000","last_unstale":"2024-03-21T13:19:58.573128+0000","last_undegraded":"2024-03-21T13:19:58.573128+0000","last_fullsized":"2024-03-21T13:19:58.573128+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:19:11.034266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:11.034266+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:26.609678+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"435'49","reported_seq":306,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563994+0000","last_change":"2024-03-21T13:17:01.334745+0000","last_active":"2024-03-21T13:19:59.563994+0000","last_peered":"2024-03-21T13:19:59.563994+0000","last_clean":"2024-03-21T13:19:59.563994+0000","last_became_active":"2024-03-21T13:12:29.525154+0000","last_became_peered":"2024-03-21T13:12:29.525154+0000","last_unstale":"2024-03-21T13:19:59.563994+0000","last_undegraded":"2024-03-21T13:19:59.563994+0000","last_fullsized":"2024-03-21T13:19:59.563994+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:17:01.334611+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:17:01.334611+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681430+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"416'11","reported_seq":1112,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566273+0000","last_change":"2024-03-21T13:19:24.784648+0000","last_active":"2024-03-21T13:19:59.566273+0000","last_peered":"2024-03-21T13:19:59.566273+0000","last_clean":"2024-03-21T13:19:59.566273+0000","last_became_active":"2024-03-21T13:17:27.639409+0000","last_became_peered":"2024-03-21T13:17:27.639409+0000","last_unstale":"2024-03-21T13:19:59.566273+0000","last_undegraded":"2024-03-21T13:19:59.566273+0000","last_fullsized":"2024-03-21T13:19:59.566273+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'11","last_scrub_stamp":"2024-03-21T13:19:24.784519+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:19:24.784519+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:50.405376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":757,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561500+0000","last_change":"2024-03-21T13:18:23.955795+0000","last_active":"2024-03-21T13:19:59.561500+0000","last_peered":"2024-03-21T13:19:59.561500+0000","last_clean":"2024-03-21T13:19:59.561500+0000","last_became_active":"2024-03-21T13:10:03.610471+0000","last_became_peered":"2024-03-21T13:10:03.610471+0000","last_unstale":"2024-03-21T13:19:59.561500+0000","last_undegraded":"2024-03-21T13:19:59.561500+0000","last_fullsized":"2024-03-21T13:19:59.561500+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:18:23.955711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:18:23.955711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":838,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563285+0000","last_change":"2024-03-21T13:17:27.659245+0000","last_active":"2024-03-21T13:19:59.563285+0000","last_peered":"2024-03-21T13:19:59.563285+0000","last_clean":"2024-03-21T13:19:59.563285+0000","last_became_active":"2024-03-21T13:17:27.658806+0000","last_became_peered":"2024-03-21T13:17:27.658806+0000","last_unstale":"2024-03-21T13:19:59.563285+0000","last_undegraded":"2024-03-21T13:19:59.563285+0000","last_fullsized":"2024-03-21T13:19:59.563285+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:05.843337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:17:05.843337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"435'42","reported_seq":921,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566096+0000","last_change":"2024-03-21T13:17:27.711943+0000","last_active":"2024-03-21T13:19:59.566096+0000","last_peered":"2024-03-21T13:19:59.566096+0000","last_clean":"2024-03-21T13:19:59.566096+0000","last_became_active":"2024-03-21T13:17:27.661667+0000","last_became_peered":"2024-03-21T13:17:27.661667+0000","last_unstale":"2024-03-21T13:19:59.566096+0000","last_undegraded":"2024-03-21T13:19:59.566096+0000","last_fullsized":"2024-03-21T13:19:59.566096+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'30","last_scrub_stamp":"2024-03-21T13:17:18.984086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:18.984086+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":850,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.571849+0000","last_change":"2024-03-21T13:19:21.898140+0000","last_active":"2024-03-21T13:19:58.571849+0000","last_peered":"2024-03-21T13:19:58.571849+0000","last_clean":"2024-03-21T13:19:58.571849+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:19:58.571849+0000","last_undegraded":"2024-03-21T13:19:58.571849+0000","last_fullsized":"2024-03-21T13:19:58.571849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:21.897981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:21.897981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:51.682684+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":275,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.593204+0000","last_change":"2024-03-21T13:19:16.579662+0000","last_active":"2024-03-21T13:19:59.593204+0000","last_peered":"2024-03-21T13:19:59.593204+0000","last_clean":"2024-03-21T13:19:59.593204+0000","last_became_active":"2024-03-21T13:17:27.653461+0000","last_became_peered":"2024-03-21T13:17:27.653461+0000","last_unstale":"2024-03-21T13:19:59.593204+0000","last_undegraded":"2024-03-21T13:19:59.593204+0000","last_fullsized":"2024-03-21T13:19:59.593204+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:16.579412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:14.280515+0000","last_clean_scrub_stamp":"2024-03-21T13:19:16.579412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:27.677859+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"416'14","reported_seq":374,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566844+0000","last_change":"2024-03-21T13:19:04.543821+0000","last_active":"2024-03-21T13:19:59.566844+0000","last_peered":"2024-03-21T13:19:59.566844+0000","last_clean":"2024-03-21T13:19:59.566844+0000","last_became_active":"2024-03-21T13:17:27.650953+0000","last_became_peered":"2024-03-21T13:17:27.650953+0000","last_unstale":"2024-03-21T13:19:59.566844+0000","last_undegraded":"2024-03-21T13:19:59.566844+0000","last_fullsized":"2024-03-21T13:19:59.566844+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'14","last_scrub_stamp":"2024-03-21T13:19:04.543702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:04.543702+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:31.957647+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":838,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.565987+0000","last_change":"2024-03-21T13:17:00.746992+0000","last_active":"2024-03-21T13:19:59.565987+0000","last_peered":"2024-03-21T13:19:59.565987+0000","last_clean":"2024-03-21T13:19:59.565987+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:19:59.565987+0000","last_undegraded":"2024-03-21T13:19:59.565987+0000","last_fullsized":"2024-03-21T13:19:59.565987+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:00.746886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:00.746886+0000","last_clean_scrub_stamp":"2024-03-21T13:17:00.746886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"435'46","reported_seq":814,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563378+0000","last_change":"2024-03-21T13:16:53.621472+0000","last_active":"2024-03-21T13:19:59.563378+0000","last_peered":"2024-03-21T13:19:59.563378+0000","last_clean":"2024-03-21T13:19:59.563378+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:19:59.563378+0000","last_undegraded":"2024-03-21T13:19:59.563378+0000","last_fullsized":"2024-03-21T13:19:59.563378+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'34","last_scrub_stamp":"2024-03-21T13:16:53.621335+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.621335+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"375'2","reported_seq":842,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559507+0000","last_change":"2024-03-21T13:18:57.637417+0000","last_active":"2024-03-21T13:19:59.559507+0000","last_peered":"2024-03-21T13:19:59.559507+0000","last_clean":"2024-03-21T13:19:59.559507+0000","last_became_active":"2024-03-21T13:12:29.502816+0000","last_became_peered":"2024-03-21T13:12:29.502816+0000","last_unstale":"2024-03-21T13:19:59.559507+0000","last_undegraded":"2024-03-21T13:19:59.559507+0000","last_fullsized":"2024-03-21T13:19:59.559507+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'2","last_scrub_stamp":"2024-03-21T13:18:57.637264+0000","last_deep_scrub":"375'2","last_deep_scrub_stamp":"2024-03-21T13:15:14.313376+0000","last_clean_scrub_stamp":"2024-03-21T13:18:57.637264+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:08.922711+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.584233+0000","last_change":"2024-03-21T13:18:23.371393+0000","last_active":"2024-03-21T13:19:59.584233+0000","last_peered":"2024-03-21T13:19:59.584233+0000","last_clean":"2024-03-21T13:19:59.584233+0000","last_became_active":"2024-03-21T13:12:29.524087+0000","last_became_peered":"2024-03-21T13:12:29.524087+0000","last_unstale":"2024-03-21T13:19:59.584233+0000","last_undegraded":"2024-03-21T13:19:59.584233+0000","last_fullsized":"2024-03-21T13:19:59.584233+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:18:23.371294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:18:23.371294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681699+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":877,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.571883+0000","last_change":"2024-03-21T13:19:00.099987+0000","last_active":"2024-03-21T13:19:58.571883+0000","last_peered":"2024-03-21T13:19:58.571883+0000","last_clean":"2024-03-21T13:19:58.571883+0000","last_became_active":"2024-03-21T13:10:03.586593+0000","last_became_peered":"2024-03-21T13:10:03.586593+0000","last_unstale":"2024-03-21T13:19:58.571883+0000","last_undegraded":"2024-03-21T13:19:58.571883+0000","last_fullsized":"2024-03-21T13:19:58.571883+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:00.099760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:00.099760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:16.317882+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"435'27","reported_seq":604,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559892+0000","last_change":"2024-03-21T13:17:48.700635+0000","last_active":"2024-03-21T13:19:59.559892+0000","last_peered":"2024-03-21T13:19:59.559892+0000","last_clean":"2024-03-21T13:19:59.559892+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:19:59.559892+0000","last_undegraded":"2024-03-21T13:19:59.559892+0000","last_fullsized":"2024-03-21T13:19:59.559892+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:17:48.700488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:48.700488+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.781072+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":413,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.577119+0000","last_change":"2024-03-21T13:19:04.371967+0000","last_active":"2024-03-21T13:19:58.577119+0000","last_peered":"2024-03-21T13:19:58.577119+0000","last_clean":"2024-03-21T13:19:58.577119+0000","last_became_active":"2024-03-21T13:10:03.602752+0000","last_became_peered":"2024-03-21T13:10:03.602752+0000","last_unstale":"2024-03-21T13:19:58.577119+0000","last_undegraded":"2024-03-21T13:19:58.577119+0000","last_fullsized":"2024-03-21T13:19:58.577119+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:04.371854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:19:04.371854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:33.417082+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":763,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.572313+0000","last_change":"2024-03-21T13:19:24.866869+0000","last_active":"2024-03-21T13:19:58.572313+0000","last_peered":"2024-03-21T13:19:58.572313+0000","last_clean":"2024-03-21T13:19:58.572313+0000","last_became_active":"2024-03-21T13:17:27.640173+0000","last_became_peered":"2024-03-21T13:17:27.640173+0000","last_unstale":"2024-03-21T13:19:58.572313+0000","last_undegraded":"2024-03-21T13:19:58.572313+0000","last_fullsized":"2024-03-21T13:19:58.572313+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:24.866683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:24.866683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:25.045107+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":854,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561593+0000","last_change":"2024-03-21T13:17:58.720904+0000","last_active":"2024-03-21T13:19:59.561593+0000","last_peered":"2024-03-21T13:19:59.561593+0000","last_clean":"2024-03-21T13:19:59.561593+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:19:59.561593+0000","last_undegraded":"2024-03-21T13:19:59.561593+0000","last_fullsized":"2024-03-21T13:19:59.561593+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:58.720766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:58.720766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.781091+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"435'53","reported_seq":444,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579177+0000","last_change":"2024-03-21T13:19:21.085451+0000","last_active":"2024-03-21T13:19:58.579177+0000","last_peered":"2024-03-21T13:19:58.579177+0000","last_clean":"2024-03-21T13:19:58.579177+0000","last_became_active":"2024-03-21T13:10:03.591832+0000","last_became_peered":"2024-03-21T13:10:03.591832+0000","last_unstale":"2024-03-21T13:19:58.579177+0000","last_undegraded":"2024-03-21T13:19:58.579177+0000","last_fullsized":"2024-03-21T13:19:58.579177+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'53","last_scrub_stamp":"2024-03-21T13:19:21.085296+0000","last_deep_scrub":"391'39","last_deep_scrub_stamp":"2024-03-21T13:16:14.266573+0000","last_clean_scrub_stamp":"2024-03-21T13:19:21.085296+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:35.762380+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":411,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579562+0000","last_change":"2024-03-21T13:19:31.172772+0000","last_active":"2024-03-21T13:19:58.579562+0000","last_peered":"2024-03-21T13:19:58.579562+0000","last_clean":"2024-03-21T13:19:58.579562+0000","last_became_active":"2024-03-21T13:12:29.499442+0000","last_became_peered":"2024-03-21T13:12:29.499442+0000","last_unstale":"2024-03-21T13:19:58.579562+0000","last_undegraded":"2024-03-21T13:19:58.579562+0000","last_fullsized":"2024-03-21T13:19:58.579562+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:31.172607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:19:31.172607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:55.730006+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"331'13","reported_seq":532,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579075+0000","last_change":"2024-03-21T13:16:21.250482+0000","last_active":"2024-03-21T13:19:58.579075+0000","last_peered":"2024-03-21T13:19:58.579075+0000","last_clean":"2024-03-21T13:19:58.579075+0000","last_became_active":"2024-03-21T13:12:29.507925+0000","last_became_peered":"2024-03-21T13:12:29.507925+0000","last_unstale":"2024-03-21T13:19:58.579075+0000","last_undegraded":"2024-03-21T13:19:58.579075+0000","last_fullsized":"2024-03-21T13:19:58.579075+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:16:21.250398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.250398+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":838,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560932+0000","last_change":"2024-03-21T13:19:27.893826+0000","last_active":"2024-03-21T13:19:59.560932+0000","last_peered":"2024-03-21T13:19:59.560932+0000","last_clean":"2024-03-21T13:19:59.560932+0000","last_became_active":"2024-03-21T13:17:27.659812+0000","last_became_peered":"2024-03-21T13:17:27.659812+0000","last_unstale":"2024-03-21T13:19:59.560932+0000","last_undegraded":"2024-03-21T13:19:59.560932+0000","last_fullsized":"2024-03-21T13:19:59.560932+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:27.893716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:27.893716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:48.260124+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"435'30","reported_seq":809,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559697+0000","last_change":"2024-03-21T13:17:16.981637+0000","last_active":"2024-03-21T13:19:59.559697+0000","last_peered":"2024-03-21T13:19:59.559697+0000","last_clean":"2024-03-21T13:19:59.559697+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:19:59.559697+0000","last_undegraded":"2024-03-21T13:19:59.559697+0000","last_fullsized":"2024-03-21T13:19:59.559697+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:17:16.981490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:16.981490+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1271,"num_read_kb":1266,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":4823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1531,"num_read_kb":1288,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":393,"num_bytes_recovered":8416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":14874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1039,"ondisk_log_size":1039,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168837,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1119640,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":92625512,"statfs":{"total":95995035648,"available":94848524288,"internally_reserved":0,"allocated":1454080,"data_stored":1023328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1145043402},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463179,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1079384,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":1077950,"kb_avail":92665768,"statfs":{"total":95995035648,"available":94889746432,"internally_reserved":0,"allocated":1454080,"data_stored":1023037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1103821254},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575140,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738665,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411615,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965508,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":964094,"kb_avail":92779644,"statfs":{"total":95995035648,"available":95006355456,"internally_reserved":0,"allocated":1433600,"data_stored":1022154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":987232709},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182682,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1173412,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":1172542,"kb_avail":92571740,"statfs":{"total":95995035648,"available":94793461760,"internally_reserved":0,"allocated":876544,"data_stored":432543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1200683463},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019156,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1185408,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":1184574,"kb_avail":92559744,"statfs":{"total":95995035648,"available":94781177856,"internally_reserved":0,"allocated":839680,"data_stored":431935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1213004229},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724815,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":397140,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93348012,"statfs":{"total":95995035648,"available":95588364288,"internally_reserved":0,"allocated":860160,"data_stored":431991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:20:07.165 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:20:07.398 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:20:07.398 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:20:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:07 smithi204 ceph-mon[33153]: from='client.16101 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:20:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:07 smithi204 ceph-mon[33153]: 3.14 scrub starts 2024-03-21T13:20:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:07 smithi204 ceph-mon[33153]: 3.14 scrub ok 2024-03-21T13:20:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:07 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:20:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:07 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:20:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:07 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:20:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:07 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:20:07.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:07 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:20:07.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[29621]: from='client.16101 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:20:07.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[29621]: 3.14 scrub starts 2024-03-21T13:20:07.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[29621]: 3.14 scrub ok 2024-03-21T13:20:07.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:20:07.910 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:20:07.910 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:20:07.910 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:20:07.910 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:20:07.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[40675]: from='client.16101 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:20:07.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[40675]: 3.14 scrub starts 2024-03-21T13:20:07.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[40675]: 3.14 scrub ok 2024-03-21T13:20:07.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:20:07.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:20:07.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:20:07.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:20:07.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:07 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:20:08.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-21T13:20:08.325 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:08.509 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:20:08.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:08 smithi163 ceph-mon[29621]: pgmap v999: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T13:20:08.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:08 smithi163 ceph-mon[29621]: 5.d scrub starts 2024-03-21T13:20:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:08 smithi163 ceph-mon[29621]: 5.d scrub ok 2024-03-21T13:20:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:08 smithi163 ceph-mon[29621]: osdmap e452: 8 total, 8 up, 6 in 2024-03-21T13:20:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:08 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3417060991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:20:08.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:08 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:20:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:08 smithi163 ceph-mon[40675]: pgmap v999: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T13:20:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:08 smithi163 ceph-mon[40675]: 5.d scrub starts 2024-03-21T13:20:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:08 smithi163 ceph-mon[40675]: 5.d scrub ok 2024-03-21T13:20:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:08 smithi163 ceph-mon[40675]: osdmap e452: 8 total, 8 up, 6 in 2024-03-21T13:20:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:08 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3417060991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:20:08.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:08 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:20:08.898 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:08 smithi204 ceph-mon[33153]: pgmap v999: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T13:20:08.898 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:08 smithi204 ceph-mon[33153]: 5.d scrub starts 2024-03-21T13:20:08.898 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:08 smithi204 ceph-mon[33153]: 5.d scrub ok 2024-03-21T13:20:08.898 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:08 smithi204 ceph-mon[33153]: osdmap e452: 8 total, 8 up, 6 in 2024-03-21T13:20:08.898 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:08 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3417060991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:20:08.898 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:08 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T13:20:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-03-21T13:20:09.743 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T13:20:09.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:09 smithi204 ceph-mon[33153]: 5.b scrub starts 2024-03-21T13:20:09.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:09 smithi204 ceph-mon[33153]: 5.b scrub ok 2024-03-21T13:20:09.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:09 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:20:09.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:09 smithi204 ceph-mon[33153]: osdmap e453: 8 total, 8 up, 6 in 2024-03-21T13:20:09.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:09 smithi204 ceph-mon[33153]: pgmap v1002: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:20:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:09 smithi163 ceph-mon[29621]: 5.b scrub starts 2024-03-21T13:20:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:09 smithi163 ceph-mon[29621]: 5.b scrub ok 2024-03-21T13:20:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:09 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:20:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:09 smithi163 ceph-mon[29621]: osdmap e453: 8 total, 8 up, 6 in 2024-03-21T13:20:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:09 smithi163 ceph-mon[29621]: pgmap v1002: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:20:10.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:09 smithi163 ceph-mon[40675]: 5.b scrub starts 2024-03-21T13:20:10.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:09 smithi163 ceph-mon[40675]: 5.b scrub ok 2024-03-21T13:20:10.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:09 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T13:20:10.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:09 smithi163 ceph-mon[40675]: osdmap e453: 8 total, 8 up, 6 in 2024-03-21T13:20:10.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:09 smithi163 ceph-mon[40675]: pgmap v1002: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:20:10.458 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:20:10.458 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:20:10.768 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:10.960 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:20:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:20:11.152 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:10 smithi204 ceph-mon[33153]: 5.2 scrub starts 2024-03-21T13:20:11.153 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:10 smithi204 ceph-mon[33153]: 5.2 scrub ok 2024-03-21T13:20:11.153 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:10 smithi204 ceph-mon[33153]: 5.15 scrub starts 2024-03-21T13:20:11.153 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:10 smithi204 ceph-mon[33153]: 5.15 scrub ok 2024-03-21T13:20:11.153 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:10 smithi204 ceph-mon[33153]: 4.1d scrub starts 2024-03-21T13:20:11.153 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:10 smithi204 ceph-mon[33153]: 4.1d scrub ok 2024-03-21T13:20:11.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-21T13:20:11.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:10 smithi163 ceph-mon[29621]: 5.2 scrub starts 2024-03-21T13:20:11.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:10 smithi163 ceph-mon[29621]: 5.2 scrub ok 2024-03-21T13:20:11.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:10 smithi163 ceph-mon[29621]: 5.15 scrub starts 2024-03-21T13:20:11.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:10 smithi163 ceph-mon[29621]: 5.15 scrub ok 2024-03-21T13:20:11.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:10 smithi163 ceph-mon[29621]: 4.1d scrub starts 2024-03-21T13:20:11.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:10 smithi163 ceph-mon[29621]: 4.1d scrub ok 2024-03-21T13:20:11.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:10 smithi163 ceph-mon[40675]: 5.2 scrub starts 2024-03-21T13:20:11.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:10 smithi163 ceph-mon[40675]: 5.2 scrub ok 2024-03-21T13:20:11.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:10 smithi163 ceph-mon[40675]: 5.15 scrub starts 2024-03-21T13:20:11.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:10 smithi163 ceph-mon[40675]: 5.15 scrub ok 2024-03-21T13:20:11.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:10 smithi163 ceph-mon[40675]: 4.1d scrub starts 2024-03-21T13:20:11.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:10 smithi163 ceph-mon[40675]: 4.1d scrub ok 2024-03-21T13:20:11.436 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:20:11.987 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":1002,"stamp":"2024-03-21T13:20:09.021724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3043,"num_read_kb":2889,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":437,"num_bytes_recovered":2372405,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1281,"ondisk_log_size":1281,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":5920476,"kb_used_data":6756,"kb_used_omap":9,"kb_used_meta":5913654,"kb_avail":556550436,"statfs":{"total":575970213888,"available":569907646464,"internally_reserved":0,"allocated":6918144,"data_stored":4364988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":6055582374},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002337"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564202+0000","last_change":"2024-03-21T13:19:02.498730+0000","last_active":"2024-03-21T13:19:59.564202+0000","last_peered":"2024-03-21T13:19:59.564202+0000","last_clean":"2024-03-21T13:19:59.564202+0000","last_became_active":"2024-03-21T13:12:29.525755+0000","last_became_peered":"2024-03-21T13:12:29.525755+0000","last_unstale":"2024-03-21T13:19:59.564202+0000","last_undegraded":"2024-03-21T13:19:59.564202+0000","last_fullsized":"2024-03-21T13:19:59.564202+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:02.498361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:19:02.498361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:05.047035+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":277,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564037+0000","last_change":"2024-03-21T13:19:25.553886+0000","last_active":"2024-03-21T13:19:59.564037+0000","last_peered":"2024-03-21T13:19:59.564037+0000","last_clean":"2024-03-21T13:19:59.564037+0000","last_became_active":"2024-03-21T13:12:29.510271+0000","last_became_peered":"2024-03-21T13:12:29.510271+0000","last_unstale":"2024-03-21T13:19:59.564037+0000","last_undegraded":"2024-03-21T13:19:59.564037+0000","last_fullsized":"2024-03-21T13:19:59.564037+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:25.553760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:25.553760+0000","last_clean_scrub_stamp":"2024-03-21T13:19:25.553760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:32.675838+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":208,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.584068+0000","last_change":"2024-03-21T13:19:10.481899+0000","last_active":"2024-03-21T13:19:59.584068+0000","last_peered":"2024-03-21T13:19:59.584068+0000","last_clean":"2024-03-21T13:19:59.584068+0000","last_became_active":"2024-03-21T13:17:27.657450+0000","last_became_peered":"2024-03-21T13:17:27.657450+0000","last_unstale":"2024-03-21T13:19:59.584068+0000","last_undegraded":"2024-03-21T13:19:59.584068+0000","last_fullsized":"2024-03-21T13:19:59.584068+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:10.481779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:10.481779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:37.511613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"435'25","reported_seq":930,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.571824+0000","last_change":"2024-03-21T13:19:06.028666+0000","last_active":"2024-03-21T13:19:58.571824+0000","last_peered":"2024-03-21T13:19:58.571824+0000","last_clean":"2024-03-21T13:19:58.571824+0000","last_became_active":"2024-03-21T13:02:14.339156+0000","last_became_peered":"2024-03-21T13:02:14.339156+0000","last_unstale":"2024-03-21T13:19:58.571824+0000","last_undegraded":"2024-03-21T13:19:58.571824+0000","last_fullsized":"2024-03-21T13:19:58.571824+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:19:06.028535+0000","last_deep_scrub":"234'19","last_deep_scrub_stamp":"2024-03-21T13:16:09.693933+0000","last_clean_scrub_stamp":"2024-03-21T13:19:06.028535+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:28.738583+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":819,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561121+0000","last_change":"2024-03-21T13:19:03.937707+0000","last_active":"2024-03-21T13:19:59.561121+0000","last_peered":"2024-03-21T13:19:59.561121+0000","last_clean":"2024-03-21T13:19:59.561121+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:19:59.561121+0000","last_undegraded":"2024-03-21T13:19:59.561121+0000","last_fullsized":"2024-03-21T13:19:59.561121+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:03.937453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:19:03.937453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:25.885052+0000","scrub_duration":275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":778,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:20:04.720614+0000","last_change":"2024-03-21T13:20:04.720638+0000","last_active":"2024-03-21T13:20:04.720614+0000","last_peered":"2024-03-21T13:20:04.720614+0000","last_clean":"2024-03-21T13:20:04.720614+0000","last_became_active":"2024-03-21T13:10:03.591543+0000","last_became_peered":"2024-03-21T13:10:03.591543+0000","last_unstale":"2024-03-21T13:20:04.720614+0000","last_undegraded":"2024-03-21T13:20:04.720614+0000","last_fullsized":"2024-03-21T13:20:04.720614+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:20:04.720538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:20:04.720538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:21:28.939212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":852,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563023+0000","last_change":"2024-03-21T13:17:03.752054+0000","last_active":"2024-03-21T13:19:59.563023+0000","last_peered":"2024-03-21T13:19:59.563023+0000","last_clean":"2024-03-21T13:19:59.563023+0000","last_became_active":"2024-03-21T13:10:03.610318+0000","last_became_peered":"2024-03-21T13:10:03.610318+0000","last_unstale":"2024-03-21T13:19:59.563023+0000","last_undegraded":"2024-03-21T13:19:59.563023+0000","last_fullsized":"2024-03-21T13:19:59.563023+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:17:03.751920+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_clean_scrub_stamp":"2024-03-21T13:17:03.751920+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"435'39","reported_seq":391,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563415+0000","last_change":"2024-03-21T13:17:27.668489+0000","last_active":"2024-03-21T13:19:59.563415+0000","last_peered":"2024-03-21T13:19:59.563415+0000","last_clean":"2024-03-21T13:19:59.563415+0000","last_became_active":"2024-03-21T13:17:27.652219+0000","last_became_peered":"2024-03-21T13:17:27.652219+0000","last_unstale":"2024-03-21T13:19:59.563415+0000","last_undegraded":"2024-03-21T13:19:59.563415+0000","last_fullsized":"2024-03-21T13:19:59.563415+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:17:01.724037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:01.724037+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265824+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":645,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560484+0000","last_change":"2024-03-21T13:17:27.645240+0000","last_active":"2024-03-21T13:19:59.560484+0000","last_peered":"2024-03-21T13:19:59.560484+0000","last_clean":"2024-03-21T13:19:59.560484+0000","last_became_active":"2024-03-21T13:17:27.641680+0000","last_became_peered":"2024-03-21T13:17:27.641680+0000","last_unstale":"2024-03-21T13:19:59.560484+0000","last_undegraded":"2024-03-21T13:19:59.560484+0000","last_fullsized":"2024-03-21T13:19:59.560484+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:20.951893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:20.951893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":275,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.574319+0000","last_change":"2024-03-21T13:19:06.558479+0000","last_active":"2024-03-21T13:19:59.574319+0000","last_peered":"2024-03-21T13:19:59.574319+0000","last_clean":"2024-03-21T13:19:59.574319+0000","last_became_active":"2024-03-21T13:17:27.659272+0000","last_became_peered":"2024-03-21T13:17:27.659272+0000","last_unstale":"2024-03-21T13:19:59.574319+0000","last_undegraded":"2024-03-21T13:19:59.574319+0000","last_fullsized":"2024-03-21T13:19:59.574319+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:06.558247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:19:06.558247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:20.044230+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.593391+0000","last_change":"2024-03-21T13:19:08.551736+0000","last_active":"2024-03-21T13:19:59.593391+0000","last_peered":"2024-03-21T13:19:59.593391+0000","last_clean":"2024-03-21T13:19:59.593391+0000","last_became_active":"2024-03-21T13:12:29.526129+0000","last_became_peered":"2024-03-21T13:12:29.526129+0000","last_unstale":"2024-03-21T13:19:59.593391+0000","last_undegraded":"2024-03-21T13:19:59.593391+0000","last_fullsized":"2024-03-21T13:19:59.593391+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:08.551622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:08.551622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:29.479156+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"435'25","reported_seq":855,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560664+0000","last_change":"2024-03-21T13:19:31.802513+0000","last_active":"2024-03-21T13:19:59.560664+0000","last_peered":"2024-03-21T13:19:59.560664+0000","last_clean":"2024-03-21T13:19:59.560664+0000","last_became_active":"2024-03-21T13:17:27.639498+0000","last_became_peered":"2024-03-21T13:17:27.639498+0000","last_unstale":"2024-03-21T13:19:59.560664+0000","last_undegraded":"2024-03-21T13:19:59.560664+0000","last_fullsized":"2024-03-21T13:19:59.560664+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:19:31.802397+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:19:31.802397+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:57.229185+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564144+0000","last_change":"2024-03-21T13:16:56.397703+0000","last_active":"2024-03-21T13:19:59.564144+0000","last_peered":"2024-03-21T13:19:59.564144+0000","last_clean":"2024-03-21T13:19:59.564144+0000","last_became_active":"2024-03-21T13:12:29.508520+0000","last_became_peered":"2024-03-21T13:12:29.508520+0000","last_unstale":"2024-03-21T13:19:59.564144+0000","last_undegraded":"2024-03-21T13:19:59.564144+0000","last_fullsized":"2024-03-21T13:19:59.564144+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:56.397585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:56.397585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681496+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":849,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562025+0000","last_change":"2024-03-21T13:17:27.641476+0000","last_active":"2024-03-21T13:19:59.562025+0000","last_peered":"2024-03-21T13:19:59.562025+0000","last_clean":"2024-03-21T13:19:59.562025+0000","last_became_active":"2024-03-21T13:17:27.641161+0000","last_became_peered":"2024-03-21T13:17:27.641161+0000","last_unstale":"2024-03-21T13:19:59.562025+0000","last_undegraded":"2024-03-21T13:19:59.562025+0000","last_fullsized":"2024-03-21T13:19:59.562025+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:18.791316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:18.791316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.565867+0000","last_change":"2024-03-21T13:19:14.479563+0000","last_active":"2024-03-21T13:19:59.565867+0000","last_peered":"2024-03-21T13:19:59.565867+0000","last_clean":"2024-03-21T13:19:59.565867+0000","last_became_active":"2024-03-21T13:12:29.524473+0000","last_became_peered":"2024-03-21T13:12:29.524473+0000","last_unstale":"2024-03-21T13:19:59.565867+0000","last_undegraded":"2024-03-21T13:19:59.565867+0000","last_fullsized":"2024-03-21T13:19:59.565867+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:14.479454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:14.479454+0000","last_clean_scrub_stamp":"2024-03-21T13:19:14.479454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:42.111248+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"435'11","reported_seq":853,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560832+0000","last_change":"2024-03-21T13:17:09.946959+0000","last_active":"2024-03-21T13:19:59.560832+0000","last_peered":"2024-03-21T13:19:59.560832+0000","last_clean":"2024-03-21T13:19:59.560832+0000","last_became_active":"2024-03-21T13:10:03.595745+0000","last_became_peered":"2024-03-21T13:10:03.595745+0000","last_unstale":"2024-03-21T13:19:59.560832+0000","last_undegraded":"2024-03-21T13:19:59.560832+0000","last_fullsized":"2024-03-21T13:19:59.560832+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'9","last_scrub_stamp":"2024-03-21T13:17:09.946815+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:17:09.946815+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":882,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:20:05.341616+0000","last_change":"2024-03-21T13:20:05.341640+0000","last_active":"2024-03-21T13:20:05.341616+0000","last_peered":"2024-03-21T13:20:05.341616+0000","last_clean":"2024-03-21T13:20:05.341616+0000","last_became_active":"2024-03-21T13:12:29.520189+0000","last_became_peered":"2024-03-21T13:12:29.520189+0000","last_unstale":"2024-03-21T13:20:05.341616+0000","last_undegraded":"2024-03-21T13:20:05.341616+0000","last_fullsized":"2024-03-21T13:20:05.341616+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:20:05.341539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:23.865785+0000","last_clean_scrub_stamp":"2024-03-21T13:20:05.341539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:21:07.625340+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":767,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.572853+0000","last_change":"2024-03-21T13:18:58.081858+0000","last_active":"2024-03-21T13:19:58.572853+0000","last_peered":"2024-03-21T13:19:58.572853+0000","last_clean":"2024-03-21T13:19:58.572853+0000","last_became_active":"2024-03-21T13:17:27.643258+0000","last_became_peered":"2024-03-21T13:17:27.643258+0000","last_unstale":"2024-03-21T13:19:58.572853+0000","last_undegraded":"2024-03-21T13:19:58.572853+0000","last_fullsized":"2024-03-21T13:19:58.572853+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:18:58.081747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:18:58.081747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:15.605977+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":637,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563965+0000","last_change":"2024-03-21T13:17:27.669084+0000","last_active":"2024-03-21T13:19:59.563965+0000","last_peered":"2024-03-21T13:19:59.563965+0000","last_clean":"2024-03-21T13:19:59.563965+0000","last_became_active":"2024-03-21T13:17:27.668691+0000","last_became_peered":"2024-03-21T13:17:27.668691+0000","last_unstale":"2024-03-21T13:19:59.563965+0000","last_undegraded":"2024-03-21T13:19:59.563965+0000","last_fullsized":"2024-03-21T13:19:59.563965+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:08.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"435'39","reported_seq":453,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.578529+0000","last_change":"2024-03-21T13:17:27.662499+0000","last_active":"2024-03-21T13:19:58.578529+0000","last_peered":"2024-03-21T13:19:58.578529+0000","last_clean":"2024-03-21T13:19:58.578529+0000","last_became_active":"2024-03-21T13:17:27.639926+0000","last_became_peered":"2024-03-21T13:17:27.639926+0000","last_unstale":"2024-03-21T13:19:58.578529+0000","last_undegraded":"2024-03-21T13:19:58.578529+0000","last_fullsized":"2024-03-21T13:19:58.578529+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:16:54.409873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.409873+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":845,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562709+0000","last_change":"2024-03-21T13:19:27.206524+0000","last_active":"2024-03-21T13:19:59.562709+0000","last_peered":"2024-03-21T13:19:59.562709+0000","last_clean":"2024-03-21T13:19:59.562709+0000","last_became_active":"2024-03-21T13:12:29.502808+0000","last_became_peered":"2024-03-21T13:12:29.502808+0000","last_unstale":"2024-03-21T13:19:59.562709+0000","last_undegraded":"2024-03-21T13:19:59.562709+0000","last_fullsized":"2024-03-21T13:19:59.562709+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:27.206409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:19:27.206409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:28.406687+0000","scrub_duration":354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":756,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559609+0000","last_change":"2024-03-21T13:17:15.990962+0000","last_active":"2024-03-21T13:19:59.559609+0000","last_peered":"2024-03-21T13:19:59.559609+0000","last_clean":"2024-03-21T13:19:59.559609+0000","last_became_active":"2024-03-21T13:10:03.612409+0000","last_became_peered":"2024-03-21T13:10:03.612409+0000","last_unstale":"2024-03-21T13:19:59.559609+0000","last_undegraded":"2024-03-21T13:19:59.559609+0000","last_fullsized":"2024-03-21T13:19:59.559609+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:15.990807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:15.990807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":896,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.572027+0000","last_change":"2024-03-21T13:19:13.983514+0000","last_active":"2024-03-21T13:19:58.572027+0000","last_peered":"2024-03-21T13:19:58.572027+0000","last_clean":"2024-03-21T13:19:58.572027+0000","last_became_active":"2024-03-21T13:17:27.630488+0000","last_became_peered":"2024-03-21T13:17:27.630488+0000","last_unstale":"2024-03-21T13:19:58.572027+0000","last_undegraded":"2024-03-21T13:19:58.572027+0000","last_fullsized":"2024-03-21T13:19:58.572027+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:13.983407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:13.983407+0000","last_clean_scrub_stamp":"2024-03-21T13:19:13.983407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:24.623704+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"435'24","reported_seq":896,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562321+0000","last_change":"2024-03-21T13:16:55.685055+0000","last_active":"2024-03-21T13:19:59.562321+0000","last_peered":"2024-03-21T13:19:59.562321+0000","last_clean":"2024-03-21T13:19:59.562321+0000","last_became_active":"2024-03-21T13:02:14.341612+0000","last_became_peered":"2024-03-21T13:02:14.341612+0000","last_unstale":"2024-03-21T13:19:59.562321+0000","last_undegraded":"2024-03-21T13:19:59.562321+0000","last_fullsized":"2024-03-21T13:19:59.562321+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-21T13:16:55.684896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:55.684896+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":248,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.597952+0000","last_change":"2024-03-21T13:17:05.780695+0000","last_active":"2024-03-21T13:19:59.597952+0000","last_peered":"2024-03-21T13:19:59.597952+0000","last_clean":"2024-03-21T13:19:59.597952+0000","last_became_active":"2024-03-21T13:13:06.420220+0000","last_became_peered":"2024-03-21T13:13:06.420220+0000","last_unstale":"2024-03-21T13:19:59.597952+0000","last_undegraded":"2024-03-21T13:19:59.597952+0000","last_fullsized":"2024-03-21T13:19:59.597952+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:17:05.780572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:05.780572+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265944+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":1102,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560360+0000","last_change":"2024-03-21T13:19:00.935061+0000","last_active":"2024-03-21T13:19:59.560360+0000","last_peered":"2024-03-21T13:19:59.560360+0000","last_clean":"2024-03-21T13:19:59.560360+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:19:59.560360+0000","last_undegraded":"2024-03-21T13:19:59.560360+0000","last_fullsized":"2024-03-21T13:19:59.560360+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-21T13:19:00.934946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:00.934946+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:16.680758+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"407'2","reported_seq":739,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:20:04.713785+0000","last_change":"2024-03-21T13:20:04.713815+0000","last_active":"2024-03-21T13:20:04.713785+0000","last_peered":"2024-03-21T13:20:04.713785+0000","last_clean":"2024-03-21T13:20:04.713785+0000","last_became_active":"2024-03-21T13:04:20.358675+0000","last_became_peered":"2024-03-21T13:04:20.358675+0000","last_unstale":"2024-03-21T13:20:04.713785+0000","last_undegraded":"2024-03-21T13:20:04.713785+0000","last_fullsized":"2024-03-21T13:20:04.713785+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'2","last_scrub_stamp":"2024-03-21T13:20:04.713700+0000","last_deep_scrub":"407'2","last_deep_scrub_stamp":"2024-03-21T13:16:58.735421+0000","last_clean_scrub_stamp":"2024-03-21T13:20:04.713700+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:21:18.664620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"435'35","reported_seq":521,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560974+0000","last_change":"2024-03-21T13:17:27.682401+0000","last_active":"2024-03-21T13:19:59.560974+0000","last_peered":"2024-03-21T13:19:59.560974+0000","last_clean":"2024-03-21T13:19:59.560974+0000","last_became_active":"2024-03-21T13:17:27.655020+0000","last_became_peered":"2024-03-21T13:17:27.655020+0000","last_unstale":"2024-03-21T13:19:59.560974+0000","last_undegraded":"2024-03-21T13:19:59.560974+0000","last_fullsized":"2024-03-21T13:19:59.560974+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-03-21T13:17:09.876547+0000","last_deep_scrub":"410'25","last_deep_scrub_stamp":"2024-03-21T13:17:09.876547+0000","last_clean_scrub_stamp":"2024-03-21T13:17:09.876547+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265994+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.593320+0000","last_change":"2024-03-21T13:19:13.594539+0000","last_active":"2024-03-21T13:19:59.593320+0000","last_peered":"2024-03-21T13:19:59.593320+0000","last_clean":"2024-03-21T13:19:59.593320+0000","last_became_active":"2024-03-21T13:12:29.527129+0000","last_became_peered":"2024-03-21T13:12:29.527129+0000","last_unstale":"2024-03-21T13:19:59.593320+0000","last_undegraded":"2024-03-21T13:19:59.593320+0000","last_fullsized":"2024-03-21T13:19:59.593320+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:13.594428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:19:13.594428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:19.950884+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":658,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560819+0000","last_change":"2024-03-21T13:17:27.650677+0000","last_active":"2024-03-21T13:19:59.560819+0000","last_peered":"2024-03-21T13:19:59.560819+0000","last_clean":"2024-03-21T13:19:59.560819+0000","last_became_active":"2024-03-21T13:17:27.649940+0000","last_became_peered":"2024-03-21T13:17:27.649940+0000","last_unstale":"2024-03-21T13:19:59.560819+0000","last_undegraded":"2024-03-21T13:19:59.560819+0000","last_fullsized":"2024-03-21T13:19:59.560819+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:12.814107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:12.814107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":855,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559252+0000","last_change":"2024-03-21T13:17:27.662093+0000","last_active":"2024-03-21T13:19:59.559252+0000","last_peered":"2024-03-21T13:19:59.559252+0000","last_clean":"2024-03-21T13:19:59.559252+0000","last_became_active":"2024-03-21T13:17:27.661238+0000","last_became_peered":"2024-03-21T13:17:27.661238+0000","last_unstale":"2024-03-21T13:19:59.559252+0000","last_undegraded":"2024-03-21T13:19:59.559252+0000","last_fullsized":"2024-03-21T13:19:59.559252+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:04.838772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:04.838772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"435'32","reported_seq":957,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560428+0000","last_change":"2024-03-21T13:18:59.931958+0000","last_active":"2024-03-21T13:19:59.560428+0000","last_peered":"2024-03-21T13:19:59.560428+0000","last_clean":"2024-03-21T13:19:59.560428+0000","last_became_active":"2024-03-21T13:17:27.652511+0000","last_became_peered":"2024-03-21T13:17:27.652511+0000","last_unstale":"2024-03-21T13:19:59.560428+0000","last_undegraded":"2024-03-21T13:19:59.560428+0000","last_fullsized":"2024-03-21T13:19:59.560428+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'32","last_scrub_stamp":"2024-03-21T13:18:59.931818+0000","last_deep_scrub":"331'24","last_deep_scrub_stamp":"2024-03-21T13:15:09.670459+0000","last_clean_scrub_stamp":"2024-03-21T13:18:59.931818+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:22.289931+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":82,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561843+0000","last_change":"2024-03-21T13:17:27.637231+0000","last_active":"2024-03-21T13:19:59.561843+0000","last_peered":"2024-03-21T13:19:59.561843+0000","last_clean":"2024-03-21T13:19:59.561843+0000","last_became_active":"2024-03-21T13:17:27.636917+0000","last_became_peered":"2024-03-21T13:17:27.636917+0000","last_unstale":"2024-03-21T13:19:59.561843+0000","last_undegraded":"2024-03-21T13:19:59.561843+0000","last_fullsized":"2024-03-21T13:19:59.561843+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:59.238664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.005026+0000","last_clean_scrub_stamp":"2024-03-21T13:16:59.238664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265745+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":840,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563596+0000","last_change":"2024-03-21T13:19:25.802629+0000","last_active":"2024-03-21T13:19:59.563596+0000","last_peered":"2024-03-21T13:19:59.563596+0000","last_clean":"2024-03-21T13:19:59.563596+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:19:59.563596+0000","last_undegraded":"2024-03-21T13:19:59.563596+0000","last_fullsized":"2024-03-21T13:19:59.563596+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:25.802407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:25.802407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:47.811294+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":192,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.584484+0000","last_change":"2024-03-21T13:17:27.660095+0000","last_active":"2024-03-21T13:19:59.584484+0000","last_peered":"2024-03-21T13:19:59.584484+0000","last_clean":"2024-03-21T13:19:59.584484+0000","last_became_active":"2024-03-21T13:17:27.657429+0000","last_became_peered":"2024-03-21T13:17:27.657429+0000","last_unstale":"2024-03-21T13:19:59.584484+0000","last_undegraded":"2024-03-21T13:19:59.584484+0000","last_fullsized":"2024-03-21T13:19:59.584484+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:13.840785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:22.955737+0000","last_clean_scrub_stamp":"2024-03-21T13:17:13.840785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"435'12","reported_seq":858,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:20:03.761072+0000","last_change":"2024-03-21T13:20:03.761102+0000","last_active":"2024-03-21T13:20:03.761072+0000","last_peered":"2024-03-21T13:20:03.761072+0000","last_clean":"2024-03-21T13:20:03.761072+0000","last_became_active":"2024-03-21T13:02:14.343738+0000","last_became_peered":"2024-03-21T13:02:14.343738+0000","last_unstale":"2024-03-21T13:20:03.761072+0000","last_undegraded":"2024-03-21T13:20:03.761072+0000","last_fullsized":"2024-03-21T13:20:03.761072+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'12","last_scrub_stamp":"2024-03-21T13:20:03.760998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:20:03.760998+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:21:17.182834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":826,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561315+0000","last_change":"2024-03-21T13:18:23.062341+0000","last_active":"2024-03-21T13:19:59.561315+0000","last_peered":"2024-03-21T13:19:59.561315+0000","last_clean":"2024-03-21T13:19:59.561315+0000","last_became_active":"2024-03-21T13:17:27.636805+0000","last_became_peered":"2024-03-21T13:17:27.636805+0000","last_unstale":"2024-03-21T13:19:59.561315+0000","last_undegraded":"2024-03-21T13:19:59.561315+0000","last_fullsized":"2024-03-21T13:19:59.561315+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:18:23.062249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:18:23.062249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":82,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561474+0000","last_change":"2024-03-21T13:17:27.645475+0000","last_active":"2024-03-21T13:19:59.561474+0000","last_peered":"2024-03-21T13:19:59.561474+0000","last_clean":"2024-03-21T13:19:59.561474+0000","last_became_active":"2024-03-21T13:17:27.644954+0000","last_became_peered":"2024-03-21T13:17:27.644954+0000","last_unstale":"2024-03-21T13:19:59.561474+0000","last_undegraded":"2024-03-21T13:19:59.561474+0000","last_fullsized":"2024-03-21T13:19:59.561474+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:05.059443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:05.059443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":277,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566771+0000","last_change":"2024-03-21T13:19:27.585259+0000","last_active":"2024-03-21T13:19:59.566771+0000","last_peered":"2024-03-21T13:19:59.566771+0000","last_clean":"2024-03-21T13:19:59.566771+0000","last_became_active":"2024-03-21T13:12:29.507706+0000","last_became_peered":"2024-03-21T13:12:29.507706+0000","last_unstale":"2024-03-21T13:19:59.566771+0000","last_undegraded":"2024-03-21T13:19:59.566771+0000","last_fullsized":"2024-03-21T13:19:59.566771+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:27.585034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:27.585034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:44.850926+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"435'27","reported_seq":585,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559892+0000","last_change":"2024-03-21T13:19:10.678826+0000","last_active":"2024-03-21T13:19:59.559892+0000","last_peered":"2024-03-21T13:19:59.559892+0000","last_clean":"2024-03-21T13:19:59.559892+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:19:59.559892+0000","last_undegraded":"2024-03-21T13:19:59.559892+0000","last_fullsized":"2024-03-21T13:19:59.559892+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'27","last_scrub_stamp":"2024-03-21T13:19:10.678656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:19:10.678656+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:15.956533+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":184,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559428+0000","last_change":"2024-03-21T13:17:27.646425+0000","last_active":"2024-03-21T13:19:59.559428+0000","last_peered":"2024-03-21T13:19:59.559428+0000","last_clean":"2024-03-21T13:19:59.559428+0000","last_became_active":"2024-03-21T13:17:27.645569+0000","last_became_peered":"2024-03-21T13:17:27.645569+0000","last_unstale":"2024-03-21T13:19:59.559428+0000","last_undegraded":"2024-03-21T13:19:59.559428+0000","last_fullsized":"2024-03-21T13:19:59.559428+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:50.025891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:50.025891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":848,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561025+0000","last_change":"2024-03-21T13:19:07.646720+0000","last_active":"2024-03-21T13:19:59.561025+0000","last_peered":"2024-03-21T13:19:59.561025+0000","last_clean":"2024-03-21T13:19:59.561025+0000","last_became_active":"2024-03-21T13:17:27.645710+0000","last_became_peered":"2024-03-21T13:17:27.645710+0000","last_unstale":"2024-03-21T13:19:59.561025+0000","last_undegraded":"2024-03-21T13:19:59.561025+0000","last_fullsized":"2024-03-21T13:19:59.561025+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:07.646564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:07.646564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:17.935079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":837,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562621+0000","last_change":"2024-03-21T13:19:11.213286+0000","last_active":"2024-03-21T13:19:59.562621+0000","last_peered":"2024-03-21T13:19:59.562621+0000","last_clean":"2024-03-21T13:19:59.562621+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:19:59.562621+0000","last_undegraded":"2024-03-21T13:19:59.562621+0000","last_fullsized":"2024-03-21T13:19:59.562621+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:11.213182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:19.383090+0000","last_clean_scrub_stamp":"2024-03-21T13:19:11.213182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:33.803636+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"435'25","reported_seq":872,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561769+0000","last_change":"2024-03-21T13:19:30.825646+0000","last_active":"2024-03-21T13:19:59.561769+0000","last_peered":"2024-03-21T13:19:59.561769+0000","last_clean":"2024-03-21T13:19:59.561769+0000","last_became_active":"2024-03-21T13:17:27.653358+0000","last_became_peered":"2024-03-21T13:17:27.653358+0000","last_unstale":"2024-03-21T13:19:59.561769+0000","last_undegraded":"2024-03-21T13:19:59.561769+0000","last_fullsized":"2024-03-21T13:19:59.561769+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:19:30.825510+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:48.085774+0000","last_clean_scrub_stamp":"2024-03-21T13:19:30.825510+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:59.841946+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":665,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562832+0000","last_change":"2024-03-21T13:17:08.823533+0000","last_active":"2024-03-21T13:19:59.562832+0000","last_peered":"2024-03-21T13:19:59.562832+0000","last_clean":"2024-03-21T13:19:59.562832+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:19:59.562832+0000","last_undegraded":"2024-03-21T13:19:59.562832+0000","last_fullsized":"2024-03-21T13:19:59.562832+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:08.823336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.823336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265960+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564432+0000","last_change":"2024-03-21T13:16:52.230103+0000","last_active":"2024-03-21T13:19:59.564432+0000","last_peered":"2024-03-21T13:19:59.564432+0000","last_clean":"2024-03-21T13:19:59.564432+0000","last_became_active":"2024-03-21T13:12:29.510100+0000","last_became_peered":"2024-03-21T13:12:29.510100+0000","last_unstale":"2024-03-21T13:19:59.564432+0000","last_undegraded":"2024-03-21T13:19:59.564432+0000","last_fullsized":"2024-03-21T13:19:59.564432+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.229670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:22.501055+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.229670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681468+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":542,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.598139+0000","last_change":"2024-03-21T13:15:59.651653+0000","last_active":"2024-03-21T13:19:59.598139+0000","last_peered":"2024-03-21T13:19:59.598139+0000","last_clean":"2024-03-21T13:19:59.598139+0000","last_became_active":"2024-03-21T13:06:45.405904+0000","last_became_peered":"2024-03-21T13:06:45.405904+0000","last_unstale":"2024-03-21T13:19:59.598139+0000","last_undegraded":"2024-03-21T13:19:59.598139+0000","last_fullsized":"2024-03-21T13:19:59.598139+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:15:59.651571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:15:59.651571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"435'27","reported_seq":114,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.572775+0000","last_change":"2024-03-21T13:19:23.826510+0000","last_active":"2024-03-21T13:19:58.572775+0000","last_peered":"2024-03-21T13:19:58.572775+0000","last_clean":"2024-03-21T13:19:58.572775+0000","last_became_active":"2024-03-21T13:17:27.653620+0000","last_became_peered":"2024-03-21T13:17:27.653620+0000","last_unstale":"2024-03-21T13:19:58.572775+0000","last_undegraded":"2024-03-21T13:19:58.572775+0000","last_fullsized":"2024-03-21T13:19:58.572775+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'27","last_scrub_stamp":"2024-03-21T13:19:23.826370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:19:23.826370+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:49.759356+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":645,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560549+0000","last_change":"2024-03-21T13:17:27.651621+0000","last_active":"2024-03-21T13:19:59.560549+0000","last_peered":"2024-03-21T13:19:59.560549+0000","last_clean":"2024-03-21T13:19:59.560549+0000","last_became_active":"2024-03-21T13:17:27.651255+0000","last_became_peered":"2024-03-21T13:17:27.651255+0000","last_unstale":"2024-03-21T13:19:59.560549+0000","last_undegraded":"2024-03-21T13:19:59.560549+0000","last_fullsized":"2024-03-21T13:19:59.560549+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:53.026764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.026764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"416'13","reported_seq":376,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.593473+0000","last_change":"2024-03-21T13:17:27.662539+0000","last_active":"2024-03-21T13:19:59.593473+0000","last_peered":"2024-03-21T13:19:59.593473+0000","last_clean":"2024-03-21T13:19:59.593473+0000","last_became_active":"2024-03-21T13:17:27.644197+0000","last_became_peered":"2024-03-21T13:17:27.644197+0000","last_unstale":"2024-03-21T13:19:59.593473+0000","last_undegraded":"2024-03-21T13:19:59.593473+0000","last_fullsized":"2024-03-21T13:19:59.593473+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'12","last_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_deep_scrub":"394'12","last_deep_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_clean_scrub_stamp":"2024-03-21T13:16:24.237800+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":850,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562793+0000","last_change":"2024-03-21T13:17:16.856970+0000","last_active":"2024-03-21T13:19:59.562793+0000","last_peered":"2024-03-21T13:19:59.562793+0000","last_clean":"2024-03-21T13:19:59.562793+0000","last_became_active":"2024-03-21T13:10:03.599265+0000","last_became_peered":"2024-03-21T13:10:03.599265+0000","last_unstale":"2024-03-21T13:19:59.562793+0000","last_undegraded":"2024-03-21T13:19:59.562793+0000","last_fullsized":"2024-03-21T13:19:59.562793+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:16.856845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:16.856845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265855+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"435'30","reported_seq":927,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562127+0000","last_change":"2024-03-21T13:18:24.919602+0000","last_active":"2024-03-21T13:19:59.562127+0000","last_peered":"2024-03-21T13:19:59.562127+0000","last_clean":"2024-03-21T13:19:59.562127+0000","last_became_active":"2024-03-21T13:10:03.613119+0000","last_became_peered":"2024-03-21T13:10:03.613119+0000","last_unstale":"2024-03-21T13:19:59.562127+0000","last_undegraded":"2024-03-21T13:19:59.562127+0000","last_fullsized":"2024-03-21T13:19:59.562127+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:18:24.919459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:18:24.919459+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":411,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.577016+0000","last_change":"2024-03-21T13:19:13.371982+0000","last_active":"2024-03-21T13:19:58.577016+0000","last_peered":"2024-03-21T13:19:58.577016+0000","last_clean":"2024-03-21T13:19:58.577016+0000","last_became_active":"2024-03-21T13:12:29.507751+0000","last_became_peered":"2024-03-21T13:12:29.507751+0000","last_unstale":"2024-03-21T13:19:58.577016+0000","last_undegraded":"2024-03-21T13:19:58.577016+0000","last_fullsized":"2024-03-21T13:19:58.577016+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:13.371894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:17.497325+0000","last_clean_scrub_stamp":"2024-03-21T13:19:13.371894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:15.436618+0000","scrub_duration":256,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"272'10","reported_seq":1111,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.773503+0000","last_change":"2024-03-21T13:19:59.773503+0000","last_active":"2024-03-21T13:19:59.773503+0000","last_peered":"2024-03-21T13:19:59.773503+0000","last_clean":"2024-03-21T13:19:59.773503+0000","last_became_active":"2024-03-21T13:10:03.594651+0000","last_became_peered":"2024-03-21T13:10:03.594651+0000","last_unstale":"2024-03-21T13:19:59.773503+0000","last_undegraded":"2024-03-21T13:19:59.773503+0000","last_fullsized":"2024-03-21T13:19:59.773503+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'10","last_scrub_stamp":"2024-03-21T13:19:59.773373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:59.773373+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:21:18.537222+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":653,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561941+0000","last_change":"2024-03-21T13:19:29.152268+0000","last_active":"2024-03-21T13:19:59.561941+0000","last_peered":"2024-03-21T13:19:59.561941+0000","last_clean":"2024-03-21T13:19:59.561941+0000","last_became_active":"2024-03-21T13:17:27.650261+0000","last_became_peered":"2024-03-21T13:17:27.650261+0000","last_unstale":"2024-03-21T13:19:59.561941+0000","last_undegraded":"2024-03-21T13:19:59.561941+0000","last_fullsized":"2024-03-21T13:19:59.561941+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:29.152059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:19:29.152059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:57.020571+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"435'21","reported_seq":288,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.593029+0000","last_change":"2024-03-21T13:16:22.227639+0000","last_active":"2024-03-21T13:19:59.593029+0000","last_peered":"2024-03-21T13:19:59.593029+0000","last_clean":"2024-03-21T13:19:59.593029+0000","last_became_active":"2024-03-21T13:12:29.510247+0000","last_became_peered":"2024-03-21T13:12:29.510247+0000","last_unstale":"2024-03-21T13:19:59.593029+0000","last_undegraded":"2024-03-21T13:19:59.593029+0000","last_fullsized":"2024-03-21T13:19:59.593029+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:16:22.227536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:22.227536+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681256+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":397,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.577042+0000","last_change":"2024-03-21T13:17:07.670896+0000","last_active":"2024-03-21T13:19:58.577042+0000","last_peered":"2024-03-21T13:19:58.577042+0000","last_clean":"2024-03-21T13:19:58.577042+0000","last_became_active":"2024-03-21T13:10:03.613841+0000","last_became_peered":"2024-03-21T13:10:03.613841+0000","last_unstale":"2024-03-21T13:19:58.577042+0000","last_undegraded":"2024-03-21T13:19:58.577042+0000","last_fullsized":"2024-03-21T13:19:58.577042+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:07.670782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:17:07.670782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169702+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":835,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563811+0000","last_change":"2024-03-21T13:17:27.659110+0000","last_active":"2024-03-21T13:19:59.563811+0000","last_peered":"2024-03-21T13:19:59.563811+0000","last_clean":"2024-03-21T13:19:59.563811+0000","last_became_active":"2024-03-21T13:17:27.658400+0000","last_became_peered":"2024-03-21T13:17:27.658400+0000","last_unstale":"2024-03-21T13:19:59.563811+0000","last_undegraded":"2024-03-21T13:19:59.563811+0000","last_fullsized":"2024-03-21T13:19:59.563811+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:13.018536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:13.018536+0000","last_clean_scrub_stamp":"2024-03-21T13:17:13.018536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":844,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563168+0000","last_change":"2024-03-21T13:16:54.653009+0000","last_active":"2024-03-21T13:19:59.563168+0000","last_peered":"2024-03-21T13:19:59.563168+0000","last_clean":"2024-03-21T13:19:59.563168+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:19:59.563168+0000","last_undegraded":"2024-03-21T13:19:59.563168+0000","last_fullsized":"2024-03-21T13:19:59.563168+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:16:54.652821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.652821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265712+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"435'21","reported_seq":699,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563355+0000","last_change":"2024-03-21T13:17:27.719115+0000","last_active":"2024-03-21T13:19:59.563355+0000","last_peered":"2024-03-21T13:19:59.563355+0000","last_clean":"2024-03-21T13:19:59.563355+0000","last_became_active":"2024-03-21T13:17:27.659084+0000","last_became_peered":"2024-03-21T13:17:27.659084+0000","last_unstale":"2024-03-21T13:19:59.563355+0000","last_undegraded":"2024-03-21T13:19:59.563355+0000","last_fullsized":"2024-03-21T13:19:59.563355+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-21T13:16:58.080100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:58.080100+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":825,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562466+0000","last_change":"2024-03-21T13:19:01.216170+0000","last_active":"2024-03-21T13:19:59.562466+0000","last_peered":"2024-03-21T13:19:59.562466+0000","last_clean":"2024-03-21T13:19:59.562466+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:19:59.562466+0000","last_undegraded":"2024-03-21T13:19:59.562466+0000","last_fullsized":"2024-03-21T13:19:59.562466+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:01.216060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:19:01.216060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:13.540476+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":277,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566693+0000","last_change":"2024-03-21T13:19:19.541389+0000","last_active":"2024-03-21T13:19:59.566693+0000","last_peered":"2024-03-21T13:19:59.566693+0000","last_clean":"2024-03-21T13:19:59.566693+0000","last_became_active":"2024-03-21T13:12:29.514113+0000","last_became_peered":"2024-03-21T13:12:29.514113+0000","last_unstale":"2024-03-21T13:19:59.566693+0000","last_undegraded":"2024-03-21T13:19:59.566693+0000","last_fullsized":"2024-03-21T13:19:59.566693+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:19.541273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:19.541273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:22.878402+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":858,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559385+0000","last_change":"2024-03-21T13:17:27.656617+0000","last_active":"2024-03-21T13:19:59.559385+0000","last_peered":"2024-03-21T13:19:59.559385+0000","last_clean":"2024-03-21T13:19:59.559385+0000","last_became_active":"2024-03-21T13:17:27.654135+0000","last_became_peered":"2024-03-21T13:17:27.654135+0000","last_unstale":"2024-03-21T13:19:59.559385+0000","last_undegraded":"2024-03-21T13:19:59.559385+0000","last_fullsized":"2024-03-21T13:19:59.559385+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:16.777565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:16.777565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.781051+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"435'18","reported_seq":896,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563469+0000","last_change":"2024-03-21T13:17:02.736704+0000","last_active":"2024-03-21T13:19:59.563469+0000","last_peered":"2024-03-21T13:19:59.563469+0000","last_clean":"2024-03-21T13:19:59.563469+0000","last_became_active":"2024-03-21T13:10:03.610810+0000","last_became_peered":"2024-03-21T13:10:03.610810+0000","last_unstale":"2024-03-21T13:19:59.563469+0000","last_undegraded":"2024-03-21T13:19:59.563469+0000","last_fullsized":"2024-03-21T13:19:59.563469+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-21T13:17:02.736579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:02.736579+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":275,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564769+0000","last_change":"2024-03-21T13:18:24.415828+0000","last_active":"2024-03-21T13:19:59.564769+0000","last_peered":"2024-03-21T13:19:59.564769+0000","last_clean":"2024-03-21T13:19:59.564769+0000","last_became_active":"2024-03-21T13:17:27.654442+0000","last_became_peered":"2024-03-21T13:17:27.654442+0000","last_unstale":"2024-03-21T13:19:59.564769+0000","last_undegraded":"2024-03-21T13:19:59.564769+0000","last_fullsized":"2024-03-21T13:19:59.564769+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:18:24.415712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:43.075256+0000","last_clean_scrub_stamp":"2024-03-21T13:18:24.415712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681728+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":886,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.565342+0000","last_change":"2024-03-21T13:17:27.677784+0000","last_active":"2024-03-21T13:19:59.565342+0000","last_peered":"2024-03-21T13:19:59.565342+0000","last_clean":"2024-03-21T13:19:59.565342+0000","last_became_active":"2024-03-21T13:17:27.656174+0000","last_became_peered":"2024-03-21T13:17:27.656174+0000","last_unstale":"2024-03-21T13:19:59.565342+0000","last_undegraded":"2024-03-21T13:19:59.565342+0000","last_fullsized":"2024-03-21T13:19:59.565342+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:16:56.698173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:56.698173+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265925+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"435'39","reported_seq":303,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559456+0000","last_change":"2024-03-21T13:16:53.176991+0000","last_active":"2024-03-21T13:19:59.559456+0000","last_peered":"2024-03-21T13:19:59.559456+0000","last_clean":"2024-03-21T13:19:59.559456+0000","last_became_active":"2024-03-21T13:12:29.513019+0000","last_became_peered":"2024-03-21T13:12:29.513019+0000","last_unstale":"2024-03-21T13:19:59.559456+0000","last_undegraded":"2024-03-21T13:19:59.559456+0000","last_fullsized":"2024-03-21T13:19:59.559456+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-21T13:16:53.176836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.176836+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681534+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":1100,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563543+0000","last_change":"2024-03-21T13:19:21.182012+0000","last_active":"2024-03-21T13:19:59.563543+0000","last_peered":"2024-03-21T13:19:59.563543+0000","last_clean":"2024-03-21T13:19:59.563543+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:19:59.563543+0000","last_undegraded":"2024-03-21T13:19:59.563543+0000","last_fullsized":"2024-03-21T13:19:59.563543+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:19:21.181898+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:19:21.181898+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:46.906471+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":826,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561212+0000","last_change":"2024-03-21T13:19:11.950569+0000","last_active":"2024-03-21T13:19:59.561212+0000","last_peered":"2024-03-21T13:19:59.561212+0000","last_clean":"2024-03-21T13:19:59.561212+0000","last_became_active":"2024-03-21T13:02:14.347313+0000","last_became_peered":"2024-03-21T13:02:14.347313+0000","last_unstale":"2024-03-21T13:19:59.561212+0000","last_undegraded":"2024-03-21T13:19:59.561212+0000","last_fullsized":"2024-03-21T13:19:59.561212+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:11.950446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:11.950446+0000","last_clean_scrub_stamp":"2024-03-21T13:19:11.950446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:17.756677+0000","scrub_duration":296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":850,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560704+0000","last_change":"2024-03-21T13:17:27.653349+0000","last_active":"2024-03-21T13:19:59.560704+0000","last_peered":"2024-03-21T13:19:59.560704+0000","last_clean":"2024-03-21T13:19:59.560704+0000","last_became_active":"2024-03-21T13:17:27.652993+0000","last_became_peered":"2024-03-21T13:17:27.652993+0000","last_unstale":"2024-03-21T13:19:59.560704+0000","last_undegraded":"2024-03-21T13:19:59.560704+0000","last_fullsized":"2024-03-21T13:19:59.560704+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.653388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.653388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265729+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"435'29","reported_seq":257,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.576332+0000","last_change":"2024-03-21T13:19:19.130198+0000","last_active":"2024-03-21T13:19:58.576332+0000","last_peered":"2024-03-21T13:19:58.576332+0000","last_clean":"2024-03-21T13:19:58.576332+0000","last_became_active":"2024-03-21T13:13:21.819111+0000","last_became_peered":"2024-03-21T13:13:21.819111+0000","last_unstale":"2024-03-21T13:19:58.576332+0000","last_undegraded":"2024-03-21T13:19:58.576332+0000","last_fullsized":"2024-03-21T13:19:58.576332+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'29","last_scrub_stamp":"2024-03-21T13:19:19.130049+0000","last_deep_scrub":"346'23","last_deep_scrub_stamp":"2024-03-21T13:13:20.232807+0000","last_clean_scrub_stamp":"2024-03-21T13:19:19.130049+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:30.549031+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":30,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":409,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.576099+0000","last_change":"2024-03-21T13:17:27.633122+0000","last_active":"2024-03-21T13:19:58.576099+0000","last_peered":"2024-03-21T13:19:58.576099+0000","last_clean":"2024-03-21T13:19:58.576099+0000","last_became_active":"2024-03-21T13:17:27.632559+0000","last_became_peered":"2024-03-21T13:17:27.632559+0000","last_unstale":"2024-03-21T13:19:58.576099+0000","last_undegraded":"2024-03-21T13:19:58.576099+0000","last_fullsized":"2024-03-21T13:19:58.576099+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:55.431004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:55.431004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169583+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":846,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560592+0000","last_change":"2024-03-21T13:19:04.089379+0000","last_active":"2024-03-21T13:19:59.560592+0000","last_peered":"2024-03-21T13:19:59.560592+0000","last_clean":"2024-03-21T13:19:59.560592+0000","last_became_active":"2024-03-21T13:10:03.598366+0000","last_became_peered":"2024-03-21T13:10:03.598366+0000","last_unstale":"2024-03-21T13:19:59.560592+0000","last_undegraded":"2024-03-21T13:19:59.560592+0000","last_fullsized":"2024-03-21T13:19:59.560592+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:04.089161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:19:04.089161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:10.879320+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":417,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.576592+0000","last_change":"2024-03-21T13:17:27.650325+0000","last_active":"2024-03-21T13:19:58.576592+0000","last_peered":"2024-03-21T13:19:58.576592+0000","last_clean":"2024-03-21T13:19:58.576592+0000","last_became_active":"2024-03-21T13:17:27.649975+0000","last_became_peered":"2024-03-21T13:17:27.649975+0000","last_unstale":"2024-03-21T13:19:58.576592+0000","last_undegraded":"2024-03-21T13:19:58.576592+0000","last_fullsized":"2024-03-21T13:19:58.576592+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:49.448804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:49.448804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169554+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":850,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.578942+0000","last_change":"2024-03-21T13:16:47.661139+0000","last_active":"2024-03-21T13:19:59.578942+0000","last_peered":"2024-03-21T13:19:59.578942+0000","last_clean":"2024-03-21T13:19:59.578942+0000","last_became_active":"2024-03-21T13:12:29.508984+0000","last_became_peered":"2024-03-21T13:12:29.508984+0000","last_unstale":"2024-03-21T13:19:59.578942+0000","last_undegraded":"2024-03-21T13:19:59.578942+0000","last_fullsized":"2024-03-21T13:19:59.578942+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:47.661063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:47.661063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265806+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"435'51","reported_seq":1001,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560830+0000","last_change":"2024-03-21T13:19:31.884114+0000","last_active":"2024-03-21T13:19:59.560830+0000","last_peered":"2024-03-21T13:19:59.560830+0000","last_clean":"2024-03-21T13:19:59.560830+0000","last_became_active":"2024-03-21T13:12:29.523359+0000","last_became_peered":"2024-03-21T13:12:29.523359+0000","last_unstale":"2024-03-21T13:19:59.560830+0000","last_undegraded":"2024-03-21T13:19:59.560830+0000","last_fullsized":"2024-03-21T13:19:59.560830+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'51","last_scrub_stamp":"2024-03-21T13:19:31.883967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:19:31.883967+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:56.608448+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":650,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.562978+0000","last_change":"2024-03-21T13:17:27.649278+0000","last_active":"2024-03-21T13:19:59.562978+0000","last_peered":"2024-03-21T13:19:59.562978+0000","last_clean":"2024-03-21T13:19:59.562978+0000","last_became_active":"2024-03-21T13:17:27.648954+0000","last_became_peered":"2024-03-21T13:17:27.648954+0000","last_unstale":"2024-03-21T13:19:59.562978+0000","last_undegraded":"2024-03-21T13:19:59.562978+0000","last_fullsized":"2024-03-21T13:19:59.562978+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:07.141148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:07.141148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.266010+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":397,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579914+0000","last_change":"2024-03-21T13:16:56.458265+0000","last_active":"2024-03-21T13:19:58.579914+0000","last_peered":"2024-03-21T13:19:58.579914+0000","last_clean":"2024-03-21T13:19:58.579914+0000","last_became_active":"2024-03-21T13:10:03.603646+0000","last_became_peered":"2024-03-21T13:10:03.603646+0000","last_unstale":"2024-03-21T13:19:58.579914+0000","last_undegraded":"2024-03-21T13:19:58.579914+0000","last_fullsized":"2024-03-21T13:19:58.579914+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:56.458050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:56.458050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169640+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.565756+0000","last_change":"2024-03-21T13:16:54.243549+0000","last_active":"2024-03-21T13:19:59.565756+0000","last_peered":"2024-03-21T13:19:59.565756+0000","last_clean":"2024-03-21T13:19:59.565756+0000","last_became_active":"2024-03-21T13:12:29.512741+0000","last_became_peered":"2024-03-21T13:12:29.512741+0000","last_unstale":"2024-03-21T13:19:59.565756+0000","last_undegraded":"2024-03-21T13:19:59.565756+0000","last_fullsized":"2024-03-21T13:19:59.565756+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:54.243395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:16:54.243395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681394+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"435'25","reported_seq":761,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.572333+0000","last_change":"2024-03-21T13:19:09.020946+0000","last_active":"2024-03-21T13:19:58.572333+0000","last_peered":"2024-03-21T13:19:58.572333+0000","last_clean":"2024-03-21T13:19:58.572333+0000","last_became_active":"2024-03-21T13:17:27.653652+0000","last_became_peered":"2024-03-21T13:17:27.653652+0000","last_unstale":"2024-03-21T13:19:58.572333+0000","last_undegraded":"2024-03-21T13:19:58.572333+0000","last_fullsized":"2024-03-21T13:19:58.572333+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:19:09.020814+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:19:09.020814+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:34.076510+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":812,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563244+0000","last_change":"2024-03-21T13:17:06.844511+0000","last_active":"2024-03-21T13:19:59.563244+0000","last_peered":"2024-03-21T13:19:59.563244+0000","last_clean":"2024-03-21T13:19:59.563244+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:19:59.563244+0000","last_undegraded":"2024-03-21T13:19:59.563244+0000","last_fullsized":"2024-03-21T13:19:59.563244+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:06.844387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:46.738659+0000","last_clean_scrub_stamp":"2024-03-21T13:17:06.844387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":283,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564255+0000","last_change":"2024-03-21T13:19:15.494560+0000","last_active":"2024-03-21T13:19:59.564255+0000","last_peered":"2024-03-21T13:19:59.564255+0000","last_clean":"2024-03-21T13:19:59.564255+0000","last_became_active":"2024-03-21T13:17:27.652655+0000","last_became_peered":"2024-03-21T13:17:27.652655+0000","last_unstale":"2024-03-21T13:19:59.564255+0000","last_undegraded":"2024-03-21T13:19:59.564255+0000","last_fullsized":"2024-03-21T13:19:59.564255+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:19:15.494435+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:19:15.494435+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:38.244906+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":465,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566021+0000","last_change":"2024-03-21T13:19:24.818885+0000","last_active":"2024-03-21T13:19:59.566021+0000","last_peered":"2024-03-21T13:19:59.566021+0000","last_clean":"2024-03-21T13:19:59.566021+0000","last_became_active":"2024-03-21T13:17:27.652986+0000","last_became_peered":"2024-03-21T13:17:27.652986+0000","last_unstale":"2024-03-21T13:19:59.566021+0000","last_undegraded":"2024-03-21T13:19:59.566021+0000","last_fullsized":"2024-03-21T13:19:59.566021+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:24.818628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:22.861136+0000","last_clean_scrub_stamp":"2024-03-21T13:19:24.818628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:53.242996+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"435'55","reported_seq":566,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:20:03.105267+0000","last_change":"2024-03-21T13:20:03.106159+0000","last_active":"2024-03-21T13:20:03.105267+0000","last_peered":"2024-03-21T13:20:03.105267+0000","last_clean":"2024-03-21T13:20:03.105267+0000","last_became_active":"2024-03-21T13:10:03.603889+0000","last_became_peered":"2024-03-21T13:10:03.603889+0000","last_unstale":"2024-03-21T13:20:03.105267+0000","last_undegraded":"2024-03-21T13:20:03.105267+0000","last_fullsized":"2024-03-21T13:20:03.105267+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'55","last_scrub_stamp":"2024-03-21T13:20:03.105201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:20:03.105201+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:21:03.928104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":469,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.569688+0000","last_change":"2024-03-21T13:19:06.882018+0000","last_active":"2024-03-21T13:19:59.569688+0000","last_peered":"2024-03-21T13:19:59.569688+0000","last_clean":"2024-03-21T13:19:59.569688+0000","last_became_active":"2024-03-21T13:17:27.643330+0000","last_became_peered":"2024-03-21T13:17:27.643330+0000","last_unstale":"2024-03-21T13:19:59.569688+0000","last_undegraded":"2024-03-21T13:19:59.569688+0000","last_fullsized":"2024-03-21T13:19:59.569688+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:06.881903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:15.968926+0000","last_clean_scrub_stamp":"2024-03-21T13:19:06.881903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:08.464724+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":827,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.579167+0000","last_change":"2024-03-21T13:16:18.683140+0000","last_active":"2024-03-21T13:19:59.579167+0000","last_peered":"2024-03-21T13:19:59.579167+0000","last_clean":"2024-03-21T13:19:59.579167+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:19:59.579167+0000","last_undegraded":"2024-03-21T13:19:59.579167+0000","last_fullsized":"2024-03-21T13:19:59.579167+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_clean_scrub_stamp":"2024-03-21T13:16:18.683041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265694+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":404,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.576155+0000","last_change":"2024-03-21T13:17:27.634210+0000","last_active":"2024-03-21T13:19:58.576155+0000","last_peered":"2024-03-21T13:19:58.576155+0000","last_clean":"2024-03-21T13:19:58.576155+0000","last_became_active":"2024-03-21T13:17:27.633871+0000","last_became_peered":"2024-03-21T13:17:27.633871+0000","last_unstale":"2024-03-21T13:19:58.576155+0000","last_undegraded":"2024-03-21T13:19:58.576155+0000","last_fullsized":"2024-03-21T13:19:58.576155+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:12.859249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.231115+0000","last_clean_scrub_stamp":"2024-03-21T13:17:12.859249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":403,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.576500+0000","last_change":"2024-03-21T13:17:27.640952+0000","last_active":"2024-03-21T13:19:58.576500+0000","last_peered":"2024-03-21T13:19:58.576500+0000","last_clean":"2024-03-21T13:19:58.576500+0000","last_became_active":"2024-03-21T13:17:27.640621+0000","last_became_peered":"2024-03-21T13:17:27.640621+0000","last_unstale":"2024-03-21T13:19:58.576500+0000","last_undegraded":"2024-03-21T13:19:58.576500+0000","last_fullsized":"2024-03-21T13:19:58.576500+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:52.465427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:16:52.465427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.169611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"435'24","reported_seq":305,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.584572+0000","last_change":"2024-03-21T13:17:27.722357+0000","last_active":"2024-03-21T13:19:59.584572+0000","last_peered":"2024-03-21T13:19:59.584572+0000","last_clean":"2024-03-21T13:19:59.584572+0000","last_became_active":"2024-03-21T13:17:27.660008+0000","last_became_peered":"2024-03-21T13:17:27.660008+0000","last_unstale":"2024-03-21T13:19:59.584572+0000","last_undegraded":"2024-03-21T13:19:59.584572+0000","last_fullsized":"2024-03-21T13:19:59.584572+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'20","last_scrub_stamp":"2024-03-21T13:17:18.348476+0000","last_deep_scrub":"243'20","last_deep_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_clean_scrub_stamp":"2024-03-21T13:17:18.348476+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681662+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":403,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579644+0000","last_change":"2024-03-21T13:19:22.107531+0000","last_active":"2024-03-21T13:19:58.579644+0000","last_peered":"2024-03-21T13:19:58.579644+0000","last_clean":"2024-03-21T13:19:58.579644+0000","last_became_active":"2024-03-21T13:12:29.506751+0000","last_became_peered":"2024-03-21T13:12:29.506751+0000","last_unstale":"2024-03-21T13:19:58.579644+0000","last_undegraded":"2024-03-21T13:19:58.579644+0000","last_fullsized":"2024-03-21T13:19:58.579644+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:22.107289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:22.107289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:45.912848+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.584398+0000","last_change":"2024-03-21T13:19:03.668928+0000","last_active":"2024-03-21T13:19:59.584398+0000","last_peered":"2024-03-21T13:19:59.584398+0000","last_clean":"2024-03-21T13:19:59.584398+0000","last_became_active":"2024-03-21T13:12:29.527083+0000","last_became_peered":"2024-03-21T13:12:29.527083+0000","last_unstale":"2024-03-21T13:19:59.584398+0000","last_undegraded":"2024-03-21T13:19:59.584398+0000","last_fullsized":"2024-03-21T13:19:59.584398+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:03.668710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:19:03.668710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:21.438873+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":1112,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560813+0000","last_change":"2024-03-21T13:19:09.154578+0000","last_active":"2024-03-21T13:19:59.560813+0000","last_peered":"2024-03-21T13:19:59.560813+0000","last_clean":"2024-03-21T13:19:59.560813+0000","last_became_active":"2024-03-21T13:17:30.671981+0000","last_became_peered":"2024-03-21T13:17:30.671981+0000","last_unstale":"2024-03-21T13:19:59.560813+0000","last_undegraded":"2024-03-21T13:19:59.560813+0000","last_fullsized":"2024-03-21T13:19:59.560813+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:19:09.154442+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:19:09.154442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:30.186517+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":343,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579656+0000","last_change":"2024-03-21T13:19:11.122953+0000","last_active":"2024-03-21T13:19:58.579656+0000","last_peered":"2024-03-21T13:19:58.579656+0000","last_clean":"2024-03-21T13:19:58.579656+0000","last_became_active":"2024-03-21T13:12:29.499989+0000","last_became_peered":"2024-03-21T13:12:29.499989+0000","last_unstale":"2024-03-21T13:19:58.579656+0000","last_undegraded":"2024-03-21T13:19:58.579656+0000","last_fullsized":"2024-03-21T13:19:58.579656+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:11.122754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:11.122754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:37.216401+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"435'48","reported_seq":1161,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.574748+0000","last_change":"2024-03-21T13:19:16.865894+0000","last_active":"2024-03-21T13:19:59.574748+0000","last_peered":"2024-03-21T13:19:59.574748+0000","last_clean":"2024-03-21T13:19:59.574748+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:19:59.574748+0000","last_undegraded":"2024-03-21T13:19:59.574748+0000","last_fullsized":"2024-03-21T13:19:59.574748+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'48","last_scrub_stamp":"2024-03-21T13:19:16.865744+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:19:16.865744+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:25.169108+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":855,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.598278+0000","last_change":"2024-03-21T13:17:02.759517+0000","last_active":"2024-03-21T13:19:59.598278+0000","last_peered":"2024-03-21T13:19:59.598278+0000","last_clean":"2024-03-21T13:19:59.598278+0000","last_became_active":"2024-03-21T13:12:29.507753+0000","last_became_peered":"2024-03-21T13:12:29.507753+0000","last_unstale":"2024-03-21T13:19:59.598278+0000","last_undegraded":"2024-03-21T13:19:59.598278+0000","last_fullsized":"2024-03-21T13:19:59.598278+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:02.759379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:17:02.759379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265761+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.565927+0000","last_change":"2024-03-21T13:19:21.577055+0000","last_active":"2024-03-21T13:19:59.565927+0000","last_peered":"2024-03-21T13:19:59.565927+0000","last_clean":"2024-03-21T13:19:59.565927+0000","last_became_active":"2024-03-21T13:12:29.514542+0000","last_became_peered":"2024-03-21T13:12:29.514542+0000","last_unstale":"2024-03-21T13:19:59.565927+0000","last_undegraded":"2024-03-21T13:19:59.565927+0000","last_fullsized":"2024-03-21T13:19:59.565927+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:21.576900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:19:21.576900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:22.445820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":678,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.571996+0000","last_change":"2024-03-21T13:19:16.959435+0000","last_active":"2024-03-21T13:19:58.571996+0000","last_peered":"2024-03-21T13:19:58.571996+0000","last_clean":"2024-03-21T13:19:58.571996+0000","last_became_active":"2024-03-21T13:17:27.639147+0000","last_became_peered":"2024-03-21T13:17:27.639147+0000","last_unstale":"2024-03-21T13:19:58.571996+0000","last_undegraded":"2024-03-21T13:19:58.571996+0000","last_fullsized":"2024-03-21T13:19:58.571996+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:16.959312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:00.086941+0000","last_clean_scrub_stamp":"2024-03-21T13:19:16.959312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:39.128234+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":2147,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:20:04.098891+0000","last_change":"2024-03-21T13:19:19.184255+0000","last_active":"2024-03-21T13:20:04.098891+0000","last_peered":"2024-03-21T13:20:04.098891+0000","last_clean":"2024-03-21T13:20:04.098891+0000","last_became_active":"2024-03-21T13:17:27.656453+0000","last_became_peered":"2024-03-21T13:17:27.656453+0000","last_unstale":"2024-03-21T13:20:04.098891+0000","last_undegraded":"2024-03-21T13:20:04.098891+0000","last_fullsized":"2024-03-21T13:20:04.098891+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:19:19.184135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:19:19.184135+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:24.075770+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1276,"num_read_kb":1271,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"435'36","reported_seq":670,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579400+0000","last_change":"2024-03-21T13:19:12.131227+0000","last_active":"2024-03-21T13:19:58.579400+0000","last_peered":"2024-03-21T13:19:58.579400+0000","last_clean":"2024-03-21T13:19:58.579400+0000","last_became_active":"2024-03-21T13:17:27.645689+0000","last_became_peered":"2024-03-21T13:17:27.645689+0000","last_unstale":"2024-03-21T13:19:58.579400+0000","last_undegraded":"2024-03-21T13:19:58.579400+0000","last_fullsized":"2024-03-21T13:19:58.579400+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'36","last_scrub_stamp":"2024-03-21T13:19:12.131071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:19:12.131071+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:35.728431+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":275,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.584319+0000","last_change":"2024-03-21T13:17:27.662358+0000","last_active":"2024-03-21T13:19:59.584319+0000","last_peered":"2024-03-21T13:19:59.584319+0000","last_clean":"2024-03-21T13:19:59.584319+0000","last_became_active":"2024-03-21T13:17:27.658711+0000","last_became_peered":"2024-03-21T13:17:27.658711+0000","last_unstale":"2024-03-21T13:19:59.584319+0000","last_undegraded":"2024-03-21T13:19:59.584319+0000","last_fullsized":"2024-03-21T13:19:59.584319+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:08.237986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:08.237986+0000","last_clean_scrub_stamp":"2024-03-21T13:17:08.237986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":737,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563466+0000","last_change":"2024-03-21T13:19:29.809130+0000","last_active":"2024-03-21T13:19:59.563466+0000","last_peered":"2024-03-21T13:19:59.563466+0000","last_clean":"2024-03-21T13:19:59.563466+0000","last_became_active":"2024-03-21T13:10:03.591088+0000","last_became_peered":"2024-03-21T13:10:03.591088+0000","last_unstale":"2024-03-21T13:19:59.563466+0000","last_undegraded":"2024-03-21T13:19:59.563466+0000","last_fullsized":"2024-03-21T13:19:59.563466+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:29.809009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:29.809009+0000","last_clean_scrub_stamp":"2024-03-21T13:19:29.809009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:52.868665+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":417,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.578120+0000","last_change":"2024-03-21T13:19:10.154315+0000","last_active":"2024-03-21T13:19:58.578120+0000","last_peered":"2024-03-21T13:19:58.578120+0000","last_clean":"2024-03-21T13:19:58.578120+0000","last_became_active":"2024-03-21T13:17:27.639129+0000","last_became_peered":"2024-03-21T13:17:27.639129+0000","last_unstale":"2024-03-21T13:19:58.578120+0000","last_undegraded":"2024-03-21T13:19:58.578120+0000","last_fullsized":"2024-03-21T13:19:58.578120+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:10.154113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:10.154113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:26.879527+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"435'39","reported_seq":326,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.574510+0000","last_change":"2024-03-21T13:17:27.724123+0000","last_active":"2024-03-21T13:19:59.574510+0000","last_peered":"2024-03-21T13:19:59.574510+0000","last_clean":"2024-03-21T13:19:59.574510+0000","last_became_active":"2024-03-21T13:17:27.658954+0000","last_became_peered":"2024-03-21T13:17:27.658954+0000","last_unstale":"2024-03-21T13:19:59.574510+0000","last_undegraded":"2024-03-21T13:19:59.574510+0000","last_fullsized":"2024-03-21T13:19:59.574510+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-21T13:17:03.309259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:03.309259+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681567+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":863,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560968+0000","last_change":"2024-03-21T13:17:27.653252+0000","last_active":"2024-03-21T13:19:59.560968+0000","last_peered":"2024-03-21T13:19:59.560968+0000","last_clean":"2024-03-21T13:19:59.560968+0000","last_became_active":"2024-03-21T13:17:27.652408+0000","last_became_peered":"2024-03-21T13:17:27.652408+0000","last_unstale":"2024-03-21T13:19:59.560968+0000","last_undegraded":"2024-03-21T13:19:59.560968+0000","last_fullsized":"2024-03-21T13:19:59.560968+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:02.830721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:17:02.830721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.781030+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":649,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559245+0000","last_change":"2024-03-21T13:17:27.646840+0000","last_active":"2024-03-21T13:19:59.559245+0000","last_peered":"2024-03-21T13:19:59.559245+0000","last_clean":"2024-03-21T13:19:59.559245+0000","last_became_active":"2024-03-21T13:17:27.646513+0000","last_became_peered":"2024-03-21T13:17:27.646513+0000","last_unstale":"2024-03-21T13:19:59.559245+0000","last_undegraded":"2024-03-21T13:19:59.559245+0000","last_fullsized":"2024-03-21T13:19:59.559245+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:16:51.069883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:16:51.069883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.593125+0000","last_change":"2024-03-21T13:19:24.587323+0000","last_active":"2024-03-21T13:19:59.593125+0000","last_peered":"2024-03-21T13:19:59.593125+0000","last_clean":"2024-03-21T13:19:59.593125+0000","last_became_active":"2024-03-21T13:12:29.512291+0000","last_became_peered":"2024-03-21T13:12:29.512291+0000","last_unstale":"2024-03-21T13:19:59.593125+0000","last_undegraded":"2024-03-21T13:19:59.593125+0000","last_fullsized":"2024-03-21T13:19:59.593125+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:24.587197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:24.587197+0000","last_clean_scrub_stamp":"2024-03-21T13:19:24.587197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:49.477367+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"435'35","reported_seq":314,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564664+0000","last_change":"2024-03-21T13:19:12.474259+0000","last_active":"2024-03-21T13:19:59.564664+0000","last_peered":"2024-03-21T13:19:59.564664+0000","last_clean":"2024-03-21T13:19:59.564664+0000","last_became_active":"2024-03-21T13:17:27.646744+0000","last_became_peered":"2024-03-21T13:17:27.646744+0000","last_unstale":"2024-03-21T13:19:59.564664+0000","last_undegraded":"2024-03-21T13:19:59.564664+0000","last_fullsized":"2024-03-21T13:19:59.564664+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'35","last_scrub_stamp":"2024-03-21T13:19:12.473978+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:19:12.473978+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:39.342421+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"416'15","reported_seq":898,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566247+0000","last_change":"2024-03-21T13:19:22.820712+0000","last_active":"2024-03-21T13:19:59.566247+0000","last_peered":"2024-03-21T13:19:59.566247+0000","last_clean":"2024-03-21T13:19:59.566247+0000","last_became_active":"2024-03-21T13:17:27.653797+0000","last_became_peered":"2024-03-21T13:17:27.653797+0000","last_unstale":"2024-03-21T13:19:59.566247+0000","last_undegraded":"2024-03-21T13:19:59.566247+0000","last_fullsized":"2024-03-21T13:19:59.566247+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'15","last_scrub_stamp":"2024-03-21T13:19:22.820593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:22.820593+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:50.485875+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":839,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.564066+0000","last_change":"2024-03-21T13:17:09.938651+0000","last_active":"2024-03-21T13:19:59.564066+0000","last_peered":"2024-03-21T13:19:59.564066+0000","last_clean":"2024-03-21T13:19:59.564066+0000","last_became_active":"2024-03-21T13:10:03.606557+0000","last_became_peered":"2024-03-21T13:10:03.606557+0000","last_unstale":"2024-03-21T13:19:59.564066+0000","last_undegraded":"2024-03-21T13:19:59.564066+0000","last_fullsized":"2024-03-21T13:19:59.564066+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:09.938560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:17:09.938560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:03.265976+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":103,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.573128+0000","last_change":"2024-03-21T13:19:11.039395+0000","last_active":"2024-03-21T13:19:58.573128+0000","last_peered":"2024-03-21T13:19:58.573128+0000","last_clean":"2024-03-21T13:19:58.573128+0000","last_became_active":"2024-03-21T13:17:27.638397+0000","last_became_peered":"2024-03-21T13:17:27.638397+0000","last_unstale":"2024-03-21T13:19:58.573128+0000","last_undegraded":"2024-03-21T13:19:58.573128+0000","last_fullsized":"2024-03-21T13:19:58.573128+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:19:11.034266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:11.034266+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:26.609678+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"435'49","reported_seq":306,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563994+0000","last_change":"2024-03-21T13:17:01.334745+0000","last_active":"2024-03-21T13:19:59.563994+0000","last_peered":"2024-03-21T13:19:59.563994+0000","last_clean":"2024-03-21T13:19:59.563994+0000","last_became_active":"2024-03-21T13:12:29.525154+0000","last_became_peered":"2024-03-21T13:12:29.525154+0000","last_unstale":"2024-03-21T13:19:59.563994+0000","last_undegraded":"2024-03-21T13:19:59.563994+0000","last_fullsized":"2024-03-21T13:19:59.563994+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'35","last_scrub_stamp":"2024-03-21T13:17:01.334611+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:17:01.334611+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681430+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"416'11","reported_seq":1113,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566317+0000","last_change":"2024-03-21T13:19:24.784648+0000","last_active":"2024-03-21T13:19:59.566317+0000","last_peered":"2024-03-21T13:19:59.566317+0000","last_clean":"2024-03-21T13:19:59.566317+0000","last_became_active":"2024-03-21T13:17:27.639409+0000","last_became_peered":"2024-03-21T13:17:27.639409+0000","last_unstale":"2024-03-21T13:19:59.566317+0000","last_undegraded":"2024-03-21T13:19:59.566317+0000","last_fullsized":"2024-03-21T13:19:59.566317+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'11","last_scrub_stamp":"2024-03-21T13:19:24.784519+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:19:24.784519+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:50.405376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":757,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561500+0000","last_change":"2024-03-21T13:18:23.955795+0000","last_active":"2024-03-21T13:19:59.561500+0000","last_peered":"2024-03-21T13:19:59.561500+0000","last_clean":"2024-03-21T13:19:59.561500+0000","last_became_active":"2024-03-21T13:10:03.610471+0000","last_became_peered":"2024-03-21T13:10:03.610471+0000","last_unstale":"2024-03-21T13:19:59.561500+0000","last_undegraded":"2024-03-21T13:19:59.561500+0000","last_fullsized":"2024-03-21T13:19:59.561500+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:18:23.955711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:08.823841+0000","last_clean_scrub_stamp":"2024-03-21T13:18:23.955711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":838,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563285+0000","last_change":"2024-03-21T13:17:27.659245+0000","last_active":"2024-03-21T13:19:59.563285+0000","last_peered":"2024-03-21T13:19:59.563285+0000","last_clean":"2024-03-21T13:19:59.563285+0000","last_became_active":"2024-03-21T13:17:27.658806+0000","last_became_peered":"2024-03-21T13:17:27.658806+0000","last_unstale":"2024-03-21T13:19:59.563285+0000","last_undegraded":"2024-03-21T13:19:59.563285+0000","last_fullsized":"2024-03-21T13:19:59.563285+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:05.843337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:17:05.843337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"435'42","reported_seq":921,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566096+0000","last_change":"2024-03-21T13:17:27.711943+0000","last_active":"2024-03-21T13:19:59.566096+0000","last_peered":"2024-03-21T13:19:59.566096+0000","last_clean":"2024-03-21T13:19:59.566096+0000","last_became_active":"2024-03-21T13:17:27.661667+0000","last_became_peered":"2024-03-21T13:17:27.661667+0000","last_unstale":"2024-03-21T13:19:59.566096+0000","last_undegraded":"2024-03-21T13:19:59.566096+0000","last_fullsized":"2024-03-21T13:19:59.566096+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'30","last_scrub_stamp":"2024-03-21T13:17:18.984086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:18.984086+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":850,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.571849+0000","last_change":"2024-03-21T13:19:21.898140+0000","last_active":"2024-03-21T13:19:58.571849+0000","last_peered":"2024-03-21T13:19:58.571849+0000","last_clean":"2024-03-21T13:19:58.571849+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:19:58.571849+0000","last_undegraded":"2024-03-21T13:19:58.571849+0000","last_fullsized":"2024-03-21T13:19:58.571849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:21.897981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:21.897981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:51.682684+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":275,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.593204+0000","last_change":"2024-03-21T13:19:16.579662+0000","last_active":"2024-03-21T13:19:59.593204+0000","last_peered":"2024-03-21T13:19:59.593204+0000","last_clean":"2024-03-21T13:19:59.593204+0000","last_became_active":"2024-03-21T13:17:27.653461+0000","last_became_peered":"2024-03-21T13:17:27.653461+0000","last_unstale":"2024-03-21T13:19:59.593204+0000","last_undegraded":"2024-03-21T13:19:59.593204+0000","last_fullsized":"2024-03-21T13:19:59.593204+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:16.579412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:14.280515+0000","last_clean_scrub_stamp":"2024-03-21T13:19:16.579412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:27.677859+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"416'14","reported_seq":375,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.566887+0000","last_change":"2024-03-21T13:19:04.543821+0000","last_active":"2024-03-21T13:19:59.566887+0000","last_peered":"2024-03-21T13:19:59.566887+0000","last_clean":"2024-03-21T13:19:59.566887+0000","last_became_active":"2024-03-21T13:17:27.650953+0000","last_became_peered":"2024-03-21T13:17:27.650953+0000","last_unstale":"2024-03-21T13:19:59.566887+0000","last_undegraded":"2024-03-21T13:19:59.566887+0000","last_fullsized":"2024-03-21T13:19:59.566887+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'14","last_scrub_stamp":"2024-03-21T13:19:04.543702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:04.543702+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:31.957647+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":838,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.565987+0000","last_change":"2024-03-21T13:17:00.746992+0000","last_active":"2024-03-21T13:19:59.565987+0000","last_peered":"2024-03-21T13:19:59.565987+0000","last_clean":"2024-03-21T13:19:59.565987+0000","last_became_active":"2024-03-21T13:02:14.345060+0000","last_became_peered":"2024-03-21T13:02:14.345060+0000","last_unstale":"2024-03-21T13:19:59.565987+0000","last_undegraded":"2024-03-21T13:19:59.565987+0000","last_fullsized":"2024-03-21T13:19:59.565987+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:00.746886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:00.746886+0000","last_clean_scrub_stamp":"2024-03-21T13:17:00.746886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"435'46","reported_seq":814,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.563378+0000","last_change":"2024-03-21T13:16:53.621472+0000","last_active":"2024-03-21T13:19:59.563378+0000","last_peered":"2024-03-21T13:19:59.563378+0000","last_clean":"2024-03-21T13:19:59.563378+0000","last_became_active":"2024-03-21T13:02:14.347690+0000","last_became_peered":"2024-03-21T13:02:14.347690+0000","last_unstale":"2024-03-21T13:19:59.563378+0000","last_undegraded":"2024-03-21T13:19:59.563378+0000","last_fullsized":"2024-03-21T13:19:59.563378+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'34","last_scrub_stamp":"2024-03-21T13:16:53.621335+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:16:53.621335+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"375'2","reported_seq":842,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559507+0000","last_change":"2024-03-21T13:18:57.637417+0000","last_active":"2024-03-21T13:19:59.559507+0000","last_peered":"2024-03-21T13:19:59.559507+0000","last_clean":"2024-03-21T13:19:59.559507+0000","last_became_active":"2024-03-21T13:12:29.502816+0000","last_became_peered":"2024-03-21T13:12:29.502816+0000","last_unstale":"2024-03-21T13:19:59.559507+0000","last_undegraded":"2024-03-21T13:19:59.559507+0000","last_fullsized":"2024-03-21T13:19:59.559507+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'2","last_scrub_stamp":"2024-03-21T13:18:57.637264+0000","last_deep_scrub":"375'2","last_deep_scrub_stamp":"2024-03-21T13:15:14.313376+0000","last_clean_scrub_stamp":"2024-03-21T13:18:57.637264+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:08.922711+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":269,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.584233+0000","last_change":"2024-03-21T13:18:23.371393+0000","last_active":"2024-03-21T13:19:59.584233+0000","last_peered":"2024-03-21T13:19:59.584233+0000","last_clean":"2024-03-21T13:19:59.584233+0000","last_became_active":"2024-03-21T13:12:29.524087+0000","last_became_peered":"2024-03-21T13:12:29.524087+0000","last_unstale":"2024-03-21T13:19:59.584233+0000","last_undegraded":"2024-03-21T13:19:59.584233+0000","last_fullsized":"2024-03-21T13:19:59.584233+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:18:23.371294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:18:23.371294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.681699+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":877,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.571883+0000","last_change":"2024-03-21T13:19:00.099987+0000","last_active":"2024-03-21T13:19:58.571883+0000","last_peered":"2024-03-21T13:19:58.571883+0000","last_clean":"2024-03-21T13:19:58.571883+0000","last_became_active":"2024-03-21T13:10:03.586593+0000","last_became_peered":"2024-03-21T13:10:03.586593+0000","last_unstale":"2024-03-21T13:19:58.571883+0000","last_undegraded":"2024-03-21T13:19:58.571883+0000","last_fullsized":"2024-03-21T13:19:58.571883+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:00.099760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:00.099760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:16.317882+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"435'27","reported_seq":604,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559892+0000","last_change":"2024-03-21T13:17:48.700635+0000","last_active":"2024-03-21T13:19:59.559892+0000","last_peered":"2024-03-21T13:19:59.559892+0000","last_clean":"2024-03-21T13:19:59.559892+0000","last_became_active":"2024-03-21T13:06:45.411709+0000","last_became_peered":"2024-03-21T13:06:45.411709+0000","last_unstale":"2024-03-21T13:19:59.559892+0000","last_undegraded":"2024-03-21T13:19:59.559892+0000","last_fullsized":"2024-03-21T13:19:59.559892+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-21T13:17:48.700488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:48.700488+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.781072+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":413,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.577119+0000","last_change":"2024-03-21T13:19:04.371967+0000","last_active":"2024-03-21T13:19:58.577119+0000","last_peered":"2024-03-21T13:19:58.577119+0000","last_clean":"2024-03-21T13:19:58.577119+0000","last_became_active":"2024-03-21T13:10:03.602752+0000","last_became_peered":"2024-03-21T13:10:03.602752+0000","last_unstale":"2024-03-21T13:19:58.577119+0000","last_undegraded":"2024-03-21T13:19:58.577119+0000","last_fullsized":"2024-03-21T13:19:58.577119+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:04.371854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:19:04.371854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:33.417082+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":763,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.572313+0000","last_change":"2024-03-21T13:19:24.866869+0000","last_active":"2024-03-21T13:19:58.572313+0000","last_peered":"2024-03-21T13:19:58.572313+0000","last_clean":"2024-03-21T13:19:58.572313+0000","last_became_active":"2024-03-21T13:17:27.640173+0000","last_became_peered":"2024-03-21T13:17:27.640173+0000","last_unstale":"2024-03-21T13:19:58.572313+0000","last_undegraded":"2024-03-21T13:19:58.572313+0000","last_fullsized":"2024-03-21T13:19:58.572313+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:24.866683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:19:24.866683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:25.045107+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":854,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.561593+0000","last_change":"2024-03-21T13:17:58.720904+0000","last_active":"2024-03-21T13:19:59.561593+0000","last_peered":"2024-03-21T13:19:59.561593+0000","last_clean":"2024-03-21T13:19:59.561593+0000","last_became_active":"2024-03-21T13:02:14.343437+0000","last_became_peered":"2024-03-21T13:02:14.343437+0000","last_unstale":"2024-03-21T13:19:59.561593+0000","last_undegraded":"2024-03-21T13:19:59.561593+0000","last_fullsized":"2024-03-21T13:19:59.561593+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:17:58.720766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:17:58.720766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:00.781091+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"435'53","reported_seq":444,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579177+0000","last_change":"2024-03-21T13:19:21.085451+0000","last_active":"2024-03-21T13:19:58.579177+0000","last_peered":"2024-03-21T13:19:58.579177+0000","last_clean":"2024-03-21T13:19:58.579177+0000","last_became_active":"2024-03-21T13:10:03.591832+0000","last_became_peered":"2024-03-21T13:10:03.591832+0000","last_unstale":"2024-03-21T13:19:58.579177+0000","last_undegraded":"2024-03-21T13:19:58.579177+0000","last_fullsized":"2024-03-21T13:19:58.579177+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'53","last_scrub_stamp":"2024-03-21T13:19:21.085296+0000","last_deep_scrub":"391'39","last_deep_scrub_stamp":"2024-03-21T13:16:14.266573+0000","last_clean_scrub_stamp":"2024-03-21T13:19:21.085296+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:35.762380+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":411,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579562+0000","last_change":"2024-03-21T13:19:31.172772+0000","last_active":"2024-03-21T13:19:58.579562+0000","last_peered":"2024-03-21T13:19:58.579562+0000","last_clean":"2024-03-21T13:19:58.579562+0000","last_became_active":"2024-03-21T13:12:29.499442+0000","last_became_peered":"2024-03-21T13:12:29.499442+0000","last_unstale":"2024-03-21T13:19:58.579562+0000","last_undegraded":"2024-03-21T13:19:58.579562+0000","last_fullsized":"2024-03-21T13:19:58.579562+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:31.172607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:19:31.172607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:55.730006+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"331'13","reported_seq":533,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:58.579120+0000","last_change":"2024-03-21T13:16:21.250482+0000","last_active":"2024-03-21T13:19:58.579120+0000","last_peered":"2024-03-21T13:19:58.579120+0000","last_clean":"2024-03-21T13:19:58.579120+0000","last_became_active":"2024-03-21T13:12:29.507925+0000","last_became_peered":"2024-03-21T13:12:29.507925+0000","last_unstale":"2024-03-21T13:19:58.579120+0000","last_undegraded":"2024-03-21T13:19:58.579120+0000","last_fullsized":"2024-03-21T13:19:58.579120+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:16:21.250398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:16:21.250398+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":838,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.560932+0000","last_change":"2024-03-21T13:19:27.893826+0000","last_active":"2024-03-21T13:19:59.560932+0000","last_peered":"2024-03-21T13:19:59.560932+0000","last_clean":"2024-03-21T13:19:59.560932+0000","last_became_active":"2024-03-21T13:17:27.659812+0000","last_became_peered":"2024-03-21T13:17:27.659812+0000","last_unstale":"2024-03-21T13:19:59.560932+0000","last_undegraded":"2024-03-21T13:19:59.560932+0000","last_fullsized":"2024-03-21T13:19:59.560932+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:19:27.893716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:19:27.893716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:20:48.260124+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"435'30","reported_seq":809,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-03-21T13:19:59.559697+0000","last_change":"2024-03-21T13:17:16.981637+0000","last_active":"2024-03-21T13:19:59.559697+0000","last_peered":"2024-03-21T13:19:59.559697+0000","last_clean":"2024-03-21T13:19:59.559697+0000","last_became_active":"2024-03-21T13:01:27.359307+0000","last_became_peered":"2024-03-21T13:01:27.359307+0000","last_unstale":"2024-03-21T13:19:59.559697+0000","last_undegraded":"2024-03-21T13:19:59.559697+0000","last_fullsized":"2024-03-21T13:19:59.559697+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-21T13:17:16.981490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:17:16.981490+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1276,"num_read_kb":1271,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":4823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1531,"num_read_kb":1288,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":393,"num_bytes_recovered":8416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":14874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1039,"ondisk_log_size":1039,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168838,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1119640,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":92625512,"statfs":{"total":95995035648,"available":94848524288,"internally_reserved":0,"allocated":1454080,"data_stored":1023328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1145043402},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463180,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1079380,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":1077950,"kb_avail":92665772,"statfs":{"total":95995035648,"available":94889750528,"internally_reserved":0,"allocated":1454080,"data_stored":1023037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1103821254},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575140,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738666,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411616,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965504,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":964094,"kb_avail":92779648,"statfs":{"total":95995035648,"available":95006359552,"internally_reserved":0,"allocated":1433600,"data_stored":1022154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":987232709},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182683,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1173408,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":1172542,"kb_avail":92571744,"statfs":{"total":95995035648,"available":94793465856,"internally_reserved":0,"allocated":876544,"data_stored":432543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1200683463},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019157,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1185404,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":1184574,"kb_avail":92559748,"statfs":{"total":95995035648,"available":94781181952,"internally_reserved":0,"allocated":839680,"data_stored":431935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1213004229},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724816,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":397140,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93348012,"statfs":{"total":95995035648,"available":95588364288,"internally_reserved":0,"allocated":860160,"data_stored":431991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:20:11.991 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-21T13:20:11.991 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T13:20:11.992 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-03-21T13:20:11.992 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.3 reset_purged_snaps_last 2024-03-21T13:20:12.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:20:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:20:12.819 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:12 smithi204 ceph-mon[33153]: 6.0 scrub starts 2024-03-21T13:20:12.819 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:12 smithi204 ceph-mon[33153]: 6.0 scrub ok 2024-03-21T13:20:12.819 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:12 smithi204 ceph-mon[33153]: from='client.16113 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:20:12.820 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:12 smithi204 ceph-mon[33153]: pgmap v1003: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:12.820 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:12 smithi204 ceph-mon[33153]: 3.10 scrub starts 2024-03-21T13:20:12.820 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:12 smithi204 ceph-mon[33153]: 3.10 scrub ok 2024-03-21T13:20:12.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:12 smithi163 ceph-mon[40675]: 6.0 scrub starts 2024-03-21T13:20:12.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:12 smithi163 ceph-mon[40675]: 6.0 scrub ok 2024-03-21T13:20:12.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:12 smithi163 ceph-mon[40675]: from='client.16113 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:20:12.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:12 smithi163 ceph-mon[40675]: pgmap v1003: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:12.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:12 smithi163 ceph-mon[40675]: 3.10 scrub starts 2024-03-21T13:20:12.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:12 smithi163 ceph-mon[40675]: 3.10 scrub ok 2024-03-21T13:20:12.926 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:12 smithi163 ceph-mon[29621]: 6.0 scrub starts 2024-03-21T13:20:12.927 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:12 smithi163 ceph-mon[29621]: 6.0 scrub ok 2024-03-21T13:20:12.927 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:12 smithi163 ceph-mon[29621]: from='client.16113 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:20:12.927 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:12 smithi163 ceph-mon[29621]: pgmap v1003: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:12.927 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:12 smithi163 ceph-mon[29621]: 3.10 scrub starts 2024-03-21T13:20:12.927 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:12 smithi163 ceph-mon[29621]: 3.10 scrub ok 2024-03-21T13:20:13.521 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:13.693 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:20:13.693 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:20:13.693 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:13 smithi204 ceph-mon[33153]: 5.8 scrub starts 2024-03-21T13:20:13.693 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:13 smithi204 ceph-mon[33153]: 5.8 scrub ok 2024-03-21T13:20:13.693 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:13 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:13.693 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:13 smithi204 ceph-mon[33153]: 4.e scrub starts 2024-03-21T13:20:13.693 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:13 smithi204 ceph-mon[33153]: 4.e scrub ok 2024-03-21T13:20:13.693 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:13 smithi204 ceph-mon[33153]: 6.10 scrub starts 2024-03-21T13:20:13.694 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:13 smithi204 ceph-mon[33153]: 6.10 scrub ok 2024-03-21T13:20:13.806 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[29621]: 5.8 scrub starts 2024-03-21T13:20:13.806 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[29621]: 5.8 scrub ok 2024-03-21T13:20:13.806 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:13.806 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[29621]: 4.e scrub starts 2024-03-21T13:20:13.806 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[29621]: 4.e scrub ok 2024-03-21T13:20:13.806 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[29621]: 6.10 scrub starts 2024-03-21T13:20:13.806 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[29621]: 6.10 scrub ok 2024-03-21T13:20:13.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[40675]: 5.8 scrub starts 2024-03-21T13:20:13.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[40675]: 5.8 scrub ok 2024-03-21T13:20:13.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:13.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[40675]: 4.e scrub starts 2024-03-21T13:20:13.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[40675]: 4.e scrub ok 2024-03-21T13:20:13.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[40675]: 6.10 scrub starts 2024-03-21T13:20:13.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:13 smithi163 ceph-mon[40675]: 6.10 scrub ok 2024-03-21T13:20:14.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-21T13:20:14.551 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-03-21T13:20:14.710 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:20:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:14 smithi204 ceph-mon[33153]: 4.f scrub starts 2024-03-21T13:20:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:14 smithi204 ceph-mon[33153]: 4.f scrub ok 2024-03-21T13:20:14.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:14 smithi204 ceph-mon[33153]: pgmap v1004: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:14.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:14 smithi204 ceph-mon[33153]: 2.1 scrub starts 2024-03-21T13:20:14.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:14 smithi204 ceph-mon[33153]: 2.1 scrub ok 2024-03-21T13:20:14.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:14 smithi204 ceph-mon[33153]: 3.c scrub starts 2024-03-21T13:20:14.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:14 smithi204 ceph-mon[33153]: 3.c scrub ok 2024-03-21T13:20:14.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:14 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1799617761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:20:14.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:14 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:20:14.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[29621]: 4.f scrub starts 2024-03-21T13:20:14.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[29621]: 4.f scrub ok 2024-03-21T13:20:14.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[29621]: pgmap v1004: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:14.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[29621]: 2.1 scrub starts 2024-03-21T13:20:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[29621]: 2.1 scrub ok 2024-03-21T13:20:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[29621]: 3.c scrub starts 2024-03-21T13:20:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[29621]: 3.c scrub ok 2024-03-21T13:20:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1799617761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:20:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:20:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[40675]: 4.f scrub starts 2024-03-21T13:20:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[40675]: 4.f scrub ok 2024-03-21T13:20:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[40675]: pgmap v1004: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[40675]: 2.1 scrub starts 2024-03-21T13:20:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[40675]: 2.1 scrub ok 2024-03-21T13:20:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[40675]: 3.c scrub starts 2024-03-21T13:20:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[40675]: 3.c scrub ok 2024-03-21T13:20:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1799617761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:20:14.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:14 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T13:20:15.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:20:15 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=cleanup t=2024-03-21T13:20:15.066547351Z level=info msg="Completed cleanup jobs" duration=38.349067ms 2024-03-21T13:20:15.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:15 smithi163 ceph-mon[29621]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:20:15.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:15 smithi163 ceph-mon[29621]: Cluster is now healthy 2024-03-21T13:20:15.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:15 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:20:15.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:15 smithi163 ceph-mon[29621]: osdmap e454: 8 total, 8 up, 6 in 2024-03-21T13:20:15.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:15 smithi163 ceph-mon[29621]: 4.1 scrub starts 2024-03-21T13:20:15.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:15 smithi163 ceph-mon[29621]: 4.1 scrub ok 2024-03-21T13:20:15.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:15 smithi163 ceph-mon[40675]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:20:15.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:15 smithi163 ceph-mon[40675]: Cluster is now healthy 2024-03-21T13:20:15.606 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:15 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:20:15.606 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:15 smithi163 ceph-mon[40675]: osdmap e454: 8 total, 8 up, 6 in 2024-03-21T13:20:15.606 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:15 smithi163 ceph-mon[40675]: 4.1 scrub starts 2024-03-21T13:20:15.606 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:15 smithi163 ceph-mon[40675]: 4.1 scrub ok 2024-03-21T13:20:15.861 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:15 smithi204 ceph-mon[33153]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T13:20:15.861 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:15 smithi204 ceph-mon[33153]: Cluster is now healthy 2024-03-21T13:20:15.861 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:15 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T13:20:15.861 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:15 smithi204 ceph-mon[33153]: osdmap e454: 8 total, 8 up, 6 in 2024-03-21T13:20:15.861 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:15 smithi204 ceph-mon[33153]: 4.1 scrub starts 2024-03-21T13:20:15.861 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:15 smithi204 ceph-mon[33153]: 4.1 scrub ok 2024-03-21T13:20:15.948 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:20:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:20:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:20:16.399 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:20:16.400 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:20:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:16 smithi204 ceph-mon[33153]: pgmap v1006: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 689 B/s rd, 0 op/s 2024-03-21T13:20:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:16 smithi204 ceph-mon[33153]: 3.a scrub starts 2024-03-21T13:20:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:16 smithi204 ceph-mon[33153]: 3.a scrub ok 2024-03-21T13:20:16.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:16 smithi204 ceph-mon[33153]: 5.1d scrub starts 2024-03-21T13:20:16.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:16 smithi204 ceph-mon[33153]: 5.1d scrub ok 2024-03-21T13:20:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:16 smithi163 ceph-mon[29621]: pgmap v1006: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 689 B/s rd, 0 op/s 2024-03-21T13:20:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:16 smithi163 ceph-mon[29621]: 3.a scrub starts 2024-03-21T13:20:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:16 smithi163 ceph-mon[29621]: 3.a scrub ok 2024-03-21T13:20:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:16 smithi163 ceph-mon[29621]: 5.1d scrub starts 2024-03-21T13:20:17.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:16 smithi163 ceph-mon[29621]: 5.1d scrub ok 2024-03-21T13:20:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:16 smithi163 ceph-mon[40675]: pgmap v1006: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 689 B/s rd, 0 op/s 2024-03-21T13:20:17.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:16 smithi163 ceph-mon[40675]: 3.a scrub starts 2024-03-21T13:20:17.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:16 smithi163 ceph-mon[40675]: 3.a scrub ok 2024-03-21T13:20:17.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:16 smithi163 ceph-mon[40675]: 5.1d scrub starts 2024-03-21T13:20:17.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:16 smithi163 ceph-mon[40675]: 5.1d scrub ok 2024-03-21T13:20:17.162 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.4 reset_purged_snaps_last 2024-03-21T13:20:17.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-21T13:20:17.409 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:17.555 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:20:17.837 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:17 smithi163 ceph-mon[29621]: 4.1b scrub starts 2024-03-21T13:20:17.838 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:17 smithi163 ceph-mon[29621]: 4.1b scrub ok 2024-03-21T13:20:17.838 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:17 smithi163 ceph-mon[29621]: 5.a scrub starts 2024-03-21T13:20:17.838 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:17 smithi163 ceph-mon[29621]: 5.a scrub ok 2024-03-21T13:20:17.838 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:17 smithi163 ceph-mon[29621]: 6.18 scrub starts 2024-03-21T13:20:17.838 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:17 smithi163 ceph-mon[29621]: 6.18 scrub ok 2024-03-21T13:20:17.838 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:17 smithi163 ceph-mon[40675]: 4.1b scrub starts 2024-03-21T13:20:17.838 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:17 smithi163 ceph-mon[40675]: 4.1b scrub ok 2024-03-21T13:20:17.838 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:17 smithi163 ceph-mon[40675]: 5.a scrub starts 2024-03-21T13:20:17.839 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:17 smithi163 ceph-mon[40675]: 5.a scrub ok 2024-03-21T13:20:17.839 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:17 smithi163 ceph-mon[40675]: 6.18 scrub starts 2024-03-21T13:20:17.839 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:17 smithi163 ceph-mon[40675]: 6.18 scrub ok 2024-03-21T13:20:17.897 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:17 smithi204 ceph-mon[33153]: 4.1b scrub starts 2024-03-21T13:20:17.897 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:17 smithi204 ceph-mon[33153]: 4.1b scrub ok 2024-03-21T13:20:17.897 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:17 smithi204 ceph-mon[33153]: 5.a scrub starts 2024-03-21T13:20:17.897 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:17 smithi204 ceph-mon[33153]: 5.a scrub ok 2024-03-21T13:20:17.897 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:17 smithi204 ceph-mon[33153]: 6.18 scrub starts 2024-03-21T13:20:17.897 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:17 smithi204 ceph-mon[33153]: 6.18 scrub ok 2024-03-21T13:20:18.693 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:18 smithi204 ceph-mon[33153]: pgmap v1007: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:18.693 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:18 smithi204 ceph-mon[33153]: 5.10 scrub starts 2024-03-21T13:20:18.693 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:18 smithi204 ceph-mon[33153]: 5.10 scrub ok 2024-03-21T13:20:18.693 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:18 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:20:18.693 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:18 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:20:18.694 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:18 smithi204 ceph-mon[33153]: osdmap e455: 8 total, 8 up, 6 in 2024-03-21T13:20:18.694 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:18 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:20:18.694 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:18 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:20:18.694 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:18 smithi204 ceph-mon[33153]: 5.5 scrub starts 2024-03-21T13:20:18.694 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:18 smithi204 ceph-mon[33153]: 5.5 scrub ok 2024-03-21T13:20:18.694 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:18 smithi204 ceph-mon[33153]: 5.12 scrub starts 2024-03-21T13:20:18.694 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:18 smithi204 ceph-mon[33153]: 5.12 scrub ok 2024-03-21T13:20:18.870 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[40675]: pgmap v1007: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:18.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[40675]: 5.10 scrub starts 2024-03-21T13:20:18.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[40675]: 5.10 scrub ok 2024-03-21T13:20:18.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:20:18.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:20:18.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[40675]: osdmap e455: 8 total, 8 up, 6 in 2024-03-21T13:20:18.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:20:18.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:20:18.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[40675]: 5.5 scrub starts 2024-03-21T13:20:18.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[40675]: 5.5 scrub ok 2024-03-21T13:20:18.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[40675]: 5.12 scrub starts 2024-03-21T13:20:18.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[40675]: 5.12 scrub ok 2024-03-21T13:20:18.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[29621]: pgmap v1007: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:18.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[29621]: 5.10 scrub starts 2024-03-21T13:20:18.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[29621]: 5.10 scrub ok 2024-03-21T13:20:18.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:20:18.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:20:18.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[29621]: osdmap e455: 8 total, 8 up, 6 in 2024-03-21T13:20:18.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:20:18.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:20:18.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[29621]: 5.5 scrub starts 2024-03-21T13:20:18.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[29621]: 5.5 scrub ok 2024-03-21T13:20:18.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[29621]: 5.12 scrub starts 2024-03-21T13:20:18.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:18 smithi163 ceph-mon[29621]: 5.12 scrub ok 2024-03-21T13:20:19.253 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:20:19.253 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:20:19.945 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:19 smithi204 ceph-mon[33153]: 3.18 scrub starts 2024-03-21T13:20:19.945 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:19 smithi204 ceph-mon[33153]: 3.18 scrub ok 2024-03-21T13:20:19.945 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:19 smithi204 ceph-mon[33153]: osdmap e456: 8 total, 8 up, 6 in 2024-03-21T13:20:19.945 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:19 smithi204 ceph-mon[33153]: 5.1f scrub starts 2024-03-21T13:20:19.945 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:19 smithi204 ceph-mon[33153]: 5.1f scrub ok 2024-03-21T13:20:19.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-21T13:20:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:19 smithi163 ceph-mon[29621]: 3.18 scrub starts 2024-03-21T13:20:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:19 smithi163 ceph-mon[29621]: 3.18 scrub ok 2024-03-21T13:20:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:19 smithi163 ceph-mon[29621]: osdmap e456: 8 total, 8 up, 6 in 2024-03-21T13:20:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:19 smithi163 ceph-mon[29621]: 5.1f scrub starts 2024-03-21T13:20:20.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:19 smithi163 ceph-mon[29621]: 5.1f scrub ok 2024-03-21T13:20:20.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:19 smithi163 ceph-mon[40675]: 3.18 scrub starts 2024-03-21T13:20:20.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:19 smithi163 ceph-mon[40675]: 3.18 scrub ok 2024-03-21T13:20:20.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:19 smithi163 ceph-mon[40675]: osdmap e456: 8 total, 8 up, 6 in 2024-03-21T13:20:20.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:19 smithi163 ceph-mon[40675]: 5.1f scrub starts 2024-03-21T13:20:20.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:19 smithi163 ceph-mon[40675]: 5.1f scrub ok 2024-03-21T13:20:20.207 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.6 reset_purged_snaps_last 2024-03-21T13:20:20.216 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:20:20.457 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:20.682 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:20 smithi204 ceph-mon[33153]: pgmap v1010: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:20:20.682 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:20 smithi204 ceph-mon[33153]: 3.9 scrub starts 2024-03-21T13:20:20.682 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:20 smithi204 ceph-mon[33153]: 3.9 scrub ok 2024-03-21T13:20:20.895 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:20 smithi163 ceph-mon[40675]: pgmap v1010: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:20:20.895 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:20 smithi163 ceph-mon[40675]: 3.9 scrub starts 2024-03-21T13:20:20.895 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:20 smithi163 ceph-mon[40675]: 3.9 scrub ok 2024-03-21T13:20:20.895 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:20 smithi163 ceph-mon[29621]: pgmap v1010: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:20:20.896 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:20 smithi163 ceph-mon[29621]: 3.9 scrub starts 2024-03-21T13:20:20.896 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:20 smithi163 ceph-mon[29621]: 3.9 scrub ok 2024-03-21T13:20:21.949 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:20:21.949 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:20:21.949 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:20:21.949 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:21.949 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:21 smithi204 ceph-mon[33153]: pgmap v1011: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:20:22.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:20:22.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:22.006 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:21 smithi163 ceph-mon[29621]: pgmap v1011: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:20:22.006 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:20:22.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:22.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:21 smithi163 ceph-mon[40675]: pgmap v1011: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:20:22.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:22 smithi204 ceph-mon[33153]: 6.5 scrub starts 2024-03-21T13:20:22.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:22 smithi204 ceph-mon[33153]: 6.5 scrub ok 2024-03-21T13:20:22.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:22 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:22.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:20:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:20:22.914 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-21T13:20:23.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:22 smithi163 ceph-mon[29621]: 6.5 scrub starts 2024-03-21T13:20:23.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:22 smithi163 ceph-mon[29621]: 6.5 scrub ok 2024-03-21T13:20:23.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:22 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:23.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:22 smithi163 ceph-mon[40675]: 6.5 scrub starts 2024-03-21T13:20:23.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:22 smithi163 ceph-mon[40675]: 6.5 scrub ok 2024-03-21T13:20:23.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:22 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:23.177 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:20:23.456 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.7 reset_purged_snaps_last 2024-03-21T13:20:23.702 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:23 smithi204 ceph-mon[33153]: 3.1e scrub starts 2024-03-21T13:20:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:23 smithi204 ceph-mon[33153]: 3.1e scrub ok 2024-03-21T13:20:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:23 smithi204 ceph-mon[33153]: pgmap v1012: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:24.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:23 smithi163 ceph-mon[29621]: 3.1e scrub starts 2024-03-21T13:20:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:23 smithi163 ceph-mon[29621]: 3.1e scrub ok 2024-03-21T13:20:24.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:23 smithi163 ceph-mon[29621]: pgmap v1012: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:23 smithi163 ceph-mon[40675]: 3.1e scrub starts 2024-03-21T13:20:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:23 smithi163 ceph-mon[40675]: 3.1e scrub ok 2024-03-21T13:20:24.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:23 smithi163 ceph-mon[40675]: pgmap v1012: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:24.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:24 smithi204 ceph-mon[33153]: 4.1f deep-scrub starts 2024-03-21T13:20:24.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:24 smithi204 ceph-mon[33153]: 4.1f deep-scrub ok 2024-03-21T13:20:24.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:24 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:20:24.753 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:24 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:20:24.897 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:20:24.897 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:20:24.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:24 smithi163 ceph-mon[29621]: 4.1f deep-scrub starts 2024-03-21T13:20:24.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:24 smithi163 ceph-mon[29621]: 4.1f deep-scrub ok 2024-03-21T13:20:24.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:24 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:20:24.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:24 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:20:24.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:24 smithi163 ceph-mon[40675]: 4.1f deep-scrub starts 2024-03-21T13:20:24.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:24 smithi163 ceph-mon[40675]: 4.1f deep-scrub ok 2024-03-21T13:20:24.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:24 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:20:24.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:24 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:20:25.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-21T13:20:25.736 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:20:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:20:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:20:25.800 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:20:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:25 smithi204 ceph-mon[33153]: 4.1c scrub starts 2024-03-21T13:20:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:25 smithi204 ceph-mon[33153]: 4.1c scrub ok 2024-03-21T13:20:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:25 smithi204 ceph-mon[33153]: 6.19 scrub starts 2024-03-21T13:20:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:25 smithi204 ceph-mon[33153]: osdmap e457: 8 total, 8 up, 6 in 2024-03-21T13:20:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:25 smithi204 ceph-mon[33153]: 6.19 scrub ok 2024-03-21T13:20:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:25 smithi204 ceph-mon[33153]: pgmap v1014: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 685 B/s rd, 0 op/s 2024-03-21T13:20:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:25 smithi163 ceph-mon[29621]: 4.1c scrub starts 2024-03-21T13:20:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:25 smithi163 ceph-mon[29621]: 4.1c scrub ok 2024-03-21T13:20:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:25 smithi163 ceph-mon[29621]: 6.19 scrub starts 2024-03-21T13:20:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:25 smithi163 ceph-mon[29621]: osdmap e457: 8 total, 8 up, 6 in 2024-03-21T13:20:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:25 smithi163 ceph-mon[29621]: 6.19 scrub ok 2024-03-21T13:20:26.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:25 smithi163 ceph-mon[29621]: pgmap v1014: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 685 B/s rd, 0 op/s 2024-03-21T13:20:26.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:25 smithi163 ceph-mon[40675]: 4.1c scrub starts 2024-03-21T13:20:26.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:25 smithi163 ceph-mon[40675]: 4.1c scrub ok 2024-03-21T13:20:26.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:25 smithi163 ceph-mon[40675]: 6.19 scrub starts 2024-03-21T13:20:26.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:25 smithi163 ceph-mon[40675]: osdmap e457: 8 total, 8 up, 6 in 2024-03-21T13:20:26.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:25 smithi163 ceph-mon[40675]: 6.19 scrub ok 2024-03-21T13:20:26.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:25 smithi163 ceph-mon[40675]: pgmap v1014: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 685 B/s rd, 0 op/s 2024-03-21T13:20:26.269 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.2 reset_purged_snaps_last 2024-03-21T13:20:26.515 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:26.935 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:26 smithi204 ceph-mon[33153]: 6.14 scrub starts 2024-03-21T13:20:26.935 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:26 smithi204 ceph-mon[33153]: 6.14 scrub ok 2024-03-21T13:20:26.935 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:26 smithi204 ceph-mon[33153]: 3.1a scrub starts 2024-03-21T13:20:26.935 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:26 smithi204 ceph-mon[33153]: 3.1a scrub ok 2024-03-21T13:20:26.935 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:26 smithi204 ceph-mon[33153]: 4.8 deep-scrub starts 2024-03-21T13:20:26.935 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:26 smithi204 ceph-mon[33153]: 4.8 deep-scrub ok 2024-03-21T13:20:26.935 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:26 smithi204 ceph-mon[33153]: 5.c deep-scrub starts 2024-03-21T13:20:26.936 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:26 smithi204 ceph-mon[33153]: 5.c deep-scrub ok 2024-03-21T13:20:26.953 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[29621]: 6.14 scrub starts 2024-03-21T13:20:26.953 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[29621]: 6.14 scrub ok 2024-03-21T13:20:26.953 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[29621]: 3.1a scrub starts 2024-03-21T13:20:26.953 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[29621]: 3.1a scrub ok 2024-03-21T13:20:26.953 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[29621]: 4.8 deep-scrub starts 2024-03-21T13:20:26.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[29621]: 4.8 deep-scrub ok 2024-03-21T13:20:26.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[29621]: 5.c deep-scrub starts 2024-03-21T13:20:26.954 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[29621]: 5.c deep-scrub ok 2024-03-21T13:20:26.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[40675]: 6.14 scrub starts 2024-03-21T13:20:26.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[40675]: 6.14 scrub ok 2024-03-21T13:20:26.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[40675]: 3.1a scrub starts 2024-03-21T13:20:26.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[40675]: 3.1a scrub ok 2024-03-21T13:20:26.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[40675]: 4.8 deep-scrub starts 2024-03-21T13:20:26.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[40675]: 4.8 deep-scrub ok 2024-03-21T13:20:26.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[40675]: 5.c deep-scrub starts 2024-03-21T13:20:26.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:26 smithi163 ceph-mon[40675]: 5.c deep-scrub ok 2024-03-21T13:20:27.654 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:20:27.654 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:20:27.933 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[29621]: 4.16 deep-scrub starts 2024-03-21T13:20:27.933 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[29621]: 4.16 deep-scrub ok 2024-03-21T13:20:27.933 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[29621]: pgmap v1015: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:27.933 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[29621]: 3.11 scrub starts 2024-03-21T13:20:27.933 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[29621]: 3.11 scrub ok 2024-03-21T13:20:27.933 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:20:27.933 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:20:27.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[40675]: 4.16 deep-scrub starts 2024-03-21T13:20:27.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[40675]: 4.16 deep-scrub ok 2024-03-21T13:20:27.934 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[40675]: pgmap v1015: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:27.934 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[40675]: 3.11 scrub starts 2024-03-21T13:20:27.934 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[40675]: 3.11 scrub ok 2024-03-21T13:20:27.934 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:20:27.934 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:27 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:20:27.959 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:27 smithi204 ceph-mon[33153]: 4.16 deep-scrub starts 2024-03-21T13:20:27.959 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:27 smithi204 ceph-mon[33153]: 4.16 deep-scrub ok 2024-03-21T13:20:27.959 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:27 smithi204 ceph-mon[33153]: pgmap v1015: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:27.959 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:27 smithi204 ceph-mon[33153]: 3.11 scrub starts 2024-03-21T13:20:27.959 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:27 smithi204 ceph-mon[33153]: 3.11 scrub ok 2024-03-21T13:20:27.960 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:27 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:20:27.960 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:27 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:20:28.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-21T13:20:28.573 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:20:28.997 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:28 smithi204 ceph-mon[33153]: 6.6 scrub starts 2024-03-21T13:20:28.997 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:28 smithi204 ceph-mon[33153]: 6.6 scrub ok 2024-03-21T13:20:29.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:28 smithi163 ceph-mon[29621]: 6.6 scrub starts 2024-03-21T13:20:29.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:28 smithi163 ceph-mon[29621]: 6.6 scrub ok 2024-03-21T13:20:29.128 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:28 smithi163 ceph-mon[40675]: 6.6 scrub starts 2024-03-21T13:20:29.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:28 smithi163 ceph-mon[40675]: 6.6 scrub ok 2024-03-21T13:20:29.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.5 reset_purged_snaps_last 2024-03-21T13:20:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:29.946 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:29 smithi204 ceph-mon[33153]: 4.5 scrub starts 2024-03-21T13:20:29.946 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:29 smithi204 ceph-mon[33153]: 4.5 scrub ok 2024-03-21T13:20:29.946 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:29 smithi204 ceph-mon[33153]: pgmap v1016: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:29.946 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:29 smithi204 ceph-mon[33153]: 3.e scrub starts 2024-03-21T13:20:29.946 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:29 smithi204 ceph-mon[33153]: 3.e scrub ok 2024-03-21T13:20:30.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:29 smithi163 ceph-mon[29621]: 4.5 scrub starts 2024-03-21T13:20:30.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:29 smithi163 ceph-mon[29621]: 4.5 scrub ok 2024-03-21T13:20:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:29 smithi163 ceph-mon[29621]: pgmap v1016: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:29 smithi163 ceph-mon[29621]: 3.e scrub starts 2024-03-21T13:20:30.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:29 smithi163 ceph-mon[29621]: 3.e scrub ok 2024-03-21T13:20:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:29 smithi163 ceph-mon[40675]: 4.5 scrub starts 2024-03-21T13:20:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:29 smithi163 ceph-mon[40675]: 4.5 scrub ok 2024-03-21T13:20:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:29 smithi163 ceph-mon[40675]: pgmap v1016: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:29 smithi163 ceph-mon[40675]: 3.e scrub starts 2024-03-21T13:20:30.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:29 smithi163 ceph-mon[40675]: 3.e scrub ok 2024-03-21T13:20:30.196 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:20:30.196 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:20:30.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-21T13:20:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:30 smithi204 ceph-mon[33153]: 4.18 scrub starts 2024-03-21T13:20:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:30 smithi204 ceph-mon[33153]: 4.18 scrub ok 2024-03-21T13:20:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:30 smithi204 ceph-mon[33153]: 4.3 scrub starts 2024-03-21T13:20:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:30 smithi204 ceph-mon[33153]: 4.3 scrub ok 2024-03-21T13:20:31.138 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:20:31.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:30 smithi163 ceph-mon[29621]: 4.18 scrub starts 2024-03-21T13:20:31.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:30 smithi163 ceph-mon[29621]: 4.18 scrub ok 2024-03-21T13:20:31.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:30 smithi163 ceph-mon[29621]: 4.3 scrub starts 2024-03-21T13:20:31.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:30 smithi163 ceph-mon[29621]: 4.3 scrub ok 2024-03-21T13:20:31.151 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:30 smithi163 ceph-mon[40675]: 4.18 scrub starts 2024-03-21T13:20:31.151 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:30 smithi163 ceph-mon[40675]: 4.18 scrub ok 2024-03-21T13:20:31.151 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:30 smithi163 ceph-mon[40675]: 4.3 scrub starts 2024-03-21T13:20:31.151 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:30 smithi163 ceph-mon[40675]: 4.3 scrub ok 2024-03-21T13:20:31.967 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:31 smithi204 ceph-mon[33153]: 4.17 scrub starts 2024-03-21T13:20:31.967 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:31 smithi204 ceph-mon[33153]: 4.17 scrub ok 2024-03-21T13:20:31.967 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:31 smithi204 ceph-mon[33153]: pgmap v1017: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:31.967 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:31 smithi204 ceph-mon[33153]: 6.12 scrub starts 2024-03-21T13:20:31.967 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:31 smithi204 ceph-mon[33153]: 6.12 scrub ok 2024-03-21T13:20:32.164 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:31 smithi163 ceph-mon[29621]: 4.17 scrub starts 2024-03-21T13:20:32.164 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:31 smithi163 ceph-mon[29621]: 4.17 scrub ok 2024-03-21T13:20:32.165 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:31 smithi163 ceph-mon[29621]: pgmap v1017: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:32.165 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:31 smithi163 ceph-mon[29621]: 6.12 scrub starts 2024-03-21T13:20:32.165 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:31 smithi163 ceph-mon[29621]: 6.12 scrub ok 2024-03-21T13:20:32.165 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:31 smithi163 ceph-mon[40675]: 4.17 scrub starts 2024-03-21T13:20:32.165 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:31 smithi163 ceph-mon[40675]: 4.17 scrub ok 2024-03-21T13:20:32.165 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:31 smithi163 ceph-mon[40675]: pgmap v1017: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:32.165 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:31 smithi163 ceph-mon[40675]: 6.12 scrub starts 2024-03-21T13:20:32.165 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:31 smithi163 ceph-mon[40675]: 6.12 scrub ok 2024-03-21T13:20:32.762 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:20:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:20:32.810 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:20:32.811 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:20:33.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:32 smithi204 ceph-mon[33153]: 3.0 scrub starts 2024-03-21T13:20:33.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:32 smithi204 ceph-mon[33153]: 3.0 scrub ok 2024-03-21T13:20:33.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:32 smithi204 ceph-mon[33153]: 3.17 deep-scrub starts 2024-03-21T13:20:33.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:32 smithi204 ceph-mon[33153]: 3.17 deep-scrub ok 2024-03-21T13:20:33.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:32 smithi204 ceph-mon[33153]: 3.8 scrub starts 2024-03-21T13:20:33.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:32 smithi204 ceph-mon[33153]: 3.8 scrub ok 2024-03-21T13:20:33.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:32 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:33.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[29621]: 3.0 scrub starts 2024-03-21T13:20:33.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[29621]: 3.0 scrub ok 2024-03-21T13:20:33.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[29621]: 3.17 deep-scrub starts 2024-03-21T13:20:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[29621]: 3.17 deep-scrub ok 2024-03-21T13:20:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[29621]: 3.8 scrub starts 2024-03-21T13:20:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[29621]: 3.8 scrub ok 2024-03-21T13:20:33.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:33.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[40675]: 3.0 scrub starts 2024-03-21T13:20:33.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[40675]: 3.0 scrub ok 2024-03-21T13:20:33.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[40675]: 3.17 deep-scrub starts 2024-03-21T13:20:33.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[40675]: 3.17 deep-scrub ok 2024-03-21T13:20:33.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[40675]: 3.8 scrub starts 2024-03-21T13:20:33.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[40675]: 3.8 scrub ok 2024-03-21T13:20:33.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:32 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:33.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-21T13:20:33.982 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:20:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:33 smithi204 ceph-mon[33153]: 5.7 scrub starts 2024-03-21T13:20:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:33 smithi204 ceph-mon[33153]: 5.7 scrub ok 2024-03-21T13:20:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:33 smithi204 ceph-mon[33153]: 4.0 scrub starts 2024-03-21T13:20:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:33 smithi204 ceph-mon[33153]: 4.0 scrub ok 2024-03-21T13:20:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:33 smithi204 ceph-mon[33153]: 6.17 deep-scrub starts 2024-03-21T13:20:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:33 smithi204 ceph-mon[33153]: pgmap v1018: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:33 smithi204 ceph-mon[33153]: 5.4 scrub starts 2024-03-21T13:20:34.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:33 smithi204 ceph-mon[33153]: 5.4 scrub ok 2024-03-21T13:20:34.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[29621]: 5.7 scrub starts 2024-03-21T13:20:34.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[29621]: 5.7 scrub ok 2024-03-21T13:20:34.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[29621]: 4.0 scrub starts 2024-03-21T13:20:34.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[29621]: 4.0 scrub ok 2024-03-21T13:20:34.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[29621]: 6.17 deep-scrub starts 2024-03-21T13:20:34.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[29621]: pgmap v1018: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:34.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[29621]: 5.4 scrub starts 2024-03-21T13:20:34.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[29621]: 5.4 scrub ok 2024-03-21T13:20:34.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[40675]: 5.7 scrub starts 2024-03-21T13:20:34.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[40675]: 5.7 scrub ok 2024-03-21T13:20:34.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[40675]: 4.0 scrub starts 2024-03-21T13:20:34.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[40675]: 4.0 scrub ok 2024-03-21T13:20:34.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[40675]: 6.17 deep-scrub starts 2024-03-21T13:20:34.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[40675]: pgmap v1018: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:34.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[40675]: 5.4 scrub starts 2024-03-21T13:20:34.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:33 smithi163 ceph-mon[40675]: 5.4 scrub ok 2024-03-21T13:20:35.091 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:34 smithi204 ceph-mon[33153]: 6.17 deep-scrub ok 2024-03-21T13:20:35.092 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:34 smithi204 ceph-mon[33153]: 3.7 scrub starts 2024-03-21T13:20:35.092 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:34 smithi204 ceph-mon[33153]: 3.7 scrub ok 2024-03-21T13:20:35.092 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:34 smithi204 ceph-mon[33153]: 4.b scrub starts 2024-03-21T13:20:35.092 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:34 smithi204 ceph-mon[33153]: 4.b scrub ok 2024-03-21T13:20:35.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:34 smithi163 ceph-mon[29621]: 6.17 deep-scrub ok 2024-03-21T13:20:35.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:34 smithi163 ceph-mon[29621]: 3.7 scrub starts 2024-03-21T13:20:35.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:34 smithi163 ceph-mon[29621]: 3.7 scrub ok 2024-03-21T13:20:35.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:34 smithi163 ceph-mon[29621]: 4.b scrub starts 2024-03-21T13:20:35.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:34 smithi163 ceph-mon[29621]: 4.b scrub ok 2024-03-21T13:20:35.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:34 smithi163 ceph-mon[40675]: 6.17 deep-scrub ok 2024-03-21T13:20:35.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:34 smithi163 ceph-mon[40675]: 3.7 scrub starts 2024-03-21T13:20:35.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:34 smithi163 ceph-mon[40675]: 3.7 scrub ok 2024-03-21T13:20:35.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:34 smithi163 ceph-mon[40675]: 4.b scrub starts 2024-03-21T13:20:35.262 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:34 smithi163 ceph-mon[40675]: 4.b scrub ok 2024-03-21T13:20:35.653 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:20:35.653 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:20:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:35 smithi204 ceph-mon[33153]: 4.7 scrub starts 2024-03-21T13:20:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:35 smithi204 ceph-mon[33153]: 4.7 scrub ok 2024-03-21T13:20:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:35 smithi204 ceph-mon[33153]: pgmap v1019: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 994 B/s rd, 0 op/s 2024-03-21T13:20:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:35 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:36.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:20:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:20:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:20:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:35 smithi163 ceph-mon[29621]: 4.7 scrub starts 2024-03-21T13:20:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:35 smithi163 ceph-mon[29621]: 4.7 scrub ok 2024-03-21T13:20:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:35 smithi163 ceph-mon[29621]: pgmap v1019: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 994 B/s rd, 0 op/s 2024-03-21T13:20:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:35 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:35 smithi163 ceph-mon[40675]: 4.7 scrub starts 2024-03-21T13:20:36.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:35 smithi163 ceph-mon[40675]: 4.7 scrub ok 2024-03-21T13:20:36.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:35 smithi163 ceph-mon[40675]: pgmap v1019: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 994 B/s rd, 0 op/s 2024-03-21T13:20:36.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:35 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:36.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-21T13:20:36.623 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:20:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:36 smithi204 ceph-mon[33153]: 3.4 scrub starts 2024-03-21T13:20:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:36 smithi204 ceph-mon[33153]: 3.4 scrub ok 2024-03-21T13:20:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:36 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:20:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:36 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:20:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:36 smithi204 ceph-mon[33153]: 3.1b scrub starts 2024-03-21T13:20:37.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:36 smithi204 ceph-mon[33153]: 3.1b scrub ok 2024-03-21T13:20:37.195 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-21T13:20:37.195 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-03-21T13:20:37.195 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd in 1 2024-03-21T13:20:37.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:36 smithi163 ceph-mon[29621]: 3.4 scrub starts 2024-03-21T13:20:37.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:36 smithi163 ceph-mon[29621]: 3.4 scrub ok 2024-03-21T13:20:37.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:36 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:20:37.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:36 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:20:37.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:36 smithi163 ceph-mon[29621]: 3.1b scrub starts 2024-03-21T13:20:37.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:36 smithi163 ceph-mon[29621]: 3.1b scrub ok 2024-03-21T13:20:37.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:36 smithi163 ceph-mon[40675]: 3.4 scrub starts 2024-03-21T13:20:37.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:36 smithi163 ceph-mon[40675]: 3.4 scrub ok 2024-03-21T13:20:37.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:36 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:20:37.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:36 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:20:37.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:36 smithi163 ceph-mon[40675]: 3.1b scrub starts 2024-03-21T13:20:37.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:36 smithi163 ceph-mon[40675]: 3.1b scrub ok 2024-03-21T13:20:37.452 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:38.147 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:37 smithi204 ceph-mon[33153]: 4.9 scrub starts 2024-03-21T13:20:38.147 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:37 smithi204 ceph-mon[33153]: 4.9 scrub ok 2024-03-21T13:20:38.147 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:37 smithi204 ceph-mon[33153]: osdmap e458: 8 total, 8 up, 6 in 2024-03-21T13:20:38.147 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:37 smithi204 ceph-mon[33153]: pgmap v1021: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:38.147 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:37 smithi204 ceph-mon[33153]: 6.16 scrub starts 2024-03-21T13:20:38.147 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:37 smithi204 ceph-mon[33153]: 6.16 scrub ok 2024-03-21T13:20:38.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:37 smithi163 ceph-mon[40675]: 4.9 scrub starts 2024-03-21T13:20:38.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:37 smithi163 ceph-mon[40675]: 4.9 scrub ok 2024-03-21T13:20:38.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:37 smithi163 ceph-mon[40675]: osdmap e458: 8 total, 8 up, 6 in 2024-03-21T13:20:38.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:37 smithi163 ceph-mon[40675]: pgmap v1021: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:38.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:37 smithi163 ceph-mon[40675]: 6.16 scrub starts 2024-03-21T13:20:38.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:37 smithi163 ceph-mon[40675]: 6.16 scrub ok 2024-03-21T13:20:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:37 smithi163 ceph-mon[29621]: 4.9 scrub starts 2024-03-21T13:20:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:37 smithi163 ceph-mon[29621]: 4.9 scrub ok 2024-03-21T13:20:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:37 smithi163 ceph-mon[29621]: osdmap e458: 8 total, 8 up, 6 in 2024-03-21T13:20:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:37 smithi163 ceph-mon[29621]: pgmap v1021: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:38.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:37 smithi163 ceph-mon[29621]: 6.16 scrub starts 2024-03-21T13:20:38.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:37 smithi163 ceph-mon[29621]: 6.16 scrub ok 2024-03-21T13:20:38.438 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-03-21T13:20:38.438 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T13:20:39.185 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:39 smithi163 ceph-mon[40675]: 6.f scrub starts 2024-03-21T13:20:39.185 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:39 smithi163 ceph-mon[40675]: 6.f scrub ok 2024-03-21T13:20:39.185 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:39 smithi163 ceph-mon[29621]: 6.f scrub starts 2024-03-21T13:20:39.185 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:39 smithi163 ceph-mon[29621]: 6.f scrub ok 2024-03-21T13:20:39.216 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:39 smithi204 ceph-mon[33153]: 6.f scrub starts 2024-03-21T13:20:39.216 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:39 smithi204 ceph-mon[33153]: 6.f scrub ok 2024-03-21T13:20:39.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:39 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a[29617]: 2024-03-21T13:20:39.513+0000 7f8e1b000700 -1 mon.a@0(leader).osd e458 definitely_dead 0 2024-03-21T13:20:40.105 INFO:teuthology.orchestra.run.smithi163.stderr:marked in osd.1. 2024-03-21T13:20:40.423 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:40 smithi163 ceph-mon[29621]: pgmap v1022: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:40.423 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:40 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3179673995' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-21T13:20:40.423 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:40 smithi163 ceph-mon[29621]: 5.f scrub starts 2024-03-21T13:20:40.423 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:40 smithi163 ceph-mon[29621]: 5.f scrub ok 2024-03-21T13:20:40.423 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:40 smithi163 ceph-mon[40675]: pgmap v1022: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:40.424 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:40 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3179673995' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-21T13:20:40.424 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:40 smithi163 ceph-mon[40675]: 5.f scrub starts 2024-03-21T13:20:40.424 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:40 smithi163 ceph-mon[40675]: 5.f scrub ok 2024-03-21T13:20:40.455 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:40 smithi204 ceph-mon[33153]: pgmap v1022: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:20:40.455 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:40 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3179673995' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-21T13:20:40.455 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:40 smithi204 ceph-mon[33153]: 5.f scrub starts 2024-03-21T13:20:40.455 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:40 smithi204 ceph-mon[33153]: 5.f scrub ok 2024-03-21T13:20:41.003 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-03-21T13:20:41.003 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-21T13:20:41.003 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-03-21T13:20:41.003 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd in 0 2024-03-21T13:20:41.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:41 smithi163 ceph-mon[40675]: 4.6 scrub starts 2024-03-21T13:20:41.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:41 smithi163 ceph-mon[40675]: 4.6 scrub ok 2024-03-21T13:20:41.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:41 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3179673995' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-03-21T13:20:41.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:41 smithi163 ceph-mon[40675]: osdmap e459: 8 total, 8 up, 7 in 2024-03-21T13:20:41.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:41 smithi163 ceph-mon[29621]: 4.6 scrub starts 2024-03-21T13:20:41.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:41 smithi163 ceph-mon[29621]: 4.6 scrub ok 2024-03-21T13:20:41.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:41 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3179673995' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-03-21T13:20:41.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:41 smithi163 ceph-mon[29621]: osdmap e459: 8 total, 8 up, 7 in 2024-03-21T13:20:41.341 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:41.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:41 smithi204 ceph-mon[33153]: 4.6 scrub starts 2024-03-21T13:20:41.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:41 smithi204 ceph-mon[33153]: 4.6 scrub ok 2024-03-21T13:20:41.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:41 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3179673995' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-03-21T13:20:41.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:41 smithi204 ceph-mon[33153]: osdmap e459: 8 total, 8 up, 7 in 2024-03-21T13:20:42.404 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:42 smithi204 ceph-mon[33153]: pgmap v1024: 132 pgs: 22 peering, 110 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:20:42.404 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:42 smithi204 ceph-mon[33153]: 3.2 scrub starts 2024-03-21T13:20:42.404 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:42 smithi204 ceph-mon[33153]: 3.2 scrub ok 2024-03-21T13:20:42.404 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:42 smithi204 ceph-mon[33153]: osdmap e460: 8 total, 8 up, 7 in 2024-03-21T13:20:42.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:42 smithi163 ceph-mon[29621]: pgmap v1024: 132 pgs: 22 peering, 110 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:20:42.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:42 smithi163 ceph-mon[29621]: 3.2 scrub starts 2024-03-21T13:20:42.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:42 smithi163 ceph-mon[29621]: 3.2 scrub ok 2024-03-21T13:20:42.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:42 smithi163 ceph-mon[29621]: osdmap e460: 8 total, 8 up, 7 in 2024-03-21T13:20:42.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:42 smithi163 ceph-mon[40675]: pgmap v1024: 132 pgs: 22 peering, 110 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:20:42.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:42 smithi163 ceph-mon[40675]: 3.2 scrub starts 2024-03-21T13:20:42.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:42 smithi163 ceph-mon[40675]: 3.2 scrub ok 2024-03-21T13:20:42.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:42 smithi163 ceph-mon[40675]: osdmap e460: 8 total, 8 up, 7 in 2024-03-21T13:20:42.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:20:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:20:43.404 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:43 smithi163 ceph-mon[29621]: 5.17 scrub starts 2024-03-21T13:20:43.404 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:43 smithi163 ceph-mon[29621]: 5.17 scrub ok 2024-03-21T13:20:43.404 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:43 smithi163 ceph-mon[29621]: 4.d scrub starts 2024-03-21T13:20:43.404 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:43 smithi163 ceph-mon[29621]: 4.d scrub ok 2024-03-21T13:20:43.404 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:43 smithi163 ceph-mon[29621]: osdmap e461: 8 total, 8 up, 7 in 2024-03-21T13:20:43.405 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:43.405 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:43 smithi163 ceph-mon[40675]: 5.17 scrub starts 2024-03-21T13:20:43.405 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:43 smithi163 ceph-mon[40675]: 5.17 scrub ok 2024-03-21T13:20:43.405 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:43 smithi163 ceph-mon[40675]: 4.d scrub starts 2024-03-21T13:20:43.405 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:43 smithi163 ceph-mon[40675]: 4.d scrub ok 2024-03-21T13:20:43.405 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:43 smithi163 ceph-mon[40675]: osdmap e461: 8 total, 8 up, 7 in 2024-03-21T13:20:43.405 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:43.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:43 smithi204 ceph-mon[33153]: 5.17 scrub starts 2024-03-21T13:20:43.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:43 smithi204 ceph-mon[33153]: 5.17 scrub ok 2024-03-21T13:20:43.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:43 smithi204 ceph-mon[33153]: 4.d scrub starts 2024-03-21T13:20:43.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:43 smithi204 ceph-mon[33153]: 4.d scrub ok 2024-03-21T13:20:43.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:43 smithi204 ceph-mon[33153]: osdmap e461: 8 total, 8 up, 7 in 2024-03-21T13:20:43.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:43.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:43 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a[29617]: 2024-03-21T13:20:43.402+0000 7f8e1b000700 -1 mon.a@0(leader).osd e462 definitely_dead 0 2024-03-21T13:20:44.128 INFO:teuthology.orchestra.run.smithi163.stderr:marked in osd.0. 2024-03-21T13:20:44.441 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[29621]: pgmap v1027: 132 pgs: 22 peering, 110 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s; 91 B/s, 4 objects/s recovering 2024-03-21T13:20:44.441 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[29621]: osdmap e462: 8 total, 8 up, 7 in 2024-03-21T13:20:44.441 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[29621]: 3.1 scrub starts 2024-03-21T13:20:44.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[29621]: 3.1 scrub ok 2024-03-21T13:20:44.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3342898057' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-21T13:20:44.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[29621]: 3.f scrub starts 2024-03-21T13:20:44.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[29621]: 3.f scrub ok 2024-03-21T13:20:44.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[29621]: 6.a deep-scrub starts 2024-03-21T13:20:44.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[29621]: 6.a deep-scrub ok 2024-03-21T13:20:44.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[40675]: pgmap v1027: 132 pgs: 22 peering, 110 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s; 91 B/s, 4 objects/s recovering 2024-03-21T13:20:44.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[40675]: osdmap e462: 8 total, 8 up, 7 in 2024-03-21T13:20:44.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[40675]: 3.1 scrub starts 2024-03-21T13:20:44.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[40675]: 3.1 scrub ok 2024-03-21T13:20:44.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3342898057' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-21T13:20:44.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[40675]: 3.f scrub starts 2024-03-21T13:20:44.443 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[40675]: 3.f scrub ok 2024-03-21T13:20:44.443 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[40675]: 6.a deep-scrub starts 2024-03-21T13:20:44.443 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:44 smithi163 ceph-mon[40675]: 6.a deep-scrub ok 2024-03-21T13:20:44.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:44 smithi204 ceph-mon[33153]: pgmap v1027: 132 pgs: 22 peering, 110 active+clean; 582 KiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s; 91 B/s, 4 objects/s recovering 2024-03-21T13:20:44.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:44 smithi204 ceph-mon[33153]: osdmap e462: 8 total, 8 up, 7 in 2024-03-21T13:20:44.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:44 smithi204 ceph-mon[33153]: 3.1 scrub starts 2024-03-21T13:20:44.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:44 smithi204 ceph-mon[33153]: 3.1 scrub ok 2024-03-21T13:20:44.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3342898057' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-21T13:20:44.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:44 smithi204 ceph-mon[33153]: 3.f scrub starts 2024-03-21T13:20:44.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:44 smithi204 ceph-mon[33153]: 3.f scrub ok 2024-03-21T13:20:44.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:44 smithi204 ceph-mon[33153]: 6.a deep-scrub starts 2024-03-21T13:20:44.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:44 smithi204 ceph-mon[33153]: 6.a deep-scrub ok 2024-03-21T13:20:44.908 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-03-21T13:20:44.908 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-21T13:20:45.196 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:45.470 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:45 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3342898057' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-03-21T13:20:45.470 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:45 smithi163 ceph-mon[29621]: osdmap e463: 8 total, 8 up, 8 in 2024-03-21T13:20:45.470 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:45 smithi163 ceph-mon[29621]: 6.1a scrub starts 2024-03-21T13:20:45.470 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:45 smithi163 ceph-mon[29621]: 6.1a scrub ok 2024-03-21T13:20:45.470 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:45 smithi163 ceph-mon[29621]: osdmap e464: 8 total, 8 up, 8 in 2024-03-21T13:20:45.471 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:45 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3342898057' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-03-21T13:20:45.471 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:45 smithi163 ceph-mon[40675]: osdmap e463: 8 total, 8 up, 8 in 2024-03-21T13:20:45.471 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:45 smithi163 ceph-mon[40675]: 6.1a scrub starts 2024-03-21T13:20:45.471 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:45 smithi163 ceph-mon[40675]: 6.1a scrub ok 2024-03-21T13:20:45.471 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:45 smithi163 ceph-mon[40675]: osdmap e464: 8 total, 8 up, 8 in 2024-03-21T13:20:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:45 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3342898057' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-03-21T13:20:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:45 smithi204 ceph-mon[33153]: osdmap e463: 8 total, 8 up, 8 in 2024-03-21T13:20:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:45 smithi204 ceph-mon[33153]: 6.1a scrub starts 2024-03-21T13:20:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:45 smithi204 ceph-mon[33153]: 6.1a scrub ok 2024-03-21T13:20:45.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:45 smithi204 ceph-mon[33153]: osdmap e464: 8 total, 8 up, 8 in 2024-03-21T13:20:45.761 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:20:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:20:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:20:46.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[29621]: 5.1c scrub starts 2024-03-21T13:20:46.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[29621]: 5.1c scrub ok 2024-03-21T13:20:46.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[29621]: pgmap v1030: 132 pgs: 25 peering, 107 active+clean; 582 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 430 B/s, 15 objects/s recovering 2024-03-21T13:20:46.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[29621]: 4.2 scrub starts 2024-03-21T13:20:46.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[29621]: 4.2 scrub ok 2024-03-21T13:20:46.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[29621]: 3.d scrub starts 2024-03-21T13:20:46.446 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[29621]: 3.d scrub ok 2024-03-21T13:20:46.446 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[29621]: osdmap e465: 8 total, 8 up, 8 in 2024-03-21T13:20:46.446 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[40675]: 5.1c scrub starts 2024-03-21T13:20:46.446 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[40675]: 5.1c scrub ok 2024-03-21T13:20:46.446 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[40675]: pgmap v1030: 132 pgs: 25 peering, 107 active+clean; 582 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 430 B/s, 15 objects/s recovering 2024-03-21T13:20:46.446 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[40675]: 4.2 scrub starts 2024-03-21T13:20:46.446 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[40675]: 4.2 scrub ok 2024-03-21T13:20:46.446 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[40675]: 3.d scrub starts 2024-03-21T13:20:46.446 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[40675]: 3.d scrub ok 2024-03-21T13:20:46.446 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:46 smithi163 ceph-mon[40675]: osdmap e465: 8 total, 8 up, 8 in 2024-03-21T13:20:46.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:46 smithi204 ceph-mon[33153]: 5.1c scrub starts 2024-03-21T13:20:46.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:46 smithi204 ceph-mon[33153]: 5.1c scrub ok 2024-03-21T13:20:46.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:46 smithi204 ceph-mon[33153]: pgmap v1030: 132 pgs: 25 peering, 107 active+clean; 582 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 430 B/s, 15 objects/s recovering 2024-03-21T13:20:46.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:46 smithi204 ceph-mon[33153]: 4.2 scrub starts 2024-03-21T13:20:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:46 smithi204 ceph-mon[33153]: 4.2 scrub ok 2024-03-21T13:20:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:46 smithi204 ceph-mon[33153]: 3.d scrub starts 2024-03-21T13:20:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:46 smithi204 ceph-mon[33153]: 3.d scrub ok 2024-03-21T13:20:46.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:46 smithi204 ceph-mon[33153]: osdmap e465: 8 total, 8 up, 8 in 2024-03-21T13:20:47.004 INFO:teuthology.orchestra.run.smithi163.stdout:mon.a: {} 2024-03-21T13:20:47.016 INFO:teuthology.orchestra.run.smithi163.stdout:mon.b: {} 2024-03-21T13:20:47.025 INFO:teuthology.orchestra.run.smithi163.stdout:mon.c: {} 2024-03-21T13:20:47.027 INFO:teuthology.orchestra.run.smithi163.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-21T13:20:47.027 INFO:teuthology.orchestra.run.smithi163.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-21T13:20:47.027 INFO:teuthology.orchestra.run.smithi163.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-21T13:20:47.395 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[40675]: 4.c scrub starts 2024-03-21T13:20:47.395 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[40675]: 4.c scrub ok 2024-03-21T13:20:47.395 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[40675]: 4.a scrub starts 2024-03-21T13:20:47.396 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[40675]: 4.a scrub ok 2024-03-21T13:20:47.396 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[40675]: 5.6 scrub starts 2024-03-21T13:20:47.396 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[40675]: 5.6 scrub ok 2024-03-21T13:20:47.396 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[40675]: 4.12 scrub starts 2024-03-21T13:20:47.396 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[40675]: 4.12 scrub ok 2024-03-21T13:20:47.396 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[40675]: 3.13 scrub starts 2024-03-21T13:20:47.396 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[40675]: 3.13 scrub ok 2024-03-21T13:20:47.396 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[40675]: osdmap e466: 8 total, 8 up, 8 in 2024-03-21T13:20:47.396 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2388307346' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:20:47.396 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[29621]: 4.c scrub starts 2024-03-21T13:20:47.396 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[29621]: 4.c scrub ok 2024-03-21T13:20:47.396 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[29621]: 4.a scrub starts 2024-03-21T13:20:47.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[29621]: 4.a scrub ok 2024-03-21T13:20:47.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[29621]: 5.6 scrub starts 2024-03-21T13:20:47.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[29621]: 5.6 scrub ok 2024-03-21T13:20:47.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[29621]: 4.12 scrub starts 2024-03-21T13:20:47.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[29621]: 4.12 scrub ok 2024-03-21T13:20:47.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[29621]: 3.13 scrub starts 2024-03-21T13:20:47.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[29621]: 3.13 scrub ok 2024-03-21T13:20:47.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[29621]: osdmap e466: 8 total, 8 up, 8 in 2024-03-21T13:20:47.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:47 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2388307346' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:20:47.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:47 smithi204 ceph-mon[33153]: 4.c scrub starts 2024-03-21T13:20:47.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:47 smithi204 ceph-mon[33153]: 4.c scrub ok 2024-03-21T13:20:47.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:47 smithi204 ceph-mon[33153]: 4.a scrub starts 2024-03-21T13:20:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:47 smithi204 ceph-mon[33153]: 4.a scrub ok 2024-03-21T13:20:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:47 smithi204 ceph-mon[33153]: 5.6 scrub starts 2024-03-21T13:20:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:47 smithi204 ceph-mon[33153]: 5.6 scrub ok 2024-03-21T13:20:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:47 smithi204 ceph-mon[33153]: 4.12 scrub starts 2024-03-21T13:20:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:47 smithi204 ceph-mon[33153]: 4.12 scrub ok 2024-03-21T13:20:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:47 smithi204 ceph-mon[33153]: 3.13 scrub starts 2024-03-21T13:20:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:47 smithi204 ceph-mon[33153]: 3.13 scrub ok 2024-03-21T13:20:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:47 smithi204 ceph-mon[33153]: osdmap e466: 8 total, 8 up, 8 in 2024-03-21T13:20:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:47 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2388307346' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:20:47.802 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-21T13:20:48.051 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:48 smithi204 ceph-mon[33153]: pgmap v1034: 132 pgs: 1 remapped+peering, 10 peering, 121 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 148 KiB/s, 37 objects/s recovering 2024-03-21T13:20:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:48 smithi204 ceph-mon[33153]: osdmap e467: 8 total, 8 up, 8 in 2024-03-21T13:20:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:48 smithi204 ceph-mon[33153]: 5.1b scrub starts 2024-03-21T13:20:49.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:48 smithi204 ceph-mon[33153]: 5.1b scrub ok 2024-03-21T13:20:49.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:48 smithi204 ceph-mon[33153]: 5.11 scrub starts 2024-03-21T13:20:49.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:48 smithi204 ceph-mon[33153]: 5.11 scrub ok 2024-03-21T13:20:49.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:48 smithi204 ceph-mon[33153]: 6.4 deep-scrub starts 2024-03-21T13:20:49.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:48 smithi204 ceph-mon[33153]: 6.4 deep-scrub ok 2024-03-21T13:20:49.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[29621]: pgmap v1034: 132 pgs: 1 remapped+peering, 10 peering, 121 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 148 KiB/s, 37 objects/s recovering 2024-03-21T13:20:49.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[29621]: osdmap e467: 8 total, 8 up, 8 in 2024-03-21T13:20:49.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[29621]: 5.1b scrub starts 2024-03-21T13:20:49.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[29621]: 5.1b scrub ok 2024-03-21T13:20:49.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[29621]: 5.11 scrub starts 2024-03-21T13:20:49.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[29621]: 5.11 scrub ok 2024-03-21T13:20:49.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[29621]: 6.4 deep-scrub starts 2024-03-21T13:20:49.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[29621]: 6.4 deep-scrub ok 2024-03-21T13:20:49.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[40675]: pgmap v1034: 132 pgs: 1 remapped+peering, 10 peering, 121 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 148 KiB/s, 37 objects/s recovering 2024-03-21T13:20:49.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[40675]: osdmap e467: 8 total, 8 up, 8 in 2024-03-21T13:20:49.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[40675]: 5.1b scrub starts 2024-03-21T13:20:49.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[40675]: 5.1b scrub ok 2024-03-21T13:20:49.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[40675]: 5.11 scrub starts 2024-03-21T13:20:49.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[40675]: 5.11 scrub ok 2024-03-21T13:20:49.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[40675]: 6.4 deep-scrub starts 2024-03-21T13:20:49.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:48 smithi163 ceph-mon[40675]: 6.4 deep-scrub ok 2024-03-21T13:20:49.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:49 smithi163 ceph-mon[29621]: 6.7 deep-scrub starts 2024-03-21T13:20:49.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:49 smithi163 ceph-mon[29621]: 6.7 deep-scrub ok 2024-03-21T13:20:49.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:49 smithi163 ceph-mon[40675]: 6.7 deep-scrub starts 2024-03-21T13:20:49.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:49 smithi163 ceph-mon[40675]: 6.7 deep-scrub ok 2024-03-21T13:20:49.937 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:49 smithi204 ceph-mon[33153]: 6.7 deep-scrub starts 2024-03-21T13:20:49.937 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:49 smithi204 ceph-mon[33153]: 6.7 deep-scrub ok 2024-03-21T13:20:50.562 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.0 to 1 (10000) 2024-03-21T13:20:50.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:50 smithi163 ceph-mon[29621]: 4.14 deep-scrub starts 2024-03-21T13:20:50.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:50 smithi163 ceph-mon[29621]: 4.14 deep-scrub ok 2024-03-21T13:20:50.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:50 smithi163 ceph-mon[29621]: pgmap v1036: 132 pgs: 1 remapped+peering, 10 peering, 121 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 145 KiB/s, 27 objects/s recovering 2024-03-21T13:20:50.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:50 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/401333809' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:20:50.940 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:50 smithi163 ceph-mon[40675]: 4.14 deep-scrub starts 2024-03-21T13:20:50.941 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:50 smithi163 ceph-mon[40675]: 4.14 deep-scrub ok 2024-03-21T13:20:50.941 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:50 smithi163 ceph-mon[40675]: pgmap v1036: 132 pgs: 1 remapped+peering, 10 peering, 121 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 145 KiB/s, 27 objects/s recovering 2024-03-21T13:20:50.941 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:50 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/401333809' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:20:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:50 smithi204 ceph-mon[33153]: 4.14 deep-scrub starts 2024-03-21T13:20:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:50 smithi204 ceph-mon[33153]: 4.14 deep-scrub ok 2024-03-21T13:20:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:50 smithi204 ceph-mon[33153]: pgmap v1036: 132 pgs: 1 remapped+peering, 10 peering, 121 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 145 KiB/s, 27 objects/s recovering 2024-03-21T13:20:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:50 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/401333809' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:20:51.376 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-21T13:20:51.629 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:51.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[29621]: 6.13 scrub starts 2024-03-21T13:20:51.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[29621]: 6.13 scrub ok 2024-03-21T13:20:51.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/401333809' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:20:51.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[29621]: osdmap e468: 8 total, 8 up, 8 in 2024-03-21T13:20:51.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[29621]: 5.e scrub starts 2024-03-21T13:20:51.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[29621]: 5.e scrub ok 2024-03-21T13:20:51.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:51.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[29621]: osdmap e469: 8 total, 8 up, 8 in 2024-03-21T13:20:51.752 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[40675]: 6.13 scrub starts 2024-03-21T13:20:51.752 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[40675]: 6.13 scrub ok 2024-03-21T13:20:51.752 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/401333809' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:20:51.752 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[40675]: osdmap e468: 8 total, 8 up, 8 in 2024-03-21T13:20:51.752 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[40675]: 5.e scrub starts 2024-03-21T13:20:51.752 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[40675]: 5.e scrub ok 2024-03-21T13:20:51.753 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:51.753 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:51 smithi163 ceph-mon[40675]: osdmap e469: 8 total, 8 up, 8 in 2024-03-21T13:20:52.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:51 smithi204 ceph-mon[33153]: 6.13 scrub starts 2024-03-21T13:20:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:51 smithi204 ceph-mon[33153]: 6.13 scrub ok 2024-03-21T13:20:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:51 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/401333809' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:20:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:51 smithi204 ceph-mon[33153]: osdmap e468: 8 total, 8 up, 8 in 2024-03-21T13:20:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:51 smithi204 ceph-mon[33153]: 5.e scrub starts 2024-03-21T13:20:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:51 smithi204 ceph-mon[33153]: 5.e scrub ok 2024-03-21T13:20:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:52.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:51 smithi204 ceph-mon[33153]: osdmap e469: 8 total, 8 up, 8 in 2024-03-21T13:20:52.900 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[29621]: 6.2 deep-scrub starts 2024-03-21T13:20:52.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[29621]: 6.2 deep-scrub ok 2024-03-21T13:20:52.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[29621]: pgmap v1038: 132 pgs: 132 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 118 KiB/s, 5 objects/s recovering 2024-03-21T13:20:52.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[29621]: 5.1a scrub starts 2024-03-21T13:20:52.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[29621]: 5.1a scrub ok 2024-03-21T13:20:52.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:20:52.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:20:52.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[29621]: 2.2 scrub starts 2024-03-21T13:20:52.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[29621]: 2.2 scrub ok 2024-03-21T13:20:52.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[40675]: 6.2 deep-scrub starts 2024-03-21T13:20:52.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[40675]: 6.2 deep-scrub ok 2024-03-21T13:20:52.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[40675]: pgmap v1038: 132 pgs: 132 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 118 KiB/s, 5 objects/s recovering 2024-03-21T13:20:52.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[40675]: 5.1a scrub starts 2024-03-21T13:20:52.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[40675]: 5.1a scrub ok 2024-03-21T13:20:52.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:20:52.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:20:52.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[40675]: 2.2 scrub starts 2024-03-21T13:20:52.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:52 smithi163 ceph-mon[40675]: 2.2 scrub ok 2024-03-21T13:20:53.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:20:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:20:53.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:52 smithi204 ceph-mon[33153]: 6.2 deep-scrub starts 2024-03-21T13:20:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:52 smithi204 ceph-mon[33153]: 6.2 deep-scrub ok 2024-03-21T13:20:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:52 smithi204 ceph-mon[33153]: pgmap v1038: 132 pgs: 132 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 118 KiB/s, 5 objects/s recovering 2024-03-21T13:20:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:52 smithi204 ceph-mon[33153]: 5.1a scrub starts 2024-03-21T13:20:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:52 smithi204 ceph-mon[33153]: 5.1a scrub ok 2024-03-21T13:20:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:52 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:20:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:52 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:20:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:52 smithi204 ceph-mon[33153]: 2.2 scrub starts 2024-03-21T13:20:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:52 smithi204 ceph-mon[33153]: 2.2 scrub ok 2024-03-21T13:20:54.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:53 smithi204 ceph-mon[33153]: 6.c scrub starts 2024-03-21T13:20:54.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:53 smithi204 ceph-mon[33153]: 6.c scrub ok 2024-03-21T13:20:54.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:54.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:53 smithi204 ceph-mon[33153]: osdmap e470: 8 total, 8 up, 8 in 2024-03-21T13:20:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:53 smithi204 ceph-mon[33153]: 3.1d scrub starts 2024-03-21T13:20:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:53 smithi204 ceph-mon[33153]: 3.1d scrub ok 2024-03-21T13:20:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:53 smithi204 ceph-mon[33153]: 6.9 scrub starts 2024-03-21T13:20:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:53 smithi204 ceph-mon[33153]: 6.9 scrub ok 2024-03-21T13:20:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:53 smithi204 ceph-mon[33153]: 5.1 scrub starts 2024-03-21T13:20:54.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:53 smithi204 ceph-mon[33153]: 5.1 scrub ok 2024-03-21T13:20:54.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[29621]: 6.c scrub starts 2024-03-21T13:20:54.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[29621]: 6.c scrub ok 2024-03-21T13:20:54.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:54.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[29621]: osdmap e470: 8 total, 8 up, 8 in 2024-03-21T13:20:54.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[29621]: 3.1d scrub starts 2024-03-21T13:20:54.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[29621]: 3.1d scrub ok 2024-03-21T13:20:54.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[29621]: 6.9 scrub starts 2024-03-21T13:20:54.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[29621]: 6.9 scrub ok 2024-03-21T13:20:54.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[29621]: 5.1 scrub starts 2024-03-21T13:20:54.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[29621]: 5.1 scrub ok 2024-03-21T13:20:54.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[40675]: 6.c scrub starts 2024-03-21T13:20:54.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[40675]: 6.c scrub ok 2024-03-21T13:20:54.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:54.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[40675]: osdmap e470: 8 total, 8 up, 8 in 2024-03-21T13:20:54.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[40675]: 3.1d scrub starts 2024-03-21T13:20:54.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[40675]: 3.1d scrub ok 2024-03-21T13:20:54.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[40675]: 6.9 scrub starts 2024-03-21T13:20:54.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[40675]: 6.9 scrub ok 2024-03-21T13:20:54.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[40675]: 5.1 scrub starts 2024-03-21T13:20:54.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:53 smithi163 ceph-mon[40675]: 5.1 scrub ok 2024-03-21T13:20:54.590 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-21T13:20:54.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[29621]: pgmap v1041: 132 pgs: 132 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 934 B/s rd, 0 op/s; 105 KiB/s, 3 objects/s recovering 2024-03-21T13:20:54.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[29621]: 3.1c scrub starts 2024-03-21T13:20:54.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[29621]: 3.1c scrub ok 2024-03-21T13:20:54.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/872785072' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:20:54.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:20:54.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:20:54.901 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:20:54.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[40675]: pgmap v1041: 132 pgs: 132 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 934 B/s rd, 0 op/s; 105 KiB/s, 3 objects/s recovering 2024-03-21T13:20:54.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[40675]: 3.1c scrub starts 2024-03-21T13:20:54.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[40675]: 3.1c scrub ok 2024-03-21T13:20:54.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/872785072' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:20:54.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:20:54.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:20:54.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:54 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:20:55.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:54 smithi204 ceph-mon[33153]: pgmap v1041: 132 pgs: 132 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 934 B/s rd, 0 op/s; 105 KiB/s, 3 objects/s recovering 2024-03-21T13:20:55.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:54 smithi204 ceph-mon[33153]: 3.1c scrub starts 2024-03-21T13:20:55.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:54 smithi204 ceph-mon[33153]: 3.1c scrub ok 2024-03-21T13:20:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:54 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/872785072' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:20:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:54 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T13:20:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:54 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:20:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:54 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:20:55.355 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 1 1 2024-03-21T13:20:55.602 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:55.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:55 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:20:55.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:55 smithi163 ceph-mon[29621]: osdmap e471: 8 total, 8 up, 8 in 2024-03-21T13:20:55.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:55 smithi163 ceph-mon[29621]: 6.1 scrub starts 2024-03-21T13:20:55.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:55 smithi163 ceph-mon[29621]: 6.1 scrub ok 2024-03-21T13:20:55.873 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:55 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:20:55.873 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:55 smithi163 ceph-mon[40675]: osdmap e471: 8 total, 8 up, 8 in 2024-03-21T13:20:55.873 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:55 smithi163 ceph-mon[40675]: 6.1 scrub starts 2024-03-21T13:20:55.873 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:55 smithi163 ceph-mon[40675]: 6.1 scrub ok 2024-03-21T13:20:55.874 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:20:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:20:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:20:56.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:55 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-21T13:20:56.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:55 smithi204 ceph-mon[33153]: osdmap e471: 8 total, 8 up, 8 in 2024-03-21T13:20:56.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:55 smithi204 ceph-mon[33153]: 6.1 scrub starts 2024-03-21T13:20:56.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:55 smithi204 ceph-mon[33153]: 6.1 scrub ok 2024-03-21T13:20:57.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:56 smithi204 ceph-mon[33153]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 129 KiB/s, 0 objects/s recovering 2024-03-21T13:20:57.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:56 smithi204 ceph-mon[33153]: 6.1c scrub starts 2024-03-21T13:20:57.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:56 smithi204 ceph-mon[33153]: 6.1c scrub ok 2024-03-21T13:20:57.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:56 smithi204 ceph-mon[33153]: 3.16 deep-scrub starts 2024-03-21T13:20:57.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:56 smithi204 ceph-mon[33153]: 3.16 deep-scrub ok 2024-03-21T13:20:57.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:56 smithi204 ceph-mon[33153]: 5.1e deep-scrub starts 2024-03-21T13:20:57.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:56 smithi204 ceph-mon[33153]: 5.1e deep-scrub ok 2024-03-21T13:20:57.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:56 smithi204 ceph-mon[33153]: osdmap e472: 8 total, 8 up, 8 in 2024-03-21T13:20:57.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[40675]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 129 KiB/s, 0 objects/s recovering 2024-03-21T13:20:57.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[40675]: 6.1c scrub starts 2024-03-21T13:20:57.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[40675]: 6.1c scrub ok 2024-03-21T13:20:57.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[40675]: 3.16 deep-scrub starts 2024-03-21T13:20:57.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[40675]: 3.16 deep-scrub ok 2024-03-21T13:20:57.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[40675]: 5.1e deep-scrub starts 2024-03-21T13:20:57.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[40675]: 5.1e deep-scrub ok 2024-03-21T13:20:57.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[40675]: osdmap e472: 8 total, 8 up, 8 in 2024-03-21T13:20:57.005 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[29621]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 129 KiB/s, 0 objects/s recovering 2024-03-21T13:20:57.005 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[29621]: 6.1c scrub starts 2024-03-21T13:20:57.005 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[29621]: 6.1c scrub ok 2024-03-21T13:20:57.005 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[29621]: 3.16 deep-scrub starts 2024-03-21T13:20:57.005 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[29621]: 3.16 deep-scrub ok 2024-03-21T13:20:57.005 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[29621]: 5.1e deep-scrub starts 2024-03-21T13:20:57.005 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[29621]: 5.1e deep-scrub ok 2024-03-21T13:20:57.005 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:56 smithi163 ceph-mon[29621]: osdmap e472: 8 total, 8 up, 8 in 2024-03-21T13:20:58.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:57 smithi204 ceph-mon[33153]: 5.13 scrub starts 2024-03-21T13:20:58.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:57 smithi204 ceph-mon[33153]: 5.13 scrub ok 2024-03-21T13:20:58.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:57 smithi204 ceph-mon[33153]: 6.d scrub starts 2024-03-21T13:20:58.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:57 smithi204 ceph-mon[33153]: 6.d scrub ok 2024-03-21T13:20:58.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:57 smithi163 ceph-mon[29621]: 5.13 scrub starts 2024-03-21T13:20:58.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:57 smithi163 ceph-mon[29621]: 5.13 scrub ok 2024-03-21T13:20:58.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:57 smithi163 ceph-mon[29621]: 6.d scrub starts 2024-03-21T13:20:58.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:57 smithi163 ceph-mon[29621]: 6.d scrub ok 2024-03-21T13:20:58.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:57 smithi163 ceph-mon[40675]: 5.13 scrub starts 2024-03-21T13:20:58.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:57 smithi163 ceph-mon[40675]: 5.13 scrub ok 2024-03-21T13:20:58.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:57 smithi163 ceph-mon[40675]: 6.d scrub starts 2024-03-21T13:20:58.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:57 smithi163 ceph-mon[40675]: 6.d scrub ok 2024-03-21T13:20:58.612 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.1 to 1 (10000) 2024-03-21T13:20:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:58 smithi204 ceph-mon[33153]: pgmap v1045: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-03-21T13:20:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:58 smithi204 ceph-mon[33153]: 6.11 scrub starts 2024-03-21T13:20:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:58 smithi204 ceph-mon[33153]: 5.16 scrub starts 2024-03-21T13:20:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:58 smithi204 ceph-mon[33153]: 5.16 scrub ok 2024-03-21T13:20:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:58 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1121783607' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T13:20:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:58 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T13:20:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:58 smithi204 ceph-mon[33153]: 4.1e scrub starts 2024-03-21T13:20:59.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:58 smithi204 ceph-mon[33153]: 4.1e scrub ok 2024-03-21T13:20:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[29621]: pgmap v1045: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-03-21T13:20:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[29621]: 6.11 scrub starts 2024-03-21T13:20:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[29621]: 5.16 scrub starts 2024-03-21T13:20:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[29621]: 5.16 scrub ok 2024-03-21T13:20:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1121783607' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T13:20:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T13:20:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[29621]: 4.1e scrub starts 2024-03-21T13:20:59.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[29621]: 4.1e scrub ok 2024-03-21T13:20:59.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[40675]: pgmap v1045: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-03-21T13:20:59.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[40675]: 6.11 scrub starts 2024-03-21T13:20:59.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[40675]: 5.16 scrub starts 2024-03-21T13:20:59.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[40675]: 5.16 scrub ok 2024-03-21T13:20:59.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1121783607' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T13:20:59.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T13:20:59.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[40675]: 4.1e scrub starts 2024-03-21T13:20:59.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:58 smithi163 ceph-mon[40675]: 4.1e scrub ok 2024-03-21T13:20:59.448 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-21T13:20:59.699 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:20:59.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:59 smithi163 ceph-mon[29621]: 6.11 scrub ok 2024-03-21T13:20:59.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:59 smithi163 ceph-mon[29621]: 4.11 deep-scrub starts 2024-03-21T13:20:59.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:59 smithi163 ceph-mon[29621]: 4.11 deep-scrub ok 2024-03-21T13:20:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:59 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-21T13:20:59.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:20:59 smithi163 ceph-mon[29621]: osdmap e473: 8 total, 8 up, 8 in 2024-03-21T13:20:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:59 smithi163 ceph-mon[40675]: 6.11 scrub ok 2024-03-21T13:20:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:59 smithi163 ceph-mon[40675]: 4.11 deep-scrub starts 2024-03-21T13:20:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:59 smithi163 ceph-mon[40675]: 4.11 deep-scrub ok 2024-03-21T13:20:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:59 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-21T13:20:59.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:20:59 smithi163 ceph-mon[40675]: osdmap e473: 8 total, 8 up, 8 in 2024-03-21T13:21:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:59 smithi204 ceph-mon[33153]: 6.11 scrub ok 2024-03-21T13:21:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:59 smithi204 ceph-mon[33153]: 4.11 deep-scrub starts 2024-03-21T13:21:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:59 smithi204 ceph-mon[33153]: 4.11 deep-scrub ok 2024-03-21T13:21:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:59 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-21T13:21:00.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:20:59 smithi204 ceph-mon[33153]: osdmap e473: 8 total, 8 up, 8 in 2024-03-21T13:21:00.989 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:00 smithi163 ceph-mon[29621]: pgmap v1047: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:00.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:00 smithi163 ceph-mon[40675]: pgmap v1047: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:00 smithi204 ceph-mon[33153]: pgmap v1047: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:01 smithi204 ceph-mon[33153]: 3.5 scrub starts 2024-03-21T13:21:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:01 smithi204 ceph-mon[33153]: 3.5 scrub ok 2024-03-21T13:21:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:01 smithi204 ceph-mon[33153]: 3.1f deep-scrub starts 2024-03-21T13:21:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:01 smithi204 ceph-mon[33153]: 3.1f deep-scrub ok 2024-03-21T13:21:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:01 smithi204 ceph-mon[33153]: 6.1d scrub starts 2024-03-21T13:21:02.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:01 smithi204 ceph-mon[33153]: 6.1d scrub ok 2024-03-21T13:21:02.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:01 smithi204 ceph-mon[33153]: osdmap e474: 8 total, 8 up, 8 in 2024-03-21T13:21:02.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[29621]: 3.5 scrub starts 2024-03-21T13:21:02.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[29621]: 3.5 scrub ok 2024-03-21T13:21:02.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[29621]: 3.1f deep-scrub starts 2024-03-21T13:21:02.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[29621]: 3.1f deep-scrub ok 2024-03-21T13:21:02.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[29621]: 6.1d scrub starts 2024-03-21T13:21:02.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[29621]: 6.1d scrub ok 2024-03-21T13:21:02.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[29621]: osdmap e474: 8 total, 8 up, 8 in 2024-03-21T13:21:02.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[40675]: 3.5 scrub starts 2024-03-21T13:21:02.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[40675]: 3.5 scrub ok 2024-03-21T13:21:02.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[40675]: 3.1f deep-scrub starts 2024-03-21T13:21:02.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[40675]: 3.1f deep-scrub ok 2024-03-21T13:21:02.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[40675]: 6.1d scrub starts 2024-03-21T13:21:02.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[40675]: 6.1d scrub ok 2024-03-21T13:21:02.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:01 smithi163 ceph-mon[40675]: osdmap e474: 8 total, 8 up, 8 in 2024-03-21T13:21:02.632 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-21T13:21:03.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:02 smithi204 ceph-mon[33153]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:21:03.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:02 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/633975518' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T13:21:03.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:02 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T13:21:03.003 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:21:02 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:21:03.007 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:02 smithi163 ceph-mon[29621]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:21:03.007 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:02 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/633975518' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T13:21:03.007 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:02 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T13:21:03.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:02 smithi163 ceph-mon[40675]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:21:03.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:02 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/633975518' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T13:21:03.007 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:02 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T13:21:03.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-21T13:21:03.636 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[29621]: 4.19 scrub starts 2024-03-21T13:21:03.636 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[29621]: 4.19 scrub ok 2024-03-21T13:21:03.636 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[29621]: 6.1b scrub starts 2024-03-21T13:21:03.636 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[29621]: 6.1b scrub ok 2024-03-21T13:21:03.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[40675]: 4.19 scrub starts 2024-03-21T13:21:03.641 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:03.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:03.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-21T13:21:03.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[29621]: osdmap e475: 8 total, 8 up, 8 in 2024-03-21T13:21:03.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[29621]: 6.1f scrub starts 2024-03-21T13:21:03.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[40675]: 4.19 scrub ok 2024-03-21T13:21:03.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[40675]: 6.1b scrub starts 2024-03-21T13:21:03.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[40675]: 6.1b scrub ok 2024-03-21T13:21:03.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:03.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-21T13:21:03.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[40675]: osdmap e475: 8 total, 8 up, 8 in 2024-03-21T13:21:03.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:03 smithi163 ceph-mon[40675]: 6.1f scrub starts 2024-03-21T13:21:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:03 smithi204 ceph-mon[33153]: 4.19 scrub starts 2024-03-21T13:21:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:03 smithi204 ceph-mon[33153]: 4.19 scrub ok 2024-03-21T13:21:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:03 smithi204 ceph-mon[33153]: 6.1b scrub starts 2024-03-21T13:21:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:03 smithi204 ceph-mon[33153]: 6.1b scrub ok 2024-03-21T13:21:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:03 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:03 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-21T13:21:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:03 smithi204 ceph-mon[33153]: osdmap e475: 8 total, 8 up, 8 in 2024-03-21T13:21:04.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:03 smithi204 ceph-mon[33153]: 6.1f scrub starts 2024-03-21T13:21:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:04 smithi204 ceph-mon[33153]: 6.1f scrub ok 2024-03-21T13:21:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:04 smithi204 ceph-mon[33153]: pgmap v1051: 132 pgs: 132 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:05.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:04 smithi163 ceph-mon[29621]: 6.1f scrub ok 2024-03-21T13:21:05.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:04 smithi163 ceph-mon[29621]: pgmap v1051: 132 pgs: 132 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:05.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:04 smithi163 ceph-mon[40675]: 6.1f scrub ok 2024-03-21T13:21:05.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:04 smithi163 ceph-mon[40675]: pgmap v1051: 132 pgs: 132 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:05.648 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:21:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:21:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:21:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:05 smithi204 ceph-mon[33153]: 5.0 scrub starts 2024-03-21T13:21:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:05 smithi204 ceph-mon[33153]: 5.0 scrub ok 2024-03-21T13:21:06.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:05 smithi204 ceph-mon[33153]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-03-21T13:21:06.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:05 smithi163 ceph-mon[29621]: 5.0 scrub starts 2024-03-21T13:21:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:05 smithi163 ceph-mon[29621]: 5.0 scrub ok 2024-03-21T13:21:06.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:05 smithi163 ceph-mon[29621]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-03-21T13:21:06.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:05 smithi163 ceph-mon[40675]: 5.0 scrub starts 2024-03-21T13:21:06.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:05 smithi163 ceph-mon[40675]: 5.0 scrub ok 2024-03-21T13:21:06.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:05 smithi163 ceph-mon[40675]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-03-21T13:21:06.552 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.2 to 1 (10000) 2024-03-21T13:21:06.946 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[29621]: 4.1a scrub starts 2024-03-21T13:21:06.946 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[29621]: 4.1a scrub ok 2024-03-21T13:21:06.946 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:06.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/936125964' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:21:06.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:21:06.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[29621]: 5.19 scrub starts 2024-03-21T13:21:06.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[29621]: 5.19 scrub ok 2024-03-21T13:21:06.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:21:06.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[29621]: osdmap e476: 8 total, 8 up, 8 in 2024-03-21T13:21:06.947 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[40675]: 4.1a scrub starts 2024-03-21T13:21:06.947 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[40675]: 4.1a scrub ok 2024-03-21T13:21:06.947 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:06.947 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/936125964' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:21:06.947 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:21:06.947 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[40675]: 5.19 scrub starts 2024-03-21T13:21:06.947 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[40675]: 5.19 scrub ok 2024-03-21T13:21:06.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:21:06.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:06 smithi163 ceph-mon[40675]: osdmap e476: 8 total, 8 up, 8 in 2024-03-21T13:21:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:06 smithi204 ceph-mon[33153]: 4.1a scrub starts 2024-03-21T13:21:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:06 smithi204 ceph-mon[33153]: 4.1a scrub ok 2024-03-21T13:21:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:06 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:06 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/936125964' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:21:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:06 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:21:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:06 smithi204 ceph-mon[33153]: 5.19 scrub starts 2024-03-21T13:21:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:06 smithi204 ceph-mon[33153]: 5.19 scrub ok 2024-03-21T13:21:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:06 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:21:07.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:06 smithi204 ceph-mon[33153]: osdmap e476: 8 total, 8 up, 8 in 2024-03-21T13:21:07.398 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-21T13:21:07.659 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:07.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:07 smithi163 ceph-mon[29621]: 4.13 scrub starts 2024-03-21T13:21:07.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:07 smithi163 ceph-mon[29621]: 4.13 scrub ok 2024-03-21T13:21:07.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:07 smithi163 ceph-mon[29621]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:07.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:07 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:21:07.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:07 smithi163 ceph-mon[29621]: 6.e scrub starts 2024-03-21T13:21:07.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:07 smithi163 ceph-mon[29621]: 6.e scrub ok 2024-03-21T13:21:07.966 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:07 smithi163 ceph-mon[40675]: 4.13 scrub starts 2024-03-21T13:21:07.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:07 smithi163 ceph-mon[40675]: 4.13 scrub ok 2024-03-21T13:21:07.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:07 smithi163 ceph-mon[40675]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:07.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:07 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:21:07.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:07 smithi163 ceph-mon[40675]: 6.e scrub starts 2024-03-21T13:21:07.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:07 smithi163 ceph-mon[40675]: 6.e scrub ok 2024-03-21T13:21:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:07 smithi204 ceph-mon[33153]: 4.13 scrub starts 2024-03-21T13:21:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:07 smithi204 ceph-mon[33153]: 4.13 scrub ok 2024-03-21T13:21:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:07 smithi204 ceph-mon[33153]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:07 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:21:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:07 smithi204 ceph-mon[33153]: 6.e scrub starts 2024-03-21T13:21:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:07 smithi204 ceph-mon[33153]: 6.e scrub ok 2024-03-21T13:21:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:08 smithi204 ceph-mon[33153]: 6.3 scrub starts 2024-03-21T13:21:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:08 smithi204 ceph-mon[33153]: 6.3 scrub ok 2024-03-21T13:21:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:21:09.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:21:09.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:21:09.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:08 smithi204 ceph-mon[33153]: 5.3 scrub starts 2024-03-21T13:21:09.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:08 smithi204 ceph-mon[33153]: 5.3 scrub ok 2024-03-21T13:21:09.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[29621]: 6.3 scrub starts 2024-03-21T13:21:09.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[29621]: 6.3 scrub ok 2024-03-21T13:21:09.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:21:09.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:21:09.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:21:09.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[29621]: 5.3 scrub starts 2024-03-21T13:21:09.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[29621]: 5.3 scrub ok 2024-03-21T13:21:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[40675]: 6.3 scrub starts 2024-03-21T13:21:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[40675]: 6.3 scrub ok 2024-03-21T13:21:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:21:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:21:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:21:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[40675]: 5.3 scrub starts 2024-03-21T13:21:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:08 smithi163 ceph-mon[40675]: 5.3 scrub ok 2024-03-21T13:21:09.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:09 smithi163 ceph-mon[40675]: 6.8 scrub starts 2024-03-21T13:21:09.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:09 smithi163 ceph-mon[40675]: 6.8 scrub ok 2024-03-21T13:21:09.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:09 smithi163 ceph-mon[40675]: 6.b scrub starts 2024-03-21T13:21:09.667 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-21T13:21:10.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:09 smithi204 ceph-mon[33153]: 6.8 scrub starts 2024-03-21T13:21:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:09 smithi204 ceph-mon[33153]: 6.8 scrub ok 2024-03-21T13:21:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:09 smithi204 ceph-mon[33153]: 6.b scrub starts 2024-03-21T13:21:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:09 smithi204 ceph-mon[33153]: pgmap v1055: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:21:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:09 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1306843701' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:21:10.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:09 smithi163 ceph-mon[40675]: pgmap v1055: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:21:10.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:09 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1306843701' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:21:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:09 smithi163 ceph-mon[29621]: 6.8 scrub starts 2024-03-21T13:21:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:09 smithi163 ceph-mon[29621]: 6.8 scrub ok 2024-03-21T13:21:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:09 smithi163 ceph-mon[29621]: 6.b scrub starts 2024-03-21T13:21:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:09 smithi163 ceph-mon[29621]: pgmap v1055: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T13:21:10.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:09 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1306843701' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T13:21:10.474 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-21T13:21:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:10.756 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[29621]: 6.b scrub ok 2024-03-21T13:21:10.756 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[29621]: 3.b deep-scrub starts 2024-03-21T13:21:10.756 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[29621]: 3.b deep-scrub ok 2024-03-21T13:21:10.757 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1306843701' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:21:10.757 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[29621]: osdmap e477: 8 total, 8 up, 8 in 2024-03-21T13:21:10.757 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[29621]: 3.3 scrub starts 2024-03-21T13:21:10.757 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[29621]: 3.3 scrub ok 2024-03-21T13:21:10.757 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[40675]: 6.b scrub ok 2024-03-21T13:21:10.757 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[40675]: 3.b deep-scrub starts 2024-03-21T13:21:10.757 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[40675]: 3.b deep-scrub ok 2024-03-21T13:21:10.757 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1306843701' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:21:10.757 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[40675]: osdmap e477: 8 total, 8 up, 8 in 2024-03-21T13:21:10.757 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[40675]: 3.3 scrub starts 2024-03-21T13:21:10.757 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:10 smithi163 ceph-mon[40675]: 3.3 scrub ok 2024-03-21T13:21:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:10 smithi204 ceph-mon[33153]: 6.b scrub ok 2024-03-21T13:21:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:10 smithi204 ceph-mon[33153]: 3.b deep-scrub starts 2024-03-21T13:21:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:10 smithi204 ceph-mon[33153]: 3.b deep-scrub ok 2024-03-21T13:21:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:10 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1306843701' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-21T13:21:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:10 smithi204 ceph-mon[33153]: osdmap e477: 8 total, 8 up, 8 in 2024-03-21T13:21:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:10 smithi204 ceph-mon[33153]: 3.3 scrub starts 2024-03-21T13:21:11.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:10 smithi204 ceph-mon[33153]: 3.3 scrub ok 2024-03-21T13:21:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:11 smithi204 ceph-mon[33153]: 3.12 scrub starts 2024-03-21T13:21:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:11 smithi204 ceph-mon[33153]: 3.12 scrub ok 2024-03-21T13:21:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:11 smithi204 ceph-mon[33153]: pgmap v1057: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:21:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:11 smithi204 ceph-mon[33153]: osdmap e478: 8 total, 8 up, 8 in 2024-03-21T13:21:12.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:11 smithi163 ceph-mon[29621]: 3.12 scrub starts 2024-03-21T13:21:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:11 smithi163 ceph-mon[29621]: 3.12 scrub ok 2024-03-21T13:21:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:11 smithi163 ceph-mon[29621]: pgmap v1057: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:21:12.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:11 smithi163 ceph-mon[29621]: osdmap e478: 8 total, 8 up, 8 in 2024-03-21T13:21:12.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:11 smithi163 ceph-mon[40675]: 3.12 scrub starts 2024-03-21T13:21:12.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:11 smithi163 ceph-mon[40675]: 3.12 scrub ok 2024-03-21T13:21:12.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:11 smithi163 ceph-mon[40675]: pgmap v1057: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:21:12.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:11 smithi163 ceph-mon[40675]: osdmap e478: 8 total, 8 up, 8 in 2024-03-21T13:21:12.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:12 smithi163 ceph-mon[40675]: 3.19 scrub starts 2024-03-21T13:21:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:12 smithi163 ceph-mon[40675]: 3.19 scrub ok 2024-03-21T13:21:12.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:12 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:12 smithi163 ceph-mon[29621]: 3.19 scrub starts 2024-03-21T13:21:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:12 smithi163 ceph-mon[29621]: 3.19 scrub ok 2024-03-21T13:21:12.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:12 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:12.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:21:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:21:12.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:12 smithi204 ceph-mon[33153]: 3.19 scrub starts 2024-03-21T13:21:12.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:12 smithi204 ceph-mon[33153]: 3.19 scrub ok 2024-03-21T13:21:12.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:12 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:13.691 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.3 to 1 (10000) 2024-03-21T13:21:14.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:13 smithi204 ceph-mon[33153]: 5.15 scrub starts 2024-03-21T13:21:14.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:13 smithi204 ceph-mon[33153]: 5.15 scrub ok 2024-03-21T13:21:14.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:13 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3094601876' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:21:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:13 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:21:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:13 smithi204 ceph-mon[33153]: pgmap v1059: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-03-21T13:21:14.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:13 smithi163 ceph-mon[29621]: 5.15 scrub starts 2024-03-21T13:21:14.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:13 smithi163 ceph-mon[29621]: 5.15 scrub ok 2024-03-21T13:21:14.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:13 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3094601876' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:21:14.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:13 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:21:14.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:13 smithi163 ceph-mon[29621]: pgmap v1059: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-03-21T13:21:14.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:13 smithi163 ceph-mon[40675]: 5.15 scrub starts 2024-03-21T13:21:14.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:13 smithi163 ceph-mon[40675]: 5.15 scrub ok 2024-03-21T13:21:14.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:13 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3094601876' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:21:14.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:13 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:21:14.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:13 smithi163 ceph-mon[40675]: pgmap v1059: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-03-21T13:21:14.513 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-21T13:21:14.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:14 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:21:14.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:14 smithi163 ceph-mon[29621]: osdmap e479: 8 total, 8 up, 8 in 2024-03-21T13:21:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:14 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:21:14.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:14 smithi163 ceph-mon[40675]: osdmap e479: 8 total, 8 up, 8 in 2024-03-21T13:21:14.764 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:14 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:21:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:14 smithi204 ceph-mon[33153]: osdmap e479: 8 total, 8 up, 8 in 2024-03-21T13:21:15.918 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:15 smithi163 ceph-mon[29621]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:15.918 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:15 smithi163 ceph-mon[29621]: 5.18 scrub starts 2024-03-21T13:21:15.918 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:15 smithi163 ceph-mon[29621]: 5.18 scrub ok 2024-03-21T13:21:15.919 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:15 smithi163 ceph-mon[40675]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:15.919 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:15 smithi163 ceph-mon[40675]: 5.18 scrub starts 2024-03-21T13:21:15.919 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:15 smithi163 ceph-mon[40675]: 5.18 scrub ok 2024-03-21T13:21:15.919 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:21:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:21:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:21:16.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:15 smithi204 ceph-mon[33153]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:16.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:15 smithi204 ceph-mon[33153]: 5.18 scrub starts 2024-03-21T13:21:16.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:15 smithi204 ceph-mon[33153]: 5.18 scrub ok 2024-03-21T13:21:17.551 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-21T13:21:17.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:17 smithi163 ceph-mon[29621]: osdmap e480: 8 total, 8 up, 8 in 2024-03-21T13:21:17.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:17 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3942092091' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:21:17.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:17 smithi163 ceph-mon[40675]: osdmap e480: 8 total, 8 up, 8 in 2024-03-21T13:21:17.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:17 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3942092091' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:21:18.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:17 smithi204 ceph-mon[33153]: osdmap e480: 8 total, 8 up, 8 in 2024-03-21T13:21:18.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:17 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3942092091' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T13:21:18.384 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-21T13:21:18.635 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:18.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[29621]: 6.15 deep-scrub starts 2024-03-21T13:21:18.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[29621]: 6.15 deep-scrub ok 2024-03-21T13:21:18.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:21:18.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:21:18.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[29621]: pgmap v1063: 132 pgs: 132 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:18.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[29621]: 6.1e scrub starts 2024-03-21T13:21:18.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3942092091' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:21:18.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[29621]: osdmap e481: 8 total, 8 up, 8 in 2024-03-21T13:21:18.758 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[29621]: 6.1e scrub ok 2024-03-21T13:21:18.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[40675]: 6.15 deep-scrub starts 2024-03-21T13:21:18.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[40675]: 6.15 deep-scrub ok 2024-03-21T13:21:18.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:21:18.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:21:18.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[40675]: pgmap v1063: 132 pgs: 132 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:18.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[40675]: 6.1e scrub starts 2024-03-21T13:21:18.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3942092091' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:21:18.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[40675]: osdmap e481: 8 total, 8 up, 8 in 2024-03-21T13:21:18.759 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:18 smithi163 ceph-mon[40675]: 6.1e scrub ok 2024-03-21T13:21:19.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:18 smithi204 ceph-mon[33153]: 6.15 deep-scrub starts 2024-03-21T13:21:19.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:18 smithi204 ceph-mon[33153]: 6.15 deep-scrub ok 2024-03-21T13:21:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:18 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:21:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:18 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:21:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:18 smithi204 ceph-mon[33153]: pgmap v1063: 132 pgs: 132 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:18 smithi204 ceph-mon[33153]: 6.1e scrub starts 2024-03-21T13:21:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:18 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3942092091' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-21T13:21:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:18 smithi204 ceph-mon[33153]: osdmap e481: 8 total, 8 up, 8 in 2024-03-21T13:21:19.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:18 smithi204 ceph-mon[33153]: 6.1e scrub ok 2024-03-21T13:21:19.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:19 smithi163 ceph-mon[29621]: 1.0 scrub starts 2024-03-21T13:21:19.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:19 smithi163 ceph-mon[29621]: 1.0 scrub ok 2024-03-21T13:21:19.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:19 smithi163 ceph-mon[29621]: 2.0 scrub starts 2024-03-21T13:21:19.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:19 smithi163 ceph-mon[29621]: 2.0 scrub ok 2024-03-21T13:21:19.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:19 smithi163 ceph-mon[40675]: 1.0 scrub starts 2024-03-21T13:21:19.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:19 smithi163 ceph-mon[40675]: 1.0 scrub ok 2024-03-21T13:21:19.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:19 smithi163 ceph-mon[40675]: 2.0 scrub starts 2024-03-21T13:21:19.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:19 smithi163 ceph-mon[40675]: 2.0 scrub ok 2024-03-21T13:21:20.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:19 smithi204 ceph-mon[33153]: 1.0 scrub starts 2024-03-21T13:21:20.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:19 smithi204 ceph-mon[33153]: 1.0 scrub ok 2024-03-21T13:21:20.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:19 smithi204 ceph-mon[33153]: 2.0 scrub starts 2024-03-21T13:21:20.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:19 smithi204 ceph-mon[33153]: 2.0 scrub ok 2024-03-21T13:21:21.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:20 smithi204 ceph-mon[33153]: pgmap v1065: 132 pgs: 132 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:20 smithi204 ceph-mon[33153]: 5.14 scrub starts 2024-03-21T13:21:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:20 smithi204 ceph-mon[33153]: 5.14 scrub ok 2024-03-21T13:21:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:20 smithi204 ceph-mon[33153]: 3.10 scrub starts 2024-03-21T13:21:21.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:20 smithi204 ceph-mon[33153]: 3.10 scrub ok 2024-03-21T13:21:21.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:20 smithi163 ceph-mon[29621]: pgmap v1065: 132 pgs: 132 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:20 smithi163 ceph-mon[29621]: 5.14 scrub starts 2024-03-21T13:21:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:20 smithi163 ceph-mon[29621]: 5.14 scrub ok 2024-03-21T13:21:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:20 smithi163 ceph-mon[29621]: 3.10 scrub starts 2024-03-21T13:21:21.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:20 smithi163 ceph-mon[29621]: 3.10 scrub ok 2024-03-21T13:21:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:20 smithi163 ceph-mon[40675]: pgmap v1065: 132 pgs: 132 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:20 smithi163 ceph-mon[40675]: 5.14 scrub starts 2024-03-21T13:21:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:20 smithi163 ceph-mon[40675]: 5.14 scrub ok 2024-03-21T13:21:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:20 smithi163 ceph-mon[40675]: 3.10 scrub starts 2024-03-21T13:21:21.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:20 smithi163 ceph-mon[40675]: 3.10 scrub ok 2024-03-21T13:21:21.550 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.4 to 1 (10000) 2024-03-21T13:21:21.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[40675]: 4.10 scrub starts 2024-03-21T13:21:21.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[40675]: 4.10 scrub ok 2024-03-21T13:21:21.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1231642333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:21:21.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:21.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-21T13:21:21.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-21T13:21:21.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[40675]: 3.a scrub starts 2024-03-21T13:21:21.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[40675]: 3.a scrub ok 2024-03-21T13:21:21.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1231642333' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:21:21.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-21T13:21:21.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-21T13:21:21.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[40675]: osdmap e482: 8 total, 8 up, 8 in 2024-03-21T13:21:21.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[29621]: 4.10 scrub starts 2024-03-21T13:21:21.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[29621]: 4.10 scrub ok 2024-03-21T13:21:21.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1231642333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:21:21.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:21.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-21T13:21:21.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-21T13:21:21.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[29621]: 3.a scrub starts 2024-03-21T13:21:21.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[29621]: 3.a scrub ok 2024-03-21T13:21:21.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1231642333' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:21:21.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-21T13:21:21.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-21T13:21:21.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:21 smithi163 ceph-mon[29621]: osdmap e482: 8 total, 8 up, 8 in 2024-03-21T13:21:22.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:21 smithi204 ceph-mon[33153]: 4.10 scrub starts 2024-03-21T13:21:22.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:21 smithi204 ceph-mon[33153]: 4.10 scrub ok 2024-03-21T13:21:22.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:21 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1231642333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:21:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-21T13:21:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-21T13:21:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:21 smithi204 ceph-mon[33153]: 3.a scrub starts 2024-03-21T13:21:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:21 smithi204 ceph-mon[33153]: 3.a scrub ok 2024-03-21T13:21:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:21 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1231642333' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:21:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-21T13:21:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-21T13:21:22.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:21 smithi204 ceph-mon[33153]: osdmap e482: 8 total, 8 up, 8 in 2024-03-21T13:21:22.393 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-21T13:21:22.651 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:22.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:22 smithi163 ceph-mon[29621]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:21:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:22 smithi163 ceph-mon[29621]: 5.a scrub starts 2024-03-21T13:21:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:22 smithi163 ceph-mon[29621]: 5.a scrub ok 2024-03-21T13:21:22.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:22 smithi163 ceph-mon[29621]: osdmap e483: 8 total, 8 up, 8 in 2024-03-21T13:21:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:22 smithi163 ceph-mon[40675]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:21:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:22 smithi163 ceph-mon[40675]: 5.a scrub starts 2024-03-21T13:21:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:22 smithi163 ceph-mon[40675]: 5.a scrub ok 2024-03-21T13:21:22.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:22 smithi163 ceph-mon[40675]: osdmap e483: 8 total, 8 up, 8 in 2024-03-21T13:21:23.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:21:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:21:23.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:22 smithi204 ceph-mon[33153]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:21:23.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:22 smithi204 ceph-mon[33153]: 5.a scrub starts 2024-03-21T13:21:23.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:22 smithi204 ceph-mon[33153]: 5.a scrub ok 2024-03-21T13:21:23.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:22 smithi204 ceph-mon[33153]: osdmap e483: 8 total, 8 up, 8 in 2024-03-21T13:21:23.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:23 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:21:23.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:23 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:21:23.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:23 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:23.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:23 smithi163 ceph-mon[40675]: 4.4 scrub starts 2024-03-21T13:21:23.917 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:23 smithi163 ceph-mon[40675]: 4.4 scrub ok 2024-03-21T13:21:23.917 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:23 smithi163 ceph-mon[40675]: osdmap e484: 8 total, 8 up, 8 in 2024-03-21T13:21:23.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:23 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:21:23.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:23 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:21:23.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:23 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:23.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:23 smithi163 ceph-mon[29621]: 4.4 scrub starts 2024-03-21T13:21:23.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:23 smithi163 ceph-mon[29621]: 4.4 scrub ok 2024-03-21T13:21:23.918 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:23 smithi163 ceph-mon[29621]: osdmap e484: 8 total, 8 up, 8 in 2024-03-21T13:21:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:23 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:21:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:23 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:21:24.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:23 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:23 smithi204 ceph-mon[33153]: 4.4 scrub starts 2024-03-21T13:21:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:23 smithi204 ceph-mon[33153]: 4.4 scrub ok 2024-03-21T13:21:24.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:23 smithi204 ceph-mon[33153]: osdmap e484: 8 total, 8 up, 8 in 2024-03-21T13:21:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:24 smithi204 ceph-mon[33153]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:25.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:24 smithi204 ceph-mon[33153]: 4.e scrub starts 2024-03-21T13:21:25.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:24 smithi204 ceph-mon[33153]: 4.e scrub ok 2024-03-21T13:21:25.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:24 smithi163 ceph-mon[29621]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:24 smithi163 ceph-mon[29621]: 4.e scrub starts 2024-03-21T13:21:25.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:24 smithi163 ceph-mon[29621]: 4.e scrub ok 2024-03-21T13:21:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:24 smithi163 ceph-mon[40675]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:24 smithi163 ceph-mon[40675]: 4.e scrub starts 2024-03-21T13:21:25.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:24 smithi163 ceph-mon[40675]: 4.e scrub ok 2024-03-21T13:21:25.599 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-21T13:21:25.969 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:25 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/715931237' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:21:25.969 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:25 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:21:25.969 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:25 smithi163 ceph-mon[40675]: 3.15 scrub starts 2024-03-21T13:21:25.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:25 smithi163 ceph-mon[40675]: 3.15 scrub ok 2024-03-21T13:21:25.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:25 smithi163 ceph-mon[40675]: 5.1d scrub starts 2024-03-21T13:21:25.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:25 smithi163 ceph-mon[40675]: 5.1d scrub ok 2024-03-21T13:21:25.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:25 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/715931237' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:21:25.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:25 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:21:25.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:25 smithi163 ceph-mon[29621]: 3.15 scrub starts 2024-03-21T13:21:25.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:25 smithi163 ceph-mon[29621]: 3.15 scrub ok 2024-03-21T13:21:25.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:25 smithi163 ceph-mon[29621]: 5.1d scrub starts 2024-03-21T13:21:25.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:25 smithi163 ceph-mon[29621]: 5.1d scrub ok 2024-03-21T13:21:25.971 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:21:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:21:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:21:26.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:25 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/715931237' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:21:26.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:25 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T13:21:26.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:25 smithi204 ceph-mon[33153]: 3.15 scrub starts 2024-03-21T13:21:26.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:25 smithi204 ceph-mon[33153]: 3.15 scrub ok 2024-03-21T13:21:26.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:25 smithi204 ceph-mon[33153]: 5.1d scrub starts 2024-03-21T13:21:26.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:25 smithi204 ceph-mon[33153]: 5.1d scrub ok 2024-03-21T13:21:26.380 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-21T13:21:26.641 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:26.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:26 smithi163 ceph-mon[29621]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:26 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:21:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:26 smithi163 ceph-mon[29621]: osdmap e485: 8 total, 8 up, 8 in 2024-03-21T13:21:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:26 smithi163 ceph-mon[29621]: 6.19 scrub starts 2024-03-21T13:21:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:26 smithi163 ceph-mon[29621]: 6.19 scrub ok 2024-03-21T13:21:26.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:26 smithi163 ceph-mon[29621]: osdmap e486: 8 total, 8 up, 8 in 2024-03-21T13:21:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:26 smithi163 ceph-mon[40675]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:26 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:21:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:26 smithi163 ceph-mon[40675]: osdmap e485: 8 total, 8 up, 8 in 2024-03-21T13:21:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:26 smithi163 ceph-mon[40675]: 6.19 scrub starts 2024-03-21T13:21:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:26 smithi163 ceph-mon[40675]: 6.19 scrub ok 2024-03-21T13:21:26.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:26 smithi163 ceph-mon[40675]: osdmap e486: 8 total, 8 up, 8 in 2024-03-21T13:21:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:26 smithi204 ceph-mon[33153]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:27.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:26 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-21T13:21:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:26 smithi204 ceph-mon[33153]: osdmap e485: 8 total, 8 up, 8 in 2024-03-21T13:21:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:26 smithi204 ceph-mon[33153]: 6.19 scrub starts 2024-03-21T13:21:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:26 smithi204 ceph-mon[33153]: 6.19 scrub ok 2024-03-21T13:21:27.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:26 smithi204 ceph-mon[33153]: osdmap e486: 8 total, 8 up, 8 in 2024-03-21T13:21:28.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:28 smithi163 ceph-mon[29621]: pgmap v1074: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 24 B/s, 2 objects/s recovering 2024-03-21T13:21:28.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:28 smithi163 ceph-mon[40675]: pgmap v1074: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 24 B/s, 2 objects/s recovering 2024-03-21T13:21:29.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:28 smithi204 ceph-mon[33153]: pgmap v1074: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 24 B/s, 2 objects/s recovering 2024-03-21T13:21:29.604 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.5 to 1 (10000) 2024-03-21T13:21:30.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:29 smithi204 ceph-mon[33153]: 4.1d scrub starts 2024-03-21T13:21:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:29 smithi204 ceph-mon[33153]: 4.1d scrub ok 2024-03-21T13:21:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:29 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4074449451' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:21:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:29 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:21:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:29 smithi204 ceph-mon[33153]: 3.c scrub starts 2024-03-21T13:21:30.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:29 smithi204 ceph-mon[33153]: 3.c scrub ok 2024-03-21T13:21:30.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:29 smithi163 ceph-mon[29621]: 4.1d scrub starts 2024-03-21T13:21:30.014 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:29 smithi163 ceph-mon[29621]: 4.1d scrub ok 2024-03-21T13:21:30.014 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:29 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4074449451' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:21:30.014 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:29 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:21:30.014 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:29 smithi163 ceph-mon[29621]: 3.c scrub starts 2024-03-21T13:21:30.014 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:29 smithi163 ceph-mon[29621]: 3.c scrub ok 2024-03-21T13:21:30.014 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:29 smithi163 ceph-mon[40675]: 4.1d scrub starts 2024-03-21T13:21:30.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:29 smithi163 ceph-mon[40675]: 4.1d scrub ok 2024-03-21T13:21:30.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:29 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4074449451' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:21:30.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:29 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:21:30.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:29 smithi163 ceph-mon[40675]: 3.c scrub starts 2024-03-21T13:21:30.015 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:29 smithi163 ceph-mon[40675]: 3.c scrub ok 2024-03-21T13:21:30.421 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-21T13:21:30.669 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[29621]: 5.9 scrub starts 2024-03-21T13:21:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[29621]: 5.9 scrub ok 2024-03-21T13:21:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[29621]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2024-03-21T13:21:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[29621]: 3.1e scrub starts 2024-03-21T13:21:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[29621]: 3.1e scrub ok 2024-03-21T13:21:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:21:30.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[29621]: osdmap e487: 8 total, 8 up, 8 in 2024-03-21T13:21:30.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[40675]: 5.9 scrub starts 2024-03-21T13:21:30.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[40675]: 5.9 scrub ok 2024-03-21T13:21:30.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[40675]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2024-03-21T13:21:30.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[40675]: 3.1e scrub starts 2024-03-21T13:21:30.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[40675]: 3.1e scrub ok 2024-03-21T13:21:30.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:21:30.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:30 smithi163 ceph-mon[40675]: osdmap e487: 8 total, 8 up, 8 in 2024-03-21T13:21:31.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:30 smithi204 ceph-mon[33153]: 5.9 scrub starts 2024-03-21T13:21:31.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:30 smithi204 ceph-mon[33153]: 5.9 scrub ok 2024-03-21T13:21:31.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:30 smithi204 ceph-mon[33153]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2024-03-21T13:21:31.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:30 smithi204 ceph-mon[33153]: 3.1e scrub starts 2024-03-21T13:21:31.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:30 smithi204 ceph-mon[33153]: 3.1e scrub ok 2024-03-21T13:21:31.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:30 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:21:31.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:30 smithi204 ceph-mon[33153]: osdmap e487: 8 total, 8 up, 8 in 2024-03-21T13:21:32.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:32 smithi163 ceph-mon[29621]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:21:32.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:32 smithi163 ceph-mon[29621]: osdmap e488: 8 total, 8 up, 8 in 2024-03-21T13:21:32.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:32 smithi163 ceph-mon[40675]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:21:32.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:32 smithi163 ceph-mon[40675]: osdmap e488: 8 total, 8 up, 8 in 2024-03-21T13:21:33.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:21:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:21:33.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:32 smithi204 ceph-mon[33153]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:21:33.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:32 smithi204 ceph-mon[33153]: osdmap e488: 8 total, 8 up, 8 in 2024-03-21T13:21:33.558 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-21T13:21:33.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:33 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:33.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:33 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1443514011' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:21:33.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:33 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:21:33.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:33 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:33.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:33 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1443514011' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:21:33.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:33 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:21:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:33 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:33 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1443514011' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:21:34.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:33 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T13:21:34.299 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-21T13:21:34.544 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:34.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[29621]: pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-03-21T13:21:34.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[29621]: 3.14 scrub starts 2024-03-21T13:21:34.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[29621]: 3.14 scrub ok 2024-03-21T13:21:34.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:21:34.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[29621]: osdmap e489: 8 total, 8 up, 8 in 2024-03-21T13:21:34.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[29621]: 4.15 scrub starts 2024-03-21T13:21:34.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[29621]: 4.15 scrub ok 2024-03-21T13:21:34.813 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[40675]: pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-03-21T13:21:34.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[40675]: 3.14 scrub starts 2024-03-21T13:21:34.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[40675]: 3.14 scrub ok 2024-03-21T13:21:34.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:21:34.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[40675]: osdmap e489: 8 total, 8 up, 8 in 2024-03-21T13:21:34.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[40675]: 4.15 scrub starts 2024-03-21T13:21:34.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:34 smithi163 ceph-mon[40675]: 4.15 scrub ok 2024-03-21T13:21:35.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:34 smithi204 ceph-mon[33153]: pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-03-21T13:21:35.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:34 smithi204 ceph-mon[33153]: 3.14 scrub starts 2024-03-21T13:21:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:34 smithi204 ceph-mon[33153]: 3.14 scrub ok 2024-03-21T13:21:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:34 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-21T13:21:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:34 smithi204 ceph-mon[33153]: osdmap e489: 8 total, 8 up, 8 in 2024-03-21T13:21:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:34 smithi204 ceph-mon[33153]: 4.15 scrub starts 2024-03-21T13:21:35.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:34 smithi204 ceph-mon[33153]: 4.15 scrub ok 2024-03-21T13:21:36.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:35 smithi204 ceph-mon[33153]: 6.6 deep-scrub starts 2024-03-21T13:21:36.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:35 smithi204 ceph-mon[33153]: 6.6 deep-scrub ok 2024-03-21T13:21:36.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:35 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:21:36.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:35 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:21:36.010 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:35 smithi163 ceph-mon[40675]: 6.6 deep-scrub starts 2024-03-21T13:21:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:35 smithi163 ceph-mon[40675]: 6.6 deep-scrub ok 2024-03-21T13:21:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:35 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:21:36.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:35 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:21:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:35 smithi163 ceph-mon[29621]: 6.6 deep-scrub starts 2024-03-21T13:21:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:35 smithi163 ceph-mon[29621]: 6.6 deep-scrub ok 2024-03-21T13:21:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:35 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:21:36.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:35 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:21:36.011 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:21:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:21:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:21:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:36 smithi163 ceph-mon[29621]: pgmap v1081: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-03-21T13:21:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:36 smithi163 ceph-mon[29621]: osdmap e490: 8 total, 8 up, 8 in 2024-03-21T13:21:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:36 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:36 smithi163 ceph-mon[29621]: 6.0 scrub starts 2024-03-21T13:21:36.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:36 smithi163 ceph-mon[29621]: 6.0 scrub ok 2024-03-21T13:21:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:36 smithi163 ceph-mon[40675]: pgmap v1081: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-03-21T13:21:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:36 smithi163 ceph-mon[40675]: osdmap e490: 8 total, 8 up, 8 in 2024-03-21T13:21:36.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:36 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:36.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:36 smithi163 ceph-mon[40675]: 6.0 scrub starts 2024-03-21T13:21:36.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:36 smithi163 ceph-mon[40675]: 6.0 scrub ok 2024-03-21T13:21:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:36 smithi204 ceph-mon[33153]: pgmap v1081: 132 pgs: 132 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-03-21T13:21:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:36 smithi204 ceph-mon[33153]: osdmap e490: 8 total, 8 up, 8 in 2024-03-21T13:21:37.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:36 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:37.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:36 smithi204 ceph-mon[33153]: 6.0 scrub starts 2024-03-21T13:21:37.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:36 smithi204 ceph-mon[33153]: 6.0 scrub ok 2024-03-21T13:21:37.579 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.6 to 1 (10000) 2024-03-21T13:21:37.876 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:37 smithi163 ceph-mon[29621]: 4.1b scrub starts 2024-03-21T13:21:37.877 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:37 smithi163 ceph-mon[29621]: 4.1b scrub ok 2024-03-21T13:21:37.877 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:37 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2261773502' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:21:37.877 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:37 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:21:37.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:37 smithi163 ceph-mon[40675]: 4.1b scrub starts 2024-03-21T13:21:37.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:37 smithi163 ceph-mon[40675]: 4.1b scrub ok 2024-03-21T13:21:37.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:37 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2261773502' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:21:37.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:37 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:21:38.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:37 smithi204 ceph-mon[33153]: 4.1b scrub starts 2024-03-21T13:21:38.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:37 smithi204 ceph-mon[33153]: 4.1b scrub ok 2024-03-21T13:21:38.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:37 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2261773502' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:21:38.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:37 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:21:38.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-21T13:21:38.434 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:38.814 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[29621]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:38.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[29621]: 5.b scrub starts 2024-03-21T13:21:38.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[29621]: 5.b scrub ok 2024-03-21T13:21:38.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:21:38.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:21:38.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:21:38.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[29621]: osdmap e491: 8 total, 8 up, 8 in 2024-03-21T13:21:38.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[40675]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:38.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[40675]: 5.b scrub starts 2024-03-21T13:21:38.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[40675]: 5.b scrub ok 2024-03-21T13:21:38.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:21:38.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:21:38.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:21:38.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:38 smithi163 ceph-mon[40675]: osdmap e491: 8 total, 8 up, 8 in 2024-03-21T13:21:39.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:38 smithi204 ceph-mon[33153]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:38 smithi204 ceph-mon[33153]: 5.b scrub starts 2024-03-21T13:21:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:38 smithi204 ceph-mon[33153]: 5.b scrub ok 2024-03-21T13:21:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:38 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:21:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:38 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:21:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:38 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:21:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:38 smithi204 ceph-mon[33153]: osdmap e491: 8 total, 8 up, 8 in 2024-03-21T13:21:39.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:39 smithi163 ceph-mon[29621]: 2.1 scrub starts 2024-03-21T13:21:39.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:39 smithi163 ceph-mon[29621]: 2.1 scrub ok 2024-03-21T13:21:39.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:39 smithi163 ceph-mon[29621]: 5.1f deep-scrub starts 2024-03-21T13:21:39.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:39 smithi163 ceph-mon[29621]: 5.1f deep-scrub ok 2024-03-21T13:21:39.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:39 smithi163 ceph-mon[29621]: 3.0 scrub starts 2024-03-21T13:21:39.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:39 smithi163 ceph-mon[29621]: 3.0 scrub ok 2024-03-21T13:21:39.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:39 smithi163 ceph-mon[40675]: 2.1 scrub starts 2024-03-21T13:21:39.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:39 smithi163 ceph-mon[40675]: 2.1 scrub ok 2024-03-21T13:21:39.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:39 smithi163 ceph-mon[40675]: 5.1f deep-scrub starts 2024-03-21T13:21:39.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:39 smithi163 ceph-mon[40675]: 5.1f deep-scrub ok 2024-03-21T13:21:39.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:39 smithi163 ceph-mon[40675]: 3.0 scrub starts 2024-03-21T13:21:39.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:39 smithi163 ceph-mon[40675]: 3.0 scrub ok 2024-03-21T13:21:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:39 smithi204 ceph-mon[33153]: 2.1 scrub starts 2024-03-21T13:21:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:39 smithi204 ceph-mon[33153]: 2.1 scrub ok 2024-03-21T13:21:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:39 smithi204 ceph-mon[33153]: 5.1f deep-scrub starts 2024-03-21T13:21:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:39 smithi204 ceph-mon[33153]: 5.1f deep-scrub ok 2024-03-21T13:21:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:39 smithi204 ceph-mon[33153]: 3.0 scrub starts 2024-03-21T13:21:40.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:39 smithi204 ceph-mon[33153]: 3.0 scrub ok 2024-03-21T13:21:40.584 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-21T13:21:40.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:40 smithi163 ceph-mon[29621]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:40.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:40 smithi163 ceph-mon[29621]: 4.1 scrub starts 2024-03-21T13:21:40.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:40 smithi163 ceph-mon[29621]: 4.1 scrub ok 2024-03-21T13:21:40.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:40 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/410198516' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:21:40.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:40 smithi163 ceph-mon[40675]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:40.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:40 smithi163 ceph-mon[40675]: 4.1 scrub starts 2024-03-21T13:21:40.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:40 smithi163 ceph-mon[40675]: 4.1 scrub ok 2024-03-21T13:21:40.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:40 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/410198516' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:21:41.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:40 smithi204 ceph-mon[33153]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:21:41.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:40 smithi204 ceph-mon[33153]: 4.1 scrub starts 2024-03-21T13:21:41.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:40 smithi204 ceph-mon[33153]: 4.1 scrub ok 2024-03-21T13:21:41.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:40 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/410198516' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T13:21:41.273 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-21T13:21:41.525 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:41.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:41 smithi163 ceph-mon[29621]: 3.18 deep-scrub starts 2024-03-21T13:21:41.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:41 smithi163 ceph-mon[29621]: 3.18 deep-scrub ok 2024-03-21T13:21:41.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:41 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/410198516' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:21:41.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:41 smithi163 ceph-mon[29621]: osdmap e492: 8 total, 8 up, 8 in 2024-03-21T13:21:41.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:41 smithi163 ceph-mon[29621]: 4.5 scrub starts 2024-03-21T13:21:41.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:41 smithi163 ceph-mon[29621]: 4.5 scrub ok 2024-03-21T13:21:41.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:41 smithi163 ceph-mon[40675]: 3.18 deep-scrub starts 2024-03-21T13:21:41.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:41 smithi163 ceph-mon[40675]: 3.18 deep-scrub ok 2024-03-21T13:21:41.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:41 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/410198516' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:21:41.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:41 smithi163 ceph-mon[40675]: osdmap e492: 8 total, 8 up, 8 in 2024-03-21T13:21:41.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:41 smithi163 ceph-mon[40675]: 4.5 scrub starts 2024-03-21T13:21:41.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:41 smithi163 ceph-mon[40675]: 4.5 scrub ok 2024-03-21T13:21:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:41 smithi204 ceph-mon[33153]: 3.18 deep-scrub starts 2024-03-21T13:21:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:41 smithi204 ceph-mon[33153]: 3.18 deep-scrub ok 2024-03-21T13:21:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:41 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/410198516' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-21T13:21:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:41 smithi204 ceph-mon[33153]: osdmap e492: 8 total, 8 up, 8 in 2024-03-21T13:21:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:41 smithi204 ceph-mon[33153]: 4.5 scrub starts 2024-03-21T13:21:42.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:41 smithi204 ceph-mon[33153]: 4.5 scrub ok 2024-03-21T13:21:42.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[29621]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:21:42.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[29621]: 5.d scrub starts 2024-03-21T13:21:42.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[29621]: 5.d scrub ok 2024-03-21T13:21:42.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[29621]: 4.16 scrub starts 2024-03-21T13:21:42.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[29621]: 4.16 scrub ok 2024-03-21T13:21:42.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[29621]: 4.9 scrub starts 2024-03-21T13:21:42.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[29621]: 4.9 scrub ok 2024-03-21T13:21:42.696 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[40675]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:21:42.696 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[40675]: 5.d scrub starts 2024-03-21T13:21:42.696 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[40675]: 5.d scrub ok 2024-03-21T13:21:42.696 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[40675]: 4.16 scrub starts 2024-03-21T13:21:42.696 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[40675]: 4.16 scrub ok 2024-03-21T13:21:42.696 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[40675]: 4.9 scrub starts 2024-03-21T13:21:42.696 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:42 smithi163 ceph-mon[40675]: 4.9 scrub ok 2024-03-21T13:21:42.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:21:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:21:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:42 smithi204 ceph-mon[33153]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:21:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:42 smithi204 ceph-mon[33153]: 5.d scrub starts 2024-03-21T13:21:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:42 smithi204 ceph-mon[33153]: 5.d scrub ok 2024-03-21T13:21:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:42 smithi204 ceph-mon[33153]: 4.16 scrub starts 2024-03-21T13:21:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:42 smithi204 ceph-mon[33153]: 4.16 scrub ok 2024-03-21T13:21:42.905 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:42 smithi204 ceph-mon[33153]: 4.9 scrub starts 2024-03-21T13:21:42.905 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:42 smithi204 ceph-mon[33153]: 4.9 scrub ok 2024-03-21T13:21:43.599 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.7 to 1 (10000) 2024-03-21T13:21:43.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[29621]: 6.14 scrub starts 2024-03-21T13:21:43.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[29621]: 6.14 scrub ok 2024-03-21T13:21:43.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[29621]: 4.0 scrub starts 2024-03-21T13:21:43.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[29621]: 4.0 scrub ok 2024-03-21T13:21:43.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:43.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[29621]: 6.16 scrub starts 2024-03-21T13:21:43.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[29621]: 6.16 scrub ok 2024-03-21T13:21:43.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[29621]: 5.12 scrub starts 2024-03-21T13:21:43.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[29621]: 5.12 scrub ok 2024-03-21T13:21:43.930 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3244064089' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:21:43.930 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[40675]: 6.14 scrub starts 2024-03-21T13:21:43.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[40675]: 6.14 scrub ok 2024-03-21T13:21:43.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[40675]: 4.0 scrub starts 2024-03-21T13:21:43.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[40675]: 4.0 scrub ok 2024-03-21T13:21:43.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:43.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[40675]: 6.16 scrub starts 2024-03-21T13:21:43.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[40675]: 6.16 scrub ok 2024-03-21T13:21:43.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[40675]: 5.12 scrub starts 2024-03-21T13:21:43.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[40675]: 5.12 scrub ok 2024-03-21T13:21:43.931 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:43 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3244064089' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:21:44.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:43 smithi204 ceph-mon[33153]: 6.14 scrub starts 2024-03-21T13:21:44.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:43 smithi204 ceph-mon[33153]: 6.14 scrub ok 2024-03-21T13:21:44.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:43 smithi204 ceph-mon[33153]: 4.0 scrub starts 2024-03-21T13:21:44.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:43 smithi204 ceph-mon[33153]: 4.0 scrub ok 2024-03-21T13:21:44.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:44.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:43 smithi204 ceph-mon[33153]: 6.16 scrub starts 2024-03-21T13:21:44.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:43 smithi204 ceph-mon[33153]: 6.16 scrub ok 2024-03-21T13:21:44.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:43 smithi204 ceph-mon[33153]: 5.12 scrub starts 2024-03-21T13:21:44.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:43 smithi204 ceph-mon[33153]: 5.12 scrub ok 2024-03-21T13:21:44.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:43 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3244064089' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:21:44.367 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-21T13:21:44.597 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[29621]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:21:44.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[29621]: 6.f scrub starts 2024-03-21T13:21:44.598 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[40675]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:21:44.598 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[40675]: 6.f scrub starts 2024-03-21T13:21:44.598 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[40675]: 6.f scrub ok 2024-03-21T13:21:44.615 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:44.878 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3244064089' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:21:44.878 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[40675]: osdmap e493: 8 total, 8 up, 8 in 2024-03-21T13:21:44.878 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[40675]: 5.2 scrub starts 2024-03-21T13:21:44.878 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[40675]: 5.2 scrub ok 2024-03-21T13:21:44.878 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[29621]: 6.f scrub ok 2024-03-21T13:21:44.878 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3244064089' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:21:44.878 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[29621]: osdmap e493: 8 total, 8 up, 8 in 2024-03-21T13:21:44.878 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[29621]: 5.2 scrub starts 2024-03-21T13:21:44.878 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:44 smithi163 ceph-mon[29621]: 5.2 scrub ok 2024-03-21T13:21:45.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:44 smithi204 ceph-mon[33153]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:21:45.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:44 smithi204 ceph-mon[33153]: 6.f scrub starts 2024-03-21T13:21:45.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:44 smithi204 ceph-mon[33153]: 6.f scrub ok 2024-03-21T13:21:45.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:44 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3244064089' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:21:45.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:44 smithi204 ceph-mon[33153]: osdmap e493: 8 total, 8 up, 8 in 2024-03-21T13:21:45.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:44 smithi204 ceph-mon[33153]: 5.2 scrub starts 2024-03-21T13:21:45.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:44 smithi204 ceph-mon[33153]: 5.2 scrub ok 2024-03-21T13:21:45.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:45 smithi163 ceph-mon[29621]: 5.10 deep-scrub starts 2024-03-21T13:21:45.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:45 smithi163 ceph-mon[29621]: 5.10 deep-scrub ok 2024-03-21T13:21:45.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:45 smithi163 ceph-mon[29621]: 6.18 scrub starts 2024-03-21T13:21:45.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:45 smithi163 ceph-mon[29621]: 6.18 scrub ok 2024-03-21T13:21:45.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:45 smithi163 ceph-mon[29621]: 3.4 scrub starts 2024-03-21T13:21:45.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:45 smithi163 ceph-mon[29621]: 3.4 scrub ok 2024-03-21T13:21:45.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:21:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:21:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:21:45.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:45 smithi163 ceph-mon[40675]: 5.10 deep-scrub starts 2024-03-21T13:21:45.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:45 smithi163 ceph-mon[40675]: 5.10 deep-scrub ok 2024-03-21T13:21:45.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:45 smithi163 ceph-mon[40675]: 6.18 scrub starts 2024-03-21T13:21:45.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:45 smithi163 ceph-mon[40675]: 6.18 scrub ok 2024-03-21T13:21:45.979 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:45 smithi163 ceph-mon[40675]: 3.4 scrub starts 2024-03-21T13:21:45.979 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:45 smithi163 ceph-mon[40675]: 3.4 scrub ok 2024-03-21T13:21:46.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:45 smithi204 ceph-mon[33153]: 5.10 deep-scrub starts 2024-03-21T13:21:46.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:45 smithi204 ceph-mon[33153]: 5.10 deep-scrub ok 2024-03-21T13:21:46.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:45 smithi204 ceph-mon[33153]: 6.18 scrub starts 2024-03-21T13:21:46.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:45 smithi204 ceph-mon[33153]: 6.18 scrub ok 2024-03-21T13:21:46.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:45 smithi204 ceph-mon[33153]: 3.4 scrub starts 2024-03-21T13:21:46.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:45 smithi204 ceph-mon[33153]: 3.4 scrub ok 2024-03-21T13:21:47.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:46 smithi204 ceph-mon[33153]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-03-21T13:21:47.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:46 smithi204 ceph-mon[33153]: 3.8 scrub starts 2024-03-21T13:21:47.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:46 smithi204 ceph-mon[33153]: 3.8 scrub ok 2024-03-21T13:21:47.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:46 smithi204 ceph-mon[33153]: 6.10 scrub starts 2024-03-21T13:21:47.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:46 smithi204 ceph-mon[33153]: 6.10 scrub ok 2024-03-21T13:21:47.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:46 smithi204 ceph-mon[33153]: 4.f scrub starts 2024-03-21T13:21:47.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:46 smithi204 ceph-mon[33153]: 4.f scrub ok 2024-03-21T13:21:47.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[29621]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-03-21T13:21:47.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[29621]: 3.8 scrub starts 2024-03-21T13:21:47.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[29621]: 3.8 scrub ok 2024-03-21T13:21:47.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[29621]: 6.10 scrub starts 2024-03-21T13:21:47.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[29621]: 6.10 scrub ok 2024-03-21T13:21:47.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[29621]: 4.f scrub starts 2024-03-21T13:21:47.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[29621]: 4.f scrub ok 2024-03-21T13:21:47.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[40675]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-03-21T13:21:47.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[40675]: 3.8 scrub starts 2024-03-21T13:21:47.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[40675]: 3.8 scrub ok 2024-03-21T13:21:47.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[40675]: 6.10 scrub starts 2024-03-21T13:21:47.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[40675]: 6.10 scrub ok 2024-03-21T13:21:47.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[40675]: 4.f scrub starts 2024-03-21T13:21:47.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:46 smithi163 ceph-mon[40675]: 4.f scrub ok 2024-03-21T13:21:47.631 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-21T13:21:48.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:47 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2449286696' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:21:48.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:47 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:21:48.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:47 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2449286696' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:21:48.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:47 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:21:48.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:47 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2449286696' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:21:48.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:47 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T13:21:48.409 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-21T13:21:48.409 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-21T13:21:48.409 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-21T13:21:48.409 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-21T13:21:48.409 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-21T13:21:48.410 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T13:21:48.663 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:48.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[29621]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:21:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:21:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:21:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:21:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[29621]: osdmap e494: 8 total, 8 up, 8 in 2024-03-21T13:21:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[29621]: 4.8 scrub starts 2024-03-21T13:21:48.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[29621]: 4.8 scrub ok 2024-03-21T13:21:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[40675]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:21:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:21:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:21:48.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:21:48.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[40675]: osdmap e494: 8 total, 8 up, 8 in 2024-03-21T13:21:48.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[40675]: 4.8 scrub starts 2024-03-21T13:21:48.762 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:48 smithi163 ceph-mon[40675]: 4.8 scrub ok 2024-03-21T13:21:49.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:48 smithi204 ceph-mon[33153]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:21:49.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:48 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:21:49.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:48 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:21:49.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:48 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-21T13:21:49.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:48 smithi204 ceph-mon[33153]: osdmap e494: 8 total, 8 up, 8 in 2024-03-21T13:21:49.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:48 smithi204 ceph-mon[33153]: 4.8 scrub starts 2024-03-21T13:21:49.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:48 smithi204 ceph-mon[33153]: 4.8 scrub ok 2024-03-21T13:21:50.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:49 smithi204 ceph-mon[33153]: 5.8 scrub starts 2024-03-21T13:21:50.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:49 smithi204 ceph-mon[33153]: 5.8 scrub ok 2024-03-21T13:21:50.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:49 smithi204 ceph-mon[33153]: 3.e scrub starts 2024-03-21T13:21:50.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:49 smithi204 ceph-mon[33153]: 3.e scrub ok 2024-03-21T13:21:50.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:49 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:21:50.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:49 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:21:50.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:49 smithi204 ceph-mon[33153]: 5.7 scrub starts 2024-03-21T13:21:50.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:49 smithi204 ceph-mon[33153]: 5.7 scrub ok 2024-03-21T13:21:50.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:49 smithi204 ceph-mon[33153]: 3.1a deep-scrub starts 2024-03-21T13:21:50.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:49 smithi204 ceph-mon[33153]: 3.1a deep-scrub ok 2024-03-21T13:21:50.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[29621]: 5.8 scrub starts 2024-03-21T13:21:50.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[29621]: 5.8 scrub ok 2024-03-21T13:21:50.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[29621]: 3.e scrub starts 2024-03-21T13:21:50.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[29621]: 3.e scrub ok 2024-03-21T13:21:50.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:21:50.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:21:50.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[29621]: 5.7 scrub starts 2024-03-21T13:21:50.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[29621]: 5.7 scrub ok 2024-03-21T13:21:50.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[29621]: 3.1a deep-scrub starts 2024-03-21T13:21:50.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[29621]: 3.1a deep-scrub ok 2024-03-21T13:21:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[40675]: 5.8 scrub starts 2024-03-21T13:21:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[40675]: 5.8 scrub ok 2024-03-21T13:21:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[40675]: 3.e scrub starts 2024-03-21T13:21:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[40675]: 3.e scrub ok 2024-03-21T13:21:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:21:50.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:21:50.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[40675]: 5.7 scrub starts 2024-03-21T13:21:50.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[40675]: 5.7 scrub ok 2024-03-21T13:21:50.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[40675]: 3.1a deep-scrub starts 2024-03-21T13:21:50.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:49 smithi163 ceph-mon[40675]: 3.1a deep-scrub ok 2024-03-21T13:21:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:21:50.808 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":495,"fsid":"d4bf79c0-e781-11ee-95cd-87774f69a715","created":"2024-03-21T12:52:56.031419+0000","modified":"2024-03-21T13:21:49.620360+0000","last_up_change":"2024-03-21T12:58:23.775302+0000","last_in_change":"2024-03-21T13:20:44.111930+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T12:56:20.557490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-21T12:58:35.713383+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":67,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-21T12:58:36.436384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-21T12:58:38.049304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-21T12:58:39.990512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-21T12:58:41.463686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c34a3b25-9e02-46e6-8f9a-d0af2615258f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":471,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6801","nonce":3086410981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3086410981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3086410981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6803","nonce":3086410981}]},"public_addr":"172.21.15.163:6801/3086410981","cluster_addr":"172.21.15.163:6802/3086410981","heartbeat_back_addr":"172.21.15.163:6804/3086410981","heartbeat_front_addr":"172.21.15.163:6803/3086410981","state":["exists","up"]},{"osd":1,"uuid":"36e05f89-c238-41f7-be65-1553b529e8ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6805","nonce":3666062784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3666062784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3666062784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6807","nonce":3666062784}]},"public_addr":"172.21.15.163:6805/3666062784","cluster_addr":"172.21.15.163:6806/3666062784","heartbeat_back_addr":"172.21.15.163:6808/3666062784","heartbeat_front_addr":"172.21.15.163:6807/3666062784","state":["exists","up"]},{"osd":2,"uuid":"968ec78d-25b8-4905-b328-d490c38ca85f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6809","nonce":4009213507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4009213507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4009213507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":4009213507}]},"public_addr":"172.21.15.163:6809/4009213507","cluster_addr":"172.21.15.163:6810/4009213507","heartbeat_back_addr":"172.21.15.163:6812/4009213507","heartbeat_front_addr":"172.21.15.163:6811/4009213507","state":["exists","up"]},{"osd":3,"uuid":"67dfa4f6-1709-48f6-971d-c6287233492d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":482,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6813","nonce":3981878189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3981878189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":3981878189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":3981878189}]},"public_addr":"172.21.15.163:6813/3981878189","cluster_addr":"172.21.15.163:6814/3981878189","heartbeat_back_addr":"172.21.15.163:6816/3981878189","heartbeat_front_addr":"172.21.15.163:6815/3981878189","state":["exists","up"]},{"osd":4,"uuid":"7fd311a9-f3c2-456d-8ab1-341cf5b33471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":485,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":719276054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":719276054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":719276054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":719276054}]},"public_addr":"172.21.15.204:6800/719276054","cluster_addr":"172.21.15.204:6801/719276054","heartbeat_back_addr":"172.21.15.204:6803/719276054","heartbeat_front_addr":"172.21.15.204:6802/719276054","state":["exists","up"]},{"osd":5,"uuid":"a87759c8-251c-415d-9387-a49f7d2f36d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1346947668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":1346947668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":1346947668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1346947668}]},"public_addr":"172.21.15.204:6804/1346947668","cluster_addr":"172.21.15.204:6805/1346947668","heartbeat_back_addr":"172.21.15.204:6807/1346947668","heartbeat_front_addr":"172.21.15.204:6806/1346947668","state":["exists","up"]},{"osd":6,"uuid":"5eb135cd-f677-4d0c-b23a-738f31615b6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":482,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":200077988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":200077988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":200077988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":200077988}]},"public_addr":"172.21.15.204:6808/200077988","cluster_addr":"172.21.15.204:6809/200077988","heartbeat_back_addr":"172.21.15.204:6811/200077988","heartbeat_front_addr":"172.21.15.204:6810/200077988","state":["exists","up"]},{"osd":7,"uuid":"4f528e10-2c93-4bc2-a54c-06ca4132ed49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":466,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":315664888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":315664888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":315664888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":315664888}]},"public_addr":"172.21.15.204:6812/315664888","cluster_addr":"172.21.15.204:6813/315664888","heartbeat_back_addr":"172.21.15.204:6815/315664888","heartbeat_front_addr":"172.21.15.204:6814/315664888","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:21:48.757373+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:20:51.586809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:21:47.524106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:21:37.401219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:21:35.208437+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:21:22.559865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:20:54.172013+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:21:16.926060+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.13","osds":[0,5,6]},{"pgid":"6.11","osds":[7,3,4]}],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.19","mappings":[{"from":4,"to":7}]},{"pgid":"4.9","mappings":[{"from":3,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.19","mappings":[{"from":3,"to":7}]},{"pgid":"6.6","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:0/3453855427":"2024-03-22T12:59:18.599967+0000","172.21.15.163:0/2571244897":"2024-03-22T12:59:18.599967+0000","172.21.15.163:6800/3073378854":"2024-03-22T12:59:18.599967+0000","172.21.15.163:0/1334591766":"2024-03-22T12:59:18.599967+0000","172.21.15.163:0/116934309":"2024-03-22T12:54:00.224135+0000","172.21.15.163:0/3890775846":"2024-03-22T12:59:18.599967+0000","172.21.15.163:0/1783921524":"2024-03-22T12:54:00.224135+0000","172.21.15.163:6800/2553440211":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/1812694077":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/1070540283":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/2354157786":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/1591817392":"2024-03-22T12:59:18.599967+0000","172.21.15.163:0/4152198951":"2024-03-22T12:54:00.224135+0000","172.21.15.163:6800/1798115633":"2024-03-22T12:54:00.224135+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T13:21:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:50 smithi204 ceph-mon[33153]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:21:51.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:50 smithi204 ceph-mon[33153]: 5.6 scrub starts 2024-03-21T13:21:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:50 smithi204 ceph-mon[33153]: 5.6 scrub ok 2024-03-21T13:21:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:50 smithi204 ceph-mon[33153]: osdmap e495: 8 total, 8 up, 8 in 2024-03-21T13:21:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:50 smithi204 ceph-mon[33153]: 3.9 scrub starts 2024-03-21T13:21:51.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:50 smithi204 ceph-mon[33153]: 3.9 scrub ok 2024-03-21T13:21:51.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:50 smithi163 ceph-mon[29621]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:21:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:50 smithi163 ceph-mon[29621]: 5.6 scrub starts 2024-03-21T13:21:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:50 smithi163 ceph-mon[29621]: 5.6 scrub ok 2024-03-21T13:21:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:50 smithi163 ceph-mon[29621]: osdmap e495: 8 total, 8 up, 8 in 2024-03-21T13:21:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:50 smithi163 ceph-mon[29621]: 3.9 scrub starts 2024-03-21T13:21:51.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:50 smithi163 ceph-mon[29621]: 3.9 scrub ok 2024-03-21T13:21:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:50 smithi163 ceph-mon[40675]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:21:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:50 smithi163 ceph-mon[40675]: 5.6 scrub starts 2024-03-21T13:21:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:50 smithi163 ceph-mon[40675]: 5.6 scrub ok 2024-03-21T13:21:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:50 smithi163 ceph-mon[40675]: osdmap e495: 8 total, 8 up, 8 in 2024-03-21T13:21:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:50 smithi163 ceph-mon[40675]: 3.9 scrub starts 2024-03-21T13:21:51.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:50 smithi163 ceph-mon[40675]: 3.9 scrub ok 2024-03-21T13:21:51.629 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-21T13:21:51.630 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T13:21:51.871 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:51.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:51 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:51.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:51 smithi163 ceph-mon[29621]: 3.17 scrub starts 2024-03-21T13:21:51.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:51 smithi163 ceph-mon[29621]: 3.17 scrub ok 2024-03-21T13:21:51.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:51 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/297384383' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:21:51.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:51 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:51.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:51 smithi163 ceph-mon[40675]: 3.17 scrub starts 2024-03-21T13:21:51.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:51 smithi163 ceph-mon[40675]: 3.17 scrub ok 2024-03-21T13:21:51.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:51 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/297384383' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:21:52.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:51 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:52.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:51 smithi204 ceph-mon[33153]: 3.17 scrub starts 2024-03-21T13:21:52.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:51 smithi204 ceph-mon[33153]: 3.17 scrub ok 2024-03-21T13:21:52.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:51 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/297384383' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:21:53.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:52 smithi204 ceph-mon[33153]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:21:53.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:52 smithi204 ceph-mon[33153]: 6.5 scrub starts 2024-03-21T13:21:53.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:52 smithi204 ceph-mon[33153]: 6.5 scrub ok 2024-03-21T13:21:53.003 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:21:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:21:53.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:52 smithi163 ceph-mon[29621]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:21:53.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:52 smithi163 ceph-mon[29621]: 6.5 scrub starts 2024-03-21T13:21:53.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:52 smithi163 ceph-mon[29621]: 6.5 scrub ok 2024-03-21T13:21:53.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:52 smithi163 ceph-mon[40675]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:21:53.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:52 smithi163 ceph-mon[40675]: 6.5 scrub starts 2024-03-21T13:21:53.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:52 smithi163 ceph-mon[40675]: 6.5 scrub ok 2024-03-21T13:21:53.818 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:21:53.818 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":495,"fsid":"d4bf79c0-e781-11ee-95cd-87774f69a715","created":"2024-03-21T12:52:56.031419+0000","modified":"2024-03-21T13:21:49.620360+0000","last_up_change":"2024-03-21T12:58:23.775302+0000","last_in_change":"2024-03-21T13:20:44.111930+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T12:56:20.557490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-21T12:58:35.713383+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":67,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-21T12:58:36.436384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-21T12:58:38.049304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-21T12:58:39.990512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-21T12:58:41.463686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c34a3b25-9e02-46e6-8f9a-d0af2615258f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":471,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6801","nonce":3086410981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3086410981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3086410981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6803","nonce":3086410981}]},"public_addr":"172.21.15.163:6801/3086410981","cluster_addr":"172.21.15.163:6802/3086410981","heartbeat_back_addr":"172.21.15.163:6804/3086410981","heartbeat_front_addr":"172.21.15.163:6803/3086410981","state":["exists","up"]},{"osd":1,"uuid":"36e05f89-c238-41f7-be65-1553b529e8ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6805","nonce":3666062784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3666062784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3666062784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6807","nonce":3666062784}]},"public_addr":"172.21.15.163:6805/3666062784","cluster_addr":"172.21.15.163:6806/3666062784","heartbeat_back_addr":"172.21.15.163:6808/3666062784","heartbeat_front_addr":"172.21.15.163:6807/3666062784","state":["exists","up"]},{"osd":2,"uuid":"968ec78d-25b8-4905-b328-d490c38ca85f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6809","nonce":4009213507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4009213507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4009213507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":4009213507}]},"public_addr":"172.21.15.163:6809/4009213507","cluster_addr":"172.21.15.163:6810/4009213507","heartbeat_back_addr":"172.21.15.163:6812/4009213507","heartbeat_front_addr":"172.21.15.163:6811/4009213507","state":["exists","up"]},{"osd":3,"uuid":"67dfa4f6-1709-48f6-971d-c6287233492d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":482,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6813","nonce":3981878189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3981878189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":3981878189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":3981878189}]},"public_addr":"172.21.15.163:6813/3981878189","cluster_addr":"172.21.15.163:6814/3981878189","heartbeat_back_addr":"172.21.15.163:6816/3981878189","heartbeat_front_addr":"172.21.15.163:6815/3981878189","state":["exists","up"]},{"osd":4,"uuid":"7fd311a9-f3c2-456d-8ab1-341cf5b33471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":485,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":719276054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":719276054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":719276054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":719276054}]},"public_addr":"172.21.15.204:6800/719276054","cluster_addr":"172.21.15.204:6801/719276054","heartbeat_back_addr":"172.21.15.204:6803/719276054","heartbeat_front_addr":"172.21.15.204:6802/719276054","state":["exists","up"]},{"osd":5,"uuid":"a87759c8-251c-415d-9387-a49f7d2f36d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1346947668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":1346947668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":1346947668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1346947668}]},"public_addr":"172.21.15.204:6804/1346947668","cluster_addr":"172.21.15.204:6805/1346947668","heartbeat_back_addr":"172.21.15.204:6807/1346947668","heartbeat_front_addr":"172.21.15.204:6806/1346947668","state":["exists","up"]},{"osd":6,"uuid":"5eb135cd-f677-4d0c-b23a-738f31615b6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":482,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":200077988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":200077988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":200077988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":200077988}]},"public_addr":"172.21.15.204:6808/200077988","cluster_addr":"172.21.15.204:6809/200077988","heartbeat_back_addr":"172.21.15.204:6811/200077988","heartbeat_front_addr":"172.21.15.204:6810/200077988","state":["exists","up"]},{"osd":7,"uuid":"4f528e10-2c93-4bc2-a54c-06ca4132ed49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":466,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":315664888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":315664888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":315664888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":315664888}]},"public_addr":"172.21.15.204:6812/315664888","cluster_addr":"172.21.15.204:6813/315664888","heartbeat_back_addr":"172.21.15.204:6815/315664888","heartbeat_front_addr":"172.21.15.204:6814/315664888","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:21:48.757373+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:20:51.586809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:21:47.524106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:21:37.401219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:21:35.208437+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:21:22.559865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:20:54.172013+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:21:16.926060+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.13","osds":[0,5,6]},{"pgid":"6.11","osds":[7,3,4]}],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.19","mappings":[{"from":4,"to":7}]},{"pgid":"4.9","mappings":[{"from":3,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.19","mappings":[{"from":3,"to":7}]},{"pgid":"6.6","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:0/3453855427":"2024-03-22T12:59:18.599967+0000","172.21.15.163:0/2571244897":"2024-03-22T12:59:18.599967+0000","172.21.15.163:6800/3073378854":"2024-03-22T12:59:18.599967+0000","172.21.15.163:0/1334591766":"2024-03-22T12:59:18.599967+0000","172.21.15.163:0/116934309":"2024-03-22T12:54:00.224135+0000","172.21.15.163:0/3890775846":"2024-03-22T12:59:18.599967+0000","172.21.15.163:0/1783921524":"2024-03-22T12:54:00.224135+0000","172.21.15.163:6800/2553440211":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/1812694077":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/1070540283":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/2354157786":"2024-03-22T12:53:32.790116+0000","172.21.15.163:0/1591817392":"2024-03-22T12:59:18.599967+0000","172.21.15.163:0/4152198951":"2024-03-22T12:54:00.224135+0000","172.21.15.163:6800/1798115633":"2024-03-22T12:54:00.224135+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T13:21:54.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:53 smithi204 ceph-mon[33153]: 4.1c scrub starts 2024-03-21T13:21:54.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:53 smithi204 ceph-mon[33153]: 4.1c scrub ok 2024-03-21T13:21:54.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:53 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:54.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:53 smithi163 ceph-mon[29621]: 4.1c scrub starts 2024-03-21T13:21:54.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:53 smithi163 ceph-mon[29621]: 4.1c scrub ok 2024-03-21T13:21:54.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:53 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:54.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:53 smithi163 ceph-mon[40675]: 4.1c scrub starts 2024-03-21T13:21:54.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:53 smithi163 ceph-mon[40675]: 4.1c scrub ok 2024-03-21T13:21:54.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:53 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:54.617 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-21T13:21:54.618 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-21T13:21:54.618 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-21T13:21:54.619 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-21T13:21:54.619 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-21T13:21:54.620 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-21T13:21:54.620 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-21T13:21:54.620 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-21T13:21:55.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:54 smithi204 ceph-mon[33153]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:21:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:54 smithi204 ceph-mon[33153]: 3.11 scrub starts 2024-03-21T13:21:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:54 smithi204 ceph-mon[33153]: 3.11 scrub ok 2024-03-21T13:21:55.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:54 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/274436720' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:21:55.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:54 smithi163 ceph-mon[29621]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:21:55.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:54 smithi163 ceph-mon[29621]: 3.11 scrub starts 2024-03-21T13:21:55.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:54 smithi163 ceph-mon[29621]: 3.11 scrub ok 2024-03-21T13:21:55.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:54 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/274436720' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:21:55.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:54 smithi163 ceph-mon[40675]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:21:55.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:54 smithi163 ceph-mon[40675]: 3.11 scrub starts 2024-03-21T13:21:55.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:54 smithi163 ceph-mon[40675]: 3.11 scrub ok 2024-03-21T13:21:55.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:54 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/274436720' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:21:55.170 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:55.170 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:55.171 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:55.171 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:55.172 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:55.186 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:55.210 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:21:55.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:55 smithi163 ceph-mon[29621]: 5.17 scrub starts 2024-03-21T13:21:55.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:55 smithi163 ceph-mon[29621]: 5.17 scrub ok 2024-03-21T13:21:55.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:55 smithi163 ceph-mon[40675]: 5.17 scrub starts 2024-03-21T13:21:55.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:55 smithi163 ceph-mon[40675]: 5.17 scrub ok 2024-03-21T13:21:55.702 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:21:55 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:21:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:21:56.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:55 smithi204 ceph-mon[33153]: 5.17 scrub starts 2024-03-21T13:21:56.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:55 smithi204 ceph-mon[33153]: 5.17 scrub ok 2024-03-21T13:21:57.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:56 smithi204 ceph-mon[33153]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:21:57.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:56 smithi204 ceph-mon[33153]: 3.f scrub starts 2024-03-21T13:21:57.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:56 smithi204 ceph-mon[33153]: 3.f scrub ok 2024-03-21T13:21:57.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:56 smithi204 ceph-mon[33153]: 4.12 deep-scrub starts 2024-03-21T13:21:57.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:56 smithi204 ceph-mon[33153]: 4.12 deep-scrub ok 2024-03-21T13:21:57.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:56 smithi204 ceph-mon[33153]: 4.1f scrub starts 2024-03-21T13:21:57.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:56 smithi204 ceph-mon[33153]: 4.1f scrub ok 2024-03-21T13:21:57.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[29621]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:21:57.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[29621]: 3.f scrub starts 2024-03-21T13:21:57.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[29621]: 3.f scrub ok 2024-03-21T13:21:57.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[29621]: 4.12 deep-scrub starts 2024-03-21T13:21:57.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[29621]: 4.12 deep-scrub ok 2024-03-21T13:21:57.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[29621]: 4.1f scrub starts 2024-03-21T13:21:57.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[29621]: 4.1f scrub ok 2024-03-21T13:21:57.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[40675]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:21:57.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[40675]: 3.f scrub starts 2024-03-21T13:21:57.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[40675]: 3.f scrub ok 2024-03-21T13:21:57.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[40675]: 4.12 deep-scrub starts 2024-03-21T13:21:57.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[40675]: 4.12 deep-scrub ok 2024-03-21T13:21:57.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[40675]: 4.1f scrub starts 2024-03-21T13:21:57.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:56 smithi163 ceph-mon[40675]: 4.1f scrub ok 2024-03-21T13:21:57.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:57 smithi163 ceph-mon[29621]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:21:57.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:57 smithi163 ceph-mon[40675]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:21:58.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:57 smithi204 ceph-mon[33153]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:21:58.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:58 smithi163 ceph-mon[29621]: 5.c scrub starts 2024-03-21T13:21:58.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:58 smithi163 ceph-mon[29621]: 5.c scrub ok 2024-03-21T13:21:58.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:58 smithi163 ceph-mon[40675]: 5.c scrub starts 2024-03-21T13:21:58.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:58 smithi163 ceph-mon[40675]: 5.c scrub ok 2024-03-21T13:21:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:58 smithi204 ceph-mon[33153]: 5.c scrub starts 2024-03-21T13:21:59.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:58 smithi204 ceph-mon[33153]: 5.c scrub ok 2024-03-21T13:21:59.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:59 smithi163 ceph-mon[29621]: 3.13 scrub starts 2024-03-21T13:21:59.756 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:59 smithi163 ceph-mon[29621]: 3.13 scrub ok 2024-03-21T13:21:59.756 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:59 smithi163 ceph-mon[29621]: 4.d scrub starts 2024-03-21T13:21:59.756 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:59 smithi163 ceph-mon[29621]: 4.d scrub ok 2024-03-21T13:21:59.756 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:21:59 smithi163 ceph-mon[29621]: pgmap v1099: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:21:59.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:59 smithi163 ceph-mon[40675]: 3.13 scrub starts 2024-03-21T13:21:59.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:59 smithi163 ceph-mon[40675]: 3.13 scrub ok 2024-03-21T13:21:59.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:59 smithi163 ceph-mon[40675]: 4.d scrub starts 2024-03-21T13:21:59.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:59 smithi163 ceph-mon[40675]: 4.d scrub ok 2024-03-21T13:21:59.756 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:21:59 smithi163 ceph-mon[40675]: pgmap v1099: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:59 smithi204 ceph-mon[33153]: 3.13 scrub starts 2024-03-21T13:22:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:59 smithi204 ceph-mon[33153]: 3.13 scrub ok 2024-03-21T13:22:00.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:59 smithi204 ceph-mon[33153]: 4.d scrub starts 2024-03-21T13:22:00.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:59 smithi204 ceph-mon[33153]: 4.d scrub ok 2024-03-21T13:22:00.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:21:59 smithi204 ceph-mon[33153]: pgmap v1099: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:00.924 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[29621]: 6.13 deep-scrub starts 2024-03-21T13:22:00.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[29621]: 6.13 deep-scrub ok 2024-03-21T13:22:00.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[29621]: 6.12 deep-scrub starts 2024-03-21T13:22:00.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[29621]: 6.12 deep-scrub ok 2024-03-21T13:22:00.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[29621]: 4.17 scrub starts 2024-03-21T13:22:00.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[29621]: 4.17 scrub ok 2024-03-21T13:22:00.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[29621]: 3.1b scrub starts 2024-03-21T13:22:00.925 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[29621]: 3.1b scrub ok 2024-03-21T13:22:00.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[40675]: 6.13 deep-scrub starts 2024-03-21T13:22:00.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[40675]: 6.13 deep-scrub ok 2024-03-21T13:22:00.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[40675]: 6.12 deep-scrub starts 2024-03-21T13:22:00.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[40675]: 6.12 deep-scrub ok 2024-03-21T13:22:00.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[40675]: 4.17 scrub starts 2024-03-21T13:22:00.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[40675]: 4.17 scrub ok 2024-03-21T13:22:00.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[40675]: 3.1b scrub starts 2024-03-21T13:22:00.926 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:00 smithi163 ceph-mon[40675]: 3.1b scrub ok 2024-03-21T13:22:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:00 smithi204 ceph-mon[33153]: 6.13 deep-scrub starts 2024-03-21T13:22:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:00 smithi204 ceph-mon[33153]: 6.13 deep-scrub ok 2024-03-21T13:22:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:00 smithi204 ceph-mon[33153]: 6.12 deep-scrub starts 2024-03-21T13:22:01.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:00 smithi204 ceph-mon[33153]: 6.12 deep-scrub ok 2024-03-21T13:22:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:00 smithi204 ceph-mon[33153]: 4.17 scrub starts 2024-03-21T13:22:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:00 smithi204 ceph-mon[33153]: 4.17 scrub ok 2024-03-21T13:22:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:00 smithi204 ceph-mon[33153]: 3.1b scrub starts 2024-03-21T13:22:01.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:00 smithi204 ceph-mon[33153]: 3.1b scrub ok 2024-03-21T13:22:01.754 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:01 smithi163 ceph-mon[29621]: 4.7 scrub starts 2024-03-21T13:22:01.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:01 smithi163 ceph-mon[29621]: 4.7 scrub ok 2024-03-21T13:22:01.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:01 smithi163 ceph-mon[29621]: 2.2 scrub starts 2024-03-21T13:22:01.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:01 smithi163 ceph-mon[29621]: 2.2 scrub ok 2024-03-21T13:22:01.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:01 smithi163 ceph-mon[29621]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:22:01.755 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:01 smithi163 ceph-mon[40675]: 4.7 scrub starts 2024-03-21T13:22:01.755 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:01 smithi163 ceph-mon[40675]: 4.7 scrub ok 2024-03-21T13:22:01.755 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:01 smithi163 ceph-mon[40675]: 2.2 scrub starts 2024-03-21T13:22:01.755 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:01 smithi163 ceph-mon[40675]: 2.2 scrub ok 2024-03-21T13:22:01.755 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:01 smithi163 ceph-mon[40675]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:22:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:01 smithi204 ceph-mon[33153]: 4.7 scrub starts 2024-03-21T13:22:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:01 smithi204 ceph-mon[33153]: 4.7 scrub ok 2024-03-21T13:22:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:01 smithi204 ceph-mon[33153]: 2.2 scrub starts 2024-03-21T13:22:02.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:01 smithi204 ceph-mon[33153]: 2.2 scrub ok 2024-03-21T13:22:02.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:01 smithi204 ceph-mon[33153]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:22:03.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:22:02 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:22:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:02 smithi204 ceph-mon[33153]: 5.4 scrub starts 2024-03-21T13:22:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:02 smithi204 ceph-mon[33153]: 5.4 scrub ok 2024-03-21T13:22:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:02 smithi204 ceph-mon[33153]: 3.7 scrub starts 2024-03-21T13:22:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:02 smithi204 ceph-mon[33153]: 3.7 scrub ok 2024-03-21T13:22:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:02 smithi204 ceph-mon[33153]: 3.1d scrub starts 2024-03-21T13:22:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:02 smithi204 ceph-mon[33153]: 3.1d scrub ok 2024-03-21T13:22:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:02 smithi204 ceph-mon[33153]: 4.c scrub starts 2024-03-21T13:22:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:02 smithi204 ceph-mon[33153]: 4.c scrub ok 2024-03-21T13:22:03.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:02 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:03.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[29621]: 5.4 scrub starts 2024-03-21T13:22:03.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[29621]: 5.4 scrub ok 2024-03-21T13:22:03.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[29621]: 3.7 scrub starts 2024-03-21T13:22:03.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[29621]: 3.7 scrub ok 2024-03-21T13:22:03.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[29621]: 3.1d scrub starts 2024-03-21T13:22:03.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[29621]: 3.1d scrub ok 2024-03-21T13:22:03.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[29621]: 4.c scrub starts 2024-03-21T13:22:03.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[29621]: 4.c scrub ok 2024-03-21T13:22:03.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:03.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[40675]: 5.4 scrub starts 2024-03-21T13:22:03.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[40675]: 5.4 scrub ok 2024-03-21T13:22:03.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[40675]: 3.7 scrub starts 2024-03-21T13:22:03.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[40675]: 3.7 scrub ok 2024-03-21T13:22:03.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[40675]: 3.1d scrub starts 2024-03-21T13:22:03.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[40675]: 3.1d scrub ok 2024-03-21T13:22:03.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[40675]: 4.c scrub starts 2024-03-21T13:22:03.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[40675]: 4.c scrub ok 2024-03-21T13:22:03.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:02 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:03 smithi204 ceph-mon[33153]: 4.6 scrub starts 2024-03-21T13:22:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:03 smithi204 ceph-mon[33153]: 4.6 scrub ok 2024-03-21T13:22:04.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:03 smithi204 ceph-mon[33153]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:22:04.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:03 smithi163 ceph-mon[29621]: 4.6 scrub starts 2024-03-21T13:22:04.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:03 smithi163 ceph-mon[29621]: 4.6 scrub ok 2024-03-21T13:22:04.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:03 smithi163 ceph-mon[29621]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:22:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:03 smithi163 ceph-mon[40675]: 4.6 scrub starts 2024-03-21T13:22:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:03 smithi163 ceph-mon[40675]: 4.6 scrub ok 2024-03-21T13:22:04.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:03 smithi163 ceph-mon[40675]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:22:04.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:04 smithi163 ceph-mon[40675]: 4.14 deep-scrub starts 2024-03-21T13:22:04.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:04 smithi163 ceph-mon[40675]: 4.14 deep-scrub ok 2024-03-21T13:22:04.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:04 smithi163 ceph-mon[40675]: 6.a scrub starts 2024-03-21T13:22:04.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:04 smithi163 ceph-mon[40675]: 6.a scrub ok 2024-03-21T13:22:04.962 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:04 smithi163 ceph-mon[29621]: 4.14 deep-scrub starts 2024-03-21T13:22:04.962 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:04 smithi163 ceph-mon[29621]: 4.14 deep-scrub ok 2024-03-21T13:22:04.962 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:04 smithi163 ceph-mon[29621]: 6.a scrub starts 2024-03-21T13:22:04.962 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:04 smithi163 ceph-mon[29621]: 6.a scrub ok 2024-03-21T13:22:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:04 smithi204 ceph-mon[33153]: 4.14 deep-scrub starts 2024-03-21T13:22:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:04 smithi204 ceph-mon[33153]: 4.14 deep-scrub ok 2024-03-21T13:22:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:04 smithi204 ceph-mon[33153]: 6.a scrub starts 2024-03-21T13:22:05.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:04 smithi204 ceph-mon[33153]: 6.a scrub ok 2024-03-21T13:22:05.788 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:05 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:22:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:22:05.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:05 smithi163 ceph-mon[40675]: 3.1c scrub starts 2024-03-21T13:22:05.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:05 smithi163 ceph-mon[40675]: 3.1c scrub ok 2024-03-21T13:22:05.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:05 smithi163 ceph-mon[40675]: pgmap v1102: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:22:05.788 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:05 smithi163 ceph-mon[29621]: 3.1c scrub starts 2024-03-21T13:22:05.789 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:05 smithi163 ceph-mon[29621]: 3.1c scrub ok 2024-03-21T13:22:05.789 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:05 smithi163 ceph-mon[29621]: pgmap v1102: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:22:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:05 smithi204 ceph-mon[33153]: 3.1c scrub starts 2024-03-21T13:22:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:05 smithi204 ceph-mon[33153]: 3.1c scrub ok 2024-03-21T13:22:06.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:05 smithi204 ceph-mon[33153]: pgmap v1102: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:22:06.958 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:06.958 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[29621]: 6.d scrub starts 2024-03-21T13:22:06.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[29621]: 5.5 scrub starts 2024-03-21T13:22:06.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[29621]: 5.5 scrub ok 2024-03-21T13:22:06.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[29621]: 6.d scrub ok 2024-03-21T13:22:06.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[29621]: 5.1c scrub starts 2024-03-21T13:22:06.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[29621]: 5.1c scrub ok 2024-03-21T13:22:06.959 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:06.959 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[40675]: 6.d scrub starts 2024-03-21T13:22:06.959 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[40675]: 5.5 scrub starts 2024-03-21T13:22:06.959 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[40675]: 5.5 scrub ok 2024-03-21T13:22:06.960 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[40675]: 6.d scrub ok 2024-03-21T13:22:06.960 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[40675]: 5.1c scrub starts 2024-03-21T13:22:06.960 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:06 smithi163 ceph-mon[40675]: 5.1c scrub ok 2024-03-21T13:22:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:06 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:06 smithi204 ceph-mon[33153]: 6.d scrub starts 2024-03-21T13:22:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:06 smithi204 ceph-mon[33153]: 5.5 scrub starts 2024-03-21T13:22:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:06 smithi204 ceph-mon[33153]: 5.5 scrub ok 2024-03-21T13:22:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:06 smithi204 ceph-mon[33153]: 6.d scrub ok 2024-03-21T13:22:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:06 smithi204 ceph-mon[33153]: 5.1c scrub starts 2024-03-21T13:22:07.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:06 smithi204 ceph-mon[33153]: 5.1c scrub ok 2024-03-21T13:22:07.957 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:07 smithi163 ceph-mon[29621]: 6.7 scrub starts 2024-03-21T13:22:07.957 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:07 smithi163 ceph-mon[29621]: 6.7 scrub ok 2024-03-21T13:22:07.957 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:07 smithi163 ceph-mon[29621]: pgmap v1103: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:22:07.957 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:07 smithi163 ceph-mon[40675]: 6.7 scrub starts 2024-03-21T13:22:07.957 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:07 smithi163 ceph-mon[40675]: 6.7 scrub ok 2024-03-21T13:22:07.957 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:07 smithi163 ceph-mon[40675]: pgmap v1103: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:22:07.994 INFO:teuthology.orchestra.run.smithi163.stdout:128849019181 2024-03-21T13:22:07.994 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-21T13:22:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:07 smithi204 ceph-mon[33153]: 6.7 scrub starts 2024-03-21T13:22:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:07 smithi204 ceph-mon[33153]: 6.7 scrub ok 2024-03-21T13:22:08.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:07 smithi204 ceph-mon[33153]: pgmap v1103: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:22:08.798 INFO:teuthology.orchestra.run.smithi163.stdout:107374182707 2024-03-21T13:22:08.798 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-21T13:22:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:08 smithi204 ceph-mon[33153]: 3.2 scrub starts 2024-03-21T13:22:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:08 smithi204 ceph-mon[33153]: 3.2 scrub ok 2024-03-21T13:22:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:22:09.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:08 smithi204 ceph-mon[33153]: 5.1b scrub starts 2024-03-21T13:22:09.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:08 smithi204 ceph-mon[33153]: 5.1b scrub ok 2024-03-21T13:22:09.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:22:09.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:22:09.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:08 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:22:09.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[29621]: 3.2 scrub starts 2024-03-21T13:22:09.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[29621]: 3.2 scrub ok 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[29621]: 5.1b scrub starts 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[29621]: 5.1b scrub ok 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[40675]: 3.2 scrub starts 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[40675]: 3.2 scrub ok 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[40675]: 5.1b scrub starts 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[40675]: 5.1b scrub ok 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:22:09.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:08 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:22:09.263 INFO:teuthology.orchestra.run.smithi163.stdout:240518168863 2024-03-21T13:22:09.263 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-21T13:22:09.709 INFO:teuthology.orchestra.run.smithi163.stdout:55834575165 2024-03-21T13:22:09.710 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-21T13:22:09.952 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[29621]: 5.f deep-scrub starts 2024-03-21T13:22:09.953 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[29621]: 5.f deep-scrub ok 2024-03-21T13:22:09.953 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[29621]: 5.11 scrub starts 2024-03-21T13:22:09.953 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[29621]: 5.11 scrub ok 2024-03-21T13:22:09.953 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[29621]: 3.6 scrub starts 2024-03-21T13:22:09.953 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[29621]: 3.6 scrub ok 2024-03-21T13:22:09.953 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[29621]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-21T13:22:09.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[40675]: 5.f deep-scrub starts 2024-03-21T13:22:09.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[40675]: 5.f deep-scrub ok 2024-03-21T13:22:09.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[40675]: 5.11 scrub starts 2024-03-21T13:22:09.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[40675]: 5.11 scrub ok 2024-03-21T13:22:09.953 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[40675]: 3.6 scrub starts 2024-03-21T13:22:09.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[40675]: 3.6 scrub ok 2024-03-21T13:22:09.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:09 smithi163 ceph-mon[40675]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-21T13:22:10.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:09 smithi204 ceph-mon[33153]: 5.f deep-scrub starts 2024-03-21T13:22:10.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:09 smithi204 ceph-mon[33153]: 5.f deep-scrub ok 2024-03-21T13:22:10.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:09 smithi204 ceph-mon[33153]: 5.11 scrub starts 2024-03-21T13:22:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:09 smithi204 ceph-mon[33153]: 5.11 scrub ok 2024-03-21T13:22:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:09 smithi204 ceph-mon[33153]: 3.6 scrub starts 2024-03-21T13:22:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:09 smithi204 ceph-mon[33153]: 3.6 scrub ok 2024-03-21T13:22:10.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:09 smithi204 ceph-mon[33153]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-21T13:22:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:10 smithi204 ceph-mon[33153]: 4.3 scrub starts 2024-03-21T13:22:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:10 smithi204 ceph-mon[33153]: 4.3 scrub ok 2024-03-21T13:22:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:10 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:22:11.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:10 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:22:11.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:10 smithi163 ceph-mon[29621]: 4.3 scrub starts 2024-03-21T13:22:11.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:10 smithi163 ceph-mon[29621]: 4.3 scrub ok 2024-03-21T13:22:11.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:10 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:22:11.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:10 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:22:11.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:10 smithi163 ceph-mon[40675]: 4.3 scrub starts 2024-03-21T13:22:11.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:10 smithi163 ceph-mon[40675]: 4.3 scrub ok 2024-03-21T13:22:11.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:10 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:22:11.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:10 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:22:11.280 INFO:teuthology.orchestra.run.smithi163.stdout:201863463205 2024-03-21T13:22:11.281 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-21T13:22:11.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[40675]: osdmap e496: 8 total, 8 up, 8 in 2024-03-21T13:22:11.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[40675]: 3.5 scrub starts 2024-03-21T13:22:11.760 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[40675]: 3.5 scrub ok 2024-03-21T13:22:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[40675]: 4.b scrub starts 2024-03-21T13:22:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[40675]: 4.b scrub ok 2024-03-21T13:22:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[40675]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[40675]: 5.e scrub starts 2024-03-21T13:22:11.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[40675]: 5.e scrub ok 2024-03-21T13:22:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[29621]: osdmap e496: 8 total, 8 up, 8 in 2024-03-21T13:22:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[29621]: 3.5 scrub starts 2024-03-21T13:22:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[29621]: 3.5 scrub ok 2024-03-21T13:22:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[29621]: 4.b scrub starts 2024-03-21T13:22:11.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[29621]: 4.b scrub ok 2024-03-21T13:22:11.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[29621]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:11.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[29621]: 5.e scrub starts 2024-03-21T13:22:11.762 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:11 smithi163 ceph-mon[29621]: 5.e scrub ok 2024-03-21T13:22:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:11 smithi204 ceph-mon[33153]: osdmap e496: 8 total, 8 up, 8 in 2024-03-21T13:22:12.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:11 smithi204 ceph-mon[33153]: 3.5 scrub starts 2024-03-21T13:22:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:11 smithi204 ceph-mon[33153]: 3.5 scrub ok 2024-03-21T13:22:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:11 smithi204 ceph-mon[33153]: 4.b scrub starts 2024-03-21T13:22:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:11 smithi204 ceph-mon[33153]: 4.b scrub ok 2024-03-21T13:22:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:11 smithi204 ceph-mon[33153]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:11 smithi204 ceph-mon[33153]: 5.e scrub starts 2024-03-21T13:22:12.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:11 smithi204 ceph-mon[33153]: 5.e scrub ok 2024-03-21T13:22:12.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:12 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:12.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:22:12 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:22:12.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:12 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:12.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:12 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:13.007 INFO:teuthology.orchestra.run.smithi163.stdout:34359738692 2024-03-21T13:22:13.008 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-21T13:22:13.823 INFO:teuthology.orchestra.run.smithi163.stdout:77309411641 2024-03-21T13:22:13.824 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-21T13:22:14.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:13 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:22:14.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:13 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:22:14.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:13 smithi204 ceph-mon[33153]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:13 smithi204 ceph-mon[33153]: 4.18 scrub starts 2024-03-21T13:22:14.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:13 smithi204 ceph-mon[33153]: 4.18 scrub ok 2024-03-21T13:22:14.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:13 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:22:14.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:13 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:22:14.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:13 smithi163 ceph-mon[29621]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:14.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:13 smithi163 ceph-mon[29621]: 4.18 scrub starts 2024-03-21T13:22:14.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:13 smithi163 ceph-mon[29621]: 4.18 scrub ok 2024-03-21T13:22:14.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:13 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:22:14.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:13 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:22:14.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:13 smithi163 ceph-mon[40675]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:14.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:13 smithi163 ceph-mon[40675]: 4.18 scrub starts 2024-03-21T13:22:14.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:13 smithi163 ceph-mon[40675]: 4.18 scrub ok 2024-03-21T13:22:14.188 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:22:14.188 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:22:14.189 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:22:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:22:14.662 INFO:teuthology.orchestra.run.smithi163.stdout:167503724842 2024-03-21T13:22:14.662 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-21T13:22:14.913 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:14 smithi163 ceph-mon[29621]: 4.2 scrub starts 2024-03-21T13:22:14.913 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:14 smithi163 ceph-mon[29621]: 4.2 scrub ok 2024-03-21T13:22:14.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:14 smithi163 ceph-mon[29621]: osdmap e497: 8 total, 8 up, 8 in 2024-03-21T13:22:14.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:14 smithi163 ceph-mon[29621]: 6.17 deep-scrub starts 2024-03-21T13:22:14.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:14 smithi163 ceph-mon[40675]: 4.2 scrub starts 2024-03-21T13:22:14.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:14 smithi163 ceph-mon[40675]: 4.2 scrub ok 2024-03-21T13:22:14.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:14 smithi163 ceph-mon[40675]: osdmap e497: 8 total, 8 up, 8 in 2024-03-21T13:22:14.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:14 smithi163 ceph-mon[40675]: 6.17 deep-scrub starts 2024-03-21T13:22:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:14 smithi204 ceph-mon[33153]: 4.2 scrub starts 2024-03-21T13:22:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:14 smithi204 ceph-mon[33153]: 4.2 scrub ok 2024-03-21T13:22:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:14 smithi204 ceph-mon[33153]: osdmap e497: 8 total, 8 up, 8 in 2024-03-21T13:22:15.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:14 smithi204 ceph-mon[33153]: 6.17 deep-scrub starts 2024-03-21T13:22:15.892 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:22:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:22:15.892 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[40675]: 6.1b scrub starts 2024-03-21T13:22:15.892 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[40675]: 6.1b scrub ok 2024-03-21T13:22:15.892 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[40675]: 6.17 deep-scrub ok 2024-03-21T13:22:15.892 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[40675]: 3.16 scrub starts 2024-03-21T13:22:15.892 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[40675]: 3.16 scrub ok 2024-03-21T13:22:15.893 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[40675]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:22:15.893 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[40675]: 6.2 scrub starts 2024-03-21T13:22:15.893 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[40675]: 6.2 scrub ok 2024-03-21T13:22:15.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[29621]: 6.1b scrub starts 2024-03-21T13:22:15.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[29621]: 6.1b scrub ok 2024-03-21T13:22:15.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[29621]: 6.17 deep-scrub ok 2024-03-21T13:22:15.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[29621]: 3.16 scrub starts 2024-03-21T13:22:15.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[29621]: 3.16 scrub ok 2024-03-21T13:22:15.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[29621]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:22:15.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[29621]: 6.2 scrub starts 2024-03-21T13:22:15.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:15 smithi163 ceph-mon[29621]: 6.2 scrub ok 2024-03-21T13:22:16.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:15 smithi204 ceph-mon[33153]: 6.1b scrub starts 2024-03-21T13:22:16.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:15 smithi204 ceph-mon[33153]: 6.1b scrub ok 2024-03-21T13:22:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:15 smithi204 ceph-mon[33153]: 6.17 deep-scrub ok 2024-03-21T13:22:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:15 smithi204 ceph-mon[33153]: 3.16 scrub starts 2024-03-21T13:22:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:15 smithi204 ceph-mon[33153]: 3.16 scrub ok 2024-03-21T13:22:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:15 smithi204 ceph-mon[33153]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:22:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:15 smithi204 ceph-mon[33153]: 6.2 scrub starts 2024-03-21T13:22:16.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:15 smithi204 ceph-mon[33153]: 6.2 scrub ok 2024-03-21T13:22:17.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:16 smithi204 ceph-mon[33153]: 4.a scrub starts 2024-03-21T13:22:17.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:16 smithi204 ceph-mon[33153]: 4.a scrub ok 2024-03-21T13:22:17.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:16 smithi204 ceph-mon[33153]: 6.1a scrub starts 2024-03-21T13:22:17.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:16 smithi204 ceph-mon[33153]: 6.1a scrub ok 2024-03-21T13:22:17.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:16 smithi163 ceph-mon[29621]: 4.a scrub starts 2024-03-21T13:22:17.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:16 smithi163 ceph-mon[29621]: 4.a scrub ok 2024-03-21T13:22:17.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:16 smithi163 ceph-mon[29621]: 6.1a scrub starts 2024-03-21T13:22:17.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:16 smithi163 ceph-mon[29621]: 6.1a scrub ok 2024-03-21T13:22:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:16 smithi163 ceph-mon[40675]: 4.a scrub starts 2024-03-21T13:22:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:16 smithi163 ceph-mon[40675]: 4.a scrub ok 2024-03-21T13:22:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:16 smithi163 ceph-mon[40675]: 6.1a scrub starts 2024-03-21T13:22:17.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:16 smithi163 ceph-mon[40675]: 6.1a scrub ok 2024-03-21T13:22:17.499 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:22:17.499 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:22:18.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:17 smithi204 ceph-mon[33153]: 3.1 scrub starts 2024-03-21T13:22:18.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:17 smithi204 ceph-mon[33153]: 3.1 scrub ok 2024-03-21T13:22:18.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:17 smithi204 ceph-mon[33153]: pgmap v1110: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:22:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:17 smithi204 ceph-mon[33153]: 6.4 scrub starts 2024-03-21T13:22:18.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:17 smithi204 ceph-mon[33153]: 6.4 scrub ok 2024-03-21T13:22:18.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:17 smithi163 ceph-mon[29621]: 3.1 scrub starts 2024-03-21T13:22:18.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:17 smithi163 ceph-mon[29621]: 3.1 scrub ok 2024-03-21T13:22:18.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:17 smithi163 ceph-mon[29621]: pgmap v1110: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:22:18.010 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:17 smithi163 ceph-mon[29621]: 6.4 scrub starts 2024-03-21T13:22:18.011 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:17 smithi163 ceph-mon[29621]: 6.4 scrub ok 2024-03-21T13:22:18.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:17 smithi163 ceph-mon[40675]: 3.1 scrub starts 2024-03-21T13:22:18.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:17 smithi163 ceph-mon[40675]: 3.1 scrub ok 2024-03-21T13:22:18.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:17 smithi163 ceph-mon[40675]: pgmap v1110: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:22:18.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:17 smithi163 ceph-mon[40675]: 6.4 scrub starts 2024-03-21T13:22:18.011 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:17 smithi163 ceph-mon[40675]: 6.4 scrub ok 2024-03-21T13:22:20.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:20 smithi204 ceph-mon[33153]: pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:22:20.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:20 smithi163 ceph-mon[29621]: pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:22:20.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:20 smithi163 ceph-mon[40675]: pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:22:21.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:21 smithi163 ceph-mon[29621]: 3.3 scrub starts 2024-03-21T13:22:21.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:21 smithi163 ceph-mon[29621]: 3.3 scrub ok 2024-03-21T13:22:21.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:21 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:21.410 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:21 smithi163 ceph-mon[40675]: 3.3 scrub starts 2024-03-21T13:22:21.410 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:21 smithi163 ceph-mon[40675]: 3.3 scrub ok 2024-03-21T13:22:21.410 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:21 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:21.416 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:22:21.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:21 smithi204 ceph-mon[33153]: 3.3 scrub starts 2024-03-21T13:22:21.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:21 smithi204 ceph-mon[33153]: 3.3 scrub ok 2024-03-21T13:22:21.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:21 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:21.632 INFO:teuthology.orchestra.run.smithi163.stdout:107374182710 2024-03-21T13:22:22.451 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[29621]: pgmap v1112: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:22.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[29621]: 5.1a scrub starts 2024-03-21T13:22:22.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[29621]: 5.1a scrub ok 2024-03-21T13:22:22.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[29621]: 5.1 scrub starts 2024-03-21T13:22:22.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[29621]: 5.1 scrub ok 2024-03-21T13:22:22.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3638451997' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T13:22:22.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[29621]: 5.16 scrub starts 2024-03-21T13:22:22.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[29621]: 5.16 scrub ok 2024-03-21T13:22:22.452 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[40675]: pgmap v1112: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:22.453 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[40675]: 5.1a scrub starts 2024-03-21T13:22:22.453 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[40675]: 5.1a scrub ok 2024-03-21T13:22:22.453 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[40675]: 5.1 scrub starts 2024-03-21T13:22:22.453 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[40675]: 5.1 scrub ok 2024-03-21T13:22:22.453 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3638451997' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T13:22:22.453 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[40675]: 5.16 scrub starts 2024-03-21T13:22:22.453 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:22 smithi163 ceph-mon[40675]: 5.16 scrub ok 2024-03-21T13:22:22.479 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182707 got 107374182710 for osd.3 2024-03-21T13:22:22.479 DEBUG:teuthology.parallel:result is None 2024-03-21T13:22:22.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:22 smithi204 ceph-mon[33153]: pgmap v1112: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:22.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:22 smithi204 ceph-mon[33153]: 5.1a scrub starts 2024-03-21T13:22:22.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:22 smithi204 ceph-mon[33153]: 5.1a scrub ok 2024-03-21T13:22:22.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:22 smithi204 ceph-mon[33153]: 5.1 scrub starts 2024-03-21T13:22:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:22 smithi204 ceph-mon[33153]: 5.1 scrub ok 2024-03-21T13:22:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:22 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3638451997' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T13:22:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:22 smithi204 ceph-mon[33153]: 5.16 scrub starts 2024-03-21T13:22:22.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:22 smithi204 ceph-mon[33153]: 5.16 scrub ok 2024-03-21T13:22:22.943 INFO:teuthology.orchestra.run.smithi163.stdout:34359738694 2024-03-21T13:22:23.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:22:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:22:23.434 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:23 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:23.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:23 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3013556512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T13:22:23.435 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:23 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:23.435 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:23 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3013556512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T13:22:23.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:23 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:23.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:23 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3013556512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T13:22:24.433 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:24 smithi163 ceph-mon[29621]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:24.434 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:24 smithi163 ceph-mon[29621]: 6.1c scrub starts 2024-03-21T13:22:24.434 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:24 smithi163 ceph-mon[29621]: 6.1c scrub ok 2024-03-21T13:22:24.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:24 smithi163 ceph-mon[40675]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:24.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:24 smithi163 ceph-mon[40675]: 6.1c scrub starts 2024-03-21T13:22:24.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:24 smithi163 ceph-mon[40675]: 6.1c scrub ok 2024-03-21T13:22:24.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:24 smithi204 ceph-mon[33153]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:24.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:24 smithi204 ceph-mon[33153]: 6.1c scrub starts 2024-03-21T13:22:24.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:24 smithi204 ceph-mon[33153]: 6.1c scrub ok 2024-03-21T13:22:24.746 INFO:teuthology.orchestra.run.smithi163.stdout:240518168866 2024-03-21T13:22:24.998 INFO:teuthology.orchestra.run.smithi163.stdout:201863463208 2024-03-21T13:22:25.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:25 smithi163 ceph-mon[29621]: 3.d scrub starts 2024-03-21T13:22:25.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:25 smithi163 ceph-mon[29621]: 3.d scrub ok 2024-03-21T13:22:25.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:25 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/4018341687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T13:22:25.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:25 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/1355011754' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T13:22:25.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:25 smithi163 ceph-mon[40675]: 3.d scrub starts 2024-03-21T13:22:25.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:25 smithi163 ceph-mon[40675]: 3.d scrub ok 2024-03-21T13:22:25.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:25 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/4018341687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T13:22:25.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:25 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/1355011754' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T13:22:25.490 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168863 got 240518168866 for osd.7 2024-03-21T13:22:25.490 DEBUG:teuthology.parallel:result is None 2024-03-21T13:22:25.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:25 smithi204 ceph-mon[33153]: 3.d scrub starts 2024-03-21T13:22:25.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:25 smithi204 ceph-mon[33153]: 3.d scrub ok 2024-03-21T13:22:25.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:25 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/4018341687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T13:22:25.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:25 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/1355011754' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T13:22:25.760 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:25 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:22:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:22:26.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:26 smithi163 ceph-mon[29621]: pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-03-21T13:22:26.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:26 smithi163 ceph-mon[29621]: 6.3 scrub starts 2024-03-21T13:22:26.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:26 smithi163 ceph-mon[29621]: 6.3 scrub ok 2024-03-21T13:22:26.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:26 smithi163 ceph-mon[40675]: pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-03-21T13:22:26.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:26 smithi163 ceph-mon[40675]: 6.3 scrub starts 2024-03-21T13:22:26.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:26 smithi163 ceph-mon[40675]: 6.3 scrub ok 2024-03-21T13:22:26.388 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863463205 got 201863463208 for osd.6 2024-03-21T13:22:26.389 DEBUG:teuthology.parallel:result is None 2024-03-21T13:22:26.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:26 smithi204 ceph-mon[33153]: pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-03-21T13:22:26.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:26 smithi204 ceph-mon[33153]: 6.3 scrub starts 2024-03-21T13:22:26.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:26 smithi204 ceph-mon[33153]: 6.3 scrub ok 2024-03-21T13:22:27.162 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738692 got 34359738694 for osd.0 2024-03-21T13:22:27.162 DEBUG:teuthology.parallel:result is None 2024-03-21T13:22:27.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:27 smithi204 ceph-mon[33153]: 4.11 scrub starts 2024-03-21T13:22:27.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:27 smithi204 ceph-mon[33153]: 4.11 scrub ok 2024-03-21T13:22:27.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:27 smithi204 ceph-mon[33153]: 3.a scrub starts 2024-03-21T13:22:27.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:27 smithi204 ceph-mon[33153]: 3.a scrub ok 2024-03-21T13:22:27.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:27 smithi204 ceph-mon[33153]: 6.11 scrub starts 2024-03-21T13:22:27.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:27 smithi204 ceph-mon[33153]: 6.11 scrub ok 2024-03-21T13:22:27.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:27 smithi163 ceph-mon[29621]: 4.11 scrub starts 2024-03-21T13:22:27.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:27 smithi163 ceph-mon[29621]: 4.11 scrub ok 2024-03-21T13:22:27.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:27 smithi163 ceph-mon[29621]: 3.a scrub starts 2024-03-21T13:22:27.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:27 smithi163 ceph-mon[29621]: 3.a scrub ok 2024-03-21T13:22:27.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:27 smithi163 ceph-mon[29621]: 6.11 scrub starts 2024-03-21T13:22:27.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:27 smithi163 ceph-mon[29621]: 6.11 scrub ok 2024-03-21T13:22:27.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:27 smithi163 ceph-mon[40675]: 4.11 scrub starts 2024-03-21T13:22:27.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:27 smithi163 ceph-mon[40675]: 4.11 scrub ok 2024-03-21T13:22:27.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:27 smithi163 ceph-mon[40675]: 3.a scrub starts 2024-03-21T13:22:27.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:27 smithi163 ceph-mon[40675]: 3.a scrub ok 2024-03-21T13:22:27.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:27 smithi163 ceph-mon[40675]: 6.11 scrub starts 2024-03-21T13:22:27.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:27 smithi163 ceph-mon[40675]: 6.11 scrub ok 2024-03-21T13:22:27.707 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:22:28.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[40675]: 6.1 scrub starts 2024-03-21T13:22:28.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[40675]: 6.1 scrub ok 2024-03-21T13:22:28.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[40675]: 4.1e scrub starts 2024-03-21T13:22:28.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[40675]: 4.1e scrub ok 2024-03-21T13:22:28.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[40675]: 6.c scrub starts 2024-03-21T13:22:28.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[40675]: 6.c scrub ok 2024-03-21T13:22:28.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[40675]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:22:28.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[29621]: 6.1 scrub starts 2024-03-21T13:22:28.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[29621]: 6.1 scrub ok 2024-03-21T13:22:28.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[29621]: 4.1e scrub starts 2024-03-21T13:22:28.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[29621]: 4.1e scrub ok 2024-03-21T13:22:28.261 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[29621]: 6.c scrub starts 2024-03-21T13:22:28.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[29621]: 6.c scrub ok 2024-03-21T13:22:28.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:28 smithi163 ceph-mon[29621]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:22:28.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:28 smithi204 ceph-mon[33153]: 6.1 scrub starts 2024-03-21T13:22:28.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:28 smithi204 ceph-mon[33153]: 6.1 scrub ok 2024-03-21T13:22:28.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:28 smithi204 ceph-mon[33153]: 4.1e scrub starts 2024-03-21T13:22:28.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:28 smithi204 ceph-mon[33153]: 4.1e scrub ok 2024-03-21T13:22:28.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:28 smithi204 ceph-mon[33153]: 6.c scrub starts 2024-03-21T13:22:28.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:28 smithi204 ceph-mon[33153]: 6.c scrub ok 2024-03-21T13:22:28.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:28 smithi204 ceph-mon[33153]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:22:28.798 INFO:teuthology.orchestra.run.smithi163.stdout:55834575169 2024-03-21T13:22:29.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:29 smithi204 ceph-mon[33153]: 5.19 scrub starts 2024-03-21T13:22:29.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:29 smithi204 ceph-mon[33153]: 5.19 scrub ok 2024-03-21T13:22:29.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:29 smithi204 ceph-mon[33153]: 3.1f scrub starts 2024-03-21T13:22:29.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:29 smithi204 ceph-mon[33153]: 3.1f scrub ok 2024-03-21T13:22:29.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:29 smithi204 ceph-mon[33153]: 5.a scrub starts 2024-03-21T13:22:29.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:29 smithi204 ceph-mon[33153]: 5.a scrub ok 2024-03-21T13:22:29.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:29 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/2400518116' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T13:22:29.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[40675]: 5.19 scrub starts 2024-03-21T13:22:29.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[40675]: 5.19 scrub ok 2024-03-21T13:22:29.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[40675]: 3.1f scrub starts 2024-03-21T13:22:29.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[40675]: 3.1f scrub ok 2024-03-21T13:22:29.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[40675]: 5.a scrub starts 2024-03-21T13:22:29.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[40675]: 5.a scrub ok 2024-03-21T13:22:29.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/2400518116' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T13:22:29.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[29621]: 5.19 scrub starts 2024-03-21T13:22:29.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[29621]: 5.19 scrub ok 2024-03-21T13:22:29.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[29621]: 3.1f scrub starts 2024-03-21T13:22:29.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[29621]: 3.1f scrub ok 2024-03-21T13:22:29.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[29621]: 5.a scrub starts 2024-03-21T13:22:29.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[29621]: 5.a scrub ok 2024-03-21T13:22:29.512 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:29 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/2400518116' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T13:22:30.041 INFO:teuthology.orchestra.run.smithi163.stdout:128849019186 2024-03-21T13:22:30.382 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[29621]: 6.9 scrub starts 2024-03-21T13:22:30.383 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[29621]: 6.9 scrub ok 2024-03-21T13:22:30.383 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[29621]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:22:30.383 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[29621]: 4.19 deep-scrub starts 2024-03-21T13:22:30.383 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[29621]: 4.19 deep-scrub ok 2024-03-21T13:22:30.383 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:22:30.383 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:22:30.383 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[29621]: 4.1a scrub starts 2024-03-21T13:22:30.383 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[29621]: 4.1a scrub ok 2024-03-21T13:22:30.383 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/997295191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T13:22:30.383 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[40675]: 6.9 scrub starts 2024-03-21T13:22:30.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[40675]: 6.9 scrub ok 2024-03-21T13:22:30.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[40675]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:22:30.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[40675]: 4.19 deep-scrub starts 2024-03-21T13:22:30.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[40675]: 4.19 deep-scrub ok 2024-03-21T13:22:30.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:22:30.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:22:30.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[40675]: 4.1a scrub starts 2024-03-21T13:22:30.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[40675]: 4.1a scrub ok 2024-03-21T13:22:30.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:30 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/997295191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T13:22:30.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:30 smithi204 ceph-mon[33153]: 6.9 scrub starts 2024-03-21T13:22:30.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:30 smithi204 ceph-mon[33153]: 6.9 scrub ok 2024-03-21T13:22:30.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:30 smithi204 ceph-mon[33153]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:22:30.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:30 smithi204 ceph-mon[33153]: 4.19 deep-scrub starts 2024-03-21T13:22:30.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:30 smithi204 ceph-mon[33153]: 4.19 deep-scrub ok 2024-03-21T13:22:30.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:30 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:22:30.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:30 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:22:30.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:30 smithi204 ceph-mon[33153]: 4.1a scrub starts 2024-03-21T13:22:30.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:30 smithi204 ceph-mon[33153]: 4.1a scrub ok 2024-03-21T13:22:30.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:30 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/997295191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T13:22:30.796 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019181 got 128849019186 for osd.4 2024-03-21T13:22:30.796 DEBUG:teuthology.parallel:result is None 2024-03-21T13:22:31.438 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:31 smithi163 ceph-mon[29621]: 5.0 scrub starts 2024-03-21T13:22:31.438 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:31 smithi163 ceph-mon[29621]: 5.0 scrub ok 2024-03-21T13:22:31.438 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:31 smithi163 ceph-mon[29621]: 2.0 scrub starts 2024-03-21T13:22:31.438 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:31 smithi163 ceph-mon[29621]: 2.0 scrub ok 2024-03-21T13:22:31.438 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:31 smithi163 ceph-mon[29621]: osdmap e498: 8 total, 8 up, 8 in 2024-03-21T13:22:31.438 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:31 smithi163 ceph-mon[40675]: 5.0 scrub starts 2024-03-21T13:22:31.438 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:31 smithi163 ceph-mon[40675]: 5.0 scrub ok 2024-03-21T13:22:31.438 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:31 smithi163 ceph-mon[40675]: 2.0 scrub starts 2024-03-21T13:22:31.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:31 smithi163 ceph-mon[40675]: 2.0 scrub ok 2024-03-21T13:22:31.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:31 smithi163 ceph-mon[40675]: osdmap e498: 8 total, 8 up, 8 in 2024-03-21T13:22:31.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:31 smithi204 ceph-mon[33153]: 5.0 scrub starts 2024-03-21T13:22:31.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:31 smithi204 ceph-mon[33153]: 5.0 scrub ok 2024-03-21T13:22:31.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:31 smithi204 ceph-mon[33153]: 2.0 scrub starts 2024-03-21T13:22:31.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:31 smithi204 ceph-mon[33153]: 2.0 scrub ok 2024-03-21T13:22:31.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:31 smithi204 ceph-mon[33153]: osdmap e498: 8 total, 8 up, 8 in 2024-03-21T13:22:31.755 INFO:teuthology.orchestra.run.smithi163.stdout:167503724846 2024-03-21T13:22:32.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:32 smithi204 ceph-mon[33153]: 3.b deep-scrub starts 2024-03-21T13:22:32.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:32 smithi204 ceph-mon[33153]: 3.b deep-scrub ok 2024-03-21T13:22:32.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:32 smithi204 ceph-mon[33153]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:32.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:32 smithi204 ceph-mon[33153]: 1.0 scrub starts 2024-03-21T13:22:32.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:32 smithi204 ceph-mon[33153]: 1.0 scrub ok 2024-03-21T13:22:32.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:32 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3941024676' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T13:22:32.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:32 smithi163 ceph-mon[29621]: 3.b deep-scrub starts 2024-03-21T13:22:32.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:32 smithi163 ceph-mon[29621]: 3.b deep-scrub ok 2024-03-21T13:22:32.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:32 smithi163 ceph-mon[29621]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:32.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:32 smithi163 ceph-mon[29621]: 1.0 scrub starts 2024-03-21T13:22:32.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:32 smithi163 ceph-mon[29621]: 1.0 scrub ok 2024-03-21T13:22:32.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:32 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3941024676' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T13:22:32.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:32 smithi163 ceph-mon[40675]: 3.b deep-scrub starts 2024-03-21T13:22:32.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:32 smithi163 ceph-mon[40675]: 3.b deep-scrub ok 2024-03-21T13:22:32.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:32 smithi163 ceph-mon[40675]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:32.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:32 smithi163 ceph-mon[40675]: 1.0 scrub starts 2024-03-21T13:22:32.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:32 smithi163 ceph-mon[40675]: 1.0 scrub ok 2024-03-21T13:22:32.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:32 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3941024676' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T13:22:32.539 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575165 got 55834575169 for osd.1 2024-03-21T13:22:32.539 DEBUG:teuthology.parallel:result is None 2024-03-21T13:22:33.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:22:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:22:33.111 INFO:teuthology.orchestra.run.smithi163.stdout:77309411646 2024-03-21T13:22:33.375 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:33 smithi163 ceph-mon[29621]: 6.1d scrub starts 2024-03-21T13:22:33.375 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:33 smithi163 ceph-mon[29621]: 6.1d scrub ok 2024-03-21T13:22:33.375 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:33 smithi163 ceph-mon[29621]: 5.13 scrub starts 2024-03-21T13:22:33.375 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:33 smithi163 ceph-mon[29621]: 5.13 scrub ok 2024-03-21T13:22:33.375 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:33 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:33.375 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:33 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3020669960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T13:22:33.375 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:33 smithi163 ceph-mon[40675]: 6.1d scrub starts 2024-03-21T13:22:33.375 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:33 smithi163 ceph-mon[40675]: 6.1d scrub ok 2024-03-21T13:22:33.376 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:33 smithi163 ceph-mon[40675]: 5.13 scrub starts 2024-03-21T13:22:33.376 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:33 smithi163 ceph-mon[40675]: 5.13 scrub ok 2024-03-21T13:22:33.376 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:33 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:33.376 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:33 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3020669960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T13:22:33.400 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724842 got 167503724846 for osd.5 2024-03-21T13:22:33.400 DEBUG:teuthology.parallel:result is None 2024-03-21T13:22:33.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:33 smithi204 ceph-mon[33153]: 6.1d scrub starts 2024-03-21T13:22:33.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:33 smithi204 ceph-mon[33153]: 6.1d scrub ok 2024-03-21T13:22:33.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:33 smithi204 ceph-mon[33153]: 5.13 scrub starts 2024-03-21T13:22:33.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:33 smithi204 ceph-mon[33153]: 5.13 scrub ok 2024-03-21T13:22:33.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:33 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:33.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:33 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3020669960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T13:22:34.154 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411641 got 77309411646 for osd.2 2024-03-21T13:22:34.154 DEBUG:teuthology.parallel:result is None 2024-03-21T13:22:34.154 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-21T13:22:34.155 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:22:34.406 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:22:34.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:34 smithi204 ceph-mon[33153]: 4.4 scrub starts 2024-03-21T13:22:34.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:34 smithi204 ceph-mon[33153]: 4.4 scrub ok 2024-03-21T13:22:34.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:34 smithi204 ceph-mon[33153]: 4.13 deep-scrub starts 2024-03-21T13:22:34.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:34 smithi204 ceph-mon[33153]: 4.13 deep-scrub ok 2024-03-21T13:22:34.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:34 smithi204 ceph-mon[33153]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:34.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:34 smithi204 ceph-mon[33153]: 6.8 scrub starts 2024-03-21T13:22:34.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:34 smithi204 ceph-mon[33153]: 6.8 scrub ok 2024-03-21T13:22:34.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:34 smithi204 ceph-mon[33153]: 6.e deep-scrub starts 2024-03-21T13:22:34.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:34 smithi204 ceph-mon[33153]: 6.e deep-scrub ok 2024-03-21T13:22:34.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[29621]: 4.4 scrub starts 2024-03-21T13:22:34.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[29621]: 4.4 scrub ok 2024-03-21T13:22:34.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[29621]: 4.13 deep-scrub starts 2024-03-21T13:22:34.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[29621]: 4.13 deep-scrub ok 2024-03-21T13:22:34.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[29621]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:34.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[29621]: 6.8 scrub starts 2024-03-21T13:22:34.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[29621]: 6.8 scrub ok 2024-03-21T13:22:34.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[29621]: 6.e deep-scrub starts 2024-03-21T13:22:34.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[29621]: 6.e deep-scrub ok 2024-03-21T13:22:34.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[40675]: 4.4 scrub starts 2024-03-21T13:22:34.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[40675]: 4.4 scrub ok 2024-03-21T13:22:34.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[40675]: 4.13 deep-scrub starts 2024-03-21T13:22:34.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[40675]: 4.13 deep-scrub ok 2024-03-21T13:22:34.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[40675]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:34.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[40675]: 6.8 scrub starts 2024-03-21T13:22:34.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[40675]: 6.8 scrub ok 2024-03-21T13:22:34.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[40675]: 6.e deep-scrub starts 2024-03-21T13:22:34.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:34 smithi163 ceph-mon[40675]: 6.e deep-scrub ok 2024-03-21T13:22:35.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:35 smithi204 ceph-mon[33153]: 5.1e scrub starts 2024-03-21T13:22:35.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:35 smithi204 ceph-mon[33153]: 5.1e scrub ok 2024-03-21T13:22:35.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:35 smithi204 ceph-mon[33153]: 5.14 deep-scrub starts 2024-03-21T13:22:35.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:35 smithi204 ceph-mon[33153]: 5.14 deep-scrub ok 2024-03-21T13:22:35.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:35 smithi204 ceph-mon[33153]: 5.15 scrub starts 2024-03-21T13:22:35.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:35 smithi204 ceph-mon[33153]: 5.15 scrub ok 2024-03-21T13:22:35.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:35 smithi163 ceph-mon[40675]: 5.1e scrub starts 2024-03-21T13:22:35.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:35 smithi163 ceph-mon[40675]: 5.1e scrub ok 2024-03-21T13:22:35.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:35 smithi163 ceph-mon[40675]: 5.14 deep-scrub starts 2024-03-21T13:22:35.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:35 smithi163 ceph-mon[40675]: 5.14 deep-scrub ok 2024-03-21T13:22:35.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:35 smithi163 ceph-mon[40675]: 5.15 scrub starts 2024-03-21T13:22:35.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:35 smithi163 ceph-mon[40675]: 5.15 scrub ok 2024-03-21T13:22:35.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:35 smithi163 ceph-mon[29621]: 5.1e scrub starts 2024-03-21T13:22:35.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:35 smithi163 ceph-mon[29621]: 5.1e scrub ok 2024-03-21T13:22:35.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:35 smithi163 ceph-mon[29621]: 5.14 deep-scrub starts 2024-03-21T13:22:35.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:35 smithi163 ceph-mon[29621]: 5.14 deep-scrub ok 2024-03-21T13:22:35.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:35 smithi163 ceph-mon[29621]: 5.15 scrub starts 2024-03-21T13:22:35.511 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:35 smithi163 ceph-mon[29621]: 5.15 scrub ok 2024-03-21T13:22:35.826 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:35 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:22:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:22:36.453 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:22:36.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:36 smithi163 ceph-mon[29621]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:36.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:36 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:36.454 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:36 smithi163 ceph-mon[40675]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:36.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:36 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:36.461 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:22:36.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:36 smithi204 ceph-mon[33153]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:36.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:36 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:37.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:37 smithi163 ceph-mon[29621]: 5.18 scrub starts 2024-03-21T13:22:37.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:37 smithi163 ceph-mon[29621]: 5.18 scrub ok 2024-03-21T13:22:37.260 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:37 smithi163 ceph-mon[29621]: from='client.16389 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:22:37.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:37 smithi163 ceph-mon[40675]: 5.18 scrub starts 2024-03-21T13:22:37.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:37 smithi163 ceph-mon[40675]: 5.18 scrub ok 2024-03-21T13:22:37.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:37 smithi163 ceph-mon[40675]: from='client.16389 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:22:37.413 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":1120,"stamp":"2024-03-21T13:22:35.059357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3174,"num_read_kb":3020,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":602,"num_bytes_recovered":3557588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1317,"ondisk_log_size":1317,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7036540,"kb_used_data":8108,"kb_used_omap":12,"kb_used_meta":7028339,"kb_avail":742924676,"statfs":{"total":767960285184,"available":760754868224,"internally_reserved":0,"allocated":8302592,"data_stored":4978788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":7197019696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002714"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":348,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287407+0000","last_change":"2024-03-21T13:22:16.284984+0000","last_active":"2024-03-21T13:22:30.287407+0000","last_peered":"2024-03-21T13:22:30.287407+0000","last_clean":"2024-03-21T13:22:30.287407+0000","last_became_active":"2024-03-21T13:21:26.559906+0000","last_became_peered":"2024-03-21T13:21:26.559906+0000","last_unstale":"2024-03-21T13:22:30.287407+0000","last_undegraded":"2024-03-21T13:22:30.287407+0000","last_fullsized":"2024-03-21T13:22:30.287407+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:16.284830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:22:16.284830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:40.235795+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":103,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286495+0000","last_change":"2024-03-21T13:22:27.514499+0000","last_active":"2024-03-21T13:22:30.286495+0000","last_peered":"2024-03-21T13:22:30.286495+0000","last_clean":"2024-03-21T13:22:30.286495+0000","last_became_active":"2024-03-21T13:20:41.135101+0000","last_became_peered":"2024-03-21T13:20:41.135101+0000","last_unstale":"2024-03-21T13:22:30.286495+0000","last_undegraded":"2024-03-21T13:22:30.286495+0000","last_fullsized":"2024-03-21T13:22:30.286495+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:27.514419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:25.553760+0000","last_clean_scrub_stamp":"2024-03-21T13:22:27.514419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:35.990872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":95,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287171+0000","last_change":"2024-03-21T13:22:27.837193+0000","last_active":"2024-03-21T13:22:30.287171+0000","last_peered":"2024-03-21T13:22:30.287171+0000","last_clean":"2024-03-21T13:22:30.287171+0000","last_became_active":"2024-03-21T13:20:45.150641+0000","last_became_peered":"2024-03-21T13:20:45.150641+0000","last_unstale":"2024-03-21T13:22:30.287171+0000","last_undegraded":"2024-03-21T13:22:30.287171+0000","last_fullsized":"2024-03-21T13:22:30.287171+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:27.837082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:00.847701+0000","last_clean_scrub_stamp":"2024-03-21T13:22:27.837082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:54.097156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"435'25","reported_seq":1057,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286233+0000","last_change":"2024-03-21T13:22:13.363690+0000","last_active":"2024-03-21T13:22:30.286233+0000","last_peered":"2024-03-21T13:22:30.286233+0000","last_clean":"2024-03-21T13:22:30.286233+0000","last_became_active":"2024-03-21T13:20:41.127564+0000","last_became_peered":"2024-03-21T13:20:41.127564+0000","last_unstale":"2024-03-21T13:22:30.286233+0000","last_undegraded":"2024-03-21T13:22:30.286233+0000","last_fullsized":"2024-03-21T13:22:30.286233+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:22:13.363563+0000","last_deep_scrub":"234'19","last_deep_scrub_stamp":"2024-03-21T13:16:09.693933+0000","last_clean_scrub_stamp":"2024-03-21T13:22:13.363563+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:21.628251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":930,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157638+0000","last_change":"2024-03-21T13:22:14.048310+0000","last_active":"2024-03-21T13:22:30.157638+0000","last_peered":"2024-03-21T13:22:30.157638+0000","last_clean":"2024-03-21T13:22:30.157638+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:22:30.157638+0000","last_undegraded":"2024-03-21T13:22:30.157638+0000","last_fullsized":"2024-03-21T13:22:30.157638+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:14.048212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:22:14.048212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:26.001544+0000","scrub_duration":242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":829,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286201+0000","last_change":"2024-03-21T13:21:26.559803+0000","last_active":"2024-03-21T13:22:30.286201+0000","last_peered":"2024-03-21T13:22:30.286201+0000","last_clean":"2024-03-21T13:22:30.286201+0000","last_became_active":"2024-03-21T13:21:26.559416+0000","last_became_peered":"2024-03-21T13:21:26.559416+0000","last_unstale":"2024-03-21T13:22:30.286201+0000","last_undegraded":"2024-03-21T13:22:30.286201+0000","last_fullsized":"2024-03-21T13:22:30.286201+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:15.094300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:21:15.094300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:34.891006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":963,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157629+0000","last_change":"2024-03-21T13:21:29.531921+0000","last_active":"2024-03-21T13:22:30.157629+0000","last_peered":"2024-03-21T13:22:30.157629+0000","last_clean":"2024-03-21T13:22:30.157629+0000","last_became_active":"2024-03-21T13:10:03.610318+0000","last_became_peered":"2024-03-21T13:10:03.610318+0000","last_unstale":"2024-03-21T13:22:30.157629+0000","last_undegraded":"2024-03-21T13:22:30.157629+0000","last_fullsized":"2024-03-21T13:22:30.157629+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:21:29.531808+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_clean_scrub_stamp":"2024-03-21T13:21:29.531808+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:36.382340+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"497'46","reported_seq":531,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156771+0000","last_change":"2024-03-21T13:22:29.193195+0000","last_active":"2024-03-21T13:22:30.156771+0000","last_peered":"2024-03-21T13:22:30.156771+0000","last_clean":"2024-03-21T13:22:30.156771+0000","last_became_active":"2024-03-21T13:20:45.148816+0000","last_became_peered":"2024-03-21T13:20:45.148816+0000","last_unstale":"2024-03-21T13:22:30.156771+0000","last_undegraded":"2024-03-21T13:22:30.156771+0000","last_fullsized":"2024-03-21T13:22:30.156771+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'46","last_scrub_stamp":"2024-03-21T13:22:29.193050+0000","last_deep_scrub":"497'46","last_deep_scrub_stamp":"2024-03-21T13:22:29.193050+0000","last_clean_scrub_stamp":"2024-03-21T13:22:29.193050+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:41.619478+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.285910+0000","last_change":"2024-03-21T13:21:44.864283+0000","last_active":"2024-03-21T13:22:30.285910+0000","last_peered":"2024-03-21T13:22:30.285910+0000","last_clean":"2024-03-21T13:22:30.285910+0000","last_became_active":"2024-03-21T13:20:45.147505+0000","last_became_peered":"2024-03-21T13:20:45.147505+0000","last_unstale":"2024-03-21T13:22:30.285910+0000","last_undegraded":"2024-03-21T13:22:30.285910+0000","last_fullsized":"2024-03-21T13:22:30.285910+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:44.864183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:21:44.864183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:48.863570+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":390,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.083595+0000","last_change":"2024-03-21T13:22:07.814634+0000","last_active":"2024-03-21T13:22:14.083595+0000","last_peered":"2024-03-21T13:22:14.083595+0000","last_clean":"2024-03-21T13:22:14.083595+0000","last_became_active":"2024-03-21T13:20:45.154567+0000","last_became_peered":"2024-03-21T13:20:45.154567+0000","last_unstale":"2024-03-21T13:22:14.083595+0000","last_undegraded":"2024-03-21T13:22:14.083595+0000","last_fullsized":"2024-03-21T13:22:14.083595+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:07.814503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:22:07.814503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.466778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":378,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.084088+0000","last_change":"2024-03-21T13:22:01.855100+0000","last_active":"2024-03-21T13:22:14.084088+0000","last_peered":"2024-03-21T13:22:14.084088+0000","last_clean":"2024-03-21T13:22:14.084088+0000","last_became_active":"2024-03-21T13:12:29.526129+0000","last_became_peered":"2024-03-21T13:12:29.526129+0000","last_unstale":"2024-03-21T13:22:14.084088+0000","last_undegraded":"2024-03-21T13:22:14.084088+0000","last_fullsized":"2024-03-21T13:22:14.084088+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:01.854997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:22:01.854997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:10.464213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"435'25","reported_seq":857,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287919+0000","last_change":"2024-03-21T13:22:29.312486+0000","last_active":"2024-03-21T13:22:30.287919+0000","last_peered":"2024-03-21T13:22:30.287919+0000","last_clean":"2024-03-21T13:22:30.287919+0000","last_became_active":"2024-03-21T13:21:26.559349+0000","last_became_peered":"2024-03-21T13:21:26.559349+0000","last_unstale":"2024-03-21T13:22:30.287919+0000","last_undegraded":"2024-03-21T13:22:30.287919+0000","last_fullsized":"2024-03-21T13:22:30.287919+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:22:29.312360+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:22:29.312360+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:55.471028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":384,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.083362+0000","last_change":"2024-03-21T13:21:25.775446+0000","last_active":"2024-03-21T13:22:14.083362+0000","last_peered":"2024-03-21T13:22:14.083362+0000","last_clean":"2024-03-21T13:22:14.083362+0000","last_became_active":"2024-03-21T13:20:41.125208+0000","last_became_peered":"2024-03-21T13:20:41.125208+0000","last_unstale":"2024-03-21T13:22:14.083362+0000","last_undegraded":"2024-03-21T13:22:14.083362+0000","last_fullsized":"2024-03-21T13:22:14.083362+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:25.775270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:21:25.775270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:43.399670+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":966,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156669+0000","last_change":"2024-03-21T13:22:21.130712+0000","last_active":"2024-03-21T13:22:30.156669+0000","last_peered":"2024-03-21T13:22:30.156669+0000","last_clean":"2024-03-21T13:22:30.156669+0000","last_became_active":"2024-03-21T13:20:41.134089+0000","last_became_peered":"2024-03-21T13:20:41.134089+0000","last_unstale":"2024-03-21T13:22:30.156669+0000","last_undegraded":"2024-03-21T13:22:30.156669+0000","last_fullsized":"2024-03-21T13:22:30.156669+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:21.130610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:21.130610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:36.931635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":384,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.084033+0000","last_change":"2024-03-21T13:22:04.888314+0000","last_active":"2024-03-21T13:22:14.084033+0000","last_peered":"2024-03-21T13:22:14.084033+0000","last_clean":"2024-03-21T13:22:14.084033+0000","last_became_active":"2024-03-21T13:20:41.124241+0000","last_became_peered":"2024-03-21T13:20:41.124241+0000","last_unstale":"2024-03-21T13:22:14.084033+0000","last_undegraded":"2024-03-21T13:22:14.084033+0000","last_fullsized":"2024-03-21T13:22:14.084033+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:04.888209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:14.479454+0000","last_clean_scrub_stamp":"2024-03-21T13:22:04.888209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:10.438139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"435'11","reported_seq":863,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288075+0000","last_change":"2024-03-21T13:21:36.513241+0000","last_active":"2024-03-21T13:22:30.288075+0000","last_peered":"2024-03-21T13:22:30.288075+0000","last_clean":"2024-03-21T13:22:30.288075+0000","last_became_active":"2024-03-21T13:21:26.558911+0000","last_became_peered":"2024-03-21T13:21:26.558911+0000","last_unstale":"2024-03-21T13:22:30.288075+0000","last_undegraded":"2024-03-21T13:22:30.288075+0000","last_fullsized":"2024-03-21T13:22:30.288075+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'11","last_scrub_stamp":"2024-03-21T13:21:36.513128+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:21:36.513128+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:42.353169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":985,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288632+0000","last_change":"2024-03-21T13:21:17.619269+0000","last_active":"2024-03-21T13:22:30.288632+0000","last_peered":"2024-03-21T13:22:30.288632+0000","last_clean":"2024-03-21T13:22:30.288632+0000","last_became_active":"2024-03-21T13:12:29.520189+0000","last_became_peered":"2024-03-21T13:12:29.520189+0000","last_unstale":"2024-03-21T13:22:30.288632+0000","last_undegraded":"2024-03-21T13:22:30.288632+0000","last_fullsized":"2024-03-21T13:22:30.288632+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:17.619101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:23.865785+0000","last_clean_scrub_stamp":"2024-03-21T13:21:17.619101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:40.350730+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":101,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287004+0000","last_change":"2024-03-21T13:21:25.288789+0000","last_active":"2024-03-21T13:22:30.287004+0000","last_peered":"2024-03-21T13:22:30.287004+0000","last_clean":"2024-03-21T13:22:30.287004+0000","last_became_active":"2024-03-21T13:20:45.138816+0000","last_became_peered":"2024-03-21T13:20:45.138816+0000","last_unstale":"2024-03-21T13:22:30.287004+0000","last_undegraded":"2024-03-21T13:22:30.287004+0000","last_fullsized":"2024-03-21T13:22:30.287004+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:25.288709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:21:25.288709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:49.499605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286318+0000","last_change":"2024-03-21T13:21:59.818575+0000","last_active":"2024-03-21T13:22:30.286318+0000","last_peered":"2024-03-21T13:22:30.286318+0000","last_clean":"2024-03-21T13:22:30.286318+0000","last_became_active":"2024-03-21T13:20:45.148149+0000","last_became_peered":"2024-03-21T13:20:45.148149+0000","last_unstale":"2024-03-21T13:22:30.286318+0000","last_undegraded":"2024-03-21T13:22:30.286318+0000","last_fullsized":"2024-03-21T13:22:30.286318+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:59.818496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:59.818496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:00.826733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"435'39","reported_seq":585,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156478+0000","last_change":"2024-03-21T13:21:52.353367+0000","last_active":"2024-03-21T13:22:30.156478+0000","last_peered":"2024-03-21T13:22:30.156478+0000","last_clean":"2024-03-21T13:22:30.156478+0000","last_became_active":"2024-03-21T13:20:41.128375+0000","last_became_peered":"2024-03-21T13:20:41.128375+0000","last_unstale":"2024-03-21T13:22:30.156478+0000","last_undegraded":"2024-03-21T13:22:30.156478+0000","last_fullsized":"2024-03-21T13:22:30.156478+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'39","last_scrub_stamp":"2024-03-21T13:21:52.353258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:52.353258+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:17.445009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":948,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.158078+0000","last_change":"2024-03-21T13:21:02.992750+0000","last_active":"2024-03-21T13:22:30.158078+0000","last_peered":"2024-03-21T13:22:30.158078+0000","last_clean":"2024-03-21T13:22:30.158078+0000","last_became_active":"2024-03-21T13:12:29.502808+0000","last_became_peered":"2024-03-21T13:12:29.502808+0000","last_unstale":"2024-03-21T13:22:30.158078+0000","last_undegraded":"2024-03-21T13:22:30.158078+0000","last_fullsized":"2024-03-21T13:22:30.158078+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:02.992650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:21:02.992650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":414,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288270+0000","last_change":"2024-03-21T13:22:06.311857+0000","last_active":"2024-03-21T13:22:30.288270+0000","last_peered":"2024-03-21T13:22:30.288270+0000","last_clean":"2024-03-21T13:22:30.288270+0000","last_became_active":"2024-03-21T13:21:26.554589+0000","last_became_peered":"2024-03-21T13:21:26.554589+0000","last_unstale":"2024-03-21T13:22:30.288270+0000","last_undegraded":"2024-03-21T13:22:30.288270+0000","last_fullsized":"2024-03-21T13:22:30.288270+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:06.311751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:06.311751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:19.011246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1013,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288061+0000","last_change":"2024-03-21T13:21:49.350135+0000","last_active":"2024-03-21T13:22:30.288061+0000","last_peered":"2024-03-21T13:22:30.288061+0000","last_clean":"2024-03-21T13:22:30.288061+0000","last_became_active":"2024-03-21T13:20:41.151350+0000","last_became_peered":"2024-03-21T13:20:41.151350+0000","last_unstale":"2024-03-21T13:22:30.288061+0000","last_undegraded":"2024-03-21T13:22:30.288061+0000","last_fullsized":"2024-03-21T13:22:30.288061+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:49.350031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:49.350031+0000","last_clean_scrub_stamp":"2024-03-21T13:21:49.350031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:58.597466+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"435'24","reported_seq":1025,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157978+0000","last_change":"2024-03-21T13:21:28.550540+0000","last_active":"2024-03-21T13:22:30.157978+0000","last_peered":"2024-03-21T13:22:30.157978+0000","last_clean":"2024-03-21T13:22:30.157978+0000","last_became_active":"2024-03-21T13:20:41.131391+0000","last_became_peered":"2024-03-21T13:20:41.131391+0000","last_unstale":"2024-03-21T13:22:30.157978+0000","last_undegraded":"2024-03-21T13:22:30.157978+0000","last_fullsized":"2024-03-21T13:22:30.157978+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'24","last_scrub_stamp":"2024-03-21T13:21:28.550447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:28.550447+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:43.585108+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":359,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156830+0000","last_change":"2024-03-21T13:22:23.173236+0000","last_active":"2024-03-21T13:22:30.156830+0000","last_peered":"2024-03-21T13:22:30.156830+0000","last_clean":"2024-03-21T13:22:30.156830+0000","last_became_active":"2024-03-21T13:13:06.420220+0000","last_became_peered":"2024-03-21T13:13:06.420220+0000","last_unstale":"2024-03-21T13:22:30.156830+0000","last_undegraded":"2024-03-21T13:22:30.156830+0000","last_fullsized":"2024-03-21T13:22:30.156830+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:22:23.173123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:22:23.173123+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.652307+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"491'9","reported_seq":1243,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286558+0000","last_change":"2024-03-21T13:21:39.207649+0000","last_active":"2024-03-21T13:22:30.286558+0000","last_peered":"2024-03-21T13:22:30.286558+0000","last_clean":"2024-03-21T13:22:30.286558+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:22:30.286558+0000","last_undegraded":"2024-03-21T13:22:30.286558+0000","last_fullsized":"2024-03-21T13:22:30.286558+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'9","last_scrub_stamp":"2024-03-21T13:21:39.207496+0000","last_deep_scrub":"491'9","last_deep_scrub_stamp":"2024-03-21T13:21:39.207496+0000","last_clean_scrub_stamp":"2024-03-21T13:21:39.207496+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:57.718125+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"407'2","reported_seq":99,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286397+0000","last_change":"2024-03-21T13:21:12.214314+0000","last_active":"2024-03-21T13:22:30.286397+0000","last_peered":"2024-03-21T13:22:30.286397+0000","last_clean":"2024-03-21T13:22:30.286397+0000","last_became_active":"2024-03-21T13:20:41.131485+0000","last_became_peered":"2024-03-21T13:20:41.131485+0000","last_unstale":"2024-03-21T13:22:30.286397+0000","last_undegraded":"2024-03-21T13:22:30.286397+0000","last_fullsized":"2024-03-21T13:22:30.286397+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'2","last_scrub_stamp":"2024-03-21T13:21:12.214180+0000","last_deep_scrub":"407'2","last_deep_scrub_stamp":"2024-03-21T13:16:58.735421+0000","last_clean_scrub_stamp":"2024-03-21T13:21:12.214180+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:33.592257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"435'35","reported_seq":107,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287164+0000","last_change":"2024-03-21T13:22:26.880768+0000","last_active":"2024-03-21T13:22:30.287164+0000","last_peered":"2024-03-21T13:22:30.287164+0000","last_clean":"2024-03-21T13:22:30.287164+0000","last_became_active":"2024-03-21T13:20:45.156727+0000","last_became_peered":"2024-03-21T13:20:45.156727+0000","last_unstale":"2024-03-21T13:22:30.287164+0000","last_undegraded":"2024-03-21T13:22:30.287164+0000","last_fullsized":"2024-03-21T13:22:30.287164+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'35","last_scrub_stamp":"2024-03-21T13:22:26.880633+0000","last_deep_scrub":"410'25","last_deep_scrub_stamp":"2024-03-21T13:17:09.876547+0000","last_clean_scrub_stamp":"2024-03-21T13:22:26.880633+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:56.742987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":103,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:31.545876+0000","last_change":"2024-03-21T13:22:31.545916+0000","last_active":"2024-03-21T13:22:31.545876+0000","last_peered":"2024-03-21T13:22:31.545876+0000","last_clean":"2024-03-21T13:22:31.545876+0000","last_became_active":"2024-03-21T13:20:41.141076+0000","last_became_peered":"2024-03-21T13:20:41.141076+0000","last_unstale":"2024-03-21T13:22:31.545876+0000","last_undegraded":"2024-03-21T13:22:31.545876+0000","last_fullsized":"2024-03-21T13:22:31.545876+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:31.545751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:22:31.545751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:24:00.930573+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287195+0000","last_change":"2024-03-21T13:20:55.922964+0000","last_active":"2024-03-21T13:22:30.287195+0000","last_peered":"2024-03-21T13:22:30.287195+0000","last_clean":"2024-03-21T13:22:30.287195+0000","last_became_active":"2024-03-21T13:20:45.155958+0000","last_became_peered":"2024-03-21T13:20:45.155958+0000","last_unstale":"2024-03-21T13:22:30.287195+0000","last_undegraded":"2024-03-21T13:22:30.287195+0000","last_fullsized":"2024-03-21T13:22:30.287195+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:20:55.922856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:55.922856+0000","last_clean_scrub_stamp":"2024-03-21T13:20:55.922856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":978,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.155612+0000","last_change":"2024-03-21T13:21:40.528839+0000","last_active":"2024-03-21T13:22:30.155612+0000","last_peered":"2024-03-21T13:22:30.155612+0000","last_clean":"2024-03-21T13:22:30.155612+0000","last_became_active":"2024-03-21T13:20:45.140132+0000","last_became_peered":"2024-03-21T13:20:45.140132+0000","last_unstale":"2024-03-21T13:22:30.155612+0000","last_undegraded":"2024-03-21T13:22:30.155612+0000","last_fullsized":"2024-03-21T13:22:30.155612+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:40.528728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:40.528728+0000","last_clean_scrub_stamp":"2024-03-21T13:21:40.528728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:51.544803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"435'32","reported_seq":1086,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288495+0000","last_change":"2024-03-21T13:21:56.036242+0000","last_active":"2024-03-21T13:22:30.288495+0000","last_peered":"2024-03-21T13:22:30.288495+0000","last_clean":"2024-03-21T13:22:30.288495+0000","last_became_active":"2024-03-21T13:20:41.151427+0000","last_became_peered":"2024-03-21T13:20:41.151427+0000","last_unstale":"2024-03-21T13:22:30.288495+0000","last_undegraded":"2024-03-21T13:22:30.288495+0000","last_fullsized":"2024-03-21T13:22:30.288495+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'32","last_scrub_stamp":"2024-03-21T13:21:56.036110+0000","last_deep_scrub":"435'32","last_deep_scrub_stamp":"2024-03-21T13:20:23.715513+0000","last_clean_scrub_stamp":"2024-03-21T13:21:56.036110+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:19.226447+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":193,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156485+0000","last_change":"2024-03-21T13:21:59.235590+0000","last_active":"2024-03-21T13:22:30.156485+0000","last_peered":"2024-03-21T13:22:30.156485+0000","last_clean":"2024-03-21T13:22:30.156485+0000","last_became_active":"2024-03-21T13:17:27.636917+0000","last_became_peered":"2024-03-21T13:17:27.636917+0000","last_unstale":"2024-03-21T13:22:30.156485+0000","last_undegraded":"2024-03-21T13:22:30.156485+0000","last_fullsized":"2024-03-21T13:22:30.156485+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:59.235488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:59.235488+0000","last_clean_scrub_stamp":"2024-03-21T13:21:59.235488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:01.098266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":951,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288812+0000","last_change":"2024-03-21T13:22:08.910813+0000","last_active":"2024-03-21T13:22:30.288812+0000","last_peered":"2024-03-21T13:22:30.288812+0000","last_clean":"2024-03-21T13:22:30.288812+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:22:30.288812+0000","last_undegraded":"2024-03-21T13:22:30.288812+0000","last_fullsized":"2024-03-21T13:22:30.288812+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:08.910711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:08.910711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:34.881831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287176+0000","last_change":"2024-03-21T13:21:50.802429+0000","last_active":"2024-03-21T13:22:30.287176+0000","last_peered":"2024-03-21T13:22:30.287176+0000","last_clean":"2024-03-21T13:22:30.287176+0000","last_became_active":"2024-03-21T13:20:45.153702+0000","last_became_peered":"2024-03-21T13:20:45.153702+0000","last_unstale":"2024-03-21T13:22:30.287176+0000","last_undegraded":"2024-03-21T13:22:30.287176+0000","last_fullsized":"2024-03-21T13:22:30.287176+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:50.802353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:31.586656+0000","last_clean_scrub_stamp":"2024-03-21T13:21:50.802353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:53.455907+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"435'12","reported_seq":973,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.155630+0000","last_change":"2024-03-21T13:21:20.577329+0000","last_active":"2024-03-21T13:22:30.155630+0000","last_peered":"2024-03-21T13:22:30.155630+0000","last_clean":"2024-03-21T13:22:30.155630+0000","last_became_active":"2024-03-21T13:20:41.129502+0000","last_became_peered":"2024-03-21T13:20:41.129502+0000","last_unstale":"2024-03-21T13:22:30.155630+0000","last_undegraded":"2024-03-21T13:22:30.155630+0000","last_fullsized":"2024-03-21T13:22:30.155630+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'12","last_scrub_stamp":"2024-03-21T13:21:20.577209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:20.577209+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:46.657785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":943,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157743+0000","last_change":"2024-03-21T13:21:58.939968+0000","last_active":"2024-03-21T13:22:30.157743+0000","last_peered":"2024-03-21T13:22:30.157743+0000","last_clean":"2024-03-21T13:22:30.157743+0000","last_became_active":"2024-03-21T13:20:45.152596+0000","last_became_peered":"2024-03-21T13:20:45.152596+0000","last_unstale":"2024-03-21T13:22:30.157743+0000","last_undegraded":"2024-03-21T13:22:30.157743+0000","last_fullsized":"2024-03-21T13:22:30.157743+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:58.939868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:58.939868+0000","last_clean_scrub_stamp":"2024-03-21T13:21:58.939868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:14.382752+0000","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":193,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156636+0000","last_change":"2024-03-21T13:21:44.630267+0000","last_active":"2024-03-21T13:22:30.156636+0000","last_peered":"2024-03-21T13:22:30.156636+0000","last_clean":"2024-03-21T13:22:30.156636+0000","last_became_active":"2024-03-21T13:17:27.644954+0000","last_became_peered":"2024-03-21T13:17:27.644954+0000","last_unstale":"2024-03-21T13:22:30.156636+0000","last_undegraded":"2024-03-21T13:22:30.156636+0000","last_fullsized":"2024-03-21T13:22:30.156636+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:44.630167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:44.630167+0000","last_clean_scrub_stamp":"2024-03-21T13:21:44.630167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:49.443582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":392,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.760340+0000","last_change":"2024-03-21T13:22:14.760359+0000","last_active":"2024-03-21T13:22:14.760340+0000","last_peered":"2024-03-21T13:22:14.760340+0000","last_clean":"2024-03-21T13:22:14.760340+0000","last_became_active":"2024-03-21T13:20:41.125565+0000","last_became_peered":"2024-03-21T13:20:41.125565+0000","last_unstale":"2024-03-21T13:22:14.760340+0000","last_undegraded":"2024-03-21T13:22:14.760340+0000","last_fullsized":"2024-03-21T13:22:14.760340+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:14.760285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:55.653437+0000","last_clean_scrub_stamp":"2024-03-21T13:22:14.760285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:16.625118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"472'32","reported_seq":701,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157330+0000","last_change":"2024-03-21T13:22:25.891852+0000","last_active":"2024-03-21T13:22:30.157330+0000","last_peered":"2024-03-21T13:22:30.157330+0000","last_clean":"2024-03-21T13:22:30.157330+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:22:30.157330+0000","last_undegraded":"2024-03-21T13:22:30.157330+0000","last_fullsized":"2024-03-21T13:22:30.157330+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'32","last_scrub_stamp":"2024-03-21T13:22:25.891762+0000","last_deep_scrub":"472'32","last_deep_scrub_stamp":"2024-03-21T13:20:57.799687+0000","last_clean_scrub_stamp":"2024-03-21T13:22:25.891762+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:43.955333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.285729+0000","last_change":"2024-03-21T13:21:45.904441+0000","last_active":"2024-03-21T13:22:30.285729+0000","last_peered":"2024-03-21T13:22:30.285729+0000","last_clean":"2024-03-21T13:22:30.285729+0000","last_became_active":"2024-03-21T13:20:45.151537+0000","last_became_peered":"2024-03-21T13:20:45.151537+0000","last_unstale":"2024-03-21T13:22:30.285729+0000","last_undegraded":"2024-03-21T13:22:30.285729+0000","last_fullsized":"2024-03-21T13:22:30.285729+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:45.904340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:21:45.904340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:10.706182+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":963,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157620+0000","last_change":"2024-03-21T13:20:55.868156+0000","last_active":"2024-03-21T13:22:30.157620+0000","last_peered":"2024-03-21T13:22:30.157620+0000","last_clean":"2024-03-21T13:22:30.157620+0000","last_became_active":"2024-03-21T13:20:45.151554+0000","last_became_peered":"2024-03-21T13:20:45.151554+0000","last_unstale":"2024-03-21T13:22:30.157620+0000","last_undegraded":"2024-03-21T13:22:30.157620+0000","last_fullsized":"2024-03-21T13:22:30.157620+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:20:55.868072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:20:55.868072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":940,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156651+0000","last_change":"2024-03-21T13:21:24.621650+0000","last_active":"2024-03-21T13:22:30.156651+0000","last_peered":"2024-03-21T13:22:30.156651+0000","last_clean":"2024-03-21T13:22:30.156651+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:22:30.156651+0000","last_undegraded":"2024-03-21T13:22:30.156651+0000","last_fullsized":"2024-03-21T13:22:30.156651+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:24.621542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:19.383090+0000","last_clean_scrub_stamp":"2024-03-21T13:21:24.621542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:43.951316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"435'25","reported_seq":113,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286725+0000","last_change":"2024-03-21T13:21:55.509191+0000","last_active":"2024-03-21T13:22:30.286725+0000","last_peered":"2024-03-21T13:22:30.286725+0000","last_clean":"2024-03-21T13:22:30.286725+0000","last_became_active":"2024-03-21T13:20:41.145892+0000","last_became_peered":"2024-03-21T13:20:41.145892+0000","last_unstale":"2024-03-21T13:22:30.286725+0000","last_undegraded":"2024-03-21T13:22:30.286725+0000","last_fullsized":"2024-03-21T13:22:30.286725+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:21:55.509099+0000","last_deep_scrub":"435'25","last_deep_scrub_stamp":"2024-03-21T13:21:55.509099+0000","last_clean_scrub_stamp":"2024-03-21T13:21:55.509099+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:11.321601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":776,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156287+0000","last_change":"2024-03-21T13:22:26.208740+0000","last_active":"2024-03-21T13:22:30.156287+0000","last_peered":"2024-03-21T13:22:30.156287+0000","last_clean":"2024-03-21T13:22:30.156287+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:22:30.156287+0000","last_undegraded":"2024-03-21T13:22:30.156287+0000","last_fullsized":"2024-03-21T13:22:30.156287+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:26.208625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:22:26.208625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:31.273307+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":95,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286787+0000","last_change":"2024-03-21T13:21:43.350151+0000","last_active":"2024-03-21T13:22:30.286787+0000","last_peered":"2024-03-21T13:22:30.286787+0000","last_clean":"2024-03-21T13:22:30.286787+0000","last_became_active":"2024-03-21T13:20:41.134901+0000","last_became_peered":"2024-03-21T13:20:41.134901+0000","last_unstale":"2024-03-21T13:22:30.286787+0000","last_undegraded":"2024-03-21T13:22:30.286787+0000","last_fullsized":"2024-03-21T13:22:30.286787+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:43.350077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:22.501055+0000","last_clean_scrub_stamp":"2024-03-21T13:21:43.350077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:43.688359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":348,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288065+0000","last_change":"2024-03-21T13:21:33.537275+0000","last_active":"2024-03-21T13:22:30.288065+0000","last_peered":"2024-03-21T13:22:30.288065+0000","last_clean":"2024-03-21T13:22:30.288065+0000","last_became_active":"2024-03-21T13:21:26.636308+0000","last_became_peered":"2024-03-21T13:21:26.636308+0000","last_unstale":"2024-03-21T13:22:30.288065+0000","last_undegraded":"2024-03-21T13:22:30.288065+0000","last_fullsized":"2024-03-21T13:22:30.288065+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:33.537174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:21:33.537174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:43.740213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"435'27","reported_seq":104,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286221+0000","last_change":"2024-03-21T13:21:06.812179+0000","last_active":"2024-03-21T13:22:30.286221+0000","last_peered":"2024-03-21T13:22:30.286221+0000","last_clean":"2024-03-21T13:22:30.286221+0000","last_became_active":"2024-03-21T13:20:45.157398+0000","last_became_peered":"2024-03-21T13:20:45.157398+0000","last_unstale":"2024-03-21T13:22:30.286221+0000","last_undegraded":"2024-03-21T13:22:30.286221+0000","last_fullsized":"2024-03-21T13:22:30.286221+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'27","last_scrub_stamp":"2024-03-21T13:21:06.812068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:06.812068+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287080+0000","last_change":"2024-03-21T13:21:42.942470+0000","last_active":"2024-03-21T13:22:30.287080+0000","last_peered":"2024-03-21T13:22:30.287080+0000","last_clean":"2024-03-21T13:22:30.287080+0000","last_became_active":"2024-03-21T13:20:45.157291+0000","last_became_peered":"2024-03-21T13:20:45.157291+0000","last_unstale":"2024-03-21T13:22:30.287080+0000","last_undegraded":"2024-03-21T13:22:30.287080+0000","last_fullsized":"2024-03-21T13:22:30.287080+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:42.942345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:21:42.942345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:01.589382+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"467'15","reported_seq":535,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:23.047290+0000","last_change":"2024-03-21T13:21:12.738809+0000","last_active":"2024-03-21T13:22:23.047290+0000","last_peered":"2024-03-21T13:22:23.047290+0000","last_clean":"2024-03-21T13:22:23.047290+0000","last_became_active":"2024-03-21T13:20:45.139449+0000","last_became_peered":"2024-03-21T13:20:45.139449+0000","last_unstale":"2024-03-21T13:22:23.047290+0000","last_undegraded":"2024-03-21T13:22:23.047290+0000","last_fullsized":"2024-03-21T13:22:23.047290+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'15","last_scrub_stamp":"2024-03-21T13:21:12.738675+0000","last_deep_scrub":"394'12","last_deep_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_clean_scrub_stamp":"2024-03-21T13:21:12.738675+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:28.373869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":961,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156772+0000","last_change":"2024-03-21T13:21:58.251095+0000","last_active":"2024-03-21T13:22:30.156772+0000","last_peered":"2024-03-21T13:22:30.156772+0000","last_clean":"2024-03-21T13:22:30.156772+0000","last_became_active":"2024-03-21T13:10:03.599265+0000","last_became_peered":"2024-03-21T13:10:03.599265+0000","last_unstale":"2024-03-21T13:22:30.156772+0000","last_undegraded":"2024-03-21T13:22:30.156772+0000","last_fullsized":"2024-03-21T13:22:30.156772+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:58.250996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:58.250996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:22.422647+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"467'35","reported_seq":1061,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157357+0000","last_change":"2024-03-21T13:22:03.763176+0000","last_active":"2024-03-21T13:22:30.157357+0000","last_peered":"2024-03-21T13:22:30.157357+0000","last_clean":"2024-03-21T13:22:30.157357+0000","last_became_active":"2024-03-21T13:20:41.131638+0000","last_became_peered":"2024-03-21T13:20:41.131638+0000","last_unstale":"2024-03-21T13:22:30.157357+0000","last_undegraded":"2024-03-21T13:22:30.157357+0000","last_fullsized":"2024-03-21T13:22:30.157357+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'35","last_scrub_stamp":"2024-03-21T13:22:03.763081+0000","last_deep_scrub":"467'35","last_deep_scrub_stamp":"2024-03-21T13:22:03.763081+0000","last_clean_scrub_stamp":"2024-03-21T13:22:03.763081+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.671429+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":348,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288663+0000","last_change":"2024-03-21T13:22:14.541903+0000","last_active":"2024-03-21T13:22:30.288663+0000","last_peered":"2024-03-21T13:22:30.288663+0000","last_clean":"2024-03-21T13:22:30.288663+0000","last_became_active":"2024-03-21T13:21:26.556903+0000","last_became_peered":"2024-03-21T13:21:26.556903+0000","last_unstale":"2024-03-21T13:22:30.288663+0000","last_undegraded":"2024-03-21T13:22:30.288663+0000","last_fullsized":"2024-03-21T13:22:30.288663+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:14.541787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:22:14.541787+0000","last_clean_scrub_stamp":"2024-03-21T13:22:14.541787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:28.646515+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"272'10","reported_seq":1243,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157435+0000","last_change":"2024-03-21T13:21:19.557781+0000","last_active":"2024-03-21T13:22:30.157435+0000","last_peered":"2024-03-21T13:22:30.157435+0000","last_clean":"2024-03-21T13:22:30.157435+0000","last_became_active":"2024-03-21T13:10:03.594651+0000","last_became_peered":"2024-03-21T13:10:03.594651+0000","last_unstale":"2024-03-21T13:22:30.157435+0000","last_undegraded":"2024-03-21T13:22:30.157435+0000","last_fullsized":"2024-03-21T13:22:30.157435+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'10","last_scrub_stamp":"2024-03-21T13:21:19.557645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:21:19.557645+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.285961+0000","last_change":"2024-03-21T13:21:10.826204+0000","last_active":"2024-03-21T13:22:30.285961+0000","last_peered":"2024-03-21T13:22:30.285961+0000","last_clean":"2024-03-21T13:22:30.285961+0000","last_became_active":"2024-03-21T13:20:45.146823+0000","last_became_peered":"2024-03-21T13:20:45.146823+0000","last_unstale":"2024-03-21T13:22:30.285961+0000","last_undegraded":"2024-03-21T13:22:30.285961+0000","last_fullsized":"2024-03-21T13:22:30.285961+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:10.826125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:21:10.826125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"435'21","reported_seq":397,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.083646+0000","last_change":"2024-03-21T13:21:33.792618+0000","last_active":"2024-03-21T13:22:14.083646+0000","last_peered":"2024-03-21T13:22:14.083646+0000","last_clean":"2024-03-21T13:22:14.083646+0000","last_became_active":"2024-03-21T13:12:29.510247+0000","last_became_peered":"2024-03-21T13:12:29.510247+0000","last_unstale":"2024-03-21T13:22:14.083646+0000","last_undegraded":"2024-03-21T13:22:14.083646+0000","last_fullsized":"2024-03-21T13:22:14.083646+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'21","last_scrub_stamp":"2024-03-21T13:21:33.792448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:33.792448+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:39.318165+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":508,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156330+0000","last_change":"2024-03-21T13:21:42.273889+0000","last_active":"2024-03-21T13:22:30.156330+0000","last_peered":"2024-03-21T13:22:30.156330+0000","last_clean":"2024-03-21T13:22:30.156330+0000","last_became_active":"2024-03-21T13:10:03.613841+0000","last_became_peered":"2024-03-21T13:10:03.613841+0000","last_unstale":"2024-03-21T13:22:30.156330+0000","last_undegraded":"2024-03-21T13:22:30.156330+0000","last_fullsized":"2024-03-21T13:22:30.156330+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:42.273796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:21:42.273796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:10.738603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":952,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.158339+0000","last_change":"2024-03-21T13:21:54.671978+0000","last_active":"2024-03-21T13:22:30.158339+0000","last_peered":"2024-03-21T13:22:30.158339+0000","last_clean":"2024-03-21T13:22:30.158339+0000","last_became_active":"2024-03-21T13:20:41.148420+0000","last_became_peered":"2024-03-21T13:20:41.148420+0000","last_unstale":"2024-03-21T13:22:30.158339+0000","last_undegraded":"2024-03-21T13:22:30.158339+0000","last_fullsized":"2024-03-21T13:22:30.158339+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:54.671834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:13.018536+0000","last_clean_scrub_stamp":"2024-03-21T13:21:54.671834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:08.190497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":955,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156715+0000","last_change":"2024-03-21T13:21:53.323165+0000","last_active":"2024-03-21T13:22:30.156715+0000","last_peered":"2024-03-21T13:22:30.156715+0000","last_clean":"2024-03-21T13:22:30.156715+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:22:30.156715+0000","last_undegraded":"2024-03-21T13:22:30.156715+0000","last_fullsized":"2024-03-21T13:22:30.156715+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:21:53.323039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:53.323039+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:20.019192+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"457'25","reported_seq":97,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287264+0000","last_change":"2024-03-21T13:21:41.901286+0000","last_active":"2024-03-21T13:22:30.287264+0000","last_peered":"2024-03-21T13:22:30.287264+0000","last_clean":"2024-03-21T13:22:30.287264+0000","last_became_active":"2024-03-21T13:20:45.156195+0000","last_became_peered":"2024-03-21T13:20:45.156195+0000","last_unstale":"2024-03-21T13:22:30.287264+0000","last_undegraded":"2024-03-21T13:22:30.287264+0000","last_fullsized":"2024-03-21T13:22:30.287264+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'25","last_scrub_stamp":"2024-03-21T13:21:41.901193+0000","last_deep_scrub":"457'25","last_deep_scrub_stamp":"2024-03-21T13:20:25.925078+0000","last_clean_scrub_stamp":"2024-03-21T13:21:41.901193+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:00.074477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":928,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156552+0000","last_change":"2024-03-21T13:21:16.763214+0000","last_active":"2024-03-21T13:22:30.156552+0000","last_peered":"2024-03-21T13:22:30.156552+0000","last_clean":"2024-03-21T13:22:30.156552+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:22:30.156552+0000","last_undegraded":"2024-03-21T13:22:30.156552+0000","last_fullsized":"2024-03-21T13:22:30.156552+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:16.763105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:16.763105+0000","last_clean_scrub_stamp":"2024-03-21T13:21:16.763105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:43.686957+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":392,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:21.667108+0000","last_change":"2024-03-21T13:22:21.667141+0000","last_active":"2024-03-21T13:22:21.667108+0000","last_peered":"2024-03-21T13:22:21.667108+0000","last_clean":"2024-03-21T13:22:21.667108+0000","last_became_active":"2024-03-21T13:20:41.124729+0000","last_became_peered":"2024-03-21T13:20:41.124729+0000","last_unstale":"2024-03-21T13:22:21.667108+0000","last_undegraded":"2024-03-21T13:22:21.667108+0000","last_fullsized":"2024-03-21T13:22:21.667108+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:21.667041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:21.667041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:35.502312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":975,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286102+0000","last_change":"2024-03-21T13:21:20.170494+0000","last_active":"2024-03-21T13:22:30.286102+0000","last_peered":"2024-03-21T13:22:30.286102+0000","last_clean":"2024-03-21T13:22:30.286102+0000","last_became_active":"2024-03-21T13:20:45.138634+0000","last_became_peered":"2024-03-21T13:20:45.138634+0000","last_unstale":"2024-03-21T13:22:30.286102+0000","last_undegraded":"2024-03-21T13:22:30.286102+0000","last_fullsized":"2024-03-21T13:22:30.286102+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:20.170392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:20.170392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:45.349703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"435'18","reported_seq":1035,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.158241+0000","last_change":"2024-03-21T13:21:59.728224+0000","last_active":"2024-03-21T13:22:30.158241+0000","last_peered":"2024-03-21T13:22:30.158241+0000","last_clean":"2024-03-21T13:22:30.158241+0000","last_became_active":"2024-03-21T13:21:22.555111+0000","last_became_peered":"2024-03-21T13:21:22.555111+0000","last_unstale":"2024-03-21T13:22:30.158241+0000","last_undegraded":"2024-03-21T13:22:30.158241+0000","last_fullsized":"2024-03-21T13:22:30.158241+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'18","last_scrub_stamp":"2024-03-21T13:21:59.728097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:59.728097+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:22.100052+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":390,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.083595+0000","last_change":"2024-03-21T13:22:03.878364+0000","last_active":"2024-03-21T13:22:14.083595+0000","last_peered":"2024-03-21T13:22:14.083595+0000","last_clean":"2024-03-21T13:22:14.083595+0000","last_became_active":"2024-03-21T13:20:45.140194+0000","last_became_peered":"2024-03-21T13:20:45.140194+0000","last_unstale":"2024-03-21T13:22:14.083595+0000","last_undegraded":"2024-03-21T13:22:14.083595+0000","last_fullsized":"2024-03-21T13:22:14.083595+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:03.878263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:43.658918+0000","last_clean_scrub_stamp":"2024-03-21T13:22:03.878263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:26.043529+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":1021,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157243+0000","last_change":"2024-03-21T13:21:55.312655+0000","last_active":"2024-03-21T13:22:30.157243+0000","last_peered":"2024-03-21T13:22:30.157243+0000","last_clean":"2024-03-21T13:22:30.157243+0000","last_became_active":"2024-03-21T13:20:45.137867+0000","last_became_peered":"2024-03-21T13:20:45.137867+0000","last_unstale":"2024-03-21T13:22:30.157243+0000","last_undegraded":"2024-03-21T13:22:30.157243+0000","last_fullsized":"2024-03-21T13:22:30.157243+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:21:55.312543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:55.312543+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:10.575745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"457'46","reported_seq":419,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.082972+0000","last_change":"2024-03-21T13:21:47.874026+0000","last_active":"2024-03-21T13:22:14.082972+0000","last_peered":"2024-03-21T13:22:14.082972+0000","last_clean":"2024-03-21T13:22:14.082972+0000","last_became_active":"2024-03-21T13:12:29.513019+0000","last_became_peered":"2024-03-21T13:12:29.513019+0000","last_unstale":"2024-03-21T13:22:14.082972+0000","last_undegraded":"2024-03-21T13:22:14.082972+0000","last_fullsized":"2024-03-21T13:22:14.082972+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'46","last_scrub_stamp":"2024-03-21T13:21:47.873868+0000","last_deep_scrub":"457'46","last_deep_scrub_stamp":"2024-03-21T13:20:25.653330+0000","last_clean_scrub_stamp":"2024-03-21T13:21:47.873868+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:06.498594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":1232,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156831+0000","last_change":"2024-03-21T13:21:28.693363+0000","last_active":"2024-03-21T13:22:30.156831+0000","last_peered":"2024-03-21T13:22:30.156831+0000","last_clean":"2024-03-21T13:22:30.156831+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:22:30.156831+0000","last_undegraded":"2024-03-21T13:22:30.156831+0000","last_fullsized":"2024-03-21T13:22:30.156831+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:21:28.693220+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:21:28.693220+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:34.892715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":935,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157681+0000","last_change":"2024-03-21T13:21:08.882742+0000","last_active":"2024-03-21T13:22:30.157681+0000","last_peered":"2024-03-21T13:22:30.157681+0000","last_clean":"2024-03-21T13:22:30.157681+0000","last_became_active":"2024-03-21T13:20:41.128472+0000","last_became_peered":"2024-03-21T13:20:41.128472+0000","last_unstale":"2024-03-21T13:22:30.157681+0000","last_undegraded":"2024-03-21T13:22:30.157681+0000","last_fullsized":"2024-03-21T13:22:30.157681+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:08.882640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:11.950446+0000","last_clean_scrub_stamp":"2024-03-21T13:21:08.882640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":967,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157282+0000","last_change":"2024-03-21T13:21:48.526108+0000","last_active":"2024-03-21T13:22:30.157282+0000","last_peered":"2024-03-21T13:22:30.157282+0000","last_clean":"2024-03-21T13:22:30.157282+0000","last_became_active":"2024-03-21T13:20:41.136377+0000","last_became_peered":"2024-03-21T13:20:41.136377+0000","last_unstale":"2024-03-21T13:22:30.157282+0000","last_undegraded":"2024-03-21T13:22:30.157282+0000","last_fullsized":"2024-03-21T13:22:30.157282+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:48.525993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:48.525993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:14.559449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"435'29","reported_seq":258,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286868+0000","last_change":"2024-03-21T13:21:42.369896+0000","last_active":"2024-03-21T13:22:30.286868+0000","last_peered":"2024-03-21T13:22:30.286868+0000","last_clean":"2024-03-21T13:22:30.286868+0000","last_became_active":"2024-03-21T13:21:26.559401+0000","last_became_peered":"2024-03-21T13:21:26.559401+0000","last_unstale":"2024-03-21T13:22:30.286868+0000","last_undegraded":"2024-03-21T13:22:30.286868+0000","last_fullsized":"2024-03-21T13:22:30.286868+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'29","last_scrub_stamp":"2024-03-21T13:21:42.369770+0000","last_deep_scrub":"346'23","last_deep_scrub_stamp":"2024-03-21T13:13:20.232807+0000","last_clean_scrub_stamp":"2024-03-21T13:21:42.369770+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:57.630500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":29,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":532,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.155760+0000","last_change":"2024-03-21T13:21:48.376986+0000","last_active":"2024-03-21T13:22:30.155760+0000","last_peered":"2024-03-21T13:22:30.155760+0000","last_clean":"2024-03-21T13:22:30.155760+0000","last_became_active":"2024-03-21T13:20:45.141107+0000","last_became_peered":"2024-03-21T13:20:45.141107+0000","last_unstale":"2024-03-21T13:22:30.155760+0000","last_undegraded":"2024-03-21T13:22:30.155760+0000","last_fullsized":"2024-03-21T13:22:30.155760+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:48.376837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:21:48.376837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:10.771072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":949,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156214+0000","last_change":"2024-03-21T13:21:07.663560+0000","last_active":"2024-03-21T13:22:30.156214+0000","last_peered":"2024-03-21T13:22:30.156214+0000","last_clean":"2024-03-21T13:22:30.156214+0000","last_became_active":"2024-03-21T13:10:03.598366+0000","last_became_peered":"2024-03-21T13:10:03.598366+0000","last_unstale":"2024-03-21T13:22:30.156214+0000","last_undegraded":"2024-03-21T13:22:30.156214+0000","last_fullsized":"2024-03-21T13:22:30.156214+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:07.663480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:21:07.663480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:33.213765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":534,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156001+0000","last_change":"2024-03-21T13:21:37.299901+0000","last_active":"2024-03-21T13:22:30.156001+0000","last_peered":"2024-03-21T13:22:30.156001+0000","last_clean":"2024-03-21T13:22:30.156001+0000","last_became_active":"2024-03-21T13:20:45.148745+0000","last_became_peered":"2024-03-21T13:20:45.148745+0000","last_unstale":"2024-03-21T13:22:30.156001+0000","last_undegraded":"2024-03-21T13:22:30.156001+0000","last_fullsized":"2024-03-21T13:22:30.156001+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:37.299812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:21:37.299812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:46.897111+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":961,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157389+0000","last_change":"2024-03-21T13:22:24.155612+0000","last_active":"2024-03-21T13:22:30.157389+0000","last_peered":"2024-03-21T13:22:30.157389+0000","last_clean":"2024-03-21T13:22:30.157389+0000","last_became_active":"2024-03-21T13:12:29.508984+0000","last_became_peered":"2024-03-21T13:12:29.508984+0000","last_unstale":"2024-03-21T13:22:30.157389+0000","last_undegraded":"2024-03-21T13:22:30.157389+0000","last_fullsized":"2024-03-21T13:22:30.157389+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:24.155505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:22:24.155505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:27.684020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"435'51","reported_seq":1165,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286745+0000","last_change":"2024-03-21T13:22:15.985194+0000","last_active":"2024-03-21T13:22:30.286745+0000","last_peered":"2024-03-21T13:22:30.286745+0000","last_clean":"2024-03-21T13:22:30.286745+0000","last_became_active":"2024-03-21T13:21:22.555176+0000","last_became_peered":"2024-03-21T13:21:22.555176+0000","last_unstale":"2024-03-21T13:22:30.286745+0000","last_undegraded":"2024-03-21T13:22:30.286745+0000","last_fullsized":"2024-03-21T13:22:30.286745+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'51","last_scrub_stamp":"2024-03-21T13:22:15.985049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:22:15.985049+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:18.645854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":95,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.285771+0000","last_change":"2024-03-21T13:22:28.946212+0000","last_active":"2024-03-21T13:22:30.285771+0000","last_peered":"2024-03-21T13:22:30.285771+0000","last_clean":"2024-03-21T13:22:30.285771+0000","last_became_active":"2024-03-21T13:20:45.147907+0000","last_became_peered":"2024-03-21T13:20:45.147907+0000","last_unstale":"2024-03-21T13:22:30.285771+0000","last_undegraded":"2024-03-21T13:22:30.285771+0000","last_fullsized":"2024-03-21T13:22:30.285771+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:28.946110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:22:28.946110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:47.442466+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":516,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156907+0000","last_change":"2024-03-21T13:22:28.133907+0000","last_active":"2024-03-21T13:22:30.156907+0000","last_peered":"2024-03-21T13:22:30.156907+0000","last_clean":"2024-03-21T13:22:30.156907+0000","last_became_active":"2024-03-21T13:10:03.603646+0000","last_became_peered":"2024-03-21T13:10:03.603646+0000","last_unstale":"2024-03-21T13:22:30.156907+0000","last_undegraded":"2024-03-21T13:22:30.156907+0000","last_fullsized":"2024-03-21T13:22:30.156907+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:28.133750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:28.133750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:42.893249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":378,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.083904+0000","last_change":"2024-03-21T13:21:28.717631+0000","last_active":"2024-03-21T13:22:14.083904+0000","last_peered":"2024-03-21T13:22:14.083904+0000","last_clean":"2024-03-21T13:22:14.083904+0000","last_became_active":"2024-03-21T13:12:29.512741+0000","last_became_peered":"2024-03-21T13:12:29.512741+0000","last_unstale":"2024-03-21T13:22:14.083904+0000","last_undegraded":"2024-03-21T13:22:14.083904+0000","last_fullsized":"2024-03-21T13:22:14.083904+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:28.717546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:28.717546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:30.185139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"435'25","reported_seq":97,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286045+0000","last_change":"2024-03-21T13:22:10.938091+0000","last_active":"2024-03-21T13:22:30.286045+0000","last_peered":"2024-03-21T13:22:30.286045+0000","last_clean":"2024-03-21T13:22:30.286045+0000","last_became_active":"2024-03-21T13:20:45.151741+0000","last_became_peered":"2024-03-21T13:20:45.151741+0000","last_unstale":"2024-03-21T13:22:30.286045+0000","last_undegraded":"2024-03-21T13:22:30.286045+0000","last_fullsized":"2024-03-21T13:22:30.286045+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:22:10.937980+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:22:10.937980+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:12.009657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":923,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157797+0000","last_change":"2024-03-21T13:21:34.470275+0000","last_active":"2024-03-21T13:22:30.157797+0000","last_peered":"2024-03-21T13:22:30.157797+0000","last_clean":"2024-03-21T13:22:30.157797+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:22:30.157797+0000","last_undegraded":"2024-03-21T13:22:30.157797+0000","last_fullsized":"2024-03-21T13:22:30.157797+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:34.470184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:34.470184+0000","last_clean_scrub_stamp":"2024-03-21T13:21:34.470184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:42.738996+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":402,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.083406+0000","last_change":"2024-03-21T13:22:00.899032+0000","last_active":"2024-03-21T13:22:14.083406+0000","last_peered":"2024-03-21T13:22:14.083406+0000","last_clean":"2024-03-21T13:22:14.083406+0000","last_became_active":"2024-03-21T13:20:41.124733+0000","last_became_peered":"2024-03-21T13:20:41.124733+0000","last_unstale":"2024-03-21T13:22:14.083406+0000","last_undegraded":"2024-03-21T13:22:14.083406+0000","last_fullsized":"2024-03-21T13:22:14.083406+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:22:00.898926+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:22:00.898926+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:22.296841+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":524,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287895+0000","last_change":"2024-03-21T13:22:20.209316+0000","last_active":"2024-03-21T13:22:30.287895+0000","last_peered":"2024-03-21T13:22:30.287895+0000","last_clean":"2024-03-21T13:22:30.287895+0000","last_became_active":"2024-03-21T13:21:26.560148+0000","last_became_peered":"2024-03-21T13:21:26.560148+0000","last_unstale":"2024-03-21T13:22:30.287895+0000","last_undegraded":"2024-03-21T13:22:30.287895+0000","last_fullsized":"2024-03-21T13:22:30.287895+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:20.209198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:22.861136+0000","last_clean_scrub_stamp":"2024-03-21T13:22:20.209198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:37.685850+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"460'56","reported_seq":149,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:32.515312+0000","last_change":"2024-03-21T13:22:32.515312+0000","last_active":"2024-03-21T13:22:32.515312+0000","last_peered":"2024-03-21T13:22:32.515312+0000","last_clean":"2024-03-21T13:22:32.515312+0000","last_became_active":"2024-03-21T13:20:41.126603+0000","last_became_peered":"2024-03-21T13:20:41.126603+0000","last_unstale":"2024-03-21T13:22:32.515312+0000","last_undegraded":"2024-03-21T13:22:32.515312+0000","last_fullsized":"2024-03-21T13:22:32.515312+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'56","last_scrub_stamp":"2024-03-21T13:22:32.515145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:22:32.515145+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:43.465033+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286974+0000","last_change":"2024-03-21T13:22:05.896267+0000","last_active":"2024-03-21T13:22:30.286974+0000","last_peered":"2024-03-21T13:22:30.286974+0000","last_clean":"2024-03-21T13:22:30.286974+0000","last_became_active":"2024-03-21T13:20:45.139450+0000","last_became_peered":"2024-03-21T13:20:45.139450+0000","last_unstale":"2024-03-21T13:22:30.286974+0000","last_undegraded":"2024-03-21T13:22:30.286974+0000","last_fullsized":"2024-03-21T13:22:30.286974+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:05.896185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:15.968926+0000","last_clean_scrub_stamp":"2024-03-21T13:22:05.896185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:25.827769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":938,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157492+0000","last_change":"2024-03-21T13:21:51.497092+0000","last_active":"2024-03-21T13:22:30.157492+0000","last_peered":"2024-03-21T13:22:30.157492+0000","last_clean":"2024-03-21T13:22:30.157492+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:22:30.157492+0000","last_undegraded":"2024-03-21T13:22:30.157492+0000","last_fullsized":"2024-03-21T13:22:30.157492+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:51.496985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_clean_scrub_stamp":"2024-03-21T13:21:51.496985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:59.582558+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":95,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286036+0000","last_change":"2024-03-21T13:21:39.405701+0000","last_active":"2024-03-21T13:22:30.286036+0000","last_peered":"2024-03-21T13:22:30.286036+0000","last_clean":"2024-03-21T13:22:30.286036+0000","last_became_active":"2024-03-21T13:20:41.146376+0000","last_became_peered":"2024-03-21T13:20:41.146376+0000","last_unstale":"2024-03-21T13:22:30.286036+0000","last_undegraded":"2024-03-21T13:22:30.286036+0000","last_fullsized":"2024-03-21T13:22:30.286036+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:39.405609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.231115+0000","last_clean_scrub_stamp":"2024-03-21T13:21:39.405609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:56.486299+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":526,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156105+0000","last_change":"2024-03-21T13:21:38.293887+0000","last_active":"2024-03-21T13:22:30.156105+0000","last_peered":"2024-03-21T13:22:30.156105+0000","last_clean":"2024-03-21T13:22:30.156105+0000","last_became_active":"2024-03-21T13:20:45.141427+0000","last_became_peered":"2024-03-21T13:20:45.141427+0000","last_unstale":"2024-03-21T13:22:30.156105+0000","last_undegraded":"2024-03-21T13:22:30.156105+0000","last_fullsized":"2024-03-21T13:22:30.156105+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:38.293801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:21:38.293801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:04.819332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"435'24","reported_seq":125,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287184+0000","last_change":"2024-03-21T13:22:00.546944+0000","last_active":"2024-03-21T13:22:30.287184+0000","last_peered":"2024-03-21T13:22:30.287184+0000","last_clean":"2024-03-21T13:22:30.287184+0000","last_became_active":"2024-03-21T13:20:45.154138+0000","last_became_peered":"2024-03-21T13:20:45.154138+0000","last_unstale":"2024-03-21T13:22:30.287184+0000","last_undegraded":"2024-03-21T13:22:30.287184+0000","last_fullsized":"2024-03-21T13:22:30.287184+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'24","last_scrub_stamp":"2024-03-21T13:22:00.546815+0000","last_deep_scrub":"243'20","last_deep_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_clean_scrub_stamp":"2024-03-21T13:22:00.546815+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:02.770535+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":514,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157171+0000","last_change":"2024-03-21T13:21:49.408138+0000","last_active":"2024-03-21T13:22:30.157171+0000","last_peered":"2024-03-21T13:22:30.157171+0000","last_clean":"2024-03-21T13:22:30.157171+0000","last_became_active":"2024-03-21T13:12:29.506751+0000","last_became_peered":"2024-03-21T13:12:29.506751+0000","last_unstale":"2024-03-21T13:22:30.157171+0000","last_undegraded":"2024-03-21T13:22:30.157171+0000","last_fullsized":"2024-03-21T13:22:30.157171+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:49.408064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:21:49.408064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:11.084702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":378,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.084352+0000","last_change":"2024-03-21T13:22:06.939476+0000","last_active":"2024-03-21T13:22:14.084352+0000","last_peered":"2024-03-21T13:22:14.084352+0000","last_clean":"2024-03-21T13:22:14.084352+0000","last_became_active":"2024-03-21T13:12:29.527083+0000","last_became_peered":"2024-03-21T13:12:29.527083+0000","last_unstale":"2024-03-21T13:22:14.084352+0000","last_undegraded":"2024-03-21T13:22:14.084352+0000","last_fullsized":"2024-03-21T13:22:14.084352+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:06.939376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:48.803220+0000","last_clean_scrub_stamp":"2024-03-21T13:22:06.939376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:16.926173+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":1239,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156417+0000","last_change":"2024-03-21T13:21:17.670584+0000","last_active":"2024-03-21T13:22:30.156417+0000","last_peered":"2024-03-21T13:22:30.156417+0000","last_clean":"2024-03-21T13:22:30.156417+0000","last_became_active":"2024-03-21T13:20:47.557485+0000","last_became_peered":"2024-03-21T13:20:47.557485+0000","last_unstale":"2024-03-21T13:22:30.156417+0000","last_undegraded":"2024-03-21T13:22:30.156417+0000","last_fullsized":"2024-03-21T13:22:30.156417+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:21:17.670482+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:21:17.670482+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":454,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156662+0000","last_change":"2024-03-21T13:22:07.209703+0000","last_active":"2024-03-21T13:22:30.156662+0000","last_peered":"2024-03-21T13:22:30.156662+0000","last_clean":"2024-03-21T13:22:30.156662+0000","last_became_active":"2024-03-21T13:12:29.499989+0000","last_became_peered":"2024-03-21T13:12:29.499989+0000","last_unstale":"2024-03-21T13:22:30.156662+0000","last_undegraded":"2024-03-21T13:22:30.156662+0000","last_fullsized":"2024-03-21T13:22:30.156662+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:07.209624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:22:07.209624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:16.425454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"435'48","reported_seq":1301,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287559+0000","last_change":"2024-03-21T13:21:41.213382+0000","last_active":"2024-03-21T13:22:30.287559+0000","last_peered":"2024-03-21T13:22:30.287559+0000","last_clean":"2024-03-21T13:22:30.287559+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:22:30.287559+0000","last_undegraded":"2024-03-21T13:22:30.287559+0000","last_fullsized":"2024-03-21T13:22:30.287559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'48","last_scrub_stamp":"2024-03-21T13:21:41.213255+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:21:41.213255+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:06.241122+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":966,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157163+0000","last_change":"2024-03-21T13:22:01.205036+0000","last_active":"2024-03-21T13:22:30.157163+0000","last_peered":"2024-03-21T13:22:30.157163+0000","last_clean":"2024-03-21T13:22:30.157163+0000","last_became_active":"2024-03-21T13:12:29.507753+0000","last_became_peered":"2024-03-21T13:12:29.507753+0000","last_unstale":"2024-03-21T13:22:30.157163+0000","last_undegraded":"2024-03-21T13:22:30.157163+0000","last_fullsized":"2024-03-21T13:22:30.157163+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:01.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:22:01.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:20.125373+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":103,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286318+0000","last_change":"2024-03-21T13:22:17.424711+0000","last_active":"2024-03-21T13:22:30.286318+0000","last_peered":"2024-03-21T13:22:30.286318+0000","last_clean":"2024-03-21T13:22:30.286318+0000","last_became_active":"2024-03-21T13:20:41.135808+0000","last_became_peered":"2024-03-21T13:20:41.135808+0000","last_unstale":"2024-03-21T13:22:30.286318+0000","last_undegraded":"2024-03-21T13:22:30.286318+0000","last_fullsized":"2024-03-21T13:22:30.286318+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:17.424625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:48.210633+0000","last_clean_scrub_stamp":"2024-03-21T13:22:17.424625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.928113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286038+0000","last_change":"2024-03-21T13:22:17.022221+0000","last_active":"2024-03-21T13:22:30.286038+0000","last_peered":"2024-03-21T13:22:30.286038+0000","last_clean":"2024-03-21T13:22:30.286038+0000","last_became_active":"2024-03-21T13:20:45.136827+0000","last_became_peered":"2024-03-21T13:20:45.136827+0000","last_unstale":"2024-03-21T13:22:30.286038+0000","last_undegraded":"2024-03-21T13:22:30.286038+0000","last_fullsized":"2024-03-21T13:22:30.286038+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:17.022147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:00.086941+0000","last_clean_scrub_stamp":"2024-03-21T13:22:17.022147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:42.708582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":2421,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156360+0000","last_change":"2024-03-21T13:22:30.144087+0000","last_active":"2024-03-21T13:22:30.156360+0000","last_peered":"2024-03-21T13:22:30.156360+0000","last_clean":"2024-03-21T13:22:30.156360+0000","last_became_active":"2024-03-21T13:20:45.137793+0000","last_became_peered":"2024-03-21T13:20:45.137793+0000","last_unstale":"2024-03-21T13:22:30.156360+0000","last_undegraded":"2024-03-21T13:22:30.156360+0000","last_fullsized":"2024-03-21T13:22:30.156360+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:22:30.143968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:22:30.143968+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:30.346973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1418,"num_read_kb":1413,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"435'36","reported_seq":101,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287291+0000","last_change":"2024-03-21T13:22:02.837829+0000","last_active":"2024-03-21T13:22:30.287291+0000","last_peered":"2024-03-21T13:22:30.287291+0000","last_clean":"2024-03-21T13:22:30.287291+0000","last_became_active":"2024-03-21T13:20:45.150533+0000","last_became_peered":"2024-03-21T13:20:45.150533+0000","last_unstale":"2024-03-21T13:22:30.287291+0000","last_undegraded":"2024-03-21T13:22:30.287291+0000","last_fullsized":"2024-03-21T13:22:30.287291+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'36","last_scrub_stamp":"2024-03-21T13:22:02.837735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:22:02.837735+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:08.633914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":396,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.084321+0000","last_change":"2024-03-21T13:21:48.901169+0000","last_active":"2024-03-21T13:22:14.084321+0000","last_peered":"2024-03-21T13:22:14.084321+0000","last_clean":"2024-03-21T13:22:14.084321+0000","last_became_active":"2024-03-21T13:20:45.139353+0000","last_became_peered":"2024-03-21T13:20:45.139353+0000","last_unstale":"2024-03-21T13:22:14.084321+0000","last_undegraded":"2024-03-21T13:22:14.084321+0000","last_fullsized":"2024-03-21T13:22:14.084321+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:48.901072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:08.237986+0000","last_clean_scrub_stamp":"2024-03-21T13:21:48.901072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:04.680186+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":103,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286496+0000","last_change":"2024-03-21T13:22:26.557554+0000","last_active":"2024-03-21T13:22:30.286496+0000","last_peered":"2024-03-21T13:22:30.286496+0000","last_clean":"2024-03-21T13:22:30.286496+0000","last_became_active":"2024-03-21T13:20:41.146817+0000","last_became_peered":"2024-03-21T13:20:41.146817+0000","last_unstale":"2024-03-21T13:22:30.286496+0000","last_undegraded":"2024-03-21T13:22:30.286496+0000","last_fullsized":"2024-03-21T13:22:30.286496+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:26.557404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:29.809009+0000","last_clean_scrub_stamp":"2024-03-21T13:22:26.557404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:27.473531+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":95,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286477+0000","last_change":"2024-03-21T13:21:45.386590+0000","last_active":"2024-03-21T13:22:30.286477+0000","last_peered":"2024-03-21T13:22:30.286477+0000","last_clean":"2024-03-21T13:22:30.286477+0000","last_became_active":"2024-03-21T13:20:41.134288+0000","last_became_peered":"2024-03-21T13:20:41.134288+0000","last_unstale":"2024-03-21T13:22:30.286477+0000","last_undegraded":"2024-03-21T13:22:30.286477+0000","last_fullsized":"2024-03-21T13:22:30.286477+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:45.386478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:45.386478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:06.855968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"435'39","reported_seq":453,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286251+0000","last_change":"2024-03-21T13:22:09.946154+0000","last_active":"2024-03-21T13:22:30.286251+0000","last_peered":"2024-03-21T13:22:30.286251+0000","last_clean":"2024-03-21T13:22:30.286251+0000","last_became_active":"2024-03-21T13:20:56.547138+0000","last_became_peered":"2024-03-21T13:20:56.547138+0000","last_unstale":"2024-03-21T13:22:30.286251+0000","last_undegraded":"2024-03-21T13:22:30.286251+0000","last_fullsized":"2024-03-21T13:22:30.286251+0000","mapping_epoch":471,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'39","last_scrub_stamp":"2024-03-21T13:22:09.946052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:22:09.946052+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.777422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":980,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286817+0000","last_change":"2024-03-21T13:21:44.246795+0000","last_active":"2024-03-21T13:22:30.286817+0000","last_peered":"2024-03-21T13:22:30.286817+0000","last_clean":"2024-03-21T13:22:30.286817+0000","last_became_active":"2024-03-21T13:20:45.139101+0000","last_became_peered":"2024-03-21T13:20:45.139101+0000","last_unstale":"2024-03-21T13:22:30.286817+0000","last_undegraded":"2024-03-21T13:22:30.286817+0000","last_fullsized":"2024-03-21T13:22:30.286817+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:44.246689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:21:44.246689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:02.536297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.285788+0000","last_change":"2024-03-21T13:21:35.932908+0000","last_active":"2024-03-21T13:22:30.285788+0000","last_peered":"2024-03-21T13:22:30.285788+0000","last_clean":"2024-03-21T13:22:30.285788+0000","last_became_active":"2024-03-21T13:20:45.135691+0000","last_became_peered":"2024-03-21T13:20:45.135691+0000","last_unstale":"2024-03-21T13:22:30.285788+0000","last_undegraded":"2024-03-21T13:22:30.285788+0000","last_fullsized":"2024-03-21T13:22:30.285788+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:35.932778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:21:35.932778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:54.303116+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":378,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.083784+0000","last_change":"2024-03-21T13:22:10.726951+0000","last_active":"2024-03-21T13:22:14.083784+0000","last_peered":"2024-03-21T13:22:14.083784+0000","last_clean":"2024-03-21T13:22:14.083784+0000","last_became_active":"2024-03-21T13:12:29.512291+0000","last_became_peered":"2024-03-21T13:12:29.512291+0000","last_unstale":"2024-03-21T13:22:14.083784+0000","last_undegraded":"2024-03-21T13:22:14.083784+0000","last_fullsized":"2024-03-21T13:22:14.083784+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:10.726838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:24.587197+0000","last_clean_scrub_stamp":"2024-03-21T13:22:10.726838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.594501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"435'35","reported_seq":115,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286817+0000","last_change":"2024-03-21T13:22:13.512153+0000","last_active":"2024-03-21T13:22:30.286817+0000","last_peered":"2024-03-21T13:22:30.286817+0000","last_clean":"2024-03-21T13:22:30.286817+0000","last_became_active":"2024-03-21T13:20:41.141582+0000","last_became_peered":"2024-03-21T13:20:41.141582+0000","last_unstale":"2024-03-21T13:22:30.286817+0000","last_undegraded":"2024-03-21T13:22:30.286817+0000","last_fullsized":"2024-03-21T13:22:30.286817+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'35","last_scrub_stamp":"2024-03-21T13:22:13.512059+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:22:13.512059+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:33.671805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"467'16","reported_seq":112,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286147+0000","last_change":"2024-03-21T13:21:07.845725+0000","last_active":"2024-03-21T13:22:30.286147+0000","last_peered":"2024-03-21T13:22:30.286147+0000","last_clean":"2024-03-21T13:22:30.286147+0000","last_became_active":"2024-03-21T13:20:45.158775+0000","last_became_peered":"2024-03-21T13:20:45.158775+0000","last_unstale":"2024-03-21T13:22:30.286147+0000","last_undegraded":"2024-03-21T13:22:30.286147+0000","last_fullsized":"2024-03-21T13:22:30.286147+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'16","last_scrub_stamp":"2024-03-21T13:21:07.845626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:21:07.845626+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":950,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156997+0000","last_change":"2024-03-21T13:22:25.311368+0000","last_active":"2024-03-21T13:22:30.156997+0000","last_peered":"2024-03-21T13:22:30.156997+0000","last_clean":"2024-03-21T13:22:30.156997+0000","last_became_active":"2024-03-21T13:10:03.606557+0000","last_became_peered":"2024-03-21T13:10:03.606557+0000","last_unstale":"2024-03-21T13:22:30.156997+0000","last_undegraded":"2024-03-21T13:22:30.156997+0000","last_fullsized":"2024-03-21T13:22:30.156997+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:25.311257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:22:25.311257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.813217+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":91,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287341+0000","last_change":"2024-03-21T13:22:08.931699+0000","last_active":"2024-03-21T13:22:30.287341+0000","last_peered":"2024-03-21T13:22:30.287341+0000","last_clean":"2024-03-21T13:22:30.287341+0000","last_became_active":"2024-03-21T13:20:45.150264+0000","last_became_peered":"2024-03-21T13:20:45.150264+0000","last_unstale":"2024-03-21T13:22:30.287341+0000","last_undegraded":"2024-03-21T13:22:30.287341+0000","last_fullsized":"2024-03-21T13:22:30.287341+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:22:08.931607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:22:08.931607+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:13.392639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"435'49","reported_seq":316,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286646+0000","last_change":"2024-03-21T13:21:39.443553+0000","last_active":"2024-03-21T13:22:30.286646+0000","last_peered":"2024-03-21T13:22:30.286646+0000","last_clean":"2024-03-21T13:22:30.286646+0000","last_became_active":"2024-03-21T13:21:26.555211+0000","last_became_peered":"2024-03-21T13:21:26.555211+0000","last_unstale":"2024-03-21T13:22:30.286646+0000","last_undegraded":"2024-03-21T13:22:30.286646+0000","last_fullsized":"2024-03-21T13:22:30.286646+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'49","last_scrub_stamp":"2024-03-21T13:21:39.443429+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:21:39.443429+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:41.501292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"460'12","reported_seq":1264,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.155821+0000","last_change":"2024-03-21T13:22:29.960749+0000","last_active":"2024-03-21T13:22:30.155821+0000","last_peered":"2024-03-21T13:22:30.155821+0000","last_clean":"2024-03-21T13:22:30.155821+0000","last_became_active":"2024-03-21T13:20:41.129168+0000","last_became_peered":"2024-03-21T13:20:41.129168+0000","last_unstale":"2024-03-21T13:22:30.155821+0000","last_undegraded":"2024-03-21T13:22:30.155821+0000","last_fullsized":"2024-03-21T13:22:30.155821+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'12","last_scrub_stamp":"2024-03-21T13:22:29.960665+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:22:29.960665+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:44.161537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":407,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287711+0000","last_change":"2024-03-21T13:22:15.284802+0000","last_active":"2024-03-21T13:22:30.287711+0000","last_peered":"2024-03-21T13:22:30.287711+0000","last_clean":"2024-03-21T13:22:30.287711+0000","last_became_active":"2024-03-21T13:21:26.553104+0000","last_became_peered":"2024-03-21T13:21:26.553104+0000","last_unstale":"2024-03-21T13:22:30.287711+0000","last_undegraded":"2024-03-21T13:22:30.287711+0000","last_fullsized":"2024-03-21T13:22:30.287711+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:15.284719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:50.909170+0000","last_clean_scrub_stamp":"2024-03-21T13:22:15.284719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:19.164991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":956,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.158347+0000","last_change":"2024-03-21T13:22:01.711873+0000","last_active":"2024-03-21T13:22:30.158347+0000","last_peered":"2024-03-21T13:22:30.158347+0000","last_clean":"2024-03-21T13:22:30.158347+0000","last_became_active":"2024-03-21T13:20:45.139123+0000","last_became_peered":"2024-03-21T13:20:45.139123+0000","last_unstale":"2024-03-21T13:22:30.158347+0000","last_undegraded":"2024-03-21T13:22:30.158347+0000","last_fullsized":"2024-03-21T13:22:30.158347+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:01.711792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:22:01.711792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:22.823736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"435'42","reported_seq":1080,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.155723+0000","last_change":"2024-03-21T13:21:42.546721+0000","last_active":"2024-03-21T13:22:30.155723+0000","last_peered":"2024-03-21T13:22:30.155723+0000","last_clean":"2024-03-21T13:22:30.155723+0000","last_became_active":"2024-03-21T13:20:45.140650+0000","last_became_peered":"2024-03-21T13:20:45.140650+0000","last_unstale":"2024-03-21T13:22:30.155723+0000","last_undegraded":"2024-03-21T13:22:30.155723+0000","last_fullsized":"2024-03-21T13:22:30.155723+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'42","last_scrub_stamp":"2024-03-21T13:21:42.546613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:42.546613+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:03.819900+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":961,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286451+0000","last_change":"2024-03-21T13:22:21.194028+0000","last_active":"2024-03-21T13:22:30.286451+0000","last_peered":"2024-03-21T13:22:30.286451+0000","last_clean":"2024-03-21T13:22:30.286451+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:22:30.286451+0000","last_undegraded":"2024-03-21T13:22:30.286451+0000","last_fullsized":"2024-03-21T13:22:30.286451+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:21.193896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:21.193896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:45.577386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":396,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.083882+0000","last_change":"2024-03-21T13:22:05.982754+0000","last_active":"2024-03-21T13:22:14.083882+0000","last_peered":"2024-03-21T13:22:14.083882+0000","last_clean":"2024-03-21T13:22:14.083882+0000","last_became_active":"2024-03-21T13:20:45.138941+0000","last_became_peered":"2024-03-21T13:20:45.138941+0000","last_unstale":"2024-03-21T13:22:14.083882+0000","last_undegraded":"2024-03-21T13:22:14.083882+0000","last_fullsized":"2024-03-21T13:22:14.083882+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:05.982640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:14.280515+0000","last_clean_scrub_stamp":"2024-03-21T13:22:05.982640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:12.581195+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"486'16","reported_seq":185,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288727+0000","last_change":"2024-03-21T13:22:11.355586+0000","last_active":"2024-03-21T13:22:30.288727+0000","last_peered":"2024-03-21T13:22:30.288727+0000","last_clean":"2024-03-21T13:22:30.288727+0000","last_became_active":"2024-03-21T13:21:26.561093+0000","last_became_peered":"2024-03-21T13:21:26.561093+0000","last_unstale":"2024-03-21T13:22:30.288727+0000","last_undegraded":"2024-03-21T13:22:30.288727+0000","last_fullsized":"2024-03-21T13:22:30.288727+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'16","last_scrub_stamp":"2024-03-21T13:22:11.355482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:11.355482+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:27.045834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":956,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.155753+0000","last_change":"2024-03-21T13:21:45.519352+0000","last_active":"2024-03-21T13:22:30.155753+0000","last_peered":"2024-03-21T13:22:30.155753+0000","last_clean":"2024-03-21T13:22:30.155753+0000","last_became_active":"2024-03-21T13:20:41.128145+0000","last_became_peered":"2024-03-21T13:20:41.128145+0000","last_unstale":"2024-03-21T13:22:30.155753+0000","last_undegraded":"2024-03-21T13:22:30.155753+0000","last_fullsized":"2024-03-21T13:22:30.155753+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:45.519268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:00.746886+0000","last_clean_scrub_stamp":"2024-03-21T13:21:45.519268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:02.875028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"435'46","reported_seq":111,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287182+0000","last_change":"2024-03-21T13:21:46.425301+0000","last_active":"2024-03-21T13:22:30.287182+0000","last_peered":"2024-03-21T13:22:30.287182+0000","last_clean":"2024-03-21T13:22:30.287182+0000","last_became_active":"2024-03-21T13:20:41.140902+0000","last_became_peered":"2024-03-21T13:20:41.140902+0000","last_unstale":"2024-03-21T13:22:30.287182+0000","last_undegraded":"2024-03-21T13:22:30.287182+0000","last_fullsized":"2024-03-21T13:22:30.287182+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'46","last_scrub_stamp":"2024-03-21T13:21:46.425201+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:21:46.425201+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:52.917930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"375'2","reported_seq":953,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157099+0000","last_change":"2024-03-21T13:22:26.934977+0000","last_active":"2024-03-21T13:22:30.157099+0000","last_peered":"2024-03-21T13:22:30.157099+0000","last_clean":"2024-03-21T13:22:30.157099+0000","last_became_active":"2024-03-21T13:12:29.502816+0000","last_became_peered":"2024-03-21T13:12:29.502816+0000","last_unstale":"2024-03-21T13:22:30.157099+0000","last_undegraded":"2024-03-21T13:22:30.157099+0000","last_fullsized":"2024-03-21T13:22:30.157099+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'2","last_scrub_stamp":"2024-03-21T13:22:26.934898+0000","last_deep_scrub":"375'2","last_deep_scrub_stamp":"2024-03-21T13:15:14.313376+0000","last_clean_scrub_stamp":"2024-03-21T13:22:26.934898+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:51.378294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":378,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-21T13:22:14.084262+0000","last_change":"2024-03-21T13:22:08.785941+0000","last_active":"2024-03-21T13:22:14.084262+0000","last_peered":"2024-03-21T13:22:14.084262+0000","last_clean":"2024-03-21T13:22:14.084262+0000","last_became_active":"2024-03-21T13:12:29.524087+0000","last_became_peered":"2024-03-21T13:12:29.524087+0000","last_unstale":"2024-03-21T13:22:14.084262+0000","last_undegraded":"2024-03-21T13:22:14.084262+0000","last_fullsized":"2024-03-21T13:22:14.084262+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:08.785846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:22:08.785846+0000","last_clean_scrub_stamp":"2024-03-21T13:22:08.785846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:30.117638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":989,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286589+0000","last_change":"2024-03-21T13:21:50.327929+0000","last_active":"2024-03-21T13:22:30.286589+0000","last_peered":"2024-03-21T13:22:30.286589+0000","last_clean":"2024-03-21T13:22:30.286589+0000","last_became_active":"2024-03-21T13:10:03.586593+0000","last_became_peered":"2024-03-21T13:10:03.586593+0000","last_unstale":"2024-03-21T13:22:30.286589+0000","last_undegraded":"2024-03-21T13:22:30.286589+0000","last_fullsized":"2024-03-21T13:22:30.286589+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:50.327810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:50.327810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:57.234648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"435'27","reported_seq":606,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288528+0000","last_change":"2024-03-21T13:21:26.561325+0000","last_active":"2024-03-21T13:22:30.288528+0000","last_peered":"2024-03-21T13:22:30.288528+0000","last_clean":"2024-03-21T13:22:30.288528+0000","last_became_active":"2024-03-21T13:21:26.560123+0000","last_became_peered":"2024-03-21T13:21:26.560123+0000","last_unstale":"2024-03-21T13:22:30.288528+0000","last_undegraded":"2024-03-21T13:22:30.288528+0000","last_fullsized":"2024-03-21T13:22:30.288528+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'27","last_scrub_stamp":"2024-03-21T13:21:24.287336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:24.287336+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:46.293226+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":524,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156748+0000","last_change":"2024-03-21T13:21:43.500715+0000","last_active":"2024-03-21T13:22:30.156748+0000","last_peered":"2024-03-21T13:22:30.156748+0000","last_clean":"2024-03-21T13:22:30.156748+0000","last_became_active":"2024-03-21T13:10:03.602752+0000","last_became_peered":"2024-03-21T13:10:03.602752+0000","last_unstale":"2024-03-21T13:22:30.156748+0000","last_undegraded":"2024-03-21T13:22:30.156748+0000","last_fullsized":"2024-03-21T13:22:30.156748+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:43.500597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:21:43.500597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:08.037562+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":101,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286566+0000","last_change":"2024-03-21T13:21:57.500678+0000","last_active":"2024-03-21T13:22:30.286566+0000","last_peered":"2024-03-21T13:22:30.286566+0000","last_clean":"2024-03-21T13:22:30.286566+0000","last_became_active":"2024-03-21T13:20:45.138762+0000","last_became_peered":"2024-03-21T13:20:45.138762+0000","last_unstale":"2024-03-21T13:22:30.286566+0000","last_undegraded":"2024-03-21T13:22:30.286566+0000","last_fullsized":"2024-03-21T13:22:30.286566+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:57.500598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:26.413887+0000","last_clean_scrub_stamp":"2024-03-21T13:21:57.500598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:21.608902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":980,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286973+0000","last_change":"2024-03-21T13:22:25.983421+0000","last_active":"2024-03-21T13:22:30.286973+0000","last_peered":"2024-03-21T13:22:30.286973+0000","last_clean":"2024-03-21T13:22:30.286973+0000","last_became_active":"2024-03-21T13:20:41.150814+0000","last_became_peered":"2024-03-21T13:20:41.150814+0000","last_unstale":"2024-03-21T13:22:30.286973+0000","last_undegraded":"2024-03-21T13:22:30.286973+0000","last_fullsized":"2024-03-21T13:22:30.286973+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:25.983325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:22:25.983325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:47.992486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"435'53","reported_seq":445,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288453+0000","last_change":"2024-03-21T13:21:58.474453+0000","last_active":"2024-03-21T13:22:30.288453+0000","last_peered":"2024-03-21T13:22:30.288453+0000","last_clean":"2024-03-21T13:22:30.288453+0000","last_became_active":"2024-03-21T13:21:26.559525+0000","last_became_peered":"2024-03-21T13:21:26.559525+0000","last_unstale":"2024-03-21T13:22:30.288453+0000","last_undegraded":"2024-03-21T13:22:30.288453+0000","last_fullsized":"2024-03-21T13:22:30.288453+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'53","last_scrub_stamp":"2024-03-21T13:21:58.474348+0000","last_deep_scrub":"391'39","last_deep_scrub_stamp":"2024-03-21T13:16:14.266573+0000","last_clean_scrub_stamp":"2024-03-21T13:21:58.474348+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:15.600281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":58,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":171,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288765+0000","last_change":"2024-03-21T13:21:26.560998+0000","last_active":"2024-03-21T13:22:30.288765+0000","last_peered":"2024-03-21T13:22:30.288765+0000","last_clean":"2024-03-21T13:22:30.288765+0000","last_became_active":"2024-03-21T13:21:26.559301+0000","last_became_peered":"2024-03-21T13:21:26.559301+0000","last_unstale":"2024-03-21T13:22:30.288765+0000","last_undegraded":"2024-03-21T13:22:30.288765+0000","last_fullsized":"2024-03-21T13:22:30.288765+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:07.200214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:21:07.200214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"331'13","reported_seq":673,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157193+0000","last_change":"2024-03-21T13:21:41.267426+0000","last_active":"2024-03-21T13:22:30.157193+0000","last_peered":"2024-03-21T13:22:30.157193+0000","last_clean":"2024-03-21T13:22:30.157193+0000","last_became_active":"2024-03-21T13:12:29.507925+0000","last_became_peered":"2024-03-21T13:12:29.507925+0000","last_unstale":"2024-03-21T13:22:30.157193+0000","last_undegraded":"2024-03-21T13:22:30.157193+0000","last_fullsized":"2024-03-21T13:22:30.157193+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:21:41.267308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:21:41.267308+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:44.982445+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":961,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.998749+0000","last_change":"2024-03-21T13:22:30.998772+0000","last_active":"2024-03-21T13:22:30.998749+0000","last_peered":"2024-03-21T13:22:30.998749+0000","last_clean":"2024-03-21T13:22:30.998749+0000","last_became_active":"2024-03-21T13:20:45.156737+0000","last_became_peered":"2024-03-21T13:20:45.156737+0000","last_unstale":"2024-03-21T13:22:30.998749+0000","last_undegraded":"2024-03-21T13:22:30.998749+0000","last_fullsized":"2024-03-21T13:22:30.998749+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:30.998672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:22:30.998672+0000","last_clean_scrub_stamp":"2024-03-21T13:22:30.998672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:49.392249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"435'30","reported_seq":819,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288206+0000","last_change":"2024-03-21T13:22:02.420671+0000","last_active":"2024-03-21T13:22:30.288206+0000","last_peered":"2024-03-21T13:22:30.288206+0000","last_clean":"2024-03-21T13:22:30.288206+0000","last_became_active":"2024-03-21T13:21:26.555282+0000","last_became_peered":"2024-03-21T13:21:26.555282+0000","last_unstale":"2024-03-21T13:22:30.288206+0000","last_undegraded":"2024-03-21T13:22:30.288206+0000","last_fullsized":"2024-03-21T13:22:30.288206+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'30","last_scrub_stamp":"2024-03-21T13:22:02.420540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:22:02.420540+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:23.114145+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1418,"num_read_kb":1413,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":5},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1520,"num_read_kb":1277,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":11744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1068,"ondisk_log_size":1068,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168868,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1119592,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":92625560,"statfs":{"total":95995035648,"available":94848573440,"internally_reserved":0,"allocated":1409024,"data_stored":988880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1145043402},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463210,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1079316,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":1077950,"kb_avail":92665836,"statfs":{"total":95995035648,"available":94889816064,"internally_reserved":0,"allocated":1388544,"data_stored":988121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1103821254},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575171,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122252,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622900,"statfs":{"total":95995035648,"available":95869849600,"internally_reserved":0,"allocated":851968,"data_stored":404681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738696,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019620,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":1018238,"kb_avail":92725532,"statfs":{"total":95995035648,"available":94950944768,"internally_reserved":0,"allocated":1404928,"data_stored":987665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1042676165},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411646,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":948256,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":92796896,"statfs":{"total":95995035648,"available":95024021504,"internally_reserved":0,"allocated":806912,"data_stored":403997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":970193349},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182713,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1173364,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":1172542,"kb_avail":92571788,"statfs":{"total":95995035648,"available":94793510912,"internally_reserved":0,"allocated":827392,"data_stored":404126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1200683463},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019187,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1172908,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":1172094,"kb_avail":92572244,"statfs":{"total":95995035648,"available":94793977856,"internally_reserved":0,"allocated":819200,"data_stored":404476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1200224709},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724846,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":401232,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93343920,"statfs":{"total":95995035648,"available":95584174080,"internally_reserved":0,"allocated":794624,"data_stored":396842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:22:37.419 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:22:37.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:37 smithi204 ceph-mon[33153]: 5.18 scrub starts 2024-03-21T13:22:37.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:37 smithi204 ceph-mon[33153]: 5.18 scrub ok 2024-03-21T13:22:37.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:37 smithi204 ceph-mon[33153]: from='client.16389 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:22:37.666 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:22:38.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:38 smithi163 ceph-mon[29621]: 6.b scrub starts 2024-03-21T13:22:38.816 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:38 smithi163 ceph-mon[29621]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:38.816 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:38 smithi163 ceph-mon[29621]: 6.b scrub ok 2024-03-21T13:22:38.816 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:38 smithi163 ceph-mon[29621]: 3.19 scrub starts 2024-03-21T13:22:38.816 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:38 smithi163 ceph-mon[29621]: 3.19 scrub ok 2024-03-21T13:22:38.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:38 smithi163 ceph-mon[40675]: 6.b scrub starts 2024-03-21T13:22:38.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:38 smithi163 ceph-mon[40675]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:38.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:38 smithi163 ceph-mon[40675]: 6.b scrub ok 2024-03-21T13:22:38.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:38 smithi163 ceph-mon[40675]: 3.19 scrub starts 2024-03-21T13:22:38.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:38 smithi163 ceph-mon[40675]: 3.19 scrub ok 2024-03-21T13:22:39.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:38 smithi204 ceph-mon[33153]: 6.b scrub starts 2024-03-21T13:22:39.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:38 smithi204 ceph-mon[33153]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:39.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:38 smithi204 ceph-mon[33153]: 6.b scrub ok 2024-03-21T13:22:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:38 smithi204 ceph-mon[33153]: 3.19 scrub starts 2024-03-21T13:22:39.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:38 smithi204 ceph-mon[33153]: 3.19 scrub ok 2024-03-21T13:22:39.593 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-21T13:22:39.596 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-03-21T13:22:39.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:39 smithi163 ceph-mon[29621]: 6.1f scrub starts 2024-03-21T13:22:39.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:39 smithi163 ceph-mon[29621]: 5.9 scrub starts 2024-03-21T13:22:39.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:39 smithi163 ceph-mon[29621]: 5.9 scrub ok 2024-03-21T13:22:39.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:39 smithi163 ceph-mon[29621]: 6.1f scrub ok 2024-03-21T13:22:39.976 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:39 smithi163 ceph-mon[40675]: 6.1f scrub starts 2024-03-21T13:22:39.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:39 smithi163 ceph-mon[40675]: 5.9 scrub starts 2024-03-21T13:22:39.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:39 smithi163 ceph-mon[40675]: 5.9 scrub ok 2024-03-21T13:22:39.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:39 smithi163 ceph-mon[40675]: 6.1f scrub ok 2024-03-21T13:22:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:39 smithi204 ceph-mon[33153]: 6.1f scrub starts 2024-03-21T13:22:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:39 smithi204 ceph-mon[33153]: 5.9 scrub starts 2024-03-21T13:22:40.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:39 smithi204 ceph-mon[33153]: 5.9 scrub ok 2024-03-21T13:22:40.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:39 smithi204 ceph-mon[33153]: 6.1f scrub ok 2024-03-21T13:22:40.413 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":1122,"stamp":"2024-03-21T13:22:39.060573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3179,"num_read_kb":3025,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":602,"num_bytes_recovered":3557588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1323,"ondisk_log_size":1323,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7036580,"kb_used_data":8156,"kb_used_omap":12,"kb_used_meta":7028339,"kb_avail":742924636,"statfs":{"total":767960285184,"available":760754827264,"internally_reserved":0,"allocated":8351744,"data_stored":5005680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":7197019696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002782"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":348,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287407+0000","last_change":"2024-03-21T13:22:16.284984+0000","last_active":"2024-03-21T13:22:30.287407+0000","last_peered":"2024-03-21T13:22:30.287407+0000","last_clean":"2024-03-21T13:22:30.287407+0000","last_became_active":"2024-03-21T13:21:26.559906+0000","last_became_peered":"2024-03-21T13:21:26.559906+0000","last_unstale":"2024-03-21T13:22:30.287407+0000","last_undegraded":"2024-03-21T13:22:30.287407+0000","last_fullsized":"2024-03-21T13:22:30.287407+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:16.284830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:35.694549+0000","last_clean_scrub_stamp":"2024-03-21T13:22:16.284830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:40.235795+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":103,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286495+0000","last_change":"2024-03-21T13:22:27.514499+0000","last_active":"2024-03-21T13:22:30.286495+0000","last_peered":"2024-03-21T13:22:30.286495+0000","last_clean":"2024-03-21T13:22:30.286495+0000","last_became_active":"2024-03-21T13:20:41.135101+0000","last_became_peered":"2024-03-21T13:20:41.135101+0000","last_unstale":"2024-03-21T13:22:30.286495+0000","last_undegraded":"2024-03-21T13:22:30.286495+0000","last_fullsized":"2024-03-21T13:22:30.286495+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:27.514419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:25.553760+0000","last_clean_scrub_stamp":"2024-03-21T13:22:27.514419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:35.990872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":95,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287171+0000","last_change":"2024-03-21T13:22:27.837193+0000","last_active":"2024-03-21T13:22:30.287171+0000","last_peered":"2024-03-21T13:22:30.287171+0000","last_clean":"2024-03-21T13:22:30.287171+0000","last_became_active":"2024-03-21T13:20:45.150641+0000","last_became_peered":"2024-03-21T13:20:45.150641+0000","last_unstale":"2024-03-21T13:22:30.287171+0000","last_undegraded":"2024-03-21T13:22:30.287171+0000","last_fullsized":"2024-03-21T13:22:30.287171+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:27.837082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:00.847701+0000","last_clean_scrub_stamp":"2024-03-21T13:22:27.837082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:54.097156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"435'25","reported_seq":1057,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286233+0000","last_change":"2024-03-21T13:22:13.363690+0000","last_active":"2024-03-21T13:22:30.286233+0000","last_peered":"2024-03-21T13:22:30.286233+0000","last_clean":"2024-03-21T13:22:30.286233+0000","last_became_active":"2024-03-21T13:20:41.127564+0000","last_became_peered":"2024-03-21T13:20:41.127564+0000","last_unstale":"2024-03-21T13:22:30.286233+0000","last_undegraded":"2024-03-21T13:22:30.286233+0000","last_fullsized":"2024-03-21T13:22:30.286233+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:22:13.363563+0000","last_deep_scrub":"234'19","last_deep_scrub_stamp":"2024-03-21T13:16:09.693933+0000","last_clean_scrub_stamp":"2024-03-21T13:22:13.363563+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:21.628251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":930,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157638+0000","last_change":"2024-03-21T13:22:14.048310+0000","last_active":"2024-03-21T13:22:30.157638+0000","last_peered":"2024-03-21T13:22:30.157638+0000","last_clean":"2024-03-21T13:22:30.157638+0000","last_became_active":"2024-03-21T12:58:43.352574+0000","last_became_peered":"2024-03-21T12:58:43.352574+0000","last_unstale":"2024-03-21T13:22:30.157638+0000","last_undegraded":"2024-03-21T13:22:30.157638+0000","last_fullsized":"2024-03-21T13:22:30.157638+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:14.048212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:05.173276+0000","last_clean_scrub_stamp":"2024-03-21T13:22:14.048212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:26.001544+0000","scrub_duration":242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":829,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286201+0000","last_change":"2024-03-21T13:21:26.559803+0000","last_active":"2024-03-21T13:22:30.286201+0000","last_peered":"2024-03-21T13:22:30.286201+0000","last_clean":"2024-03-21T13:22:30.286201+0000","last_became_active":"2024-03-21T13:21:26.559416+0000","last_became_peered":"2024-03-21T13:21:26.559416+0000","last_unstale":"2024-03-21T13:22:30.286201+0000","last_undegraded":"2024-03-21T13:22:30.286201+0000","last_fullsized":"2024-03-21T13:22:30.286201+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:15.094300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:16.731434+0000","last_clean_scrub_stamp":"2024-03-21T13:21:15.094300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:34.891006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"108'2","reported_seq":963,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157629+0000","last_change":"2024-03-21T13:21:29.531921+0000","last_active":"2024-03-21T13:22:30.157629+0000","last_peered":"2024-03-21T13:22:30.157629+0000","last_clean":"2024-03-21T13:22:30.157629+0000","last_became_active":"2024-03-21T13:10:03.610318+0000","last_became_peered":"2024-03-21T13:10:03.610318+0000","last_unstale":"2024-03-21T13:22:30.157629+0000","last_undegraded":"2024-03-21T13:22:30.157629+0000","last_fullsized":"2024-03-21T13:22:30.157629+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-03-21T13:21:29.531808+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-03-21T13:14:11.126531+0000","last_clean_scrub_stamp":"2024-03-21T13:21:29.531808+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:36.382340+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"497'46","reported_seq":531,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156771+0000","last_change":"2024-03-21T13:22:29.193195+0000","last_active":"2024-03-21T13:22:30.156771+0000","last_peered":"2024-03-21T13:22:30.156771+0000","last_clean":"2024-03-21T13:22:30.156771+0000","last_became_active":"2024-03-21T13:20:45.148816+0000","last_became_peered":"2024-03-21T13:20:45.148816+0000","last_unstale":"2024-03-21T13:22:30.156771+0000","last_undegraded":"2024-03-21T13:22:30.156771+0000","last_fullsized":"2024-03-21T13:22:30.156771+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'46","last_scrub_stamp":"2024-03-21T13:22:29.193050+0000","last_deep_scrub":"497'46","last_deep_scrub_stamp":"2024-03-21T13:22:29.193050+0000","last_clean_scrub_stamp":"2024-03-21T13:22:29.193050+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:41.619478+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.285910+0000","last_change":"2024-03-21T13:21:44.864283+0000","last_active":"2024-03-21T13:22:30.285910+0000","last_peered":"2024-03-21T13:22:30.285910+0000","last_clean":"2024-03-21T13:22:30.285910+0000","last_became_active":"2024-03-21T13:20:45.147505+0000","last_became_peered":"2024-03-21T13:20:45.147505+0000","last_unstale":"2024-03-21T13:22:30.285910+0000","last_undegraded":"2024-03-21T13:22:30.285910+0000","last_fullsized":"2024-03-21T13:22:30.285910+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:44.864183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:21:44.864183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:48.863570+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":392,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287013+0000","last_change":"2024-03-21T13:22:07.814634+0000","last_active":"2024-03-21T13:22:30.287013+0000","last_peered":"2024-03-21T13:22:30.287013+0000","last_clean":"2024-03-21T13:22:30.287013+0000","last_became_active":"2024-03-21T13:20:45.154567+0000","last_became_peered":"2024-03-21T13:20:45.154567+0000","last_unstale":"2024-03-21T13:22:30.287013+0000","last_undegraded":"2024-03-21T13:22:30.287013+0000","last_fullsized":"2024-03-21T13:22:30.287013+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:07.814503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:42.076564+0000","last_clean_scrub_stamp":"2024-03-21T13:22:07.814503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.466778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":380,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287576+0000","last_change":"2024-03-21T13:22:01.855100+0000","last_active":"2024-03-21T13:22:30.287576+0000","last_peered":"2024-03-21T13:22:30.287576+0000","last_clean":"2024-03-21T13:22:30.287576+0000","last_became_active":"2024-03-21T13:12:29.526129+0000","last_became_peered":"2024-03-21T13:12:29.526129+0000","last_unstale":"2024-03-21T13:22:30.287576+0000","last_undegraded":"2024-03-21T13:22:30.287576+0000","last_fullsized":"2024-03-21T13:22:30.287576+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:01.854997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:22:01.854997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:10.464213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"435'25","reported_seq":857,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287919+0000","last_change":"2024-03-21T13:22:29.312486+0000","last_active":"2024-03-21T13:22:30.287919+0000","last_peered":"2024-03-21T13:22:30.287919+0000","last_clean":"2024-03-21T13:22:30.287919+0000","last_became_active":"2024-03-21T13:21:26.559349+0000","last_became_peered":"2024-03-21T13:21:26.559349+0000","last_unstale":"2024-03-21T13:22:30.287919+0000","last_undegraded":"2024-03-21T13:22:30.287919+0000","last_fullsized":"2024-03-21T13:22:30.287919+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:22:29.312360+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-21T13:00:35.880161+0000","last_clean_scrub_stamp":"2024-03-21T13:22:29.312360+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:55.471028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":386,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286604+0000","last_change":"2024-03-21T13:21:25.775446+0000","last_active":"2024-03-21T13:22:30.286604+0000","last_peered":"2024-03-21T13:22:30.286604+0000","last_clean":"2024-03-21T13:22:30.286604+0000","last_became_active":"2024-03-21T13:20:41.125208+0000","last_became_peered":"2024-03-21T13:20:41.125208+0000","last_unstale":"2024-03-21T13:22:30.286604+0000","last_undegraded":"2024-03-21T13:22:30.286604+0000","last_fullsized":"2024-03-21T13:22:30.286604+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:25.775270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:21:25.775270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:43.399670+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":966,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156669+0000","last_change":"2024-03-21T13:22:21.130712+0000","last_active":"2024-03-21T13:22:30.156669+0000","last_peered":"2024-03-21T13:22:30.156669+0000","last_clean":"2024-03-21T13:22:30.156669+0000","last_became_active":"2024-03-21T13:20:41.134089+0000","last_became_peered":"2024-03-21T13:20:41.134089+0000","last_unstale":"2024-03-21T13:22:30.156669+0000","last_undegraded":"2024-03-21T13:22:30.156669+0000","last_fullsized":"2024-03-21T13:22:30.156669+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:21.130610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:21.130610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:36.931635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":386,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287064+0000","last_change":"2024-03-21T13:22:04.888314+0000","last_active":"2024-03-21T13:22:30.287064+0000","last_peered":"2024-03-21T13:22:30.287064+0000","last_clean":"2024-03-21T13:22:30.287064+0000","last_became_active":"2024-03-21T13:20:41.124241+0000","last_became_peered":"2024-03-21T13:20:41.124241+0000","last_unstale":"2024-03-21T13:22:30.287064+0000","last_undegraded":"2024-03-21T13:22:30.287064+0000","last_fullsized":"2024-03-21T13:22:30.287064+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:04.888209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:14.479454+0000","last_clean_scrub_stamp":"2024-03-21T13:22:04.888209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:10.438139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"435'11","reported_seq":863,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288075+0000","last_change":"2024-03-21T13:21:36.513241+0000","last_active":"2024-03-21T13:22:30.288075+0000","last_peered":"2024-03-21T13:22:30.288075+0000","last_clean":"2024-03-21T13:22:30.288075+0000","last_became_active":"2024-03-21T13:21:26.558911+0000","last_became_peered":"2024-03-21T13:21:26.558911+0000","last_unstale":"2024-03-21T13:22:30.288075+0000","last_undegraded":"2024-03-21T13:22:30.288075+0000","last_fullsized":"2024-03-21T13:22:30.288075+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'11","last_scrub_stamp":"2024-03-21T13:21:36.513128+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-21T12:59:53.952401+0000","last_clean_scrub_stamp":"2024-03-21T13:21:36.513128+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:42.353169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":985,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288632+0000","last_change":"2024-03-21T13:21:17.619269+0000","last_active":"2024-03-21T13:22:30.288632+0000","last_peered":"2024-03-21T13:22:30.288632+0000","last_clean":"2024-03-21T13:22:30.288632+0000","last_became_active":"2024-03-21T13:12:29.520189+0000","last_became_peered":"2024-03-21T13:12:29.520189+0000","last_unstale":"2024-03-21T13:22:30.288632+0000","last_undegraded":"2024-03-21T13:22:30.288632+0000","last_fullsized":"2024-03-21T13:22:30.288632+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:17.619101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:12:23.865785+0000","last_clean_scrub_stamp":"2024-03-21T13:21:17.619101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:40.350730+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":101,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287004+0000","last_change":"2024-03-21T13:21:25.288789+0000","last_active":"2024-03-21T13:22:30.287004+0000","last_peered":"2024-03-21T13:22:30.287004+0000","last_clean":"2024-03-21T13:22:30.287004+0000","last_became_active":"2024-03-21T13:20:45.138816+0000","last_became_peered":"2024-03-21T13:20:45.138816+0000","last_unstale":"2024-03-21T13:22:30.287004+0000","last_undegraded":"2024-03-21T13:22:30.287004+0000","last_fullsized":"2024-03-21T13:22:30.287004+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:25.288709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:55.014453+0000","last_clean_scrub_stamp":"2024-03-21T13:21:25.288709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:49.499605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286318+0000","last_change":"2024-03-21T13:21:59.818575+0000","last_active":"2024-03-21T13:22:30.286318+0000","last_peered":"2024-03-21T13:22:30.286318+0000","last_clean":"2024-03-21T13:22:30.286318+0000","last_became_active":"2024-03-21T13:20:45.148149+0000","last_became_peered":"2024-03-21T13:20:45.148149+0000","last_unstale":"2024-03-21T13:22:30.286318+0000","last_undegraded":"2024-03-21T13:22:30.286318+0000","last_fullsized":"2024-03-21T13:22:30.286318+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:59.818496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:59.818496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:00.826733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"435'39","reported_seq":586,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156523+0000","last_change":"2024-03-21T13:21:52.353367+0000","last_active":"2024-03-21T13:22:30.156523+0000","last_peered":"2024-03-21T13:22:30.156523+0000","last_clean":"2024-03-21T13:22:30.156523+0000","last_became_active":"2024-03-21T13:20:41.128375+0000","last_became_peered":"2024-03-21T13:20:41.128375+0000","last_unstale":"2024-03-21T13:22:30.156523+0000","last_undegraded":"2024-03-21T13:22:30.156523+0000","last_fullsized":"2024-03-21T13:22:30.156523+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'39","last_scrub_stamp":"2024-03-21T13:21:52.353258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:52.353258+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:17.445009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":948,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.158078+0000","last_change":"2024-03-21T13:21:02.992750+0000","last_active":"2024-03-21T13:22:30.158078+0000","last_peered":"2024-03-21T13:22:30.158078+0000","last_clean":"2024-03-21T13:22:30.158078+0000","last_became_active":"2024-03-21T13:12:29.502808+0000","last_became_peered":"2024-03-21T13:12:29.502808+0000","last_unstale":"2024-03-21T13:22:30.158078+0000","last_undegraded":"2024-03-21T13:22:30.158078+0000","last_fullsized":"2024-03-21T13:22:30.158078+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:02.992650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:21:02.992650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":414,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288270+0000","last_change":"2024-03-21T13:22:06.311857+0000","last_active":"2024-03-21T13:22:30.288270+0000","last_peered":"2024-03-21T13:22:30.288270+0000","last_clean":"2024-03-21T13:22:30.288270+0000","last_became_active":"2024-03-21T13:21:26.554589+0000","last_became_peered":"2024-03-21T13:21:26.554589+0000","last_unstale":"2024-03-21T13:22:30.288270+0000","last_undegraded":"2024-03-21T13:22:30.288270+0000","last_fullsized":"2024-03-21T13:22:30.288270+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:06.311751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:06.311751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:19.011246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1013,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288061+0000","last_change":"2024-03-21T13:21:49.350135+0000","last_active":"2024-03-21T13:22:30.288061+0000","last_peered":"2024-03-21T13:22:30.288061+0000","last_clean":"2024-03-21T13:22:30.288061+0000","last_became_active":"2024-03-21T13:20:41.151350+0000","last_became_peered":"2024-03-21T13:20:41.151350+0000","last_unstale":"2024-03-21T13:22:30.288061+0000","last_undegraded":"2024-03-21T13:22:30.288061+0000","last_fullsized":"2024-03-21T13:22:30.288061+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:49.350031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:49.350031+0000","last_clean_scrub_stamp":"2024-03-21T13:21:49.350031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:58.597466+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"435'24","reported_seq":1025,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157978+0000","last_change":"2024-03-21T13:21:28.550540+0000","last_active":"2024-03-21T13:22:30.157978+0000","last_peered":"2024-03-21T13:22:30.157978+0000","last_clean":"2024-03-21T13:22:30.157978+0000","last_became_active":"2024-03-21T13:20:41.131391+0000","last_became_peered":"2024-03-21T13:20:41.131391+0000","last_unstale":"2024-03-21T13:22:30.157978+0000","last_undegraded":"2024-03-21T13:22:30.157978+0000","last_fullsized":"2024-03-21T13:22:30.157978+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'24","last_scrub_stamp":"2024-03-21T13:21:28.550447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:28.550447+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:43.585108+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":359,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156830+0000","last_change":"2024-03-21T13:22:23.173236+0000","last_active":"2024-03-21T13:22:30.156830+0000","last_peered":"2024-03-21T13:22:30.156830+0000","last_clean":"2024-03-21T13:22:30.156830+0000","last_became_active":"2024-03-21T13:13:06.420220+0000","last_became_peered":"2024-03-21T13:13:06.420220+0000","last_unstale":"2024-03-21T13:22:30.156830+0000","last_undegraded":"2024-03-21T13:22:30.156830+0000","last_fullsized":"2024-03-21T13:22:30.156830+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-21T13:22:23.173123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:22:23.173123+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.652307+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"491'9","reported_seq":1244,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286614+0000","last_change":"2024-03-21T13:21:39.207649+0000","last_active":"2024-03-21T13:22:30.286614+0000","last_peered":"2024-03-21T13:22:30.286614+0000","last_clean":"2024-03-21T13:22:30.286614+0000","last_became_active":"2024-03-21T12:58:41.166818+0000","last_became_peered":"2024-03-21T12:58:41.166818+0000","last_unstale":"2024-03-21T13:22:30.286614+0000","last_undegraded":"2024-03-21T13:22:30.286614+0000","last_fullsized":"2024-03-21T13:22:30.286614+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'9","last_scrub_stamp":"2024-03-21T13:21:39.207496+0000","last_deep_scrub":"491'9","last_deep_scrub_stamp":"2024-03-21T13:21:39.207496+0000","last_clean_scrub_stamp":"2024-03-21T13:21:39.207496+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:57.718125+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"407'2","reported_seq":99,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286397+0000","last_change":"2024-03-21T13:21:12.214314+0000","last_active":"2024-03-21T13:22:30.286397+0000","last_peered":"2024-03-21T13:22:30.286397+0000","last_clean":"2024-03-21T13:22:30.286397+0000","last_became_active":"2024-03-21T13:20:41.131485+0000","last_became_peered":"2024-03-21T13:20:41.131485+0000","last_unstale":"2024-03-21T13:22:30.286397+0000","last_undegraded":"2024-03-21T13:22:30.286397+0000","last_fullsized":"2024-03-21T13:22:30.286397+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'2","last_scrub_stamp":"2024-03-21T13:21:12.214180+0000","last_deep_scrub":"407'2","last_deep_scrub_stamp":"2024-03-21T13:16:58.735421+0000","last_clean_scrub_stamp":"2024-03-21T13:21:12.214180+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:33.592257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"435'35","reported_seq":107,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287164+0000","last_change":"2024-03-21T13:22:26.880768+0000","last_active":"2024-03-21T13:22:30.287164+0000","last_peered":"2024-03-21T13:22:30.287164+0000","last_clean":"2024-03-21T13:22:30.287164+0000","last_became_active":"2024-03-21T13:20:45.156727+0000","last_became_peered":"2024-03-21T13:20:45.156727+0000","last_unstale":"2024-03-21T13:22:30.287164+0000","last_undegraded":"2024-03-21T13:22:30.287164+0000","last_fullsized":"2024-03-21T13:22:30.287164+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'35","last_scrub_stamp":"2024-03-21T13:22:26.880633+0000","last_deep_scrub":"410'25","last_deep_scrub_stamp":"2024-03-21T13:17:09.876547+0000","last_clean_scrub_stamp":"2024-03-21T13:22:26.880633+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:56.742987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":103,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:31.545876+0000","last_change":"2024-03-21T13:22:31.545916+0000","last_active":"2024-03-21T13:22:31.545876+0000","last_peered":"2024-03-21T13:22:31.545876+0000","last_clean":"2024-03-21T13:22:31.545876+0000","last_became_active":"2024-03-21T13:20:41.141076+0000","last_became_peered":"2024-03-21T13:20:41.141076+0000","last_unstale":"2024-03-21T13:22:31.545876+0000","last_undegraded":"2024-03-21T13:22:31.545876+0000","last_fullsized":"2024-03-21T13:22:31.545876+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:31.545751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:22:31.545751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:24:00.930573+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":95,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:33.863715+0000","last_change":"2024-03-21T13:22:33.863737+0000","last_active":"2024-03-21T13:22:33.863715+0000","last_peered":"2024-03-21T13:22:33.863715+0000","last_clean":"2024-03-21T13:22:33.863715+0000","last_became_active":"2024-03-21T13:20:45.155958+0000","last_became_peered":"2024-03-21T13:20:45.155958+0000","last_unstale":"2024-03-21T13:22:33.863715+0000","last_undegraded":"2024-03-21T13:22:33.863715+0000","last_fullsized":"2024-03-21T13:22:33.863715+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:33.863649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:55.922856+0000","last_clean_scrub_stamp":"2024-03-21T13:22:33.863649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:42.095973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":978,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.155612+0000","last_change":"2024-03-21T13:21:40.528839+0000","last_active":"2024-03-21T13:22:30.155612+0000","last_peered":"2024-03-21T13:22:30.155612+0000","last_clean":"2024-03-21T13:22:30.155612+0000","last_became_active":"2024-03-21T13:20:45.140132+0000","last_became_peered":"2024-03-21T13:20:45.140132+0000","last_unstale":"2024-03-21T13:22:30.155612+0000","last_undegraded":"2024-03-21T13:22:30.155612+0000","last_fullsized":"2024-03-21T13:22:30.155612+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:40.528728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:40.528728+0000","last_clean_scrub_stamp":"2024-03-21T13:21:40.528728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:51.544803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"435'32","reported_seq":1086,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288495+0000","last_change":"2024-03-21T13:21:56.036242+0000","last_active":"2024-03-21T13:22:30.288495+0000","last_peered":"2024-03-21T13:22:30.288495+0000","last_clean":"2024-03-21T13:22:30.288495+0000","last_became_active":"2024-03-21T13:20:41.151427+0000","last_became_peered":"2024-03-21T13:20:41.151427+0000","last_unstale":"2024-03-21T13:22:30.288495+0000","last_undegraded":"2024-03-21T13:22:30.288495+0000","last_fullsized":"2024-03-21T13:22:30.288495+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'32","last_scrub_stamp":"2024-03-21T13:21:56.036110+0000","last_deep_scrub":"435'32","last_deep_scrub_stamp":"2024-03-21T13:20:23.715513+0000","last_clean_scrub_stamp":"2024-03-21T13:21:56.036110+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:19.226447+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":193,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156485+0000","last_change":"2024-03-21T13:21:59.235590+0000","last_active":"2024-03-21T13:22:30.156485+0000","last_peered":"2024-03-21T13:22:30.156485+0000","last_clean":"2024-03-21T13:22:30.156485+0000","last_became_active":"2024-03-21T13:17:27.636917+0000","last_became_peered":"2024-03-21T13:17:27.636917+0000","last_unstale":"2024-03-21T13:22:30.156485+0000","last_undegraded":"2024-03-21T13:22:30.156485+0000","last_fullsized":"2024-03-21T13:22:30.156485+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:59.235488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:59.235488+0000","last_clean_scrub_stamp":"2024-03-21T13:21:59.235488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:01.098266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":951,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288812+0000","last_change":"2024-03-21T13:22:08.910813+0000","last_active":"2024-03-21T13:22:30.288812+0000","last_peered":"2024-03-21T13:22:30.288812+0000","last_clean":"2024-03-21T13:22:30.288812+0000","last_became_active":"2024-03-21T12:58:41.166899+0000","last_became_peered":"2024-03-21T12:58:41.166899+0000","last_unstale":"2024-03-21T13:22:30.288812+0000","last_undegraded":"2024-03-21T13:22:30.288812+0000","last_fullsized":"2024-03-21T13:22:30.288812+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:08.910711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:08.910711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:34.881831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287176+0000","last_change":"2024-03-21T13:21:50.802429+0000","last_active":"2024-03-21T13:22:30.287176+0000","last_peered":"2024-03-21T13:22:30.287176+0000","last_clean":"2024-03-21T13:22:30.287176+0000","last_became_active":"2024-03-21T13:20:45.153702+0000","last_became_peered":"2024-03-21T13:20:45.153702+0000","last_unstale":"2024-03-21T13:22:30.287176+0000","last_undegraded":"2024-03-21T13:22:30.287176+0000","last_fullsized":"2024-03-21T13:22:30.287176+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:50.802353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:31.586656+0000","last_clean_scrub_stamp":"2024-03-21T13:21:50.802353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:53.455907+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"435'12","reported_seq":973,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.155630+0000","last_change":"2024-03-21T13:21:20.577329+0000","last_active":"2024-03-21T13:22:30.155630+0000","last_peered":"2024-03-21T13:22:30.155630+0000","last_clean":"2024-03-21T13:22:30.155630+0000","last_became_active":"2024-03-21T13:20:41.129502+0000","last_became_peered":"2024-03-21T13:20:41.129502+0000","last_unstale":"2024-03-21T13:22:30.155630+0000","last_undegraded":"2024-03-21T13:22:30.155630+0000","last_fullsized":"2024-03-21T13:22:30.155630+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'12","last_scrub_stamp":"2024-03-21T13:21:20.577209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:20.577209+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:46.657785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":943,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157743+0000","last_change":"2024-03-21T13:21:58.939968+0000","last_active":"2024-03-21T13:22:30.157743+0000","last_peered":"2024-03-21T13:22:30.157743+0000","last_clean":"2024-03-21T13:22:30.157743+0000","last_became_active":"2024-03-21T13:20:45.152596+0000","last_became_peered":"2024-03-21T13:20:45.152596+0000","last_unstale":"2024-03-21T13:22:30.157743+0000","last_undegraded":"2024-03-21T13:22:30.157743+0000","last_fullsized":"2024-03-21T13:22:30.157743+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:58.939868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:58.939868+0000","last_clean_scrub_stamp":"2024-03-21T13:21:58.939868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:14.382752+0000","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":193,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156636+0000","last_change":"2024-03-21T13:21:44.630267+0000","last_active":"2024-03-21T13:22:30.156636+0000","last_peered":"2024-03-21T13:22:30.156636+0000","last_clean":"2024-03-21T13:22:30.156636+0000","last_became_active":"2024-03-21T13:17:27.644954+0000","last_became_peered":"2024-03-21T13:17:27.644954+0000","last_unstale":"2024-03-21T13:22:30.156636+0000","last_undegraded":"2024-03-21T13:22:30.156636+0000","last_fullsized":"2024-03-21T13:22:30.156636+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:44.630167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:44.630167+0000","last_clean_scrub_stamp":"2024-03-21T13:21:44.630167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:49.443582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":394,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286928+0000","last_change":"2024-03-21T13:22:14.760359+0000","last_active":"2024-03-21T13:22:30.286928+0000","last_peered":"2024-03-21T13:22:30.286928+0000","last_clean":"2024-03-21T13:22:30.286928+0000","last_became_active":"2024-03-21T13:20:41.125565+0000","last_became_peered":"2024-03-21T13:20:41.125565+0000","last_unstale":"2024-03-21T13:22:30.286928+0000","last_undegraded":"2024-03-21T13:22:30.286928+0000","last_fullsized":"2024-03-21T13:22:30.286928+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:14.760285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:55.653437+0000","last_clean_scrub_stamp":"2024-03-21T13:22:14.760285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:16.625118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"472'32","reported_seq":701,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157330+0000","last_change":"2024-03-21T13:22:25.891852+0000","last_active":"2024-03-21T13:22:30.157330+0000","last_peered":"2024-03-21T13:22:30.157330+0000","last_clean":"2024-03-21T13:22:30.157330+0000","last_became_active":"2024-03-21T13:06:29.484776+0000","last_became_peered":"2024-03-21T13:06:29.484776+0000","last_unstale":"2024-03-21T13:22:30.157330+0000","last_undegraded":"2024-03-21T13:22:30.157330+0000","last_fullsized":"2024-03-21T13:22:30.157330+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'32","last_scrub_stamp":"2024-03-21T13:22:25.891762+0000","last_deep_scrub":"472'32","last_deep_scrub_stamp":"2024-03-21T13:20:57.799687+0000","last_clean_scrub_stamp":"2024-03-21T13:22:25.891762+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:43.955333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.285729+0000","last_change":"2024-03-21T13:21:45.904441+0000","last_active":"2024-03-21T13:22:30.285729+0000","last_peered":"2024-03-21T13:22:30.285729+0000","last_clean":"2024-03-21T13:22:30.285729+0000","last_became_active":"2024-03-21T13:20:45.151537+0000","last_became_peered":"2024-03-21T13:20:45.151537+0000","last_unstale":"2024-03-21T13:22:30.285729+0000","last_undegraded":"2024-03-21T13:22:30.285729+0000","last_fullsized":"2024-03-21T13:22:30.285729+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:45.904340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:21:45.904340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:10.706182+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":971,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:31.952753+0000","last_change":"2024-03-21T13:22:31.952773+0000","last_active":"2024-03-21T13:22:31.952753+0000","last_peered":"2024-03-21T13:22:31.952753+0000","last_clean":"2024-03-21T13:22:31.952753+0000","last_became_active":"2024-03-21T13:20:45.151554+0000","last_became_peered":"2024-03-21T13:20:45.151554+0000","last_unstale":"2024-03-21T13:22:31.952753+0000","last_undegraded":"2024-03-21T13:22:31.952753+0000","last_fullsized":"2024-03-21T13:22:31.952753+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:31.952693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:31.952693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:52.689063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":940,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156651+0000","last_change":"2024-03-21T13:21:24.621650+0000","last_active":"2024-03-21T13:22:30.156651+0000","last_peered":"2024-03-21T13:22:30.156651+0000","last_clean":"2024-03-21T13:22:30.156651+0000","last_became_active":"2024-03-21T12:58:37.733623+0000","last_became_peered":"2024-03-21T12:58:37.733623+0000","last_unstale":"2024-03-21T13:22:30.156651+0000","last_undegraded":"2024-03-21T13:22:30.156651+0000","last_fullsized":"2024-03-21T13:22:30.156651+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:24.621542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:19.383090+0000","last_clean_scrub_stamp":"2024-03-21T13:21:24.621542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:43.951316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"435'25","reported_seq":113,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286725+0000","last_change":"2024-03-21T13:21:55.509191+0000","last_active":"2024-03-21T13:22:30.286725+0000","last_peered":"2024-03-21T13:22:30.286725+0000","last_clean":"2024-03-21T13:22:30.286725+0000","last_became_active":"2024-03-21T13:20:41.145892+0000","last_became_peered":"2024-03-21T13:20:41.145892+0000","last_unstale":"2024-03-21T13:22:30.286725+0000","last_undegraded":"2024-03-21T13:22:30.286725+0000","last_fullsized":"2024-03-21T13:22:30.286725+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:21:55.509099+0000","last_deep_scrub":"435'25","last_deep_scrub_stamp":"2024-03-21T13:21:55.509099+0000","last_clean_scrub_stamp":"2024-03-21T13:21:55.509099+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:11.321601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":776,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156287+0000","last_change":"2024-03-21T13:22:26.208740+0000","last_active":"2024-03-21T13:22:30.156287+0000","last_peered":"2024-03-21T13:22:30.156287+0000","last_clean":"2024-03-21T13:22:30.156287+0000","last_became_active":"2024-03-21T13:03:37.387187+0000","last_became_peered":"2024-03-21T13:03:37.387187+0000","last_unstale":"2024-03-21T13:22:30.156287+0000","last_undegraded":"2024-03-21T13:22:30.156287+0000","last_fullsized":"2024-03-21T13:22:30.156287+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:26.208625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:22:26.208625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:31.273307+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":95,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286787+0000","last_change":"2024-03-21T13:21:43.350151+0000","last_active":"2024-03-21T13:22:30.286787+0000","last_peered":"2024-03-21T13:22:30.286787+0000","last_clean":"2024-03-21T13:22:30.286787+0000","last_became_active":"2024-03-21T13:20:41.134901+0000","last_became_peered":"2024-03-21T13:20:41.134901+0000","last_unstale":"2024-03-21T13:22:30.286787+0000","last_undegraded":"2024-03-21T13:22:30.286787+0000","last_fullsized":"2024-03-21T13:22:30.286787+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:43.350077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:22.501055+0000","last_clean_scrub_stamp":"2024-03-21T13:21:43.350077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:43.688359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":348,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288065+0000","last_change":"2024-03-21T13:21:33.537275+0000","last_active":"2024-03-21T13:22:30.288065+0000","last_peered":"2024-03-21T13:22:30.288065+0000","last_clean":"2024-03-21T13:22:30.288065+0000","last_became_active":"2024-03-21T13:21:26.636308+0000","last_became_peered":"2024-03-21T13:21:26.636308+0000","last_unstale":"2024-03-21T13:22:30.288065+0000","last_undegraded":"2024-03-21T13:22:30.288065+0000","last_fullsized":"2024-03-21T13:22:30.288065+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:33.537174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:45.833304+0000","last_clean_scrub_stamp":"2024-03-21T13:21:33.537174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:43.740213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"498'32","reported_seq":117,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:32.906292+0000","last_change":"2024-03-21T13:22:32.906329+0000","last_active":"2024-03-21T13:22:32.906292+0000","last_peered":"2024-03-21T13:22:32.906292+0000","last_clean":"2024-03-21T13:22:32.906292+0000","last_became_active":"2024-03-21T13:20:45.157398+0000","last_became_peered":"2024-03-21T13:20:45.157398+0000","last_unstale":"2024-03-21T13:22:32.906292+0000","last_undegraded":"2024-03-21T13:22:32.906292+0000","last_fullsized":"2024-03-21T13:22:32.906292+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'32","last_scrub_stamp":"2024-03-21T13:22:32.906236+0000","last_deep_scrub":"498'32","last_deep_scrub_stamp":"2024-03-21T13:22:32.906236+0000","last_clean_scrub_stamp":"2024-03-21T13:22:32.906236+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:52.410830+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287080+0000","last_change":"2024-03-21T13:21:42.942470+0000","last_active":"2024-03-21T13:22:30.287080+0000","last_peered":"2024-03-21T13:22:30.287080+0000","last_clean":"2024-03-21T13:22:30.287080+0000","last_became_active":"2024-03-21T13:20:45.157291+0000","last_became_peered":"2024-03-21T13:20:45.157291+0000","last_unstale":"2024-03-21T13:22:30.287080+0000","last_undegraded":"2024-03-21T13:22:30.287080+0000","last_fullsized":"2024-03-21T13:22:30.287080+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:42.942345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:21:42.942345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:01.589382+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"467'15","reported_seq":546,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:34.690866+0000","last_change":"2024-03-21T13:22:34.691169+0000","last_active":"2024-03-21T13:22:34.690866+0000","last_peered":"2024-03-21T13:22:34.690866+0000","last_clean":"2024-03-21T13:22:34.690866+0000","last_became_active":"2024-03-21T13:20:45.139449+0000","last_became_peered":"2024-03-21T13:20:45.139449+0000","last_unstale":"2024-03-21T13:22:34.690866+0000","last_undegraded":"2024-03-21T13:22:34.690866+0000","last_fullsized":"2024-03-21T13:22:34.690866+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'15","last_scrub_stamp":"2024-03-21T13:22:34.690770+0000","last_deep_scrub":"394'12","last_deep_scrub_stamp":"2024-03-21T13:16:24.237800+0000","last_clean_scrub_stamp":"2024-03-21T13:22:34.690770+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:40.298810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":961,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156772+0000","last_change":"2024-03-21T13:21:58.251095+0000","last_active":"2024-03-21T13:22:30.156772+0000","last_peered":"2024-03-21T13:22:30.156772+0000","last_clean":"2024-03-21T13:22:30.156772+0000","last_became_active":"2024-03-21T13:10:03.599265+0000","last_became_peered":"2024-03-21T13:10:03.599265+0000","last_unstale":"2024-03-21T13:22:30.156772+0000","last_undegraded":"2024-03-21T13:22:30.156772+0000","last_fullsized":"2024-03-21T13:22:30.156772+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:58.250996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:58.250996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:22.422647+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"467'35","reported_seq":1061,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157357+0000","last_change":"2024-03-21T13:22:03.763176+0000","last_active":"2024-03-21T13:22:30.157357+0000","last_peered":"2024-03-21T13:22:30.157357+0000","last_clean":"2024-03-21T13:22:30.157357+0000","last_became_active":"2024-03-21T13:20:41.131638+0000","last_became_peered":"2024-03-21T13:20:41.131638+0000","last_unstale":"2024-03-21T13:22:30.157357+0000","last_undegraded":"2024-03-21T13:22:30.157357+0000","last_fullsized":"2024-03-21T13:22:30.157357+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'35","last_scrub_stamp":"2024-03-21T13:22:03.763081+0000","last_deep_scrub":"467'35","last_deep_scrub_stamp":"2024-03-21T13:22:03.763081+0000","last_clean_scrub_stamp":"2024-03-21T13:22:03.763081+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.671429+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":348,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288663+0000","last_change":"2024-03-21T13:22:14.541903+0000","last_active":"2024-03-21T13:22:30.288663+0000","last_peered":"2024-03-21T13:22:30.288663+0000","last_clean":"2024-03-21T13:22:30.288663+0000","last_became_active":"2024-03-21T13:21:26.556903+0000","last_became_peered":"2024-03-21T13:21:26.556903+0000","last_unstale":"2024-03-21T13:22:30.288663+0000","last_undegraded":"2024-03-21T13:22:30.288663+0000","last_fullsized":"2024-03-21T13:22:30.288663+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:14.541787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:22:14.541787+0000","last_clean_scrub_stamp":"2024-03-21T13:22:14.541787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:28.646515+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"498'11","reported_seq":1253,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:33.927492+0000","last_change":"2024-03-21T13:22:33.927540+0000","last_active":"2024-03-21T13:22:33.927492+0000","last_peered":"2024-03-21T13:22:33.927492+0000","last_clean":"2024-03-21T13:22:33.927492+0000","last_became_active":"2024-03-21T13:10:03.594651+0000","last_became_peered":"2024-03-21T13:10:03.594651+0000","last_unstale":"2024-03-21T13:22:33.927492+0000","last_undegraded":"2024-03-21T13:22:33.927492+0000","last_fullsized":"2024-03-21T13:22:33.927492+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'11","last_scrub_stamp":"2024-03-21T13:22:33.927396+0000","last_deep_scrub":"498'11","last_deep_scrub_stamp":"2024-03-21T13:22:33.927396+0000","last_clean_scrub_stamp":"2024-03-21T13:22:33.927396+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:37.415265+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.285961+0000","last_change":"2024-03-21T13:21:10.826204+0000","last_active":"2024-03-21T13:22:30.285961+0000","last_peered":"2024-03-21T13:22:30.285961+0000","last_clean":"2024-03-21T13:22:30.285961+0000","last_became_active":"2024-03-21T13:20:45.146823+0000","last_became_peered":"2024-03-21T13:20:45.146823+0000","last_unstale":"2024-03-21T13:22:30.285961+0000","last_undegraded":"2024-03-21T13:22:30.285961+0000","last_fullsized":"2024-03-21T13:22:30.285961+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:10.826125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:21.365173+0000","last_clean_scrub_stamp":"2024-03-21T13:21:10.826125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"435'21","reported_seq":399,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287277+0000","last_change":"2024-03-21T13:21:33.792618+0000","last_active":"2024-03-21T13:22:30.287277+0000","last_peered":"2024-03-21T13:22:30.287277+0000","last_clean":"2024-03-21T13:22:30.287277+0000","last_became_active":"2024-03-21T13:12:29.510247+0000","last_became_peered":"2024-03-21T13:12:29.510247+0000","last_unstale":"2024-03-21T13:22:30.287277+0000","last_undegraded":"2024-03-21T13:22:30.287277+0000","last_fullsized":"2024-03-21T13:22:30.287277+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'21","last_scrub_stamp":"2024-03-21T13:21:33.792448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:33.792448+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:39.318165+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":508,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156330+0000","last_change":"2024-03-21T13:21:42.273889+0000","last_active":"2024-03-21T13:22:30.156330+0000","last_peered":"2024-03-21T13:22:30.156330+0000","last_clean":"2024-03-21T13:22:30.156330+0000","last_became_active":"2024-03-21T13:10:03.613841+0000","last_became_peered":"2024-03-21T13:10:03.613841+0000","last_unstale":"2024-03-21T13:22:30.156330+0000","last_undegraded":"2024-03-21T13:22:30.156330+0000","last_fullsized":"2024-03-21T13:22:30.156330+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:42.273796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:51.730413+0000","last_clean_scrub_stamp":"2024-03-21T13:21:42.273796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:10.738603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":952,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.158339+0000","last_change":"2024-03-21T13:21:54.671978+0000","last_active":"2024-03-21T13:22:30.158339+0000","last_peered":"2024-03-21T13:22:30.158339+0000","last_clean":"2024-03-21T13:22:30.158339+0000","last_became_active":"2024-03-21T13:20:41.148420+0000","last_became_peered":"2024-03-21T13:20:41.148420+0000","last_unstale":"2024-03-21T13:22:30.158339+0000","last_undegraded":"2024-03-21T13:22:30.158339+0000","last_fullsized":"2024-03-21T13:22:30.158339+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:54.671834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:13.018536+0000","last_clean_scrub_stamp":"2024-03-21T13:21:54.671834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:08.190497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"64'1","reported_seq":955,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156715+0000","last_change":"2024-03-21T13:21:53.323165+0000","last_active":"2024-03-21T13:22:30.156715+0000","last_peered":"2024-03-21T13:22:30.156715+0000","last_clean":"2024-03-21T13:22:30.156715+0000","last_became_active":"2024-03-21T12:58:37.735223+0000","last_became_peered":"2024-03-21T12:58:37.735223+0000","last_unstale":"2024-03-21T13:22:30.156715+0000","last_undegraded":"2024-03-21T13:22:30.156715+0000","last_fullsized":"2024-03-21T13:22:30.156715+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:21:53.323039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:53.323039+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:20.019192+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"457'25","reported_seq":97,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287264+0000","last_change":"2024-03-21T13:21:41.901286+0000","last_active":"2024-03-21T13:22:30.287264+0000","last_peered":"2024-03-21T13:22:30.287264+0000","last_clean":"2024-03-21T13:22:30.287264+0000","last_became_active":"2024-03-21T13:20:45.156195+0000","last_became_peered":"2024-03-21T13:20:45.156195+0000","last_unstale":"2024-03-21T13:22:30.287264+0000","last_undegraded":"2024-03-21T13:22:30.287264+0000","last_fullsized":"2024-03-21T13:22:30.287264+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'25","last_scrub_stamp":"2024-03-21T13:21:41.901193+0000","last_deep_scrub":"457'25","last_deep_scrub_stamp":"2024-03-21T13:20:25.925078+0000","last_clean_scrub_stamp":"2024-03-21T13:21:41.901193+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:00.074477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":928,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156552+0000","last_change":"2024-03-21T13:21:16.763214+0000","last_active":"2024-03-21T13:22:30.156552+0000","last_peered":"2024-03-21T13:22:30.156552+0000","last_clean":"2024-03-21T13:22:30.156552+0000","last_became_active":"2024-03-21T12:58:43.353126+0000","last_became_peered":"2024-03-21T12:58:43.353126+0000","last_unstale":"2024-03-21T13:22:30.156552+0000","last_undegraded":"2024-03-21T13:22:30.156552+0000","last_fullsized":"2024-03-21T13:22:30.156552+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:16.763105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:16.763105+0000","last_clean_scrub_stamp":"2024-03-21T13:21:16.763105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:43.686957+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":394,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286806+0000","last_change":"2024-03-21T13:22:21.667141+0000","last_active":"2024-03-21T13:22:30.286806+0000","last_peered":"2024-03-21T13:22:30.286806+0000","last_clean":"2024-03-21T13:22:30.286806+0000","last_became_active":"2024-03-21T13:20:41.124729+0000","last_became_peered":"2024-03-21T13:20:41.124729+0000","last_unstale":"2024-03-21T13:22:30.286806+0000","last_undegraded":"2024-03-21T13:22:30.286806+0000","last_fullsized":"2024-03-21T13:22:30.286806+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:21.667041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:21.667041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:35.502312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":975,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286102+0000","last_change":"2024-03-21T13:21:20.170494+0000","last_active":"2024-03-21T13:22:30.286102+0000","last_peered":"2024-03-21T13:22:30.286102+0000","last_clean":"2024-03-21T13:22:30.286102+0000","last_became_active":"2024-03-21T13:20:45.138634+0000","last_became_peered":"2024-03-21T13:20:45.138634+0000","last_unstale":"2024-03-21T13:22:30.286102+0000","last_undegraded":"2024-03-21T13:22:30.286102+0000","last_fullsized":"2024-03-21T13:22:30.286102+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:20.170392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:20.170392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:45.349703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"435'18","reported_seq":1035,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.158241+0000","last_change":"2024-03-21T13:21:59.728224+0000","last_active":"2024-03-21T13:22:30.158241+0000","last_peered":"2024-03-21T13:22:30.158241+0000","last_clean":"2024-03-21T13:22:30.158241+0000","last_became_active":"2024-03-21T13:21:22.555111+0000","last_became_peered":"2024-03-21T13:21:22.555111+0000","last_unstale":"2024-03-21T13:22:30.158241+0000","last_undegraded":"2024-03-21T13:22:30.158241+0000","last_fullsized":"2024-03-21T13:22:30.158241+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'18","last_scrub_stamp":"2024-03-21T13:21:59.728097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:59.728097+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:22.100052+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":392,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287107+0000","last_change":"2024-03-21T13:22:03.878364+0000","last_active":"2024-03-21T13:22:30.287107+0000","last_peered":"2024-03-21T13:22:30.287107+0000","last_clean":"2024-03-21T13:22:30.287107+0000","last_became_active":"2024-03-21T13:20:45.140194+0000","last_became_peered":"2024-03-21T13:20:45.140194+0000","last_unstale":"2024-03-21T13:22:30.287107+0000","last_undegraded":"2024-03-21T13:22:30.287107+0000","last_fullsized":"2024-03-21T13:22:30.287107+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:03.878263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:43.658918+0000","last_clean_scrub_stamp":"2024-03-21T13:22:03.878263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:26.043529+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'2","reported_seq":1021,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157243+0000","last_change":"2024-03-21T13:21:55.312655+0000","last_active":"2024-03-21T13:22:30.157243+0000","last_peered":"2024-03-21T13:22:30.157243+0000","last_clean":"2024-03-21T13:22:30.157243+0000","last_became_active":"2024-03-21T13:20:45.137867+0000","last_became_peered":"2024-03-21T13:20:45.137867+0000","last_unstale":"2024-03-21T13:22:30.157243+0000","last_undegraded":"2024-03-21T13:22:30.157243+0000","last_fullsized":"2024-03-21T13:22:30.157243+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-03-21T13:21:55.312543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:55.312543+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:10.575745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"457'46","reported_seq":421,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286108+0000","last_change":"2024-03-21T13:21:47.874026+0000","last_active":"2024-03-21T13:22:30.286108+0000","last_peered":"2024-03-21T13:22:30.286108+0000","last_clean":"2024-03-21T13:22:30.286108+0000","last_became_active":"2024-03-21T13:12:29.513019+0000","last_became_peered":"2024-03-21T13:12:29.513019+0000","last_unstale":"2024-03-21T13:22:30.286108+0000","last_undegraded":"2024-03-21T13:22:30.286108+0000","last_fullsized":"2024-03-21T13:22:30.286108+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'46","last_scrub_stamp":"2024-03-21T13:21:47.873868+0000","last_deep_scrub":"457'46","last_deep_scrub_stamp":"2024-03-21T13:20:25.653330+0000","last_clean_scrub_stamp":"2024-03-21T13:21:47.873868+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:06.498594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":1234,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:33.049868+0000","last_change":"2024-03-21T13:21:28.693363+0000","last_active":"2024-03-21T13:22:33.049868+0000","last_peered":"2024-03-21T13:22:33.049868+0000","last_clean":"2024-03-21T13:22:33.049868+0000","last_became_active":"2024-03-21T12:58:41.166011+0000","last_became_peered":"2024-03-21T12:58:41.166011+0000","last_unstale":"2024-03-21T13:22:33.049868+0000","last_undegraded":"2024-03-21T13:22:33.049868+0000","last_fullsized":"2024-03-21T13:22:33.049868+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-03-21T13:21:28.693220+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-03-21T13:00:20.176653+0000","last_clean_scrub_stamp":"2024-03-21T13:21:28.693220+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":935,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157681+0000","last_change":"2024-03-21T13:21:08.882742+0000","last_active":"2024-03-21T13:22:30.157681+0000","last_peered":"2024-03-21T13:22:30.157681+0000","last_clean":"2024-03-21T13:22:30.157681+0000","last_became_active":"2024-03-21T13:20:41.128472+0000","last_became_peered":"2024-03-21T13:20:41.128472+0000","last_unstale":"2024-03-21T13:22:30.157681+0000","last_undegraded":"2024-03-21T13:22:30.157681+0000","last_fullsized":"2024-03-21T13:22:30.157681+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:08.882640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:11.950446+0000","last_clean_scrub_stamp":"2024-03-21T13:21:08.882640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":967,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157282+0000","last_change":"2024-03-21T13:21:48.526108+0000","last_active":"2024-03-21T13:22:30.157282+0000","last_peered":"2024-03-21T13:22:30.157282+0000","last_clean":"2024-03-21T13:22:30.157282+0000","last_became_active":"2024-03-21T13:20:41.136377+0000","last_became_peered":"2024-03-21T13:20:41.136377+0000","last_unstale":"2024-03-21T13:22:30.157282+0000","last_undegraded":"2024-03-21T13:22:30.157282+0000","last_fullsized":"2024-03-21T13:22:30.157282+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:48.525993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:48.525993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:14.559449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"435'29","reported_seq":258,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286868+0000","last_change":"2024-03-21T13:21:42.369896+0000","last_active":"2024-03-21T13:22:30.286868+0000","last_peered":"2024-03-21T13:22:30.286868+0000","last_clean":"2024-03-21T13:22:30.286868+0000","last_became_active":"2024-03-21T13:21:26.559401+0000","last_became_peered":"2024-03-21T13:21:26.559401+0000","last_unstale":"2024-03-21T13:22:30.286868+0000","last_undegraded":"2024-03-21T13:22:30.286868+0000","last_fullsized":"2024-03-21T13:22:30.286868+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'29","last_scrub_stamp":"2024-03-21T13:21:42.369770+0000","last_deep_scrub":"346'23","last_deep_scrub_stamp":"2024-03-21T13:13:20.232807+0000","last_clean_scrub_stamp":"2024-03-21T13:21:42.369770+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:57.630500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":29,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":532,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.155760+0000","last_change":"2024-03-21T13:21:48.376986+0000","last_active":"2024-03-21T13:22:30.155760+0000","last_peered":"2024-03-21T13:22:30.155760+0000","last_clean":"2024-03-21T13:22:30.155760+0000","last_became_active":"2024-03-21T13:20:45.141107+0000","last_became_peered":"2024-03-21T13:20:45.141107+0000","last_unstale":"2024-03-21T13:22:30.155760+0000","last_undegraded":"2024-03-21T13:22:30.155760+0000","last_fullsized":"2024-03-21T13:22:30.155760+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:48.376837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:21:48.376837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:10.771072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":958,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:33.230318+0000","last_change":"2024-03-21T13:22:33.230318+0000","last_active":"2024-03-21T13:22:33.230318+0000","last_peered":"2024-03-21T13:22:33.230318+0000","last_clean":"2024-03-21T13:22:33.230318+0000","last_became_active":"2024-03-21T13:10:03.598366+0000","last_became_peered":"2024-03-21T13:10:03.598366+0000","last_unstale":"2024-03-21T13:22:33.230318+0000","last_undegraded":"2024-03-21T13:22:33.230318+0000","last_fullsized":"2024-03-21T13:22:33.230318+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:33.230225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:08:14.371718+0000","last_clean_scrub_stamp":"2024-03-21T13:22:33.230225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:52.763600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":534,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156001+0000","last_change":"2024-03-21T13:21:37.299901+0000","last_active":"2024-03-21T13:22:30.156001+0000","last_peered":"2024-03-21T13:22:30.156001+0000","last_clean":"2024-03-21T13:22:30.156001+0000","last_became_active":"2024-03-21T13:20:45.148745+0000","last_became_peered":"2024-03-21T13:20:45.148745+0000","last_unstale":"2024-03-21T13:22:30.156001+0000","last_undegraded":"2024-03-21T13:22:30.156001+0000","last_fullsized":"2024-03-21T13:22:30.156001+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:37.299812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:21:37.299812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:46.897111+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":961,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157389+0000","last_change":"2024-03-21T13:22:24.155612+0000","last_active":"2024-03-21T13:22:30.157389+0000","last_peered":"2024-03-21T13:22:30.157389+0000","last_clean":"2024-03-21T13:22:30.157389+0000","last_became_active":"2024-03-21T13:12:29.508984+0000","last_became_peered":"2024-03-21T13:12:29.508984+0000","last_unstale":"2024-03-21T13:22:30.157389+0000","last_undegraded":"2024-03-21T13:22:30.157389+0000","last_fullsized":"2024-03-21T13:22:30.157389+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:24.155505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:22:24.155505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:27.684020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"435'51","reported_seq":1165,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286745+0000","last_change":"2024-03-21T13:22:15.985194+0000","last_active":"2024-03-21T13:22:30.286745+0000","last_peered":"2024-03-21T13:22:30.286745+0000","last_clean":"2024-03-21T13:22:30.286745+0000","last_became_active":"2024-03-21T13:21:22.555176+0000","last_became_peered":"2024-03-21T13:21:22.555176+0000","last_unstale":"2024-03-21T13:22:30.286745+0000","last_undegraded":"2024-03-21T13:22:30.286745+0000","last_fullsized":"2024-03-21T13:22:30.286745+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'51","last_scrub_stamp":"2024-03-21T13:22:15.985049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:22:15.985049+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:18.645854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":95,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.285771+0000","last_change":"2024-03-21T13:22:28.946212+0000","last_active":"2024-03-21T13:22:30.285771+0000","last_peered":"2024-03-21T13:22:30.285771+0000","last_clean":"2024-03-21T13:22:30.285771+0000","last_became_active":"2024-03-21T13:20:45.147907+0000","last_became_peered":"2024-03-21T13:20:45.147907+0000","last_unstale":"2024-03-21T13:22:30.285771+0000","last_undegraded":"2024-03-21T13:22:30.285771+0000","last_fullsized":"2024-03-21T13:22:30.285771+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:28.946110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:22:28.946110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:47.442466+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":516,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156907+0000","last_change":"2024-03-21T13:22:28.133907+0000","last_active":"2024-03-21T13:22:30.156907+0000","last_peered":"2024-03-21T13:22:30.156907+0000","last_clean":"2024-03-21T13:22:30.156907+0000","last_became_active":"2024-03-21T13:10:03.603646+0000","last_became_peered":"2024-03-21T13:10:03.603646+0000","last_unstale":"2024-03-21T13:22:30.156907+0000","last_undegraded":"2024-03-21T13:22:30.156907+0000","last_fullsized":"2024-03-21T13:22:30.156907+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:28.133750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:28.133750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:42.893249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":380,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286947+0000","last_change":"2024-03-21T13:21:28.717631+0000","last_active":"2024-03-21T13:22:30.286947+0000","last_peered":"2024-03-21T13:22:30.286947+0000","last_clean":"2024-03-21T13:22:30.286947+0000","last_became_active":"2024-03-21T13:12:29.512741+0000","last_became_peered":"2024-03-21T13:12:29.512741+0000","last_unstale":"2024-03-21T13:22:30.286947+0000","last_undegraded":"2024-03-21T13:22:30.286947+0000","last_fullsized":"2024-03-21T13:22:30.286947+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:28.717546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:28.717546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"435'25","reported_seq":97,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286045+0000","last_change":"2024-03-21T13:22:10.938091+0000","last_active":"2024-03-21T13:22:30.286045+0000","last_peered":"2024-03-21T13:22:30.286045+0000","last_clean":"2024-03-21T13:22:30.286045+0000","last_became_active":"2024-03-21T13:20:45.151741+0000","last_became_peered":"2024-03-21T13:20:45.151741+0000","last_unstale":"2024-03-21T13:22:30.286045+0000","last_undegraded":"2024-03-21T13:22:30.286045+0000","last_fullsized":"2024-03-21T13:22:30.286045+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'25","last_scrub_stamp":"2024-03-21T13:22:10.937980+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-21T12:59:46.421144+0000","last_clean_scrub_stamp":"2024-03-21T13:22:10.937980+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:12.009657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":924,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157831+0000","last_change":"2024-03-21T13:21:34.470275+0000","last_active":"2024-03-21T13:22:30.157831+0000","last_peered":"2024-03-21T13:22:30.157831+0000","last_clean":"2024-03-21T13:22:30.157831+0000","last_became_active":"2024-03-21T12:58:43.180395+0000","last_became_peered":"2024-03-21T12:58:43.180395+0000","last_unstale":"2024-03-21T13:22:30.157831+0000","last_undegraded":"2024-03-21T13:22:30.157831+0000","last_fullsized":"2024-03-21T13:22:30.157831+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:34.470184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:21:34.470184+0000","last_clean_scrub_stamp":"2024-03-21T13:21:34.470184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:42.738996+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":404,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286830+0000","last_change":"2024-03-21T13:22:00.899032+0000","last_active":"2024-03-21T13:22:30.286830+0000","last_peered":"2024-03-21T13:22:30.286830+0000","last_clean":"2024-03-21T13:22:30.286830+0000","last_became_active":"2024-03-21T13:20:41.124733+0000","last_became_peered":"2024-03-21T13:20:41.124733+0000","last_unstale":"2024-03-21T13:22:30.286830+0000","last_undegraded":"2024-03-21T13:22:30.286830+0000","last_fullsized":"2024-03-21T13:22:30.286830+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2024-03-21T13:22:00.898926+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2024-03-21T13:05:43.961057+0000","last_clean_scrub_stamp":"2024-03-21T13:22:00.898926+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:22.296841+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0.00048040600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":524,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287895+0000","last_change":"2024-03-21T13:22:20.209316+0000","last_active":"2024-03-21T13:22:30.287895+0000","last_peered":"2024-03-21T13:22:30.287895+0000","last_clean":"2024-03-21T13:22:30.287895+0000","last_became_active":"2024-03-21T13:21:26.560148+0000","last_became_peered":"2024-03-21T13:21:26.560148+0000","last_unstale":"2024-03-21T13:22:30.287895+0000","last_undegraded":"2024-03-21T13:22:30.287895+0000","last_fullsized":"2024-03-21T13:22:30.287895+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:20.209198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:22.861136+0000","last_clean_scrub_stamp":"2024-03-21T13:22:20.209198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:37.685850+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"460'56","reported_seq":149,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:32.515312+0000","last_change":"2024-03-21T13:22:32.515312+0000","last_active":"2024-03-21T13:22:32.515312+0000","last_peered":"2024-03-21T13:22:32.515312+0000","last_clean":"2024-03-21T13:22:32.515312+0000","last_became_active":"2024-03-21T13:20:41.126603+0000","last_became_peered":"2024-03-21T13:20:41.126603+0000","last_unstale":"2024-03-21T13:22:32.515312+0000","last_undegraded":"2024-03-21T13:22:32.515312+0000","last_fullsized":"2024-03-21T13:22:32.515312+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'56","last_scrub_stamp":"2024-03-21T13:22:32.515145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:22:32.515145+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:43.465033+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286974+0000","last_change":"2024-03-21T13:22:05.896267+0000","last_active":"2024-03-21T13:22:30.286974+0000","last_peered":"2024-03-21T13:22:30.286974+0000","last_clean":"2024-03-21T13:22:30.286974+0000","last_became_active":"2024-03-21T13:20:45.139450+0000","last_became_peered":"2024-03-21T13:20:45.139450+0000","last_unstale":"2024-03-21T13:22:30.286974+0000","last_undegraded":"2024-03-21T13:22:30.286974+0000","last_fullsized":"2024-03-21T13:22:30.286974+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:05.896185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:13:15.968926+0000","last_clean_scrub_stamp":"2024-03-21T13:22:05.896185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:25.827769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":938,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157492+0000","last_change":"2024-03-21T13:21:51.497092+0000","last_active":"2024-03-21T13:22:30.157492+0000","last_peered":"2024-03-21T13:22:30.157492+0000","last_clean":"2024-03-21T13:22:30.157492+0000","last_became_active":"2024-03-21T12:58:43.352648+0000","last_became_peered":"2024-03-21T12:58:43.352648+0000","last_unstale":"2024-03-21T13:22:30.157492+0000","last_undegraded":"2024-03-21T13:22:30.157492+0000","last_fullsized":"2024-03-21T13:22:30.157492+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:51.496985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:18.683041+0000","last_clean_scrub_stamp":"2024-03-21T13:21:51.496985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:59.582558+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":95,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286036+0000","last_change":"2024-03-21T13:21:39.405701+0000","last_active":"2024-03-21T13:22:30.286036+0000","last_peered":"2024-03-21T13:22:30.286036+0000","last_clean":"2024-03-21T13:22:30.286036+0000","last_became_active":"2024-03-21T13:20:41.146376+0000","last_became_peered":"2024-03-21T13:20:41.146376+0000","last_unstale":"2024-03-21T13:22:30.286036+0000","last_undegraded":"2024-03-21T13:22:30.286036+0000","last_fullsized":"2024-03-21T13:22:30.286036+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:39.405609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:12.231115+0000","last_clean_scrub_stamp":"2024-03-21T13:21:39.405609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:56.486299+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":527,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156141+0000","last_change":"2024-03-21T13:21:38.293887+0000","last_active":"2024-03-21T13:22:30.156141+0000","last_peered":"2024-03-21T13:22:30.156141+0000","last_clean":"2024-03-21T13:22:30.156141+0000","last_became_active":"2024-03-21T13:20:45.141427+0000","last_became_peered":"2024-03-21T13:20:45.141427+0000","last_unstale":"2024-03-21T13:22:30.156141+0000","last_undegraded":"2024-03-21T13:22:30.156141+0000","last_fullsized":"2024-03-21T13:22:30.156141+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:38.293801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:21:38.293801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:04.819332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"435'24","reported_seq":125,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287184+0000","last_change":"2024-03-21T13:22:00.546944+0000","last_active":"2024-03-21T13:22:30.287184+0000","last_peered":"2024-03-21T13:22:30.287184+0000","last_clean":"2024-03-21T13:22:30.287184+0000","last_became_active":"2024-03-21T13:20:45.154138+0000","last_became_peered":"2024-03-21T13:20:45.154138+0000","last_unstale":"2024-03-21T13:22:30.287184+0000","last_undegraded":"2024-03-21T13:22:30.287184+0000","last_fullsized":"2024-03-21T13:22:30.287184+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'24","last_scrub_stamp":"2024-03-21T13:22:00.546815+0000","last_deep_scrub":"243'20","last_deep_scrub_stamp":"2024-03-21T13:09:10.683382+0000","last_clean_scrub_stamp":"2024-03-21T13:22:00.546815+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:02.770535+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":514,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157171+0000","last_change":"2024-03-21T13:21:49.408138+0000","last_active":"2024-03-21T13:22:30.157171+0000","last_peered":"2024-03-21T13:22:30.157171+0000","last_clean":"2024-03-21T13:22:30.157171+0000","last_became_active":"2024-03-21T13:12:29.506751+0000","last_became_peered":"2024-03-21T13:12:29.506751+0000","last_unstale":"2024-03-21T13:22:30.157171+0000","last_undegraded":"2024-03-21T13:22:30.157171+0000","last_fullsized":"2024-03-21T13:22:30.157171+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:49.408064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:21:49.408064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:11.084702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":380,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288342+0000","last_change":"2024-03-21T13:22:06.939476+0000","last_active":"2024-03-21T13:22:30.288342+0000","last_peered":"2024-03-21T13:22:30.288342+0000","last_clean":"2024-03-21T13:22:30.288342+0000","last_became_active":"2024-03-21T13:12:29.527083+0000","last_became_peered":"2024-03-21T13:12:29.527083+0000","last_unstale":"2024-03-21T13:22:30.288342+0000","last_undegraded":"2024-03-21T13:22:30.288342+0000","last_fullsized":"2024-03-21T13:22:30.288342+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:06.939376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:48.803220+0000","last_clean_scrub_stamp":"2024-03-21T13:22:06.939376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:16.926173+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'130","reported_seq":1247,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:31.187192+0000","last_change":"2024-03-21T13:22:31.187243+0000","last_active":"2024-03-21T13:22:31.187192+0000","last_peered":"2024-03-21T13:22:31.187192+0000","last_clean":"2024-03-21T13:22:31.187192+0000","last_became_active":"2024-03-21T13:20:47.557485+0000","last_became_peered":"2024-03-21T13:20:47.557485+0000","last_unstale":"2024-03-21T13:22:31.187192+0000","last_undegraded":"2024-03-21T13:22:31.187192+0000","last_fullsized":"2024-03-21T13:22:31.187192+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'130","last_scrub_stamp":"2024-03-21T13:22:31.187102+0000","last_deep_scrub":"79'130","last_deep_scrub_stamp":"2024-03-21T13:08:07.454016+0000","last_clean_scrub_stamp":"2024-03-21T13:22:31.187102+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:34.653514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":454,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156662+0000","last_change":"2024-03-21T13:22:07.209703+0000","last_active":"2024-03-21T13:22:30.156662+0000","last_peered":"2024-03-21T13:22:30.156662+0000","last_clean":"2024-03-21T13:22:30.156662+0000","last_became_active":"2024-03-21T13:12:29.499989+0000","last_became_peered":"2024-03-21T13:12:29.499989+0000","last_unstale":"2024-03-21T13:22:30.156662+0000","last_undegraded":"2024-03-21T13:22:30.156662+0000","last_fullsized":"2024-03-21T13:22:30.156662+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:07.209624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:22:07.209624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:16.425454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"435'48","reported_seq":1302,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287610+0000","last_change":"2024-03-21T13:21:41.213382+0000","last_active":"2024-03-21T13:22:30.287610+0000","last_peered":"2024-03-21T13:22:30.287610+0000","last_clean":"2024-03-21T13:22:30.287610+0000","last_became_active":"2024-03-21T12:58:39.751558+0000","last_became_peered":"2024-03-21T12:58:39.751558+0000","last_unstale":"2024-03-21T13:22:30.287610+0000","last_undegraded":"2024-03-21T13:22:30.287610+0000","last_fullsized":"2024-03-21T13:22:30.287610+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'48","last_scrub_stamp":"2024-03-21T13:21:41.213255+0000","last_deep_scrub":"183'24","last_deep_scrub_stamp":"2024-03-21T13:05:42.033266+0000","last_clean_scrub_stamp":"2024-03-21T13:21:41.213255+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:06.241122+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":966,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157163+0000","last_change":"2024-03-21T13:22:01.205036+0000","last_active":"2024-03-21T13:22:30.157163+0000","last_peered":"2024-03-21T13:22:30.157163+0000","last_clean":"2024-03-21T13:22:30.157163+0000","last_became_active":"2024-03-21T13:12:29.507753+0000","last_became_peered":"2024-03-21T13:12:29.507753+0000","last_unstale":"2024-03-21T13:22:30.157163+0000","last_undegraded":"2024-03-21T13:22:30.157163+0000","last_fullsized":"2024-03-21T13:22:30.157163+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:01.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:01:40.968175+0000","last_clean_scrub_stamp":"2024-03-21T13:22:01.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:20.125373+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":103,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286318+0000","last_change":"2024-03-21T13:22:17.424711+0000","last_active":"2024-03-21T13:22:30.286318+0000","last_peered":"2024-03-21T13:22:30.286318+0000","last_clean":"2024-03-21T13:22:30.286318+0000","last_became_active":"2024-03-21T13:20:41.135808+0000","last_became_peered":"2024-03-21T13:20:41.135808+0000","last_unstale":"2024-03-21T13:22:30.286318+0000","last_undegraded":"2024-03-21T13:22:30.286318+0000","last_fullsized":"2024-03-21T13:22:30.286318+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:17.424625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:48.210633+0000","last_clean_scrub_stamp":"2024-03-21T13:22:17.424625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.928113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286038+0000","last_change":"2024-03-21T13:22:17.022221+0000","last_active":"2024-03-21T13:22:30.286038+0000","last_peered":"2024-03-21T13:22:30.286038+0000","last_clean":"2024-03-21T13:22:30.286038+0000","last_became_active":"2024-03-21T13:20:45.136827+0000","last_became_peered":"2024-03-21T13:20:45.136827+0000","last_unstale":"2024-03-21T13:22:30.286038+0000","last_undegraded":"2024-03-21T13:22:30.286038+0000","last_fullsized":"2024-03-21T13:22:30.286038+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:17.022147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:00.086941+0000","last_clean_scrub_stamp":"2024-03-21T13:22:17.022147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:42.708582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"72'5","reported_seq":2426,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:34.307440+0000","last_change":"2024-03-21T13:22:30.144087+0000","last_active":"2024-03-21T13:22:34.307440+0000","last_peered":"2024-03-21T13:22:34.307440+0000","last_clean":"2024-03-21T13:22:34.307440+0000","last_became_active":"2024-03-21T13:20:45.137793+0000","last_became_peered":"2024-03-21T13:20:45.137793+0000","last_unstale":"2024-03-21T13:22:34.307440+0000","last_undegraded":"2024-03-21T13:22:34.307440+0000","last_fullsized":"2024-03-21T13:22:34.307440+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T13:22:30.143968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:35.713454+0000","last_clean_scrub_stamp":"2024-03-21T13:22:30.143968+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:30.346973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0026949840000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1423,"num_read_kb":1418,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"435'36","reported_seq":101,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287291+0000","last_change":"2024-03-21T13:22:02.837829+0000","last_active":"2024-03-21T13:22:30.287291+0000","last_peered":"2024-03-21T13:22:30.287291+0000","last_clean":"2024-03-21T13:22:30.287291+0000","last_became_active":"2024-03-21T13:20:45.150533+0000","last_became_peered":"2024-03-21T13:20:45.150533+0000","last_unstale":"2024-03-21T13:22:30.287291+0000","last_undegraded":"2024-03-21T13:22:30.287291+0000","last_fullsized":"2024-03-21T13:22:30.287291+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'36","last_scrub_stamp":"2024-03-21T13:22:02.837735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:22:02.837735+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:08.633914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":398,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287671+0000","last_change":"2024-03-21T13:21:48.901169+0000","last_active":"2024-03-21T13:22:30.287671+0000","last_peered":"2024-03-21T13:22:30.287671+0000","last_clean":"2024-03-21T13:22:30.287671+0000","last_became_active":"2024-03-21T13:20:45.139353+0000","last_became_peered":"2024-03-21T13:20:45.139353+0000","last_unstale":"2024-03-21T13:22:30.287671+0000","last_undegraded":"2024-03-21T13:22:30.287671+0000","last_fullsized":"2024-03-21T13:22:30.287671+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:48.901072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:08.237986+0000","last_clean_scrub_stamp":"2024-03-21T13:21:48.901072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:04.680186+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":103,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286496+0000","last_change":"2024-03-21T13:22:26.557554+0000","last_active":"2024-03-21T13:22:30.286496+0000","last_peered":"2024-03-21T13:22:30.286496+0000","last_clean":"2024-03-21T13:22:30.286496+0000","last_became_active":"2024-03-21T13:20:41.146817+0000","last_became_peered":"2024-03-21T13:20:41.146817+0000","last_unstale":"2024-03-21T13:22:30.286496+0000","last_undegraded":"2024-03-21T13:22:30.286496+0000","last_fullsized":"2024-03-21T13:22:30.286496+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:26.557404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:29.809009+0000","last_clean_scrub_stamp":"2024-03-21T13:22:26.557404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:27.473531+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":95,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286477+0000","last_change":"2024-03-21T13:21:45.386590+0000","last_active":"2024-03-21T13:22:30.286477+0000","last_peered":"2024-03-21T13:22:30.286477+0000","last_clean":"2024-03-21T13:22:30.286477+0000","last_became_active":"2024-03-21T13:20:41.134288+0000","last_became_peered":"2024-03-21T13:20:41.134288+0000","last_unstale":"2024-03-21T13:22:30.286477+0000","last_undegraded":"2024-03-21T13:22:30.286477+0000","last_fullsized":"2024-03-21T13:22:30.286477+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:45.386478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:45.386478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:06.855968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"435'39","reported_seq":453,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286251+0000","last_change":"2024-03-21T13:22:09.946154+0000","last_active":"2024-03-21T13:22:30.286251+0000","last_peered":"2024-03-21T13:22:30.286251+0000","last_clean":"2024-03-21T13:22:30.286251+0000","last_became_active":"2024-03-21T13:20:56.547138+0000","last_became_peered":"2024-03-21T13:20:56.547138+0000","last_unstale":"2024-03-21T13:22:30.286251+0000","last_undegraded":"2024-03-21T13:22:30.286251+0000","last_fullsized":"2024-03-21T13:22:30.286251+0000","mapping_epoch":471,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'39","last_scrub_stamp":"2024-03-21T13:22:09.946052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:22:09.946052+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.777422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":980,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286817+0000","last_change":"2024-03-21T13:21:44.246795+0000","last_active":"2024-03-21T13:22:30.286817+0000","last_peered":"2024-03-21T13:22:30.286817+0000","last_clean":"2024-03-21T13:22:30.286817+0000","last_became_active":"2024-03-21T13:20:45.139101+0000","last_became_peered":"2024-03-21T13:20:45.139101+0000","last_unstale":"2024-03-21T13:22:30.286817+0000","last_undegraded":"2024-03-21T13:22:30.286817+0000","last_fullsized":"2024-03-21T13:22:30.286817+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:44.246689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:21:44.246689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:02.536297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":87,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.285788+0000","last_change":"2024-03-21T13:21:35.932908+0000","last_active":"2024-03-21T13:22:30.285788+0000","last_peered":"2024-03-21T13:22:30.285788+0000","last_clean":"2024-03-21T13:22:30.285788+0000","last_became_active":"2024-03-21T13:20:45.135691+0000","last_became_peered":"2024-03-21T13:20:45.135691+0000","last_unstale":"2024-03-21T13:22:30.285788+0000","last_undegraded":"2024-03-21T13:22:30.285788+0000","last_fullsized":"2024-03-21T13:22:30.285788+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:35.932778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:21:35.932778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:54.303116+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":380,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287324+0000","last_change":"2024-03-21T13:22:10.726951+0000","last_active":"2024-03-21T13:22:30.287324+0000","last_peered":"2024-03-21T13:22:30.287324+0000","last_clean":"2024-03-21T13:22:30.287324+0000","last_became_active":"2024-03-21T13:12:29.512291+0000","last_became_peered":"2024-03-21T13:12:29.512291+0000","last_unstale":"2024-03-21T13:22:30.287324+0000","last_undegraded":"2024-03-21T13:22:30.287324+0000","last_fullsized":"2024-03-21T13:22:30.287324+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:10.726838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:19:24.587197+0000","last_clean_scrub_stamp":"2024-03-21T13:22:10.726838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.594501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"435'35","reported_seq":115,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286817+0000","last_change":"2024-03-21T13:22:13.512153+0000","last_active":"2024-03-21T13:22:30.286817+0000","last_peered":"2024-03-21T13:22:30.286817+0000","last_clean":"2024-03-21T13:22:30.286817+0000","last_became_active":"2024-03-21T13:20:41.141582+0000","last_became_peered":"2024-03-21T13:20:41.141582+0000","last_unstale":"2024-03-21T13:22:30.286817+0000","last_undegraded":"2024-03-21T13:22:30.286817+0000","last_fullsized":"2024-03-21T13:22:30.286817+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'35","last_scrub_stamp":"2024-03-21T13:22:13.512059+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-03-21T12:59:45.653197+0000","last_clean_scrub_stamp":"2024-03-21T13:22:13.512059+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:33.671805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"467'16","reported_seq":113,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286194+0000","last_change":"2024-03-21T13:21:07.845725+0000","last_active":"2024-03-21T13:22:30.286194+0000","last_peered":"2024-03-21T13:22:30.286194+0000","last_clean":"2024-03-21T13:22:30.286194+0000","last_became_active":"2024-03-21T13:20:45.158775+0000","last_became_peered":"2024-03-21T13:20:45.158775+0000","last_unstale":"2024-03-21T13:22:30.286194+0000","last_undegraded":"2024-03-21T13:22:30.286194+0000","last_fullsized":"2024-03-21T13:22:30.286194+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'16","last_scrub_stamp":"2024-03-21T13:21:07.845626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:21:07.845626+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":950,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156997+0000","last_change":"2024-03-21T13:22:25.311368+0000","last_active":"2024-03-21T13:22:30.156997+0000","last_peered":"2024-03-21T13:22:30.156997+0000","last_clean":"2024-03-21T13:22:30.156997+0000","last_became_active":"2024-03-21T13:10:03.606557+0000","last_became_peered":"2024-03-21T13:10:03.606557+0000","last_unstale":"2024-03-21T13:22:30.156997+0000","last_undegraded":"2024-03-21T13:22:30.156997+0000","last_fullsized":"2024-03-21T13:22:30.156997+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:25.311257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:42.156230+0000","last_clean_scrub_stamp":"2024-03-21T13:22:25.311257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:32.813217+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":91,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287341+0000","last_change":"2024-03-21T13:22:08.931699+0000","last_active":"2024-03-21T13:22:30.287341+0000","last_peered":"2024-03-21T13:22:30.287341+0000","last_clean":"2024-03-21T13:22:30.287341+0000","last_became_active":"2024-03-21T13:20:45.150264+0000","last_became_peered":"2024-03-21T13:20:45.150264+0000","last_unstale":"2024-03-21T13:22:30.287341+0000","last_undegraded":"2024-03-21T13:22:30.287341+0000","last_fullsized":"2024-03-21T13:22:30.287341+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-21T13:22:08.931607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:22:08.931607+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:13.392639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"435'49","reported_seq":316,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286646+0000","last_change":"2024-03-21T13:21:39.443553+0000","last_active":"2024-03-21T13:22:30.286646+0000","last_peered":"2024-03-21T13:22:30.286646+0000","last_clean":"2024-03-21T13:22:30.286646+0000","last_became_active":"2024-03-21T13:21:26.555211+0000","last_became_peered":"2024-03-21T13:21:26.555211+0000","last_unstale":"2024-03-21T13:22:30.286646+0000","last_undegraded":"2024-03-21T13:22:30.286646+0000","last_fullsized":"2024-03-21T13:22:30.286646+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'49","last_scrub_stamp":"2024-03-21T13:21:39.443429+0000","last_deep_scrub":"87'21","last_deep_scrub_stamp":"2024-03-21T12:59:52.916260+0000","last_clean_scrub_stamp":"2024-03-21T13:21:39.443429+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:41.501292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"460'12","reported_seq":1265,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.155865+0000","last_change":"2024-03-21T13:22:29.960749+0000","last_active":"2024-03-21T13:22:30.155865+0000","last_peered":"2024-03-21T13:22:30.155865+0000","last_clean":"2024-03-21T13:22:30.155865+0000","last_became_active":"2024-03-21T13:20:41.129168+0000","last_became_peered":"2024-03-21T13:20:41.129168+0000","last_unstale":"2024-03-21T13:22:30.155865+0000","last_undegraded":"2024-03-21T13:22:30.155865+0000","last_fullsized":"2024-03-21T13:22:30.155865+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'12","last_scrub_stamp":"2024-03-21T13:22:29.960665+0000","last_deep_scrub":"226'10","last_deep_scrub_stamp":"2024-03-21T13:08:16.739190+0000","last_clean_scrub_stamp":"2024-03-21T13:22:29.960665+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:44.161537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":407,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287711+0000","last_change":"2024-03-21T13:22:15.284802+0000","last_active":"2024-03-21T13:22:30.287711+0000","last_peered":"2024-03-21T13:22:30.287711+0000","last_clean":"2024-03-21T13:22:30.287711+0000","last_became_active":"2024-03-21T13:21:26.553104+0000","last_became_peered":"2024-03-21T13:21:26.553104+0000","last_unstale":"2024-03-21T13:22:30.287711+0000","last_undegraded":"2024-03-21T13:22:30.287711+0000","last_fullsized":"2024-03-21T13:22:30.287711+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:15.284719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:50.909170+0000","last_clean_scrub_stamp":"2024-03-21T13:22:15.284719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:19.164991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":956,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.158347+0000","last_change":"2024-03-21T13:22:01.711873+0000","last_active":"2024-03-21T13:22:30.158347+0000","last_peered":"2024-03-21T13:22:30.158347+0000","last_clean":"2024-03-21T13:22:30.158347+0000","last_became_active":"2024-03-21T13:20:45.139123+0000","last_became_peered":"2024-03-21T13:20:45.139123+0000","last_unstale":"2024-03-21T13:22:30.158347+0000","last_undegraded":"2024-03-21T13:22:30.158347+0000","last_fullsized":"2024-03-21T13:22:30.158347+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:01.711792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:59:43.240674+0000","last_clean_scrub_stamp":"2024-03-21T13:22:01.711792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:22.823736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"435'42","reported_seq":1080,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.155723+0000","last_change":"2024-03-21T13:21:42.546721+0000","last_active":"2024-03-21T13:22:30.155723+0000","last_peered":"2024-03-21T13:22:30.155723+0000","last_clean":"2024-03-21T13:22:30.155723+0000","last_became_active":"2024-03-21T13:20:45.140650+0000","last_became_peered":"2024-03-21T13:20:45.140650+0000","last_unstale":"2024-03-21T13:22:30.155723+0000","last_undegraded":"2024-03-21T13:22:30.155723+0000","last_fullsized":"2024-03-21T13:22:30.155723+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'42","last_scrub_stamp":"2024-03-21T13:21:42.546613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:42.546613+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:03.819900+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":961,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286451+0000","last_change":"2024-03-21T13:22:21.194028+0000","last_active":"2024-03-21T13:22:30.286451+0000","last_peered":"2024-03-21T13:22:30.286451+0000","last_clean":"2024-03-21T13:22:30.286451+0000","last_became_active":"2024-03-21T12:58:41.164376+0000","last_became_peered":"2024-03-21T12:58:41.164376+0000","last_unstale":"2024-03-21T13:22:30.286451+0000","last_undegraded":"2024-03-21T13:22:30.286451+0000","last_fullsized":"2024-03-21T13:22:30.286451+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:21.193896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:21.193896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:45.577386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":398,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287394+0000","last_change":"2024-03-21T13:22:05.982754+0000","last_active":"2024-03-21T13:22:30.287394+0000","last_peered":"2024-03-21T13:22:30.287394+0000","last_clean":"2024-03-21T13:22:30.287394+0000","last_became_active":"2024-03-21T13:20:45.138941+0000","last_became_peered":"2024-03-21T13:20:45.138941+0000","last_unstale":"2024-03-21T13:22:30.287394+0000","last_undegraded":"2024-03-21T13:22:30.287394+0000","last_fullsized":"2024-03-21T13:22:30.287394+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:05.982640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:16:14.280515+0000","last_clean_scrub_stamp":"2024-03-21T13:22:05.982640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:12.581195+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"486'16","reported_seq":186,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288781+0000","last_change":"2024-03-21T13:22:11.355586+0000","last_active":"2024-03-21T13:22:30.288781+0000","last_peered":"2024-03-21T13:22:30.288781+0000","last_clean":"2024-03-21T13:22:30.288781+0000","last_became_active":"2024-03-21T13:21:26.561093+0000","last_became_peered":"2024-03-21T13:21:26.561093+0000","last_unstale":"2024-03-21T13:22:30.288781+0000","last_undegraded":"2024-03-21T13:22:30.288781+0000","last_fullsized":"2024-03-21T13:22:30.288781+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'16","last_scrub_stamp":"2024-03-21T13:22:11.355482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:22:11.355482+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:27.045834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":956,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.155753+0000","last_change":"2024-03-21T13:21:45.519352+0000","last_active":"2024-03-21T13:22:30.155753+0000","last_peered":"2024-03-21T13:22:30.155753+0000","last_clean":"2024-03-21T13:22:30.155753+0000","last_became_active":"2024-03-21T13:20:41.128145+0000","last_became_peered":"2024-03-21T13:20:41.128145+0000","last_unstale":"2024-03-21T13:22:30.155753+0000","last_undegraded":"2024-03-21T13:22:30.155753+0000","last_fullsized":"2024-03-21T13:22:30.155753+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:45.519268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:17:00.746886+0000","last_clean_scrub_stamp":"2024-03-21T13:21:45.519268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:02.875028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"435'46","reported_seq":111,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287182+0000","last_change":"2024-03-21T13:21:46.425301+0000","last_active":"2024-03-21T13:22:30.287182+0000","last_peered":"2024-03-21T13:22:30.287182+0000","last_clean":"2024-03-21T13:22:30.287182+0000","last_became_active":"2024-03-21T13:20:41.140902+0000","last_became_peered":"2024-03-21T13:20:41.140902+0000","last_unstale":"2024-03-21T13:22:30.287182+0000","last_undegraded":"2024-03-21T13:22:30.287182+0000","last_fullsized":"2024-03-21T13:22:30.287182+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'46","last_scrub_stamp":"2024-03-21T13:21:46.425201+0000","last_deep_scrub":"228'22","last_deep_scrub_stamp":"2024-03-21T13:08:22.736371+0000","last_clean_scrub_stamp":"2024-03-21T13:21:46.425201+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:52.917930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"375'2","reported_seq":953,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157099+0000","last_change":"2024-03-21T13:22:26.934977+0000","last_active":"2024-03-21T13:22:30.157099+0000","last_peered":"2024-03-21T13:22:30.157099+0000","last_clean":"2024-03-21T13:22:30.157099+0000","last_became_active":"2024-03-21T13:12:29.502816+0000","last_became_peered":"2024-03-21T13:12:29.502816+0000","last_unstale":"2024-03-21T13:22:30.157099+0000","last_undegraded":"2024-03-21T13:22:30.157099+0000","last_fullsized":"2024-03-21T13:22:30.157099+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'2","last_scrub_stamp":"2024-03-21T13:22:26.934898+0000","last_deep_scrub":"375'2","last_deep_scrub_stamp":"2024-03-21T13:15:14.313376+0000","last_clean_scrub_stamp":"2024-03-21T13:22:26.934898+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:51.378294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":380,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.287663+0000","last_change":"2024-03-21T13:22:08.785941+0000","last_active":"2024-03-21T13:22:30.287663+0000","last_peered":"2024-03-21T13:22:30.287663+0000","last_clean":"2024-03-21T13:22:30.287663+0000","last_became_active":"2024-03-21T13:12:29.524087+0000","last_became_peered":"2024-03-21T13:12:29.524087+0000","last_unstale":"2024-03-21T13:22:30.287663+0000","last_undegraded":"2024-03-21T13:22:30.287663+0000","last_fullsized":"2024-03-21T13:22:30.287663+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:08.785846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:22:08.785846+0000","last_clean_scrub_stamp":"2024-03-21T13:22:08.785846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:30.117638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":989,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286589+0000","last_change":"2024-03-21T13:21:50.327929+0000","last_active":"2024-03-21T13:22:30.286589+0000","last_peered":"2024-03-21T13:22:30.286589+0000","last_clean":"2024-03-21T13:22:30.286589+0000","last_became_active":"2024-03-21T13:10:03.586593+0000","last_became_peered":"2024-03-21T13:10:03.586593+0000","last_unstale":"2024-03-21T13:22:30.286589+0000","last_undegraded":"2024-03-21T13:22:30.286589+0000","last_fullsized":"2024-03-21T13:22:30.286589+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:50.327810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:21:50.327810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:57.234648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"435'27","reported_seq":606,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288528+0000","last_change":"2024-03-21T13:21:26.561325+0000","last_active":"2024-03-21T13:22:30.288528+0000","last_peered":"2024-03-21T13:22:30.288528+0000","last_clean":"2024-03-21T13:22:30.288528+0000","last_became_active":"2024-03-21T13:21:26.560123+0000","last_became_peered":"2024-03-21T13:21:26.560123+0000","last_unstale":"2024-03-21T13:22:30.288528+0000","last_undegraded":"2024-03-21T13:22:30.288528+0000","last_fullsized":"2024-03-21T13:22:30.288528+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'27","last_scrub_stamp":"2024-03-21T13:21:24.287336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:21:24.287336+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:46.293226+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":524,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.156748+0000","last_change":"2024-03-21T13:21:43.500715+0000","last_active":"2024-03-21T13:22:30.156748+0000","last_peered":"2024-03-21T13:22:30.156748+0000","last_clean":"2024-03-21T13:22:30.156748+0000","last_became_active":"2024-03-21T13:10:03.602752+0000","last_became_peered":"2024-03-21T13:10:03.602752+0000","last_unstale":"2024-03-21T13:22:30.156748+0000","last_undegraded":"2024-03-21T13:22:30.156748+0000","last_fullsized":"2024-03-21T13:22:30.156748+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:43.500597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:00:19.681413+0000","last_clean_scrub_stamp":"2024-03-21T13:21:43.500597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:08.037562+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":101,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286566+0000","last_change":"2024-03-21T13:21:57.500678+0000","last_active":"2024-03-21T13:22:30.286566+0000","last_peered":"2024-03-21T13:22:30.286566+0000","last_clean":"2024-03-21T13:22:30.286566+0000","last_became_active":"2024-03-21T13:20:45.138762+0000","last_became_peered":"2024-03-21T13:20:45.138762+0000","last_unstale":"2024-03-21T13:22:30.286566+0000","last_undegraded":"2024-03-21T13:22:30.286566+0000","last_fullsized":"2024-03-21T13:22:30.286566+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:21:57.500598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:20:26.413887+0000","last_clean_scrub_stamp":"2024-03-21T13:21:57.500598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:21.608902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":980,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.286973+0000","last_change":"2024-03-21T13:22:25.983421+0000","last_active":"2024-03-21T13:22:30.286973+0000","last_peered":"2024-03-21T13:22:30.286973+0000","last_clean":"2024-03-21T13:22:30.286973+0000","last_became_active":"2024-03-21T13:20:41.150814+0000","last_became_peered":"2024-03-21T13:20:41.150814+0000","last_unstale":"2024-03-21T13:22:30.286973+0000","last_undegraded":"2024-03-21T13:22:30.286973+0000","last_fullsized":"2024-03-21T13:22:30.286973+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:25.983325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:36.715100+0000","last_clean_scrub_stamp":"2024-03-21T13:22:25.983325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:47.992486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"435'53","reported_seq":445,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288453+0000","last_change":"2024-03-21T13:21:58.474453+0000","last_active":"2024-03-21T13:22:30.288453+0000","last_peered":"2024-03-21T13:22:30.288453+0000","last_clean":"2024-03-21T13:22:30.288453+0000","last_became_active":"2024-03-21T13:21:26.559525+0000","last_became_peered":"2024-03-21T13:21:26.559525+0000","last_unstale":"2024-03-21T13:22:30.288453+0000","last_undegraded":"2024-03-21T13:22:30.288453+0000","last_fullsized":"2024-03-21T13:22:30.288453+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'53","last_scrub_stamp":"2024-03-21T13:21:58.474348+0000","last_deep_scrub":"391'39","last_deep_scrub_stamp":"2024-03-21T13:16:14.266573+0000","last_clean_scrub_stamp":"2024-03-21T13:21:58.474348+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:15.600281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":58,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":179,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:33.241472+0000","last_change":"2024-03-21T13:22:33.241496+0000","last_active":"2024-03-21T13:22:33.241472+0000","last_peered":"2024-03-21T13:22:33.241472+0000","last_clean":"2024-03-21T13:22:33.241472+0000","last_became_active":"2024-03-21T13:21:26.559301+0000","last_became_peered":"2024-03-21T13:21:26.559301+0000","last_unstale":"2024-03-21T13:22:33.241472+0000","last_undegraded":"2024-03-21T13:22:33.241472+0000","last_fullsized":"2024-03-21T13:22:33.241472+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:33.241387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:22:33.241387+0000","last_clean_scrub_stamp":"2024-03-21T13:22:33.241387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:33.803178+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"331'13","reported_seq":674,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.157242+0000","last_change":"2024-03-21T13:21:41.267426+0000","last_active":"2024-03-21T13:22:30.157242+0000","last_peered":"2024-03-21T13:22:30.157242+0000","last_clean":"2024-03-21T13:22:30.157242+0000","last_became_active":"2024-03-21T13:12:29.507925+0000","last_became_peered":"2024-03-21T13:12:29.507925+0000","last_unstale":"2024-03-21T13:22:30.157242+0000","last_undegraded":"2024-03-21T13:22:30.157242+0000","last_fullsized":"2024-03-21T13:22:30.157242+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'13","last_scrub_stamp":"2024-03-21T13:21:41.267308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:40.734631+0000","last_clean_scrub_stamp":"2024-03-21T13:21:41.267308+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:22:44.982445+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":961,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.998749+0000","last_change":"2024-03-21T13:22:30.998772+0000","last_active":"2024-03-21T13:22:30.998749+0000","last_peered":"2024-03-21T13:22:30.998749+0000","last_clean":"2024-03-21T13:22:30.998749+0000","last_became_active":"2024-03-21T13:20:45.156737+0000","last_became_peered":"2024-03-21T13:20:45.156737+0000","last_unstale":"2024-03-21T13:22:30.998749+0000","last_undegraded":"2024-03-21T13:22:30.998749+0000","last_fullsized":"2024-03-21T13:22:30.998749+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:22:30.998672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:22:30.998672+0000","last_clean_scrub_stamp":"2024-03-21T13:22:30.998672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:49.392249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"435'30","reported_seq":819,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-21T13:22:30.288206+0000","last_change":"2024-03-21T13:22:02.420671+0000","last_active":"2024-03-21T13:22:30.288206+0000","last_peered":"2024-03-21T13:22:30.288206+0000","last_clean":"2024-03-21T13:22:30.288206+0000","last_became_active":"2024-03-21T13:21:26.555282+0000","last_became_peered":"2024-03-21T13:21:26.555282+0000","last_unstale":"2024-03-21T13:22:30.288206+0000","last_undegraded":"2024-03-21T13:22:30.288206+0000","last_fullsized":"2024-03-21T13:22:30.288206+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'30","last_scrub_stamp":"2024-03-21T13:22:02.420540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:58:38.718880+0000","last_clean_scrub_stamp":"2024-03-21T13:22:02.420540+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:23:23.114145+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1423,"num_read_kb":1418,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":5},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1520,"num_read_kb":1277,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":11744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1073,"ondisk_log_size":1073,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168869,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1119604,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":92625548,"statfs":{"total":95995035648,"available":94848561152,"internally_reserved":0,"allocated":1421312,"data_stored":995603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1145043402},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463211,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1079332,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":1077950,"kb_avail":92665820,"statfs":{"total":95995035648,"available":94889799680,"internally_reserved":0,"allocated":1400832,"data_stored":994844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1103821254},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575171,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122252,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622900,"statfs":{"total":95995035648,"available":95869849600,"internally_reserved":0,"allocated":851968,"data_stored":404681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738697,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019632,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":1018238,"kb_avail":92725520,"statfs":{"total":95995035648,"available":94950932480,"internally_reserved":0,"allocated":1417216,"data_stored":994388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1042676165},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411647,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":948252,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":92796900,"statfs":{"total":95995035648,"available":95024025600,"internally_reserved":0,"allocated":806912,"data_stored":403997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":970193349},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182714,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1173360,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":1172542,"kb_avail":92571792,"statfs":{"total":95995035648,"available":94793515008,"internally_reserved":0,"allocated":827392,"data_stored":404126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1200683463},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019188,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1172904,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":1172094,"kb_avail":92572248,"statfs":{"total":95995035648,"available":94793981952,"internally_reserved":0,"allocated":819200,"data_stored":404476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1200224709},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724847,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":401244,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93343908,"statfs":{"total":95995035648,"available":95584161792,"internally_reserved":0,"allocated":806912,"data_stored":403565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:22:40.426 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-21T13:22:40.426 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-21T13:22:40.671 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:22:40.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:40 smithi163 ceph-mon[29621]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:40.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:40 smithi163 ceph-mon[29621]: from='client.26279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:22:40.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:40 smithi163 ceph-mon[29621]: 5.3 scrub starts 2024-03-21T13:22:40.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:40 smithi163 ceph-mon[29621]: 5.3 scrub ok 2024-03-21T13:22:40.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:40 smithi163 ceph-mon[40675]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:40.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:40 smithi163 ceph-mon[40675]: from='client.26279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:22:40.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:40 smithi163 ceph-mon[40675]: 5.3 scrub starts 2024-03-21T13:22:40.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:40 smithi163 ceph-mon[40675]: 5.3 scrub ok 2024-03-21T13:22:41.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:40 smithi204 ceph-mon[33153]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:41.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:40 smithi204 ceph-mon[33153]: from='client.26279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:22:41.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:40 smithi204 ceph-mon[33153]: 5.3 scrub starts 2024-03-21T13:22:41.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:40 smithi204 ceph-mon[33153]: 5.3 scrub ok 2024-03-21T13:22:41.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:41 smithi163 ceph-mon[29621]: purged_snaps scrub starts 2024-03-21T13:22:41.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:41 smithi163 ceph-mon[29621]: purged_snaps scrub ok 2024-03-21T13:22:41.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:41 smithi163 ceph-mon[29621]: osdmap e499: 8 total, 8 up, 8 in 2024-03-21T13:22:41.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:41 smithi163 ceph-mon[29621]: 3.c scrub starts 2024-03-21T13:22:41.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:41 smithi163 ceph-mon[29621]: 3.c scrub ok 2024-03-21T13:22:41.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:41 smithi163 ceph-mon[40675]: purged_snaps scrub starts 2024-03-21T13:22:41.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:41 smithi163 ceph-mon[40675]: purged_snaps scrub ok 2024-03-21T13:22:41.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:41 smithi163 ceph-mon[40675]: osdmap e499: 8 total, 8 up, 8 in 2024-03-21T13:22:41.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:41 smithi163 ceph-mon[40675]: 3.c scrub starts 2024-03-21T13:22:41.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:41 smithi163 ceph-mon[40675]: 3.c scrub ok 2024-03-21T13:22:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:41 smithi204 ceph-mon[33153]: purged_snaps scrub starts 2024-03-21T13:22:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:41 smithi204 ceph-mon[33153]: purged_snaps scrub ok 2024-03-21T13:22:42.002 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:41 smithi204 ceph-mon[33153]: osdmap e499: 8 total, 8 up, 8 in 2024-03-21T13:22:42.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:41 smithi204 ceph-mon[33153]: 3.c scrub starts 2024-03-21T13:22:42.003 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:41 smithi204 ceph-mon[33153]: 3.c scrub ok 2024-03-21T13:22:42.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[40675]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:42.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[40675]: 4.15 scrub starts 2024-03-21T13:22:42.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[40675]: 4.15 scrub ok 2024-03-21T13:22:42.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[40675]: from='client.? 172.21.15.163:0/3645913313' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-21T13:22:42.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-21T13:22:42.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[40675]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-21T13:22:42.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:22:42.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:22:42.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:22:42.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[40675]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:22:42.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[29621]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:42.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[29621]: 4.15 scrub starts 2024-03-21T13:22:42.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[29621]: 4.15 scrub ok 2024-03-21T13:22:42.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[29621]: from='client.? 172.21.15.163:0/3645913313' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-21T13:22:42.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-21T13:22:42.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[29621]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-21T13:22:42.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:22:42.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:22:42.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:22:42.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-mon[29621]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:22:42.903 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:42] ENGINE Bus STOPPING 2024-03-21T13:22:42.903 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:42] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:22:42.903 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:42] ENGINE Bus STOPPED 2024-03-21T13:22:42.903 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:42] ENGINE Bus STARTING 2024-03-21T13:22:42.903 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:42] ENGINE Serving on http://:::9283 2024-03-21T13:22:42.903 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:42 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:42] ENGINE Bus STARTED 2024-03-21T13:22:42.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:22:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug there is no tcmu-runner data available 2024-03-21T13:22:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:42 smithi204 ceph-mon[33153]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:42 smithi204 ceph-mon[33153]: 4.15 scrub starts 2024-03-21T13:22:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:42 smithi204 ceph-mon[33153]: 4.15 scrub ok 2024-03-21T13:22:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:42 smithi204 ceph-mon[33153]: from='client.? 172.21.15.163:0/3645913313' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-21T13:22:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:42 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-21T13:22:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:42 smithi204 ceph-mon[33153]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-21T13:22:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:42 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:22:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:42 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:22:42.904 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:42 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:22:42.905 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:42 smithi204 ceph-mon[33153]: from='mgr.14595 172.21.15.163:0/4110179474' entity='mgr.y' 2024-03-21T13:22:43.263 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-21T13:22:43.273 INFO:tasks.cephadm:Teardown begin 2024-03-21T13:22:43.273 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:22:43.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:22:43.387 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-21T13:22:43.387 DEBUG:teuthology.orchestra.run.smithi163:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-21T13:22:43.427 DEBUG:teuthology.orchestra.run.smithi204:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-21T13:22:43.444 INFO:tasks.cephadm:Stopping all daemons... 2024-03-21T13:22:43.445 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-21T13:22:43.445 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.a 2024-03-21T13:22:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:43 smithi204 ceph-mon[33153]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:43 smithi204 ceph-mon[33153]: 3.12 deep-scrub starts 2024-03-21T13:22:43.752 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:43 smithi204 ceph-mon[33153]: 3.12 deep-scrub ok 2024-03-21T13:22:43.760 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:43 smithi163 systemd[1]: Stopping Ceph mon.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:22:43.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-mon[29621]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:43.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-mon[29621]: 3.12 deep-scrub starts 2024-03-21T13:22:43.761 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-mon[29621]: 3.12 deep-scrub ok 2024-03-21T13:22:43.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-mon[40675]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:43.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-mon[40675]: 3.12 deep-scrub starts 2024-03-21T13:22:43.761 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-mon[40675]: 3.12 deep-scrub ok 2024-03-21T13:22:44.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a[29617]: 2024-03-21T13:22:43.837+0000 7f8e2080b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:22:44.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a[29617]: 2024-03-21T13:22:43.837+0000 7f8e2080b700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-21T13:22:44.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:43 smithi163 podman[210148]: 2024-03-21 13:22:43.889533056 +0000 UTC m=+0.189867255 container died 46f326bcd8aaa2147b8c7d2a503206e204feb8959dc77dc2aed060e667598c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-21T13:22:44.231 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:43] ENGINE Bus STOPPING 2024-03-21T13:22:44.231 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:22:44.231 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:43] ENGINE Bus STOPPED 2024-03-21T13:22:44.231 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:43] ENGINE Bus STARTING 2024-03-21T13:22:44.231 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:43] ENGINE Serving on http://:::9283 2024-03-21T13:22:44.231 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:43 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:43] ENGINE Bus STARTED 2024-03-21T13:22:44.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:44 smithi163 podman[210148]: 2024-03-21 13:22:44.338445607 +0000 UTC m=+0.638779819 container cleanup 46f326bcd8aaa2147b8c7d2a503206e204feb8959dc77dc2aed060e667598c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-21T13:22:44.510 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:44 smithi163 bash[210148]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a 2024-03-21T13:22:44.798 INFO:journalctl@ceph.mon.a.smithi163.stdout:Mar 21 13:22:44 smithi163 podman[210178]: 2024-03-21 13:22:44.681434499 +0000 UTC m=+0.788422845 container remove 46f326bcd8aaa2147b8c7d2a503206e204feb8959dc77dc2aed060e667598c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-a, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-21T13:22:44.802 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.a.service' 2024-03-21T13:22:44.872 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:22:44.872 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-21T13:22:44.872 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-21T13:22:44.872 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.c 2024-03-21T13:22:45.185 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:44 smithi163 systemd[1]: Stopping Ceph mon.c for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:22:45.511 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-c[40671]: 2024-03-21T13:22:45.184+0000 7fbb853a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:22:45.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-c[40671]: 2024-03-21T13:22:45.184+0000 7fbb853a4700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-21T13:22:45.696 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:45 smithi163 podman[210288]: 2024-03-21 13:22:45.337704649 +0000 UTC m=+0.278063371 container died 4cd4267d4f7d15ff70a972c35873ce847efaaa263b241c1f1c0312f4b5a0297d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:22:45.696 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:45] ENGINE Bus STOPPING 2024-03-21T13:22:45.696 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:22:45.696 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:45] ENGINE Bus STOPPED 2024-03-21T13:22:45.696 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:45] ENGINE Bus STARTING 2024-03-21T13:22:45.696 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:45] ENGINE Serving on http://:::9283 2024-03-21T13:22:45.696 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: [21/Mar/2024:13:22:45] ENGINE Bus STARTED 2024-03-21T13:22:45.986 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:45 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: ::ffff:172.21.15.204 - - [21/Mar/2024:13:22:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:22:45.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:45 smithi163 podman[210288]: 2024-03-21 13:22:45.726740126 +0000 UTC m=+0.667098857 container cleanup 4cd4267d4f7d15ff70a972c35873ce847efaaa263b241c1f1c0312f4b5a0297d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-c, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0) 2024-03-21T13:22:45.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Mar 21 13:22:45 smithi163 bash[210288]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-c 2024-03-21T13:22:46.152 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.c.service' 2024-03-21T13:22:46.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:22:46.192 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-21T13:22:46.192 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-21T13:22:46.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.b 2024-03-21T13:22:46.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:46 smithi204 systemd[1]: Stopping Ceph mon.b for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:22:46.848 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:46 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-b[33137]: 2024-03-21T13:22:46.582+0000 7fce45f7e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:22:46.848 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:46 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-b[33137]: 2024-03-21T13:22:46.582+0000 7fce45f7e700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-21T13:22:47.131 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:46 smithi204 podman[151072]: 2024-03-21 13:22:46.848365939 +0000 UTC m=+0.367049963 container died defe8b351948e7ae80e2ec0459d01a01a81d49c88aed69b47f142dca27d3925d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-b, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-21T13:22:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:47 smithi204 podman[151072]: 2024-03-21 13:22:47.325109197 +0000 UTC m=+0.843793220 container cleanup defe8b351948e7ae80e2ec0459d01a01a81d49c88aed69b47f142dca27d3925d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:22:47.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Mar 21 13:22:47 smithi204 bash[151072]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mon-b 2024-03-21T13:22:47.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mon.b.service' 2024-03-21T13:22:47.745 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:22:47.745 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-21T13:22:47.745 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-21T13:22:47.745 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mgr.y 2024-03-21T13:22:48.048 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:47 smithi163 systemd[1]: Stopping Ceph mgr.y for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:22:48.390 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:48 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T13:22:48.046+0000 7fa35ea47700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:22:48.390 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:48 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y[29853]: 2024-03-21T13:22:48.046+0000 7fa35ea47700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-21T13:22:48.390 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:48 smithi163 podman[210420]: 2024-03-21 13:22:48.078538536 +0000 UTC m=+0.137428622 container died 3e827e3422520045d7a0789121ca254510860ac74ea8672debd1c48414cdcec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-21T13:22:48.983 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:48 smithi163 podman[210420]: 2024-03-21 13:22:48.648653357 +0000 UTC m=+0.707543446 container cleanup 3e827e3422520045d7a0789121ca254510860ac74ea8672debd1c48414cdcec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-21T13:22:48.983 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Mar 21 13:22:48 smithi163 bash[210420]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-y 2024-03-21T13:22:49.127 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-21T13:22:49.165 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:22:49.166 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-21T13:22:49.166 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-21T13:22:49.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mgr.x 2024-03-21T13:22:49.502 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 13:22:49 smithi204 systemd[1]: Stopping Ceph mgr.x for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:22:49.806 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 13:22:49 smithi204 podman[151195]: 2024-03-21 13:22:49.538081669 +0000 UTC m=+0.175674154 container died 0d0e832d97240c0ba84b7863d5e665e9164584e8fae32e49b81b2eaccb93c346 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-21T13:22:50.181 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 13:22:49 smithi204 podman[151195]: 2024-03-21 13:22:49.905355835 +0000 UTC m=+0.542948317 container cleanup 0d0e832d97240c0ba84b7863d5e665e9164584e8fae32e49b81b2eaccb93c346 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:22:50.181 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Mar 21 13:22:49 smithi204 bash[151195]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-mgr-x 2024-03-21T13:22:50.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-21T13:22:50.316 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:22:50.316 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-21T13:22:50.316 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-21T13:22:50.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.0 2024-03-21T13:22:50.695 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:50 smithi163 systemd[1]: Stopping Ceph osd.0 for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:22:51.010 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:50 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0[44656]: 2024-03-21T13:22:50.695+0000 7fb7d25fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:22:51.011 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:50 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0[44656]: 2024-03-21T13:22:50.695+0000 7fb7d25fb700 -1 osd.0 499 *** Got signal Terminated *** 2024-03-21T13:22:51.011 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:50 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0[44656]: 2024-03-21T13:22:50.695+0000 7fb7d25fb700 -1 osd.0 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:22:56.510 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:56 smithi163 podman[210541]: 2024-03-21 13:22:56.288783525 +0000 UTC m=+5.771106286 container died 5594de8973e08c9af9200fdc4cf4973ebe4dc1c53135b9b9f03e3fc15c888c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-21T13:22:56.938 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:56 smithi163 podman[210541]: 2024-03-21 13:22:56.695251023 +0000 UTC m=+6.177573794 container cleanup 5594de8973e08c9af9200fdc4cf4973ebe4dc1c53135b9b9f03e3fc15c888c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-21T13:22:56.938 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:56 smithi163 bash[210541]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0 2024-03-21T13:22:57.257 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:56 smithi163 podman[210561]: 2024-03-21 13:22:56.938196618 +0000 UTC m=+0.648694821 container remove 5594de8973e08c9af9200fdc4cf4973ebe4dc1c53135b9b9f03e3fc15c888c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-03-21T13:22:57.510 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:57 smithi163 podman[210653]: 2024-03-21 13:22:57.160432195 +0000 UTC m=+0.018735588 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:22:57.511 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:57 smithi163 podman[210653]: 2024-03-21 13:22:57.305769439 +0000 UTC m=+0.164072831 container create 63b353161ca6ab69e99a6f06778d138d1bb57f53d1b52fccc7539f0743aae31f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-21T13:22:57.947 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:57 smithi163 podman[210653]: 2024-03-21 13:22:57.56685485 +0000 UTC m=+0.425158243 container init 63b353161ca6ab69e99a6f06778d138d1bb57f53d1b52fccc7539f0743aae31f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:22:57.947 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:57 smithi163 podman[210653]: 2024-03-21 13:22:57.570373877 +0000 UTC m=+0.428677274 container start 63b353161ca6ab69e99a6f06778d138d1bb57f53d1b52fccc7539f0743aae31f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:22:57.947 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:57 smithi163 podman[210653]: 2024-03-21 13:22:57.650065586 +0000 UTC m=+0.508368992 container attach 63b353161ca6ab69e99a6f06778d138d1bb57f53d1b52fccc7539f0743aae31f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-21T13:22:58.216 INFO:journalctl@ceph.osd.0.smithi163.stdout:Mar 21 13:22:57 smithi163 podman[210653]: 2024-03-21 13:22:57.947622178 +0000 UTC m=+0.805925571 container died 63b353161ca6ab69e99a6f06778d138d1bb57f53d1b52fccc7539f0743aae31f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-21T13:22:58.776 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.0.service' 2024-03-21T13:22:58.820 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:22:58.820 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-21T13:22:58.820 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-21T13:22:58.820 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.1 2024-03-21T13:22:59.010 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:22:58 smithi163 systemd[1]: Stopping Ceph osd.1 for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:22:59.510 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:22:59 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1[50228]: 2024-03-21T13:22:59.182+0000 7f48920d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:22:59.511 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:22:59 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1[50228]: 2024-03-21T13:22:59.182+0000 7f48920d0700 -1 osd.1 499 *** Got signal Terminated *** 2024-03-21T13:22:59.511 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:22:59 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1[50228]: 2024-03-21T13:22:59.182+0000 7f48920d0700 -1 osd.1 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:23:04.866 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:23:04 smithi163 podman[210760]: 2024-03-21 13:23:04.566095257 +0000 UTC m=+5.525092319 container died be8c730f5703ca98d4afda4969d55f207347b5699a1d29333fe39ad5e23d7fb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:23:05.260 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:23:05 smithi163 podman[210760]: 2024-03-21 13:23:05.074091441 +0000 UTC m=+6.033088492 container cleanup be8c730f5703ca98d4afda4969d55f207347b5699a1d29333fe39ad5e23d7fb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-21T13:23:05.261 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:23:05 smithi163 bash[210760]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1 2024-03-21T13:23:05.760 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:23:05 smithi163 podman[210779]: 2024-03-21 13:23:05.392496823 +0000 UTC m=+0.820475364 container remove be8c730f5703ca98d4afda4969d55f207347b5699a1d29333fe39ad5e23d7fb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:23:06.096 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:23:05 smithi163 podman[210870]: 2024-03-21 13:23:05.740895693 +0000 UTC m=+0.018059546 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:23:06.096 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:23:05 smithi163 podman[210870]: 2024-03-21 13:23:05.860329731 +0000 UTC m=+0.137493627 container create 857598d9f6c7220e4b9ecd17e3f0d18cf8f9e086828f12d3296b86930dd52b6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-21T13:23:06.096 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:23:06 smithi163 podman[210870]: 2024-03-21 13:23:06.096456195 +0000 UTC m=+0.373620065 container init 857598d9f6c7220e4b9ecd17e3f0d18cf8f9e086828f12d3296b86930dd52b6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-21T13:23:06.387 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:23:06 smithi163 podman[210870]: 2024-03-21 13:23:06.099982787 +0000 UTC m=+0.377146641 container start 857598d9f6c7220e4b9ecd17e3f0d18cf8f9e086828f12d3296b86930dd52b6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:23:06.387 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:23:06 smithi163 podman[210870]: 2024-03-21 13:23:06.187692818 +0000 UTC m=+0.464856689 container attach 857598d9f6c7220e4b9ecd17e3f0d18cf8f9e086828f12d3296b86930dd52b6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-03-21T13:23:06.750 INFO:journalctl@ceph.osd.1.smithi163.stdout:Mar 21 13:23:06 smithi163 podman[210870]: 2024-03-21 13:23:06.388120158 +0000 UTC m=+0.665284018 container died 857598d9f6c7220e4b9ecd17e3f0d18cf8f9e086828f12d3296b86930dd52b6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-21T13:23:07.063 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.1.service' 2024-03-21T13:23:07.105 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:23:07.105 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-21T13:23:07.105 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-21T13:23:07.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.2 2024-03-21T13:23:07.486 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:07 smithi163 systemd[1]: Stopping Ceph osd.2 for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:23:07.761 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:07 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2[56029]: 2024-03-21T13:23:07.485+0000 7fd062484700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:23:07.761 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:07 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2[56029]: 2024-03-21T13:23:07.485+0000 7fd062484700 -1 osd.2 499 *** Got signal Terminated *** 2024-03-21T13:23:07.761 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:07 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2[56029]: 2024-03-21T13:23:07.485+0000 7fd062484700 -1 osd.2 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:23:08.002 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:23:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:23:07.704Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.163:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.163:8765: connect: connection refused" 2024-03-21T13:23:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:23:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:23:07.708Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.163:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.163:8765: connect: connection refused" 2024-03-21T13:23:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:23:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:23:07.709Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.163:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.163:8765: connect: connection refused" 2024-03-21T13:23:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:23:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:23:07.714Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.163:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.163:8765: connect: connection refused" 2024-03-21T13:23:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:23:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:23:07.717Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.163:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.163:8765: connect: connection refused" 2024-03-21T13:23:13.336 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:13 smithi163 podman[210977]: 2024-03-21 13:23:13.033947596 +0000 UTC m=+5.705421945 container died d2b10e7c2fba2543508ddc9d889b745c66092e710f7100571a4b72a3d2dae8cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-21T13:23:13.760 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:13 smithi163 podman[210977]: 2024-03-21 13:23:13.502361075 +0000 UTC m=+6.173835431 container cleanup d2b10e7c2fba2543508ddc9d889b745c66092e710f7100571a4b72a3d2dae8cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-21T13:23:13.760 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:13 smithi163 bash[210977]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2 2024-03-21T13:23:14.172 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:13 smithi163 podman[210995]: 2024-03-21 13:23:13.853945953 +0000 UTC m=+0.818595983 container remove d2b10e7c2fba2543508ddc9d889b745c66092e710f7100571a4b72a3d2dae8cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-21T13:23:14.510 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:14 smithi163 podman[211088]: 2024-03-21 13:23:14.075590431 +0000 UTC m=+0.017780797 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:23:14.511 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:14 smithi163 podman[211088]: 2024-03-21 13:23:14.179701298 +0000 UTC m=+0.121891684 container create a1314f20b2842bddb97b4346bd41ba9e0b921fea958455908843b1850d8a7987 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:23:14.511 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:14 smithi163 podman[211088]: 2024-03-21 13:23:14.415876506 +0000 UTC m=+0.358066860 container init a1314f20b2842bddb97b4346bd41ba9e0b921fea958455908843b1850d8a7987 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-21T13:23:14.511 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:14 smithi163 podman[211088]: 2024-03-21 13:23:14.419318653 +0000 UTC m=+0.361509016 container start a1314f20b2842bddb97b4346bd41ba9e0b921fea958455908843b1850d8a7987 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, org.label-schema.schema-version=1.0) 2024-03-21T13:23:15.010 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:14 smithi163 podman[211088]: 2024-03-21 13:23:14.515259675 +0000 UTC m=+0.457450040 container attach a1314f20b2842bddb97b4346bd41ba9e0b921fea958455908843b1850d8a7987 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-21T13:23:15.011 INFO:journalctl@ceph.osd.2.smithi163.stdout:Mar 21 13:23:14 smithi163 podman[211088]: 2024-03-21 13:23:14.720403116 +0000 UTC m=+0.662593456 container died a1314f20b2842bddb97b4346bd41ba9e0b921fea958455908843b1850d8a7987 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-03-21T13:23:15.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.2.service' 2024-03-21T13:23:15.441 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:23:15.441 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-21T13:23:15.441 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-21T13:23:15.442 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.3 2024-03-21T13:23:15.680 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:15 smithi163 systemd[1]: Stopping Ceph osd.3 for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:23:16.010 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3[62023]: 2024-03-21T13:23:15.789+0000 7f8ba21ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:23:16.011 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3[62023]: 2024-03-21T13:23:15.789+0000 7f8ba21ec700 -1 osd.3 499 *** Got signal Terminated *** 2024-03-21T13:23:16.011 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:15 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3[62023]: 2024-03-21T13:23:15.789+0000 7f8ba21ec700 -1 osd.3 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:23:17.012 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:16 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:16.758+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:18.002 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:17 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:17.735+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:18.003 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:17 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:17.896+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:54.883680+0000 front 2024-03-21T13:22:54.883753+0000 (oldest deadline 2024-03-21T13:23:17.183517+0000) 2024-03-21T13:23:18.003 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:17 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:17.916+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:18.260 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:17 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3[62023]: 2024-03-21T13:23:17.857+0000 7f8b9a7d8700 -1 osd.3 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:52.387842+0000 front 2024-03-21T13:22:52.387998+0000 (oldest deadline 2024-03-21T13:23:17.687999+0000) 2024-03-21T13:23:19.213 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:18 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:18.877+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:54.883680+0000 front 2024-03-21T13:22:54.883753+0000 (oldest deadline 2024-03-21T13:23:17.183517+0000) 2024-03-21T13:23:19.214 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:18 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:18.751+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:19.214 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:18 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:18.952+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:19.260 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:18 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3[62023]: 2024-03-21T13:23:18.831+0000 7f8b9a7d8700 -1 osd.3 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:52.387842+0000 front 2024-03-21T13:22:52.387998+0000 (oldest deadline 2024-03-21T13:23:17.687999+0000) 2024-03-21T13:23:20.130 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:19 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:19.833+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:54.883680+0000 front 2024-03-21T13:22:54.883753+0000 (oldest deadline 2024-03-21T13:23:17.183517+0000) 2024-03-21T13:23:20.130 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:19 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:19.937+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:20.130 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:19 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:19.783+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:20.260 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:19 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3[62023]: 2024-03-21T13:23:19.840+0000 7f8b9a7d8700 -1 osd.3 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:52.387842+0000 front 2024-03-21T13:22:52.387998+0000 (oldest deadline 2024-03-21T13:23:17.687999+0000) 2024-03-21T13:23:20.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:20 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:20.129+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:21.127 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:20 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:20.825+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:54.883680+0000 front 2024-03-21T13:22:54.883753+0000 (oldest deadline 2024-03-21T13:23:17.183517+0000) 2024-03-21T13:23:21.128 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:20 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:20.909+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:21.128 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:20 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:20.778+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:21.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:21.127+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:21.697 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:21 smithi163 podman[211192]: 2024-03-21 13:23:21.407157767 +0000 UTC m=+5.755557171 container died 55f1ae82d47ba606d3348c03e737f064d198a587b1240b2732284821a2dcc152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-21T13:23:22.010 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:21 smithi163 podman[211192]: 2024-03-21 13:23:21.872073484 +0000 UTC m=+6.220472904 container cleanup 55f1ae82d47ba606d3348c03e737f064d198a587b1240b2732284821a2dcc152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:23:22.010 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:21 smithi163 bash[211192]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3 2024-03-21T13:23:22.116 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:21.819+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:54.883680+0000 front 2024-03-21T13:22:54.883753+0000 (oldest deadline 2024-03-21T13:23:17.183517+0000) 2024-03-21T13:23:22.117 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:21.764+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:22.117 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:21 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:21.945+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:22.452 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:22 smithi163 podman[211221]: 2024-03-21 13:23:22.148349677 +0000 UTC m=+0.737656333 container remove 55f1ae82d47ba606d3348c03e737f064d198a587b1240b2732284821a2dcc152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True) 2024-03-21T13:23:22.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:22.116+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:22.705 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:22 smithi163 podman[211315]: 2024-03-21 13:23:22.355211903 +0000 UTC m=+0.018191513 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:23:22.705 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:22 smithi163 podman[211315]: 2024-03-21 13:23:22.457271952 +0000 UTC m=+0.120251549 container create aa7dc64acfa49f2639d9747880e6bb4adea0bf1599b28a1b6f4162bb92240f29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-21T13:23:22.706 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:22 smithi163 podman[211315]: 2024-03-21 13:23:22.702237715 +0000 UTC m=+0.365217318 container init aa7dc64acfa49f2639d9747880e6bb4adea0bf1599b28a1b6f4162bb92240f29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:23:22.706 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:22 smithi163 podman[211315]: 2024-03-21 13:23:22.705767301 +0000 UTC m=+0.368746896 container start aa7dc64acfa49f2639d9747880e6bb4adea0bf1599b28a1b6f4162bb92240f29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:23:22.997 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:22 smithi163 podman[211315]: 2024-03-21 13:23:22.802018411 +0000 UTC m=+0.464998017 container attach aa7dc64acfa49f2639d9747880e6bb4adea0bf1599b28a1b6f4162bb92240f29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux ) 2024-03-21T13:23:23.077 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:22.771+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:54.883680+0000 front 2024-03-21T13:22:54.883753+0000 (oldest deadline 2024-03-21T13:23:17.183517+0000) 2024-03-21T13:23:23.077 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:22.942+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:23.077 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:22 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:22.777+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:23.260 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 21 13:23:22 smithi163 podman[211315]: 2024-03-21 13:23:22.998401437 +0000 UTC m=+0.661381042 container died aa7dc64acfa49f2639d9747880e6bb4adea0bf1599b28a1b6f4162bb92240f29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-3-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-21T13:23:23.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:23 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:23.076+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:23.664 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.3.service' 2024-03-21T13:23:23.706 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:23:23.707 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-21T13:23:23.707 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-21T13:23:23.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.4 2024-03-21T13:23:24.002 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:23 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:23.823+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:24.002 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:23 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:23.981+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:24.003 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:23 smithi204 systemd[1]: Stopping Ceph osd.4 for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:23:24.003 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:23 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:23.797+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:54.883680+0000 front 2024-03-21T13:22:54.883753+0000 (oldest deadline 2024-03-21T13:23:17.183517+0000) 2024-03-21T13:23:24.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:24 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:24.055+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:24.503 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:24 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:24.024+0000 7fcb7ddcc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:23:24.503 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:24 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:24.024+0000 7fcb7ddcc700 -1 osd.4 499 *** Got signal Terminated *** 2024-03-21T13:23:24.503 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:24 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:24.024+0000 7fcb7ddcc700 -1 osd.4 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:23:25.252 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:24 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:24.822+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:25.252 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:24 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:24.789+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:54.883680+0000 front 2024-03-21T13:22:54.883753+0000 (oldest deadline 2024-03-21T13:23:17.183517+0000) 2024-03-21T13:23:25.253 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:25 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:25.034+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:25.253 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:24 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:24.962+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:25.253 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:24 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:24.962+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:26.252 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:25 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:25.801+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:54.883680+0000 front 2024-03-21T13:22:54.883753+0000 (oldest deadline 2024-03-21T13:23:17.183517+0000) 2024-03-21T13:23:26.252 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:25 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:25.800+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:26.253 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:26.001+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:26.253 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:26.002+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:26.253 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:26.002+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:27.252 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:27 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:27.004+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:27.253 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:26.775+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:54.883680+0000 front 2024-03-21T13:22:54.883753+0000 (oldest deadline 2024-03-21T13:23:17.183517+0000) 2024-03-21T13:23:27.253 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:26.775+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:01.284554+0000 front 2024-03-21T13:23:01.284649+0000 (oldest deadline 2024-03-21T13:23:26.584482+0000) 2024-03-21T13:23:27.253 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:26.800+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:27.253 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:26 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:26.800+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:27.253 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:27 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:27.001+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:27.253 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:27 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:27.001+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:28.045 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:27 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:27.753+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:54.883680+0000 front 2024-03-21T13:22:54.883753+0000 (oldest deadline 2024-03-21T13:23:17.183517+0000) 2024-03-21T13:23:28.045 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:27 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:27.753+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:01.284554+0000 front 2024-03-21T13:23:01.284649+0000 (oldest deadline 2024-03-21T13:23:26.584482+0000) 2024-03-21T13:23:28.045 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:27 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:27.843+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:28.045 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:27 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:27.843+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:28.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:28 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:28.045+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:28.503 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:28 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:28.045+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:28.503 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:28 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:28.045+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:29.002 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:28 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:28.704+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:54.883680+0000 front 2024-03-21T13:22:54.883753+0000 (oldest deadline 2024-03-21T13:23:17.183517+0000) 2024-03-21T13:23:29.003 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:28 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4[38173]: 2024-03-21T13:23:28.704+0000 7fcb763b8700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:01.284554+0000 front 2024-03-21T13:23:01.284649+0000 (oldest deadline 2024-03-21T13:23:26.584482+0000) 2024-03-21T13:23:29.003 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:28 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:28.873+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:29.003 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:28 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:28.873+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:29.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:29 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:29.082+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:29.503 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:29 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:29.082+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:29.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:29 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:29.055+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:29.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:29 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:29.055+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:29.910 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:29 smithi204 podman[151332]: 2024-03-21 13:23:29.61938642 +0000 UTC m=+5.712051276 container died b1c822ea800f2c1b235f9c8defbe1e010800d21075fb98f5118aba45a45d8b85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:23:30.252 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:30 smithi204 podman[151332]: 2024-03-21 13:23:30.008761736 +0000 UTC m=+6.101426592 container cleanup b1c822ea800f2c1b235f9c8defbe1e010800d21075fb98f5118aba45a45d8b85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-21T13:23:30.253 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:30 smithi204 bash[151332]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4 2024-03-21T13:23:30.253 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:29 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:29.913+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:30.253 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:29 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:29.913+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:30.254 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:30 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:30.097+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:30.254 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:30 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:30.097+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:30.255 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:30 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:30.032+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:30.255 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:30 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:30.032+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:30.674 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:30 smithi204 podman[151351]: 2024-03-21 13:23:30.327057488 +0000 UTC m=+0.679155671 container remove b1c822ea800f2c1b235f9c8defbe1e010800d21075fb98f5118aba45a45d8b85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-21T13:23:30.937 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:30 smithi204 podman[151443]: 2024-03-21 13:23:30.577611876 +0000 UTC m=+0.018451663 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:23:30.937 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:30 smithi204 podman[151443]: 2024-03-21 13:23:30.686245382 +0000 UTC m=+0.127085163 container create 0c08cc090b33472761bf28dca4b053803cabd0eb746dc6433454bd230cac493b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-21T13:23:30.937 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:30 smithi204 podman[151443]: 2024-03-21 13:23:30.880375007 +0000 UTC m=+0.321214794 container init 0c08cc090b33472761bf28dca4b053803cabd0eb746dc6433454bd230cac493b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:23:30.937 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:30 smithi204 podman[151443]: 2024-03-21 13:23:30.883424074 +0000 UTC m=+0.324263856 container start 0c08cc090b33472761bf28dca4b053803cabd0eb746dc6433454bd230cac493b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-21T13:23:31.252 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:30 smithi204 podman[151443]: 2024-03-21 13:23:30.980627517 +0000 UTC m=+0.421467300 container attach 0c08cc090b33472761bf28dca4b053803cabd0eb746dc6433454bd230cac493b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:23:31.253 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:30 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:30.936+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:31.253 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:30 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:30.936+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:31.253 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:31.120+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:31.253 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:31.120+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:31.254 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:31.005+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:31.254 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:31.005+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:31.894 INFO:journalctl@ceph.osd.4.smithi204.stdout:Mar 21 13:23:31 smithi204 podman[151443]: 2024-03-21 13:23:31.621108972 +0000 UTC m=+1.061948749 container died 0c08cc090b33472761bf28dca4b053803cabd0eb746dc6433454bd230cac493b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:23:32.253 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:31.893+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:32.253 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:31 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:31.893+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:32.253 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:32.133+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:32.253 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:32.133+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:32.253 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:32.039+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:32.253 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:32.039+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:32.278 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.4.service' 2024-03-21T13:23:32.321 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:23:32.321 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-21T13:23:32.322 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-21T13:23:32.322 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.5 2024-03-21T13:23:32.654 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:32 smithi204 systemd[1]: Stopping Ceph osd.5 for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:23:33.002 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:32.653+0000 7f38ac9d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:23:33.002 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:32.653+0000 7f38ac9d0700 -1 osd.5 499 *** Got signal Terminated *** 2024-03-21T13:23:33.002 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:32.653+0000 7f38ac9d0700 -1 osd.5 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:23:33.002 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:32.858+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:33.003 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:32 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:32.859+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:33.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:33 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:33.159+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:33.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:33 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:33.159+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:33.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:33 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:33.052+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:33.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:33 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:33.052+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:34.168 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:34 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:34.060+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:34.169 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:34 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:34.060+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:34.169 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:34 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:34.060+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:34.169 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:33 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:33.873+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:34.169 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:33 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:33.873+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:34.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:34 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:34.167+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:34.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:34 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:34.167+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:35.201 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:34 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:34.874+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:35.201 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:34 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:34.874+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:35.201 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:35 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:35.055+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:35.202 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:35 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:35.055+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:35.202 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:35 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:35.055+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:35.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:35 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:35.201+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:35.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:35 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:35.201+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:35.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:35 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:35.201+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:12.273389+0000 front 2024-03-21T13:23:12.273365+0000 (oldest deadline 2024-03-21T13:23:34.573182+0000) 2024-03-21T13:23:36.247 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:36 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:36.021+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:36.248 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:36 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:36.021+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:36.248 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:36 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:36.021+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:36.248 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:35 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:35.909+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:36.248 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:35 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:35.909+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:36.248 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:35 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:35.909+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:11.727218+0000 front 2024-03-21T13:23:11.727274+0000 (oldest deadline 2024-03-21T13:23:35.227131+0000) 2024-03-21T13:23:36.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:36 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:36.247+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:36.503 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:36 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:36.247+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:36.503 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:36 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:36.247+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:12.273389+0000 front 2024-03-21T13:23:12.273365+0000 (oldest deadline 2024-03-21T13:23:34.573182+0000) 2024-03-21T13:23:37.235 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:36 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:36.898+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:37.235 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:36 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:36.898+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:37.235 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:36 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:36.898+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:11.727218+0000 front 2024-03-21T13:23:11.727274+0000 (oldest deadline 2024-03-21T13:23:35.227131+0000) 2024-03-21T13:23:37.235 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:37 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:37.046+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:37.236 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:37 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:37.046+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:37.236 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:37 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:37.046+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:37.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:37 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:37.235+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:37.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:37 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:37.235+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:37.503 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:37 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:37.235+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:12.273389+0000 front 2024-03-21T13:23:12.273365+0000 (oldest deadline 2024-03-21T13:23:34.573182+0000) 2024-03-21T13:23:38.489 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:38.232+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:38.489 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:38.232+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:38.489 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:38.232+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:12.273389+0000 front 2024-03-21T13:23:12.273365+0000 (oldest deadline 2024-03-21T13:23:34.573182+0000) 2024-03-21T13:23:38.490 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:38.018+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:38.490 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:38.018+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:38.490 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:38.018+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:38.491 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: *** Caught signal (Segmentation fault) ** 2024-03-21T13:23:38.491 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: in thread 7f3893f9a700 thread_name:safe_timer 2024-03-21T13:23:38.491 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev) 2024-03-21T13:23:38.491 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38b3e26d20] 2024-03-21T13:23:38.491 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55b0a91accda] 2024-03-21T13:23:38.491 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55b0a923ab68] 2024-03-21T13:23:38.491 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55b0a8e1eda3] 2024-03-21T13:23:38.492 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55b0a90abad1] 2024-03-21T13:23:38.492 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55b0a8d99d74] 2024-03-21T13:23:38.492 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55b0a8d9bdca] 2024-03-21T13:23:38.492 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55b0a8d9dd33] 2024-03-21T13:23:38.492 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55b0a8cfefba] 2024-03-21T13:23:38.492 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x55b0a935b505] 2024-03-21T13:23:38.492 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55b0a935cd31] 2024-03-21T13:23:38.492 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f38b3e1c1ca] 2024-03-21T13:23:38.492 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 13: clone() 2024-03-21T13:23:38.492 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:38.059+0000 7f3893f9a700 -1 *** Caught signal (Segmentation fault) ** 2024-03-21T13:23:38.492 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: in thread 7f3893f9a700 thread_name:safe_timer 2024-03-21T13:23:38.492 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev) 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38b3e26d20] 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55b0a91accda] 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55b0a923ab68] 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55b0a8e1eda3] 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55b0a90abad1] 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55b0a8d99d74] 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55b0a8d9bdca] 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55b0a8d9dd33] 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55b0a8cfefba] 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x55b0a935b505] 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55b0a935cd31] 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f38b3e1c1ca] 2024-03-21T13:23:38.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 13: clone() 2024-03-21T13:23:38.494 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-21T13:23:38.494 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:38.494 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -3847> 2024-03-21T13:23:16.758+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.494 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -3139> 2024-03-21T13:23:17.735+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.494 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -2994> 2024-03-21T13:23:18.751+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.494 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -2848> 2024-03-21T13:23:19.783+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.494 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -2806> 2024-03-21T13:23:20.778+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.494 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -2552> 2024-03-21T13:23:21.764+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.494 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -2422> 2024-03-21T13:23:22.777+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.494 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -2352> 2024-03-21T13:23:23.823+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.494 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -2302> 2024-03-21T13:23:24.822+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.494 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1707> 2024-03-21T13:23:25.800+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.495 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1591> 2024-03-21T13:23:26.800+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.495 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1590> 2024-03-21T13:23:26.800+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:38.495 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1485> 2024-03-21T13:23:27.843+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.495 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1484> 2024-03-21T13:23:27.843+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:38.495 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1403> 2024-03-21T13:23:28.873+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.495 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1402> 2024-03-21T13:23:28.873+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:38.495 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1166> 2024-03-21T13:23:29.913+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.495 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1165> 2024-03-21T13:23:29.913+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:38.496 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1099> 2024-03-21T13:23:30.936+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.496 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1098> 2024-03-21T13:23:30.936+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:38.496 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1037> 2024-03-21T13:23:31.893+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.496 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1036> 2024-03-21T13:23:31.893+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:38.496 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1001> 2024-03-21T13:23:32.653+0000 7f38ac9d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:23:38.496 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -1000> 2024-03-21T13:23:32.653+0000 7f38ac9d0700 -1 osd.5 499 *** Got signal Terminated *** 2024-03-21T13:23:38.496 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -998> 2024-03-21T13:23:32.653+0000 7f38ac9d0700 -1 osd.5 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:23:38.496 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -955> 2024-03-21T13:23:32.858+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.496 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -954> 2024-03-21T13:23:32.859+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:38.496 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -472> 2024-03-21T13:23:33.873+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.496 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -471> 2024-03-21T13:23:33.873+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:38.496 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -418> 2024-03-21T13:23:34.874+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.496 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -417> 2024-03-21T13:23:34.874+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:38.497 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -365> 2024-03-21T13:23:35.909+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.497 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -364> 2024-03-21T13:23:35.909+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:38.497 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -363> 2024-03-21T13:23:35.909+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:11.727218+0000 front 2024-03-21T13:23:11.727274+0000 (oldest deadline 2024-03-21T13:23:35.227131+0000) 2024-03-21T13:23:38.497 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -267> 2024-03-21T13:23:36.898+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:50.624695+0000 front 2024-03-21T13:22:50.624754+0000 (oldest deadline 2024-03-21T13:23:16.524671+0000) 2024-03-21T13:23:38.497 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -266> 2024-03-21T13:23:36.898+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:04.126071+0000 front 2024-03-21T13:23:04.126226+0000 (oldest deadline 2024-03-21T13:23:26.426074+0000) 2024-03-21T13:23:38.497 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: -265> 2024-03-21T13:23:36.898+0000 7f38a4fbc700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:11.727218+0000 front 2024-03-21T13:23:11.727274+0000 (oldest deadline 2024-03-21T13:23:35.227131+0000) 2024-03-21T13:23:38.497 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 0> 2024-03-21T13:23:38.059+0000 7f3893f9a700 -1 *** Caught signal (Segmentation fault) ** 2024-03-21T13:23:38.497 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: in thread 7f3893f9a700 thread_name:safe_timer 2024-03-21T13:23:38.497 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:38.497 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev) 2024-03-21T13:23:38.497 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38b3e26d20] 2024-03-21T13:23:38.497 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55b0a91accda] 2024-03-21T13:23:38.497 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55b0a923ab68] 2024-03-21T13:23:38.498 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55b0a8e1eda3] 2024-03-21T13:23:38.498 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55b0a90abad1] 2024-03-21T13:23:38.498 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55b0a8d99d74] 2024-03-21T13:23:38.498 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55b0a8d9bdca] 2024-03-21T13:23:38.498 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55b0a8d9dd33] 2024-03-21T13:23:38.498 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55b0a8cfefba] 2024-03-21T13:23:38.498 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x55b0a935b505] 2024-03-21T13:23:38.498 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55b0a935cd31] 2024-03-21T13:23:38.502 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f38b3e1c1ca] 2024-03-21T13:23:38.502 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 13: clone() 2024-03-21T13:23:38.502 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-21T13:23:38.502 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5[43727]: 2024-03-21T13:23:38.502 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 podman[151558]: 2024-03-21 13:23:38.204261469 +0000 UTC m=+5.651418672 container died 3fcae0bda113c924c807c26e31e4f9fe5adccc00ade7ed39f11117d9e13f6cb6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:23:38.752 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 podman[151558]: 2024-03-21 13:23:38.563057183 +0000 UTC m=+6.010214394 container cleanup 3fcae0bda113c924c807c26e31e4f9fe5adccc00ade7ed39f11117d9e13f6cb6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:23:38.753 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 bash[151558]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5 2024-03-21T13:23:39.038 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 podman[151575]: 2024-03-21 13:23:38.781329293 +0000 UTC m=+0.571568112 container remove 3fcae0bda113c924c807c26e31e4f9fe5adccc00ade7ed39f11117d9e13f6cb6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-21T13:23:39.038 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:38 smithi204 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.5.service: Main process exited, code=exited, status=139/n/a 2024-03-21T13:23:39.318 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:39 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:39.037+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:39.318 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:39 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:39.037+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:39.318 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:39 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:39.037+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:39.319 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:39 smithi204 podman[151665]: 2024-03-21 13:23:38.995794188 +0000 UTC m=+0.017714948 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:23:39.319 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:39 smithi204 podman[151665]: 2024-03-21 13:23:39.10750872 +0000 UTC m=+0.129429543 container create ff7529c41964ad9d95258ab2e2df70c1c42e046005512c009dc00c63df451b28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:23:39.319 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:39 smithi204 podman[151665]: 2024-03-21 13:23:39.3186769 +0000 UTC m=+0.340597668 container init ff7529c41964ad9d95258ab2e2df70c1c42e046005512c009dc00c63df451b28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:23:39.319 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:39 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:39.230+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:39.319 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:39 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:39.230+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:39.319 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:39 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:39.230+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:12.273389+0000 front 2024-03-21T13:23:12.273365+0000 (oldest deadline 2024-03-21T13:23:34.573182+0000) 2024-03-21T13:23:39.618 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:39 smithi204 podman[151665]: 2024-03-21 13:23:39.322330915 +0000 UTC m=+0.344251676 container start ff7529c41964ad9d95258ab2e2df70c1c42e046005512c009dc00c63df451b28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2) 2024-03-21T13:23:39.618 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:39 smithi204 podman[151665]: 2024-03-21 13:23:39.426612776 +0000 UTC m=+0.448533549 container attach ff7529c41964ad9d95258ab2e2df70c1c42e046005512c009dc00c63df451b28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:23:39.938 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 21 13:23:39 smithi204 podman[151665]: 2024-03-21 13:23:39.618261309 +0000 UTC m=+0.640182078 container died ff7529c41964ad9d95258ab2e2df70c1c42e046005512c009dc00c63df451b28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-21T13:23:40.196 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:40 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:40.015+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:40.197 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:40 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:40.015+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:40.197 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:40 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:40.015+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:40.255 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.5.service' 2024-03-21T13:23:40.299 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:23:40.299 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-21T13:23:40.299 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-21T13:23:40.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.6 2024-03-21T13:23:40.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:40 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:40.196+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:40.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:40 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:40.196+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:40.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:40 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:40.196+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:12.273389+0000 front 2024-03-21T13:23:12.273365+0000 (oldest deadline 2024-03-21T13:23:34.573182+0000) 2024-03-21T13:23:40.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:40 smithi204 systemd[1]: Stopping Ceph osd.6 for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:23:40.975 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:40 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:40.633+0000 7f32704da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:23:40.975 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:40 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:40.633+0000 7f32704da700 -1 osd.6 499 *** Got signal Terminated *** 2024-03-21T13:23:40.975 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:40 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:40.633+0000 7f32704da700 -1 osd.6 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:23:41.253 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:41.203+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:41.253 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:41.203+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:41.253 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:41 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:41.203+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:12.273389+0000 front 2024-03-21T13:23:12.273365+0000 (oldest deadline 2024-03-21T13:23:34.573182+0000) 2024-03-21T13:23:41.253 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:40 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:40.975+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:41.253 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:40 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:40.975+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:41.253 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:40 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:40.975+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:42.339 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:42.194+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:42.339 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:42.194+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:42.339 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:42.194+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:12.273389+0000 front 2024-03-21T13:23:12.273365+0000 (oldest deadline 2024-03-21T13:23:34.573182+0000) 2024-03-21T13:23:42.339 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:42.023+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:42.339 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:42.023+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:42.340 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:42 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:42.023+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:43.187 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:43 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:43.034+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:43.187 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:43 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:43.034+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:43.188 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:43 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:43.034+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:43.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:43 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:43.186+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:43.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:43 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:43.186+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:43.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:43 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:43.186+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:12.273389+0000 front 2024-03-21T13:23:12.273365+0000 (oldest deadline 2024-03-21T13:23:34.573182+0000) 2024-03-21T13:23:43.503 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:43 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:43.187+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-03-21T13:23:19.774532+0000 front 2024-03-21T13:23:19.774543+0000 (oldest deadline 2024-03-21T13:23:42.674533+0000) 2024-03-21T13:23:44.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:44 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:44.168+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:44.503 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:44 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:44.168+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:44.503 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:44 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:44.168+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:12.273389+0000 front 2024-03-21T13:23:12.273365+0000 (oldest deadline 2024-03-21T13:23:34.573182+0000) 2024-03-21T13:23:44.503 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:44 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:44.168+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-03-21T13:23:19.774532+0000 front 2024-03-21T13:23:19.774543+0000 (oldest deadline 2024-03-21T13:23:42.674533+0000) 2024-03-21T13:23:44.504 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:44 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:44.025+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:44.504 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:44 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:44.025+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:44.504 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:44 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:44.025+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:45.502 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:45 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:45.129+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:51.369989+0000 front 2024-03-21T13:22:51.370096+0000 (oldest deadline 2024-03-21T13:23:17.270003+0000) 2024-03-21T13:23:45.503 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:45 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:45.129+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.471654+0000 front 2024-03-21T13:23:02.471675+0000 (oldest deadline 2024-03-21T13:23:24.771579+0000) 2024-03-21T13:23:45.503 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:45 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:45.129+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:12.273389+0000 front 2024-03-21T13:23:12.273365+0000 (oldest deadline 2024-03-21T13:23:34.573182+0000) 2024-03-21T13:23:45.503 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:45 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6[49601]: 2024-03-21T13:23:45.129+0000 7f3268ac6700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-03-21T13:23:19.774532+0000 front 2024-03-21T13:23:19.774543+0000 (oldest deadline 2024-03-21T13:23:42.674533+0000) 2024-03-21T13:23:45.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:45 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:45.009+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:45.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:45 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:45.009+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:45.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:45 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:45.009+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:45.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:45 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:45.009+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-03-21T13:23:19.179648+0000 front 2024-03-21T13:23:19.179662+0000 (oldest deadline 2024-03-21T13:23:44.479643+0000) 2024-03-21T13:23:46.393 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:46 smithi204 podman[151772]: 2024-03-21 13:23:46.10416508 +0000 UTC m=+5.577933296 container died 2a1eae6608f4a6565c10c131ed92c50f23fd8315080a969cae15811e59cf3aef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-21T13:23:46.393 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:46 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:46.050+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:46.393 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:46 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:46.050+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:46.393 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:46 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:46.050+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:46.393 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:46 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:46.050+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-03-21T13:23:19.179648+0000 front 2024-03-21T13:23:19.179662+0000 (oldest deadline 2024-03-21T13:23:44.479643+0000) 2024-03-21T13:23:46.726 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:46 smithi204 podman[151772]: 2024-03-21 13:23:46.492774373 +0000 UTC m=+5.966542598 container cleanup 2a1eae6608f4a6565c10c131ed92c50f23fd8315080a969cae15811e59cf3aef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:23:46.727 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:46 smithi204 bash[151772]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6 2024-03-21T13:23:47.002 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:46 smithi204 podman[151790]: 2024-03-21 13:23:46.727227584 +0000 UTC m=+0.618243960 container remove 2a1eae6608f4a6565c10c131ed92c50f23fd8315080a969cae15811e59cf3aef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:23:47.339 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:47 smithi204 podman[151880]: 2024-03-21 13:23:46.959430526 +0000 UTC m=+0.018870154 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:23:47.339 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:47 smithi204 podman[151880]: 2024-03-21 13:23:47.061548858 +0000 UTC m=+0.120988518 container create 7c4de31808db3e243ba9d2953ddbf2d6eff3185e0b42b4bee97802ecc196392a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-21T13:23:47.339 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:47 smithi204 podman[151880]: 2024-03-21 13:23:47.339291587 +0000 UTC m=+0.398731214 container init 7c4de31808db3e243ba9d2953ddbf2d6eff3185e0b42b4bee97802ecc196392a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-21T13:23:47.339 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:47 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:47.068+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:47.339 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:47 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:47.068+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:47.339 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:47 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:47.068+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:47.339 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:47 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:47.068+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-03-21T13:23:19.179648+0000 front 2024-03-21T13:23:19.179662+0000 (oldest deadline 2024-03-21T13:23:44.479643+0000) 2024-03-21T13:23:47.640 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:47 smithi204 podman[151880]: 2024-03-21 13:23:47.342645644 +0000 UTC m=+0.402085259 container start 7c4de31808db3e243ba9d2953ddbf2d6eff3185e0b42b4bee97802ecc196392a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-21T13:23:47.640 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:47 smithi204 podman[151880]: 2024-03-21 13:23:47.430968586 +0000 UTC m=+0.490408325 container attach 7c4de31808db3e243ba9d2953ddbf2d6eff3185e0b42b4bee97802ecc196392a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:23:47.984 INFO:journalctl@ceph.osd.6.smithi204.stdout:Mar 21 13:23:47 smithi204 podman[151880]: 2024-03-21 13:23:47.640927239 +0000 UTC m=+0.700366862 container died 7c4de31808db3e243ba9d2953ddbf2d6eff3185e0b42b4bee97802ecc196392a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-03-21T13:23:48.252 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:48 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:48.095+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:48.252 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:48 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:48.095+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:48.252 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:48 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:48.095+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:48.252 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:48 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:48.095+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-03-21T13:23:19.179648+0000 front 2024-03-21T13:23:19.179662+0000 (oldest deadline 2024-03-21T13:23:44.479643+0000) 2024-03-21T13:23:48.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.6.service' 2024-03-21T13:23:48.429 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:23:48.430 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-21T13:23:48.430 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-21T13:23:48.430 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.7 2024-03-21T13:23:48.650 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:48 smithi204 systemd[1]: Stopping Ceph osd.7 for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:23:49.002 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:48 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:48.778+0000 7ffa50b11700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:23:49.002 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:48 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:48.778+0000 7ffa50b11700 -1 osd.7 499 *** Got signal Terminated *** 2024-03-21T13:23:49.002 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:48 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:48.779+0000 7ffa50b11700 -1 osd.7 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:23:49.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:49 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:49.109+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:49.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:49 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:49.109+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:49.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:49 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:49.109+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:49.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:49 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:49.109+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-03-21T13:23:19.179648+0000 front 2024-03-21T13:23:19.179662+0000 (oldest deadline 2024-03-21T13:23:44.479643+0000) 2024-03-21T13:23:50.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:50 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:50.067+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:50.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:50 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:50.067+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:50.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:50 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:50.067+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:50.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:50 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:50.067+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-03-21T13:23:19.179648+0000 front 2024-03-21T13:23:19.179662+0000 (oldest deadline 2024-03-21T13:23:44.479643+0000) 2024-03-21T13:23:50.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:50 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:50.067+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.204:6802 osd.4 since back 2024-03-21T13:23:24.480078+0000 front 2024-03-21T13:23:24.480065+0000 (oldest deadline 2024-03-21T13:23:49.780018+0000) 2024-03-21T13:23:51.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:51.078+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:51.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:51.078+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:51.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:51.078+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:51.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:51.078+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-03-21T13:23:19.179648+0000 front 2024-03-21T13:23:19.179662+0000 (oldest deadline 2024-03-21T13:23:44.479643+0000) 2024-03-21T13:23:51.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:51 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:51.078+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.204:6802 osd.4 since back 2024-03-21T13:23:24.480078+0000 front 2024-03-21T13:23:24.480065+0000 (oldest deadline 2024-03-21T13:23:49.780018+0000) 2024-03-21T13:23:52.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:52.033+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:52.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:52.033+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:52.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:52.033+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:52.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:52.033+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-03-21T13:23:19.179648+0000 front 2024-03-21T13:23:19.179662+0000 (oldest deadline 2024-03-21T13:23:44.479643+0000) 2024-03-21T13:23:52.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:52 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:52.033+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.204:6802 osd.4 since back 2024-03-21T13:23:24.480078+0000 front 2024-03-21T13:23:24.480065+0000 (oldest deadline 2024-03-21T13:23:49.780018+0000) 2024-03-21T13:23:53.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:53 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:53.002+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-03-21T13:22:53.575913+0000 front 2024-03-21T13:22:53.576133+0000 (oldest deadline 2024-03-21T13:23:19.376565+0000) 2024-03-21T13:23:53.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:53 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:53.002+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-03-21T13:23:02.777921+0000 front 2024-03-21T13:23:02.778003+0000 (oldest deadline 2024-03-21T13:23:28.677952+0000) 2024-03-21T13:23:53.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:53 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:53.002+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-03-21T13:23:08.678570+0000 front 2024-03-21T13:23:08.678454+0000 (oldest deadline 2024-03-21T13:23:33.378350+0000) 2024-03-21T13:23:53.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:53 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:53.002+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-03-21T13:23:19.179648+0000 front 2024-03-21T13:23:19.179662+0000 (oldest deadline 2024-03-21T13:23:44.479643+0000) 2024-03-21T13:23:53.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:53 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7[55758]: 2024-03-21T13:23:53.002+0000 7ffa4d105700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.204:6802 osd.4 since back 2024-03-21T13:23:24.480078+0000 front 2024-03-21T13:23:24.480065+0000 (oldest deadline 2024-03-21T13:23:49.780018+0000) 2024-03-21T13:23:54.752 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:54 smithi204 podman[151985]: 2024-03-21 13:23:54.466126271 +0000 UTC m=+5.810161323 container died ce943b1fea7434f96197ae8ea2be99a1579439def6295e38743d57f49aa4ce31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-03-21T13:23:55.252 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:55 smithi204 podman[151985]: 2024-03-21 13:23:55.012914388 +0000 UTC m=+6.356949477 container cleanup ce943b1fea7434f96197ae8ea2be99a1579439def6295e38743d57f49aa4ce31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-21T13:23:55.252 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:55 smithi204 bash[151985]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7 2024-03-21T13:23:55.612 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:55 smithi204 podman[152014]: 2024-03-21 13:23:55.297364069 +0000 UTC m=+0.834628295 container remove ce943b1fea7434f96197ae8ea2be99a1579439def6295e38743d57f49aa4ce31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:23:55.942 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:55 smithi204 podman[152107]: 2024-03-21 13:23:55.515779574 +0000 UTC m=+0.019035146 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:23:55.942 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:55 smithi204 podman[152107]: 2024-03-21 13:23:55.681804646 +0000 UTC m=+0.185060255 container create dcc83fea3de33f5e2314f782625447043e8092aaac0b310054d7eedf1cea8426 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:23:55.942 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:55 smithi204 podman[152107]: 2024-03-21 13:23:55.942763369 +0000 UTC m=+0.446018936 container init dcc83fea3de33f5e2314f782625447043e8092aaac0b310054d7eedf1cea8426 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:23:56.238 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:55 smithi204 podman[152107]: 2024-03-21 13:23:55.945889858 +0000 UTC m=+0.449145430 container start dcc83fea3de33f5e2314f782625447043e8092aaac0b310054d7eedf1cea8426 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-21T13:23:56.238 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:56 smithi204 podman[152107]: 2024-03-21 13:23:56.042798331 +0000 UTC m=+0.546053905 container attach dcc83fea3de33f5e2314f782625447043e8092aaac0b310054d7eedf1cea8426 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:23:56.502 INFO:journalctl@ceph.osd.7.smithi204.stdout:Mar 21 13:23:56 smithi204 podman[152107]: 2024-03-21 13:23:56.239066266 +0000 UTC m=+0.742321837 container died dcc83fea3de33f5e2314f782625447043e8092aaac0b310054d7eedf1cea8426 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-21T13:23:56.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@osd.7.service' 2024-03-21T13:23:56.885 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:23:56.886 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-21T13:23:56.886 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-21T13:23:56.886 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@rgw.foo.a 2024-03-21T13:23:57.186 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 13:23:56 smithi163 systemd[1]: Stopping Ceph rgw.foo.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:23:57.445 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 13:23:57 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-rgw-foo-a[67126]: 2024-03-21T13:23:57.185+0000 7f2508a71700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:23:57.445 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 13:23:57 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-rgw-foo-a[67126]: 2024-03-21T13:23:57.185+0000 7f25172f6ac0 -1 shutting down 2024-03-21T13:24:07.511 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 13:24:07 smithi163 bash[211423]: time="2024-03-21T13:24:07Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-21T13:24:07.511 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 13:24:07 smithi163 podman[211423]: 2024-03-21 13:24:07.210528051 +0000 UTC m=+10.128984480 container died f432467663204361a881d26a340a14ac5eb73de04addb629c08dbb685ec9b8ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:24:07.885 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 13:24:07 smithi163 podman[211423]: 2024-03-21 13:24:07.802257097 +0000 UTC m=+10.720713569 container cleanup f432467663204361a881d26a340a14ac5eb73de04addb629c08dbb685ec9b8ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:24:07.885 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 13:24:07 smithi163 bash[211423]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-rgw-foo-a 2024-03-21T13:24:08.002 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:07.704Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.163:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.163:8765: connect: connection refused" 2024-03-21T13:24:08.002 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:07.707Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.163:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.163:8765: connect: connection refused" 2024-03-21T13:24:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:07.709Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.163:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.163:8765: connect: connection refused" 2024-03-21T13:24:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:07.714Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.163:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.163:8765: connect: connection refused" 2024-03-21T13:24:08.003 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:07 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:07.718Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.163:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.163:8765: connect: connection refused" 2024-03-21T13:24:08.190 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 13:24:08 smithi163 podman[211450]: 2024-03-21 13:24:08.08670777 +0000 UTC m=+0.866360556 container remove f432467663204361a881d26a340a14ac5eb73de04addb629c08dbb685ec9b8ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:24:08.190 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Mar 21 13:24:08 smithi163 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-21T13:24:08.204 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@rgw.foo.a.service' 2024-03-21T13:24:08.244 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:24:08.244 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-21T13:24:08.244 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-21T13:24:08.244 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@prometheus.a 2024-03-21T13:24:08.565 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 systemd[1]: Stopping Ceph prometheus.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:24:08.565 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:08.565Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T13:24:08.857 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:08.565Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-21T13:24:08.857 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:08.565Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-21T13:24:08.857 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:08.565Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-21T13:24:08.857 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:08.565Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T13:24:08.858 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:08.565Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-21T13:24:08.858 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:08.565Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T13:24:08.858 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:08.565Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-21T13:24:08.858 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:08.574Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-21T13:24:08.858 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:08.575Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T13:24:08.858 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:08.575Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-21T13:24:08.858 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a[67851]: ts=2024-03-21T13:24:08.575Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-21T13:24:08.858 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 podman[152213]: 2024-03-21 13:24:08.585766945 +0000 UTC m=+0.146078570 container died 617428b1bc18ebf7103465de0cd8d6697e9e2408141382f534b454e3362eecaa (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:24:09.240 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 podman[152213]: 2024-03-21 13:24:08.973515838 +0000 UTC m=+0.533827531 container cleanup 617428b1bc18ebf7103465de0cd8d6697e9e2408141382f534b454e3362eecaa (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:24:09.241 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Mar 21 13:24:08 smithi204 bash[152213]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T13:24:09.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-21T13:24:09.427 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:24:09.427 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-21T13:24:09.427 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-21T13:24:09.620 INFO:teuthology.orchestra.run.smithi163.stdout:Deleting cluster with fsid: d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T13:24:12.011 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:24:11 smithi163 systemd[1]: Stopping Ceph alertmanager.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:24:12.466 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:24:12 smithi163 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a[80908]: ts=2024-03-21T13:24:12.165Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-21T13:24:12.466 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:24:12 smithi163 podman[211927]: 2024-03-21 13:24:12.206539961 +0000 UTC m=+0.151157404 container died 68f866c321577c22fe67c34073adfedea9d496cc88e981d15e00587341ed1b7a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:24:13.125 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:24:12 smithi163 podman[211927]: 2024-03-21 13:24:12.840842599 +0000 UTC m=+0.785460042 container cleanup 68f866c321577c22fe67c34073adfedea9d496cc88e981d15e00587341ed1b7a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:24:13.126 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:24:12 smithi163 bash[211927]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T13:24:13.126 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:24:13 smithi163 podman[211941]: 2024-03-21 13:24:13.050536219 +0000 UTC m=+0.838804596 container remove 68f866c321577c22fe67c34073adfedea9d496cc88e981d15e00587341ed1b7a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:24:13.126 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:24:13 smithi163 podman[211941]: 2024-03-21 13:24:13.126402407 +0000 UTC m=+0.914670786 volume remove 53557f7936d28c2bf92aa378bb2f0e2757f2a32d7b7397f26f140d856a4c33c2 2024-03-21T13:24:13.760 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:24:13 smithi163 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T13:24:13.761 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:24:13 smithi163 systemd[1]: Stopped Ceph alertmanager.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T13:24:13.761 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Mar 21 13:24:13 smithi163 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.702s CPU time. 2024-03-21T13:24:13.761 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:24:13 smithi163 systemd[1]: Stopping Ceph node-exporter.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:24:14.120 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:24:13 smithi163 podman[212054]: 2024-03-21 13:24:13.856113843 +0000 UTC m=+0.108794075 container died ad11847ba7c4e03477d31c86aee11c3ac9dc24a199d5159f69ae0e4c6f40add8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:24:14.510 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:24:14 smithi163 podman[212054]: 2024-03-21 13:24:14.303789823 +0000 UTC m=+0.556470051 container cleanup ad11847ba7c4e03477d31c86aee11c3ac9dc24a199d5159f69ae0e4c6f40add8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:24:14.510 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:24:14 smithi163 bash[212054]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a 2024-03-21T13:24:14.861 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:24:14 smithi163 podman[212068]: 2024-03-21 13:24:14.588302724 +0000 UTC m=+0.736223031 container remove ad11847ba7c4e03477d31c86aee11c3ac9dc24a199d5159f69ae0e4c6f40add8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:24:14.861 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:24:14 smithi163 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-21T13:24:14.861 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:24:14 smithi163 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-21T13:24:14.861 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:24:14 smithi163 systemd[1]: Stopped Ceph node-exporter.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T13:24:14.861 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Mar 21 13:24:14 smithi163 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 4.715s CPU time. 2024-03-21T13:24:15.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-21T13:24:15.545 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T13:24:17.761 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:24:17 smithi204 systemd[1]: Stopping Ceph iscsi.iscsi.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:24:18.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:24:17 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a[60448]: debug Shutdown received 2024-03-21T13:24:28.062 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:24:27 smithi204 bash[152623]: time="2024-03-21T13:24:27Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-21T13:24:28.062 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:24:27 smithi204 podman[152623]: 2024-03-21 13:24:27.778566574 +0000 UTC m=+10.128471459 container died 759279b9e79c034149d225cbad143ad50dfd756b476ba0a53509ba5a4af685c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-21T13:24:28.428 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:24:28 smithi204 podman[152623]: 2024-03-21 13:24:28.14399971 +0000 UTC m=+10.493904604 container cleanup 759279b9e79c034149d225cbad143ad50dfd756b476ba0a53509ba5a4af685c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:24:28.428 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:24:28 smithi204 bash[152623]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a 2024-03-21T13:24:28.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:24:28 smithi204 podman[152651]: 2024-03-21 13:24:28.428563098 +0000 UTC m=+0.647356972 container remove 759279b9e79c034149d225cbad143ad50dfd756b476ba0a53509ba5a4af685c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-21T13:24:28.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:24:28 smithi204 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-21T13:24:28.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:24:28 smithi204 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-21T13:24:28.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:24:28 smithi204 systemd[1]: Stopped Ceph iscsi.iscsi.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T13:24:28.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Mar 21 13:24:28 smithi204 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Consumed 3.595s CPU time. 2024-03-21T13:24:30.180 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:24:30 smithi204 systemd[1]: Stopping Ceph grafana.a for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:24:30.502 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:24:30 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=server t=2024-03-21T13:24:30.303076604Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-21T13:24:30.502 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:24:30 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=ticker t=2024-03-21T13:24:30.303511071Z level=info msg=stopped last_tick=2024-03-21T13:24:30Z 2024-03-21T13:24:30.502 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:24:30 smithi204 ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a[64034]: logger=tracing t=2024-03-21T13:24:30.303522193Z level=info msg="Closing tracing" 2024-03-21T13:24:30.502 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:24:30 smithi204 podman[152911]: 2024-03-21 13:24:30.323570849 +0000 UTC m=+0.117921548 container died 146318eeea1aeb9cf3a13aa4bcc912d2a5d79144e080ccb0f6d0930ad768e87b (image=quay.io/ceph/grafana:9.4.12, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:24:30.970 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:24:30 smithi204 podman[152911]: 2024-03-21 13:24:30.67763352 +0000 UTC m=+0.471984226 container cleanup 146318eeea1aeb9cf3a13aa4bcc912d2a5d79144e080ccb0f6d0930ad768e87b (image=quay.io/ceph/grafana:9.4.12, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:24:30.970 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:24:30 smithi204 bash[152911]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a 2024-03-21T13:24:31.252 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:24:30 smithi204 podman[152926]: 2024-03-21 13:24:30.970633661 +0000 UTC m=+0.647548998 container remove 146318eeea1aeb9cf3a13aa4bcc912d2a5d79144e080ccb0f6d0930ad768e87b (image=quay.io/ceph/grafana:9.4.12, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:24:31.252 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:24:31 smithi204 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-21T13:24:31.252 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:24:31 smithi204 systemd[1]: Stopped Ceph grafana.a for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T13:24:31.252 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Mar 21 13:24:31 smithi204 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@grafana.a.service: Consumed 17.536s CPU time. 2024-03-21T13:24:31.534 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:24:31 smithi204 systemd[1]: Stopping Ceph node-exporter.b for d4bf79c0-e781-11ee-95cd-87774f69a715... 2024-03-21T13:24:31.873 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:24:31 smithi204 podman[153037]: 2024-03-21 13:24:31.534181361 +0000 UTC m=+0.116719129 container died 0be4949f37bf2656b78bb5202b6304bf8df71c6218d56b03fae6c287fd729966 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:24:31.873 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:24:31 smithi204 podman[153037]: 2024-03-21 13:24:31.873604879 +0000 UTC m=+0.456142653 container cleanup 0be4949f37bf2656b78bb5202b6304bf8df71c6218d56b03fae6c287fd729966 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:24:32.166 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:24:31 smithi204 bash[153037]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b 2024-03-21T13:24:32.495 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:24:32 smithi204 podman[153051]: 2024-03-21 13:24:32.166762547 +0000 UTC m=+0.637113666 container remove 0be4949f37bf2656b78bb5202b6304bf8df71c6218d56b03fae6c287fd729966 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d4bf79c0-e781-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:24:32.495 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:24:32 smithi204 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-21T13:24:32.495 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:24:32 smithi204 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-21T13:24:32.495 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:24:32 smithi204 systemd[1]: Stopped Ceph node-exporter.b for d4bf79c0-e781-11ee-95cd-87774f69a715. 2024-03-21T13:24:32.495 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Mar 21 13:24:32 smithi204 systemd[1]: ceph-d4bf79c0-e781-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 4.412s CPU time. 2024-03-21T13:24:33.108 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:24:33.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:24:33.174 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-21T13:24:33.176 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614953/remote/smithi163/crash 2024-03-21T13:24:33.177 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/crash -- . 2024-03-21T13:24:33.235 INFO:teuthology.orchestra.run.smithi163.stderr:tar: /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-21T13:24:33.235 INFO:teuthology.orchestra.run.smithi163.stderr:tar: Error is not recoverable: exiting now 2024-03-21T13:24:33.237 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614953/remote/smithi204/crash 2024-03-21T13:24:33.238 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/crash -- . 2024-03-21T13:24:33.284 INFO:teuthology.orchestra.run.smithi204.stderr:tar: /var/lib/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-21T13:24:33.284 INFO:teuthology.orchestra.run.smithi204.stderr:tar: Error is not recoverable: exiting now 2024-03-21T13:24:33.285 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-21T13:24:33.285 DEBUG:teuthology.orchestra.run.smithi163:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-21T13:24:33.380 INFO:teuthology.orchestra.run.smithi163.stdout:2024-03-21T13:00:40.634647+0000 mon.a (mon.0) 845 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:24:33.382 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-21T13:24:33.383 DEBUG:teuthology.orchestra.run.smithi163:> sudo egrep '\[SEC\]' /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-21T13:24:33.416 DEBUG:teuthology.orchestra.run.smithi163:> sudo egrep '\[ERR\]' /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-21T13:24:33.489 DEBUG:teuthology.orchestra.run.smithi163:> sudo egrep '\[WRN\]' /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-21T13:24:33.557 INFO:teuthology.orchestra.run.smithi163.stdout:2024-03-21T13:00:40.634647+0000 mon.a (mon.0) 845 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T13:24:33.559 INFO:tasks.cephadm:Compressing logs... 2024-03-21T13:24:33.559 DEBUG:teuthology.orchestra.run.smithi163:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T13:24:33.601 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T13:24:33.660 INFO:teuthology.orchestra.run.smithi163.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-21T13:24:33.660 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-21T13:24:33.663 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-21T13:24:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-21T13:24:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-21T13:24:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mon.a.log: /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-21T13:24:33.667 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-21T13:24:33.667 INFO:teuthology.orchestra.run.smithi163.stderr: 59.3% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-21T13:24:33.667 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-21T13:24:33.667 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-21T13:24:33.667 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-21T13:24:33.667 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.log 2024-03-21T13:24:33.667 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-21T13:24:33.667 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-21T13:24:33.668 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-21T13:24:33.675 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mon.c.log: 91.8% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-21T13:24:33.676 INFO:teuthology.orchestra.run.smithi204.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-21T13:24:33.676 INFO:teuthology.orchestra.run.smithi204.stderr:: No such file or directory 2024-03-21T13:24:33.688 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-21T13:24:33.688 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-21T13:24:33.688 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-21T13:24:33.689 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-21T13:24:33.689 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-21T13:24:33.691 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-21T13:24:33.691 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/tcmu-runner.log 2024-03-21T13:24:33.691 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.5.log: /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-21T13:24:33.691 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.log 2024-03-21T13:24:33.691 INFO:teuthology.orchestra.run.smithi204.stderr: 66.9% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/tcmu-runner.log.gz 2024-03-21T13:24:33.691 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-21T13:24:33.691 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-21T13:24:33.749 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mon.b.log: 81.5% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-21T13:24:33.774 INFO:teuthology.orchestra.run.smithi163.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-21T13:24:33.789 INFO:teuthology.orchestra.run.smithi163.stderr: 94.1% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-21T13:24:33.810 INFO:teuthology.orchestra.run.smithi204.stderr: 90.3% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-21T13:24:33.852 INFO:teuthology.orchestra.run.smithi204.stderr: 92.2% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-21T13:24:33.853 INFO:teuthology.orchestra.run.smithi204.stderr: 91.2% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-21T13:24:33.865 INFO:teuthology.orchestra.run.smithi163.stderr: 89.1% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-21T13:24:33.912 INFO:teuthology.orchestra.run.smithi204.stderr: 85.4% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-21T13:24:34.016 INFO:teuthology.orchestra.run.smithi204.stderr: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-21T13:24:34.653 INFO:teuthology.orchestra.run.smithi163.stderr: 91.2% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-21T13:24:35.094 INFO:teuthology.orchestra.run.smithi163.stderr: 89.7% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-21T13:24:36.855 INFO:teuthology.orchestra.run.smithi204.stderr: 91.1% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-21T13:24:39.849 INFO:teuthology.orchestra.run.smithi163.stderr: 91.2% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-21T13:24:53.329 INFO:teuthology.orchestra.run.smithi163.stderr: 90.4% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-21T13:25:03.556 INFO:teuthology.orchestra.run.smithi163.stderr: 94.1% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-21T13:26:50.310 INFO:teuthology.orchestra.run.smithi204.stderr: 93.4% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-21T13:27:20.101 INFO:teuthology.orchestra.run.smithi163.stderr: 93.3% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-21T13:30:03.992 INFO:teuthology.orchestra.run.smithi163.stderr: 92.9% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-21T13:31:45.810 INFO:teuthology.orchestra.run.smithi163.stderr: 92.6% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-21T13:31:45.812 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-21T13:31:45.812 INFO:teuthology.orchestra.run.smithi163.stderr:real 7m12.198s 2024-03-21T13:31:45.812 INFO:teuthology.orchestra.run.smithi163.stderr:user 5m52.246s 2024-03-21T13:31:45.812 INFO:teuthology.orchestra.run.smithi163.stderr:sys 0m15.605s 2024-03-21T13:39:37.547 INFO:teuthology.orchestra.run.smithi204.stderr: 93.1% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-21T13:43:22.970 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-21T13:43:50.439 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/d4bf79c0-e781-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-21T13:43:50.441 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-21T13:43:50.442 INFO:teuthology.orchestra.run.smithi204.stderr:real 19m16.826s 2024-03-21T13:43:50.442 INFO:teuthology.orchestra.run.smithi204.stderr:user 6m40.921s 2024-03-21T13:43:50.442 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m20.074s 2024-03-21T13:43:50.442 INFO:tasks.cephadm:Archiving logs... 2024-03-21T13:43:50.442 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614953/remote/smithi163/log 2024-03-21T13:43:50.444 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T13:44:47.340 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614953/remote/smithi204/log 2024-03-21T13:44:47.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T13:45:49.035 INFO:tasks.cephadm:Removing cluster... 2024-03-21T13:45:49.036 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 --force 2024-03-21T13:45:49.459 INFO:teuthology.orchestra.run.smithi163.stdout:Deleting cluster with fsid: d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T13:45:50.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4bf79c0-e781-11ee-95cd-87774f69a715 --force 2024-03-21T13:45:50.866 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: d4bf79c0-e781-11ee-95cd-87774f69a715 2024-03-21T13:45:51.642 INFO:tasks.cephadm:Removing cephadm ... 2024-03-21T13:45:51.642 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-21T13:45:51.658 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-21T13:45:51.682 INFO:tasks.cephadm:Teardown complete 2024-03-21T13:45:51.683 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-21T13:45:51.698 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-21T13:45:51.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-21T13:45:51.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-21T13:45:51.823 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-21T13:45:51.832 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-21T13:45:51.833 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T13:45:51.835 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T13:45:51.846 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-03-21T13:45:51.850 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-03-21T13:45:51.896 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T13:45:51.896 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-03-21T13:45:51.896 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 52 +2681us[+2681us] +/- 122ms 2024-03-21T13:45:51.896 INFO:teuthology.orchestra.run.smithi163.stdout:^* hv02.front.sepia.ceph.com 3 6 377 53 -1377us[-1358us] +/- 63ms 2024-03-21T13:45:51.897 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +15us[ +15us] +/- 77ms 2024-03-21T13:45:51.897 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:45:51.906 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T13:45:51.906 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-03-21T13:45:51.906 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 +2723us[+2714us] +/- 122ms 2024-03-21T13:45:51.906 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv02.front.sepia.ceph.com 3 6 377 53 -1320us[-1329us] +/- 63ms 2024-03-21T13:45:51.906 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +79us[ +70us] +/- 77ms 2024-03-21T13:45:51.906 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:45:51.907 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-21T13:45:51.917 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-21T13:45:51.918 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-21T13:45:51.947 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-21T13:45:52.032 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-21T13:45:52.149 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-21T13:45:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:45:52.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-21T13:45:52.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:45:52.207 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-21T13:45:52.217 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-21T13:45:52.317 INFO:teuthology.task.internal:Duration was 3612.263385 seconds 2024-03-21T13:45:52.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-21T13:45:52.347 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-21T13:45:52.347 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T13:45:52.350 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T13:45:52.424 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T13:45:52.440 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T13:45:52.846 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-21T13:45:52.847 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2024-03-21T13:45:52.847 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T13:45:52.867 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-03-21T13:45:52.867 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T13:45:52.891 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-21T13:45:52.892 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T13:45:52.909 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T13:45:52.966 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-21T13:45:52.966 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T13:45:52.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T13:45:54.294 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-21T13:45:54.304 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-21T13:45:54.305 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T13:45:54.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T13:45:54.651 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-21T13:45:54.661 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T13:45:54.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T13:45:54.697 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2024-03-21T13:45:54.734 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-03-21T13:45:54.788 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T13:45:54.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:45:54.803 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T13:45:54.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:45:54.845 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-21T13:45:54.855 INFO:teuthology.task.internal:Transferring archived files... 2024-03-21T13:45:54.856 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614953/remote/smithi163 2024-03-21T13:45:54.857 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T13:45:54.922 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614953/remote/smithi204 2024-03-21T13:45:54.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T13:45:54.963 INFO:teuthology.task.internal:Removing archive directory... 2024-03-21T13:45:54.963 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T13:45:54.966 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T13:45:55.015 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-21T13:45:55.048 INFO:teuthology.task.internal:Not uploading archives. 2024-03-21T13:45:55.048 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-21T13:45:55.091 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-21T13:45:55.091 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T13:45:55.095 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T13:45:55.107 INFO:teuthology.orchestra.run.smithi163.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 13:45 /home/ubuntu/cephtest 2024-03-21T13:45:55.115 INFO:teuthology.orchestra.run.smithi204.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 13:45 /home/ubuntu/cephtest 2024-03-21T13:45:55.117 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-21T13:45:55.166 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-21T13:45:55.226 INFO:teuthology.nuke:Checking targets against current locks 2024-03-21T13:45:55.254 DEBUG:teuthology.nuke:shortname: smithi163 2024-03-21T13:45:55.255 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T13:45:55.281 DEBUG:teuthology.nuke:shortname: smithi204 2024-03-21T13:45:55.281 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T13:45:55.285 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614953', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 12:35:15.305809', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN589b3RczkarYEMJ7Gi2FAuMmWJFf7GwYb25ciTKTGKJ+++Bt6IO38ss1jDZ4R7lrqEun5K7RMeuMfdZ85ma8U='} 2024-03-21T13:45:55.308 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614953', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 12:35:15.304290', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJ2UNdgj78yltC2wNujQGNpQzoyP4jJ6xFd72mvl9HnPOFewfqDGbEfEQqNGhpwO++brsqfqGzM12X2DLDrI+Q='} 2024-03-21T13:45:55.344 INFO:teuthology.orchestra.console.smithi163:Power off 2024-03-21T13:45:55.345 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T13:45:55.368 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off 2024-03-21T13:45:55.369 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:45:55.383 INFO:teuthology.orchestra.console.smithi204:Power off 2024-03-21T13:45:55.383 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T13:45:55.393 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-03-21T13:45:55.407 DEBUG:teuthology.orchestra.console.smithi204:power off output: Chassis Power Control: Down/Off 2024-03-21T13:45:55.407 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:45:55.432 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-21T13:45:59.395 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:45:59.433 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:45:59.505 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-03-21T13:45:59.541 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-21T13:46:03.511 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:46:03.542 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:46:03.621 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-03-21T13:46:03.652 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off 2024-03-21T13:46:03.752 INFO:teuthology.orchestra.console.smithi204:Power off completed 2024-03-21T13:46:07.622 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:46:07.731 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2024-03-21T13:46:07.832 INFO:teuthology.orchestra.console.smithi163:Power off completed 2024-03-21T13:46:07.936 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} duration: 3612.2633848190308 failure_reason: '"2024-03-21T13:00:40.634647+0000 mon.a (mon.0) 845 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log' flavor: default owner: scheduled_gabrioux@teuthology success: false 2024-03-21T13:46:07.936 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T13:46:08.024 INFO:teuthology.run:FAIL